mac-capture: Add toggle to enable higher quality SCK transparency

[jcm93]

Description

Adds a toggle that allows macOS Screen Capture sources to use the BGRA pixel format, enabling higher quality transparency, since the default format, l10r, only includes 2 bits to represent the alpha channel, leading to poor alpha composite quality. New toggle visible below:

Motivation and Context

macOS Screen Captures have had inferior alpha compositing since the introduction of the source because of l10r's low resolution transparency. For example:

l10r:

BGRA:

BGRA will feature less color accuracy given the two fewer bits per channel for RGB. This mostly takes the form of increased banding within gradients that make full use of the 10-bit color range, often encountered with macOS's use of the P3 color space on most Apple displays.

Higher resolution alpha, however, enables various creative usecases with the ability to cleanly composite application and window captures with correct shadows and transparency in front of arbitrary backgrounds created in OBS.

How Has This Been Tested?

Tested on Apple Silicon, macOS 15.7.1.

Types of changes

• New feature (non-breaking change which adds functionality)
• Tweak (non-breaking change to improve existing functionality)

Checklist:

• My code has been run through clang-format.
• I have read the contributing document.
• My code is not on the master branch.
• The code has been tested.
• All commit messages are properly formatted and commits squashed where appropriate.
• I have included updates to all appropriate documentation.