Productize true-headless HDR through Gamescope Portal/PipeWire capture

[papi-ux]

Status

Target: Polaris v1.3.3 planning
v1.3.2: explicitly out of scope; this issue does not block or add claims to that release.

The original DRM-lease/forged-EDID idea in this issue has been superseded by a working true-headless architecture demonstrated on an NVIDIA host:

Moonlight client
  → Polaris
  → XDG Desktop Portal ScreenCast
  → PipeWire from headless Gamescope
  → LINEAR DMA-BUF
  → Vulkan copy
  → CUDA/NVENC

The downstream stack has produced stable 4K60 HDR/10-bit results on tested NVIDIA hardware and useful SDR/fallback controls. That proves the direction is viable; it does not make the downstream patch stack upstream-ready as one large merge.

Maintainer handoff

• Distro-agnostic architecture and session wiring:
  https://github.com/luxus/polaris-hdr-linux-patches/blob/main/docs/upstream-handoff-papi-ux.md
• Phase-gated Polaris patches:
  https://github.com/luxus/polaris-hdr-linux-patches/blob/main/polaris/README.md
• Detailed issue handoff comment:
  Productize true-headless HDR through Gamescope Portal/PipeWire capture #152 (comment)

The external patches are reference implementations and test inputs, not an automatic merge source. Each phase must be reconciled against current Polaris master, reviewed independently, and verified with Polaris-owned tests and hardware evidence.

Polaris v1.3.3 productization plan

Phase 0 — current-master source map and architecture contract

• Compare current Polaris master, the stale perf/issue-152-pipewire-capture branch, and the downstream phase patches.
• Identify what is already upstream, what is obsolete, and what still needs an upstream-shaped implementation.
• Freeze interfaces for transport selection, buffer lifetime, conversion-path reporting, session HDR intent, Gamescope ownership, and fallback behavior.
• Keep Phase 1 independent from Gamescope-specific lifecycle, private D-Bus, WSI policy, and HDR.

Phase 1 — Portal/PipeWire foundation with reliable SHM fallback

• Implement/refresh real XDG Portal ScreenCast → PipeWire capture.
• Choose transport from actual PipeWire allocation, not requested descriptors.
• Handle add_buffer / remove_buffer lifetime correctly.
• Offer DMA-BUF only when same-GPU and modifier/import requirements are proven.
• Preserve MemFd/MemPtr as a first-class working path.
• Keep an explicit forced-SHM diagnostic escape hatch such as POLARIS_PORTAL_DMABUF=0.
• Report truthful diagnostics:
  • capture_transport=shm|dmabuf
  • frame_residency=cpu|gpu
• A refused or failed DMA-BUF path must continue through SHM rather than crash or black-screen.

Phase 2 — NVIDIA DMA-BUF → Vulkan → CUDA/NVENC fast path

• Gate the fast path on runtime capability detection and exact same-GPU ownership.
• Support the proven LINEAR one-plane BGRx/BGRA path first; add 10-bit formats only with Phase 4.
• Invalidate imports when PipeWire removes/recycles buffers.
• Document synchronization and ownership assumptions explicitly.
• Report conversion honestly:
  • convert_path=vulkan_cuda
  • convert_path=mmap_cuda for loud sticky fallback
• Do not label mmap/CPU fallback as GPU-native.
• Do not revive desktop cuImportExternalMemory(DMABUF_FD) or cuGraphicsEGLRegisterImage as the generic NVIDIA path without new evidence.

Phase 3 — first-class experimental Gamescope Stream mode

Polaris should own the happy path rather than requiring users to assemble shell scripts and user services:

• Start, attach to, health-check, and stop headless Gamescope.
• Support attach and nested presentation modes with explicit policy.
• Pin Gamescope, PipeWire capture, Vulkan conversion, CUDA, and NVENC to the same GPU.
• Own Steam/game start → wait → stop and cleanup behavior.
• Keep session markers, environment files, restore tokens, HDR intent, and process ownership scoped to the active stream.
• Support a private ScreenCast D-Bus only when desktop coexistence requires it.
• Keep Polaris itself on the normal session bus so Avahi, tray, and desktop services continue working.
• Do not globally replace KDE/GNOME ScreenCast routing or break KRDP/desktop capture.

Phase 4 — HDR contract and encoded-signal truth

Keep these states separate:

• hdr_capable
• hdr_requested
• explicit session HDR policy
• negotiated capture format/colorimetry
• encoder pixel format and metadata

Enable HDR only when all layers agree:

1. the active client/session explicitly requests HDR;
2. Gamescope is launched/configured for that session's HDR policy;
3. Portal/PipeWire negotiates the expected 10-bit format and colorimetry;
4. conversion produces P010;
5. the encoder emits matching Rec.2020/PQ/VUI/SEI/mastering metadata.

Hard invariants:

• Device-database HDR capability alone must never force session HDR.
• Detect P010 through hardware-frame sw_format, not frame->format when frames are CUDA-backed.
• SDR remains an independent 8-bit NV12 path.
• Hybrid PQ-shaped capture with SDR NV12 encode is forbidden.
• Force state must be scoped to the owning session and cleaned up deterministically.

Verification gates

Each accepted phase needs its own focused tests, independent review, and local checkpoint before the next phase begins.

Minimum release-readiness evidence for this feature:

• CUDA and no-CUDA builds/tests pass on current master.
• Forced SHM streams continuously and reports CPU residency truthfully.
• DMA-BUF refusal/import failure falls back without crash, black screen, or false telemetry.
• NVIDIA fast path and sticky fallback are exercised on a real RTX-class host.
• Gamescope start/wait/stop leaves no Steam, Gamescope, portal, session-marker, or audio-routing residue.
• Host KDE/GNOME portal behavior remains intact when Gamescope Stream is idle or disabled.
• HDR and SDR color-pattern tests match the encoded bitstream metadata.
• Supported-client matrix records codec, resolution, refresh, HDR state, transport, conversion path, and cleanup result.
• Zero new Polaris coredumps; privacy-safe logs only.

Compatibility boundary

Initial fast-path confidence is NVIDIA-first because the working Vulkan→CUDA→NVENC evidence is NVIDIA-specific. AMD/VAAPI support requires its own design and hardware matrix; it must not be implied by this issue's NVIDIA evidence.

The reported iPhone/VoidLink HDR color problem remains a compatibility caveat until encoded-sample testing separates host bitstream correctness from client decode/render behavior. macOS/iPad/tvOS success does not automatically clear every iOS client path.

Definition of done for this issue

This issue is complete when Polaris has an opt-in, accurately named Gamescope Stream mode with:

• reliable Portal/PipeWire SHM capture;
• capability-gated, truthful GPU fast paths;
• Polaris-owned Gamescope/portal/session lifecycle;
• deterministic cleanup and desktop coexistence;
• HDR enabled only through an end-to-end session/format/encode contract;
• documented support and fallback boundaries backed by real hardware/client evidence.

Historical context

The disconnected-connector, forged-EDID, and DRM-lease approach remains useful historical research, but it is no longer the primary design target. The current product direction is true-headless Gamescope Portal capture with incremental upstreaming and honest fallback behavior.

[IceDBorn]

I managed true headless hdr, by patching gamescope to provide hdr metadata for the pipewire capture path and by utilizing a desktop portal for gamescope, made by Jovian-Experiments for SteamOS like experience under NixOS. Here's the commit and source.

[papi-ux]

That’s a substantial improvement over the connector/EDID/DRM-lease approach, holy hell, thank you! Headless gamescope PipeWire node + private gamescope portal appears to remove the physical-output requirement, while the xBGR changes provide the metadata Sunshine needs for HDR. v interesting.

I'm gonna reframe this issue around the true-headless path. Before treating it as production-ready, I’d like to validate the negotiated PipeWire format, primaries/transfer metadata, and encoder path on more than one GPU/driverstack + a fresh Sunshine/gamescope log from a known-good HDR session would be extremely useful! Great stuff. I gotta try out Nix now.

[luxus]

i got it working with my 4090 and nixos as well.. but with enabled hdr the streaming suffers cpu copy and lower framerates ..
the option from grok, would be Polaris portal DMA-BUF + CUDA + HDR .. which means a larger patch on polaris

[IceDBorn]

For some reason, mine looks smoother with pipewire capture instead of kms :p

[luxus]

maybe depends on the game.. i was testing it with armored core 6, where it was hard to hit 60fps.. it didnt look bad, but i would like to have gpu-native working..
my clanker is working on it :D if i got it working i will provide a PR with a clean patch for polaris to have it working hopefully

[luxus]

the path was very painful...
https://gist.github.com/luxus/e2cf68243c3f23b9934cea8ade4339bb
i guess you can your clanker ask to take a look ..

biggest issues...
encode pressure is around 8.5-9.0ms
and i don't like the hdr colors, but it looked like that before i even tried to switch to cuda
so it is working.. but for sure not a in a state that would be useful..
but maybe useful as reference or inspiration :D

[papi-ux]

yuup, had my clanker go through all three patches as well. Some v useful pieces in there, especially the render node preference and NVIDIA import path. A few parts need reworking around PipeWire buffer lifetime, modifiers,and HDR metadata though, so I don't want to merge the whole set as-is YET

I pushed the safer capture/DMA-BUF foundation and intentionally did SDR first. Luckily passed the full local test suite plus a real fallback/reconnect smoke.

If youre around, could you test that exact commit on the 4090/gamescope setup without the gist patches applied then send some fresh logs? If it falls back, the log hopefully should tell us why! Thanks for all of this!

[luxus]

i guess i will have the time maybe tomorrow or the day after tomorrow..
here are some additional words of my clanker and a repo with the files:

Thanks for the SDR-first work on perf/issue-152-pipewire-capture and for digging through the gist. Here’s a denser field report of what we actually learned on the gamescope + portal HDR path (RTX 4090 + NixOS, hybrid AMD secondary GPU). This is research / inspiration, not a “merge this blob” claim — fully agree buffer lifetime, modifiers, and HDR metadata need a careful upstream design.

Patch archive (rebased notes + three polaris + four gamescope patches):
https://github.com/luxus/polaris-hdr-linux-patches

Same content was the gist earlier: https://gist.github.com/luxus/e2cf68243c3f23b9934cea8ade4339bb

Upstream context we built on: IceDBorn’s gamescope portal + PipeWire HDR metadata work (same issue).

Setup (gamescope path)

Piece	What we used
Host	NixOS, Plasma Wayland daily desktop left alone
GPU	4090 NVENC + AMD iGPU present (must pin NVIDIA)
Stream runtime	Idle gamescope (HDR/headless-ish) + xdg-desktop-portal-gamescope
Polaris	Portal capture (capture = portal), CUDA build
Client	Moonlight / ATV-class, HEVC, often 4K60, HDR / 10-bit flags
Default non-HDR	Stock Polaris labwc headless remains the “just works” SDR path

Architecture we ended on:

polaris-hdr-idle: gamescope (NVIDIA pin, HDR flags)
        │  PipeWire Video/Source
        ▼
xdg-desktop-portal-gamescope  ◄── portals.conf / XDG_CURRENT_DESKTOP=gamescope
        │
        ▼
polaris portal_grab → EGL import → CUDA → NVENC HEVC Main10
        │
app prep: steam -gamepadui into that gamescope session

What was relatively easy

1. Proving the MemPtr baseline (G0)
   Single portal EnumFormat, MemFd/SHM: video + audio + client HDR toggle worked. That was the “don’t break known-good” floor.

2. Seeing what gamescope advertises (G1)
   pw-cli EnumFormat on the gamescope Video/Source node showed BGRx (± modifier), NV12 (± modifier), and xBGR_210LE for HDR. So the format menu was already rich; the problem wasn’t “gamescope only has 8-bit.”

3. Producer fix once we stopped thrashing polaris (G2)
   Unpatched gamescope only set ParamBuffers dataType=DmaBuf when a modifier was fixated. Clients that negotiated plain formats stuck on MemFd forever.
   Our pipewire-prefer-dmabuf patch: always advertise DmaBuf|MemFd|MemPtr, and in add_buffer prefer DmaBuf when the type mask allows it. After that, probes with a real focus window got FIRST_FRAME type=DmaBuf.

4. Stacking IceDOS-style metadata patches
   PipeWire HDR metadata + color management + headless colorimetry patches gave us log lines for 10-bit + HDR-tagged HEVC once capture was alive. Useful as a reference even if metadata values are still wrong for “looks like HDMI.”

5. Dual-GPU pin (mechanical but clear)
   Same physical GPU for gamescope, PipeWire dmabuf, EGL, and NVENC. Runtime discovery of NVIDIA renderD* + PCI id; export to labwc/WLR and to gamescope --prefer-vk-device. Do not need to blacklist amdgpu if pin is correct.

What hurt (order we actually burned time)

1. Producer before consumer (hard lesson)

We spent too long changing polaris portal negotiation while gamescope never offered DmaBuf on the formats we chose.

Learning: Log SPA_DATA_* on first frame. No type=DmaBuf ⇒ do not “fix” GPU encode. POLARIS_PORTAL_DMABUF=0 (force MemFd/MemPtr) is a valid escape when GPU descriptors + MemPtr would black-screen.

2. Idle gamescope produces zero frames

paint_pipewire() only emits when a focus window has new commits. A bare idle (sleep infinity style) looks like a broken portal; it’s empty compositor.

Learning: G2 probes need something painting (glxgears, Steam Big Picture, game). Session start must launch a real client into that gamescope.

3. Portal implementation selection

If the user session is KDE, xdg-desktop-portal often keeps KDE ScreenCast even when the app sets XDG_CURRENT_DESKTOP=gamescope.

Learning: Need all of: gamescope portal package, portals.conf preferred ScreenCast=gamescope, polaris unit not inheriting KDE desktop env in a way that overrides, and restart of xdg-desktop-portal after flip. Switching labwc ↔ gamescope portal without cleaning restore tokens also left sticky state.

4. DMA-BUF import on NVIDIA (GL 0x502)

Even with NVIDIA-only buffers, EGLImageTargetTexture2DOES failed with GL 0x502 when the portal set modifier=DRM_FORMAT_MOD_INVALID (no modifier prop on format). Gamescope linear dmabufs want DRM_FORMAT_MOD_LINEAR attrs, not “invalid ⇒ omit.”

CUDA external memory import of those dmabufs failed for us; path that worked:

• EGL image from dmabuf
• TexStorageEXT-style texture
• GL→CUDA / existing encode device path

Patches in the archive: egl-dmabuf-import, cuda-gl-dmabuf, plus portal first-frame logging / dataType preference.

5. Buffer type mismatch ⇒ black (not “slow”)

If the consumer allocates GPU image descriptors but PipeWire only delivers MemPtr, painting must not pretend zero-copy. We hit “MemPtr frames with GPU descriptors — dropping until reinit” style failures.

Learning: GPU path only when SPA_DATA_DmaBuf (and import succeeds). Otherwise system-memory path. Your branch’s “honest fallback + visible path” is exactly right.

6. Performance: convert ≠ encode

Once DmaBuf + GPU path was healthy on gamescope HDR:

Stage	Ballpark (4K-class HDR session)
RGB10 DmaBuf → P010 convert	~0.7–1.0 ms
NVENC encode (UI “encode ms”)	~8.2–8.8 ms (samples up to ~9)

So:

• Zero-copy does not fix 120 Hz by itself; encode is the budget.
• ATV/clients at 60 are fine; 120 is encode-bound on this content.
• We briefly instrumented convert separately then removed it — not useful for the color problem.

7. HDR color quality (separate from transport)

With 10-bit XB30 DmaBuf and HDR-tagged HEVC in logs, the picture still looked washed / flat vs PC→TV HDMI. That was true before the CUDA path.

We treat that as: hardcoded / stub HDR metadata + Linux matrix-only convert vs real PQ/Rec.2020 semantics of whatever gamescope actually writes — not “need more zero-copy.” Notes: docs/polaris-hdr-color.md in the archive. Capture can be “correct DmaBuf” while the signal is still wrong for the client.

8. Stack weight

Idle gamescope + private portal + polaris env flip + Steam into foreign WAYLAND_DISPLAY is a lot of moving parts vs labwc headless. Fine for opt-in HDR research; poor default next to stock Headless Stream.

Gated checklist we used (and would re-use on your branch)

Gate	Pass criterion
G0	MemPtr portal: stable video
G1	EnumFormat dump shows expected formats
G2	FIRST_FRAME type=DmaBuf with focus window
G3	Portal accepts DmaBuf, MemPtr still works if DmaBuf absent
G4	GPU encode only on real DmaBuf; no black on MemPtr
G5	Moonlight A/B; dashboard/log path honest
G6	Small patches + docs; no dual dead paths

Producer (G2) before polaris GPU (G3–G4) was the discipline we learned the hard way.

Comparison that helped us stop thrashing

Same host, different runtimes (not apples-to-apples HDR):

• labwc headless (daily): DMA-BUF, typically 8-bit bgra8 capture, client may still request 10-bit → P010 SDR upconvert; encode UI ~4.5–5 ms @ 4K60.
• gamescope HDR experiment: DMA-BUF XB30, HDR-tagged path; encode ~8–9 ms + cheap convert.

So “gamescope is slower” was largely HDR / 10-bit capture + Main10 encode load, not “DmaBuf failed so everything is MemPtr.” When MemPtr was forced, pain was correctness/black more than “a few extra ms.”

What we’d hope from perf/issue-152-pipewire-capture

Happy to test that commit without our gist patches on the 4090, and report:

1. First-frame SPA_DATA_*, fourcc, modifier
2. Whether GPU path sticks or falls back (and the log reason)
3. Reconnect after stream end
4. Optional: HDR client flags vs SDR encode ms

Useful pieces from our spike (your call what’s already covered): render-node preference, linear modifier default, “GPU only if DmaBuf,” TexStorageEXT import, and not merging CUDA extmem.

Thanks again for taking the careful path — SDR foundation + honest fallback is what we wished we’d started with.

[papi-ux]

​Oh hell yeah, this is exactly what I needed. One clarification for the test: keep the gamescope pipewire-prefer-dmabuf patch enabled and remove only the Polaris patches, otherwise stock gamescope may stay on MemFd before my branch can prove the GPU path. Please run with a focus window active and include the render_node, negotiated format/modifier, capture_transport, and frame_residency lines. The branch intentionally falls back to SHM if it can't prove capture and encode are on the same GPU.

[luxus]

okay.. i will take a look on it.. i will patch the commit in the 152 branch on the master... i will take a look how minimal it would be to add gpu instead of cpu copy as well..

[luxus]

Update (2026-07-14) — gamescope portal HDR path is working end-to-end on our host

Following up on the earlier DMA-BUF / PipeWire notes: we kept iterating on a portal + headless gamescope stack (not DRM lease / forged EDID). On an NVIDIA host (lea, 4K60 Moonlight client) that path is now stable, HDR-tagged, and looks good — including color, not just logs.

What works (current known-good)

Layer	Result
Capture	XDG portal ScreenCast → gamescope (xdg-desktop-portal-gamescope), capture_transport=dmabuf, GPU-resident frames
Convert	convert_path=vulkan_cuda: LINEAR one-plane BGRx/BGRA DMA-BUF → Vulkan import → GPU copy → OPAQUE_FD → CUDA → existing RGBA→NV12 kernel → NVENC (no GL convert, no silent mmap)
Fallback	Sticky loud mmap_cuda only if the Vulkan bridge fails (honest CPU-residency stats; not reported as fake gpu_native)
Escape	POLARIS_PORTAL_DMABUF=0 → portal SHM / CUDA RAM path still available
HDR	Client profile HDR → Rec.2020 + SMPTE 2084 PQ, 10-bit, usable mastering metadata; user-confirmed good color + stable frames
Present	Steam/Proton XWayland attach into idle gamescope; do not set ENABLE_GAMESCOPE_WSI / ENABLE_HDR_WSI on that path (WSI env caused CreateSwapchainKHR / layer-hook failures and washed present earlier)

SDR clients (HDR off in profile) and forced-SDR apps still work on the same host.

What we deliberately did not need for this result

• DRM lease + forged EDID (interesting lead from the issue body; not required for portal gamescope on this machine)
• Desktop NVIDIA cuImportExternalMemory(DMABUF_FD) or cuGraphicsEGLRegisterImage — both fail on desktop (Tegra/Jetson-oriented); Vulkan DMA-BUF import + OPAQUE_FD export is the path that works
• Sunshine-style GL BGRA→NV12 as the steady-state portal convert (works, but was ~8–10 ms-class convert tax @ 4K; we left that for non-portal capture)

Architecture sketch

gamescope (headless HDR idle / force-HDR when client wants HDR)
    → portal / PipeWire LINEAR DmaBuf (BGRx/BGRA)
    → Vulkan (VK_EXT_external_memory_dma_buf) + GPU copy
    → CUDA OPAQUE_FD map + kernel → NVENC
    → Moonlight client (HDR or SDR profile)

Patch / packaging tree

Research packaging and the topic patch series live in:

https://github.com/luxus/polaris-hdr-linux-patches

Polaris topics applied on top of master (roughly):

1. Portal PipeWire + same-GPU DmaBuf negotiation + CUDA path (+ prefer xBGR_210LE)
2. Portal HDR metadata + force-file gate for session HDR
3. Web UI session persist
4. Force 8-bit NV12 for non-HDR streams
5. LINEAR DmaBuf → Vulkan→CUDA bridge (05-portal-dmabuf-vulkan-cuda) with loud mmap fallback

gamescope side: PipeWire HDR metadata, headless colorimetry, prefer-dmabuf. Portal package: stream-size fix.

Issue tracker there has the closed transport/encode work and remaining product items (Web UI path labels, first-class “Gamescope Stream” mode vs mislabeled “Mirror Desktop”, optional nested WSI).

Takeaways that may help upstream / #152 design

1. Portal + idle gamescope is a practical HDR host path without virtual HDR in labwc and without connector lease/EDID forges — at least on single-GPU NVIDIA with a focusable gamescope session.
2. Capture residency ≠ convert path. Logging capture_transport=dmabuf while convert is mmap/CPU will lie to dashboards; expose convert_path (we use vulkan_cuda / mmap_cuda / SHM-ram).
3. Desktop NVIDIA has no direct dmabuf→CUDA import for gamescope LINEAR buffers; Vulkan bridge (or GL bind) is the realistic GPU path.
4. WSI env is not “turn on HDR” for the attach topology; keep WSI for a real nested gamescope child if you need it, separate from portal capture color.
5. Encode UI ms still ≈ convert + NVENC; GPU convert removes the bad paths but 4K60 Main10 budget is still mostly encode.

Happy to help test any upstream-shaped subset of the portal / convert_path / Vulkan bridge pieces against perf/issue-152-pipewire-capture or master when useful. Thanks for the careful SDR-foundation / honest-fallback direction earlier — that matched how we unblocked this.