Investigate AMD headless labwc stutter / high host processing on RX 9060 XT

[papi-ux]

Bug Description

A user reports that after the headless startup crash was resolved, Polaris can now connect and stream from a private labwc/headless runtime on CachyOS + AMD RX 9060 XT, but the stream is stuttery and Moonlight reports host processing around ~14 ms. The same host/client/network reportedly averages ~2 ms host processing on vanilla Sunshine with spikes around ~4 ms.

This looks like a potential Polaris headless/AMD VAAPI performance regression or tuning issue rather than a network/pairing issue. The attached log shows the session starts successfully and stays up until the user manually force-stops it.

Environment

• OS: CachyOS, current around 2026-05-18/19
• GPU: AMD Radeon RX 9060 XT / gfx1200
• Kernel/DRM in log: 7.0.9-1-cachyos, DRM 3.64
• Mesa in log: Mesa Gallium driver 26.1.0-arch2.1 for AMD Radeon RX 9060 XT
• Client: Moonlight over wired LAN, local Steam Deck profile in log
• Polaris version in provided log: 1.0.18 commit c2bd3f4c719a92aad1f7ed611b8c9483b0f4781c
• Config highlights:
  • headless_mode = enabled
  • linux_use_cage_compositor = enabled
  • linux_prefer_gpu_native_capture = disabled per user report
  • adaptive_bitrate_enabled = enabled
  • output_name = 1
  • client profile loads HDR enabled for STEAM DECK LOCAL

Steps / Scenario

1. Pair Moonlight client.
2. Start a Headless Stream session through the private labwc runtime.
3. Stream starts successfully after the previous std::clamp crash fix.
4. User observes high/stuttery host processing in Moonlight compared with vanilla Sunshine on the same system.

Expected Behavior

Headless Stream on AMD VAAPI should not add a large host-processing penalty versus a comparable Sunshine setup when the stream is on a local wired LAN and using GPU capture/encode paths.

Actual Behavior

User reports:

• Polaris: host processing around ~14 ms and visibly stuttery.
• Vanilla Sunshine: ~2 ms average host processing, ~4 ms spikes, no stutter.

Evidence from Log

Pastebin supplied by user: https://pastebin.com/x9ifdiFn

Key lines from the log:

Polaris version: 1.0.18 commit: c2bd3f4c719a92aad1f7ed611b8c9483b0f4781c
config: 'headless_mode' = enabled
config: 'linux_use_cage_compositor' = enabled
config: 'output_name' = 1
...
labwc: Starting in headless mode — resolution=2560x1440@60Hz
labwc: Ready — headless compositor active, socket=wayland-1 startup_ms=105
...
wlr: Using ext-image-copy-capture DMA-BUF for headless labwc
Creating encoder [hevc_vaapi]
Color coding: SDR (Rec. 709)
Color depth: 10-bit
Color range: JPEG
HDR decision: client_dynamic_range=1 display_hdr=false hdr_metadata_available=false stream_hdr_enabled=false
vaapi vendor: Mesa Gallium driver 26.1.0-arch2.1 for AMD Radeon RX 9060 XT (radeonsi, gfx1200, ACO, DRM 3.64, 7.0.9-1-cachyos)
Using frame converter [avcodec_encode_device] target_device=vaapi target_residency=gpu target_format=p010
New streaming session started [active sessions: 1]
wlr: capture_transport=dmabuf frame_residency=gpu frame_format=bgra8

Other notable log details:

• Client requested 107500kbps, Polaris hosted at 20000kbps from the device/client profile.
• The stream used HEVC VAAPI 10-bit/P010 even though stream_hdr_enabled=false because the client profile/HDR request made client_dynamic_range=1.
• There are repeated dashboard preview capture failures against the active cage socket, but the log text says preview capture is separate from active stream. Still worth checking whether repeated grim preview attempts while the dashboard is open can create measurable contention on this AMD/labwc path.
• AI optimizer failed because claude CLI exited with code 32512, then fell back to cached/device heuristics. Probably unrelated to frame pacing, but present in the log.
• Session ended via Force stop from system tray; no host crash in this run.

Initial Triage / Hypotheses

Not enough data yet to prove a Polaris bug, but this is worth tracking as a potential Polaris performance issue because the user has a same-host Sunshine baseline.

Areas to investigate:

1. AMD headless DMA-BUF + VAAPI path may still have a conversion/synchronization cost despite logging GPU residency (bgra8 capture into VAAPI P010 encode path).
2. HEVC Main10/P010 may be slower on this new RDNA/gfx1200 stack than H.264 or HEVC 8-bit. Need ask user to retest with HDR/client HDR disabled and/or H.264/8-bit.
3. Dashboard preview polling may be hitting grim against the cage socket repeatedly while a stream is active. It should be separate, but we should verify it cannot contend with the active ext-image-copy capture path.
4. Adaptive bitrate / cached client profile selected 20 Mbps and tune=2; likely not the source of 14 ms host processing, but compare with Auto Quality disabled.

Requested Follow-up Data

• Retest on the current public release (v1.1.0 or newer), since the provided log is 1.0.18.
• Whether the Polaris web dashboard / Display Preview was open during the stutter test; if yes, retest with the dashboard closed or preview hidden.
• Moonlight settings: resolution/FPS, codec, HDR on/off, bitrate.
• Polaris /polaris/v1/session/status or /polaris/v1/stream-policy during an active stuttery session, especially capture path, gpu_native/cpu_copy, codec, color depth, encode time, FPS, and latency fields.
• A comparison log for a forced 8-bit/non-HDR H.264 or HEVC run if available.

Related Context

This report followed an earlier std::clamp abort during session startup on 1.0.13. User reports that updating resolved the crash and allowed virtual/headless streaming to start.

[IceDBorn]

I've tested headless streaming yesterday on my machine, everything worked as they would work under sunshine.

Distro: NixOS (packaged it myself, might contribute a flake.nix if you want, then I'll try PRing it into nixpkgs)
CPU: Ryzen 5 3600
GPU: RX 6900 XT
RAM: 16GB (32GB with swap)
DE: Cosmic
Kernel: 7.0.9 cachyos lto
Mesa: 26.1.0

[papi-ux]

Hell yes @IceDBorn , thank you for testing this out!

[8bitbuddhist]

I've tested headless streaming yesterday on my machine, everything worked as they would work under sunshine.

Distro: NixOS (packaged it myself, might contribute a flake.nix if you want, then I'll try PRing it into nixpkgs) CPU: Ryzen 5 3600 GPU: RX 6900 XT RAM: 16GB (32GB with swap) DE: Cosmic Kernel: 7.0.9 cachyos lto Mesa: 26.1.0

I'd love to try out your Nix package @IceDBorn! Mostly because I'm struggling to package myself 😅

[IceDBorn]

@8bitbuddhist Here you go

Polaris did not meet my needs though. I was in need of HDR, so, I hacked a solution together, using an unplugged port of my GPU, an injected fake valve index EDID with my desired resolution, refresh rate and HDR metadata into that port and a lease client, which requests the fake injected monitor from KDE plasma and launches steam inside of gamescope. Here's the code, @papi-ux could take a look and hack something together for polaris himself. That's semi-headless to be exact, for true headless, 2 issues have to close. One in sunshine's side and one in gamescope's side.

[papi-ux]

Thanks for the links and the HDR notes, it's super useful.

Current read: the RX 9060 XT stutter report and the HDR workaround are related only in the sense that both hit the Linux headless/display-contract boundary. For true HDR, your semi-headless path makes sense: Polaris/Moonlight need a display path that actually exposes HDR metadata. Pure Private Headless currently reports SDR/no HDR metadata, so Polaris should treat that as 10-bit SDR rather than pretending it is HDR. I just tightened that truth path in Nova/Polaris so users see the downgrade clearly instead of getting mysterious washed-out color.

For this specific performance issue, I still want a fresh v1.1.x+ retest before calling it fixed or closing it. Best comparison matrix would be: dashboard closed vs open, HDR/10-bit off vs on, HEVC 8-bit vs HEVC Main10, and H.264 if available, with a fresh Polaris log plus the active session status/policy JSON. The key thing I want to isolate is whether AMD headless DMA-BUF -> VAAPI P010 is adding sync/conversion cost on gfx1200, or whether preview/dashboard polling or the selected 10

Also yes, a clean flake/package PR would be welcome separately. I would keep packaging separate from this perf bug so this issue does not become a hydra.