[Bug]: another client’s screenshot capture changes pixel-click coordinates

[injaneity]

Primary area

Cua Driver

Summary

Another client's screenshot capture can overwrite the coordinate transform used to interpret an earlier screenshot, causing a pixel click to land in the wrong region of an unchanged window. Track the defect and architectural follow-up here rather than continue the independent session-registry implementation in #3628.

Reproduction

1. Connect two persistent MCP clients, A and B, to the same daemon.
2. Use a disposable native fixture with a visible target and a mouse-event receiver. Keep the window geometry unchanged.
3. Set A's session max_image_dimension to 200 and B's to 0 (native resolution).
4. A calls get_window_state for the fixture and selects a target coordinate in its delivered image. A clicks it as a baseline.
5. A captures again. B captures the same window at native resolution. Without refreshing, A clicks the identical image coordinate.
6. A refreshes its screenshot and repeats the click.

Use explicitly selected foreground delivery to the disposable fixture for the demonstrated native reproduction. Background delivery was separately refused and is not evidence of the wrong-position result.

Expected behavior

A pixel action must not silently reinterpret coordinates using another capture's transform. It should use the frame associated with the observation or explicitly refuse stale/unavailable context, according to the reviewed compatibility contract.

Actual behavior and evidence

The window remained 1470×932 points at (40,30), with native backing scale 1.0. A received a 200×127 image and B received 1470×932. The selected A-image point was (99.5,62.5).

Sequence	Native receiver landing, window-top coordinates	Outcome
A capture → A click	(731.325,459.375)	Target
A capture → B native capture → A click	(99.5,62.5)	Wrong region
A refresh → A click	(731.325,459.375)	Target

The correct resize correction is 1470/200 = 7.35. The native receiver's mouse events establish the misdelivery; driver responses reported an unverifiable effect and are not success evidence. The reproduction used a disposable fixture, not existing user browser tabs.

This is installed-version diagnostic evidence, not exact-main or candidate certification. No claim is made about the cause of any older incident lacking its complete request/image sequence.

Environment

• Component: installed macOS cua-driver 0.23.2.
• Native application: disposable AppKit fixture; two persistent MCP connections to one daemon.
• Binary SHA-256: 67ccfc99e69ebb5881fdfc3787d85abcd8cc2beb7423f255549557623cab6907.
• Reported source SHA: null. Exact OS build and architecture are not asserted by this report.
• Source inspection/characterization baseline: b1170930589691fd1101f6943410f32e57bda8b6.

Mechanism and architectural relationship

The macOS resize registry is keyed by PID/window rather than observation ownership. A resized capture stores a ratio; an unscaled capture clears it; pixel actions consult that mutable latest value. Linux and Windows used PID-only keys, additionally permitting cross-window replacement. Native wrong-position reproduction is macOS-only so far; source inspection is not Windows/Linux native certification.

Related records:

• cua-driver-rs (Windows): get_window_state omits the screenshot resize ratio, and the coordinate-space docs have gaps (agent cursor not visible in window captures) #1882 covers missing resize metadata and coordinate documentation. Metadata alone does not prevent transform replacement.
• RFC [RFC]: simplify Cua Driver state and actions using pi-computer-use concepts #3473 and its active implementation test(cua-driver): characterize snapshot invariants before refactoring #3616 consolidate snapshot identity and lookup payload ownership. Screenshot transforms are interpretation data with the same ownership concern.
• fix(cua-driver): isolate screenshot transforms between clients #3628 is the superseded session-isolation prototype, retained for evidence and reusable regression coverage.

Proposed direction for review: make screenshot interpretation data part of the existing snapshot publication/resolution/retirement contract, rather than add another independently authoritative mutable registry. Identify which old storage/conversion paths disappear. Keep platform-native geometry validation and delivery mechanics in their adapters.

This proposal does not authorize scope expansion of #3616. Its selected scope preserves public contracts and newer-read replacement. New public image identifiers, historical actionable screenshots, geometry-freshness changes, and changed refusal behavior require explicit review. The architectural decision remains in #3473; this issue records the defect and acceptance evidence. Do not start a competing implementation while #3616 is active without recording why it is being superseded.

Preserved prototype and validation

Prototype commits remain in #3628 on branch fix/session-owned-screenshot-transforms, final head 21fad867081f6b69d91418c39572aededdbace22.

It introduced shared session/window resize ownership, cleanup, missing-context refusal, core/trusted-dispatch tests, and same-daemon two-client regressions in existing AppKit, WPF, and supported GTK3 pixel rows. Session isolation contains the demonstrated interference but still allows another capture in the same session to replace the earlier transform.

Recorded validation:

• Core suite: 620 passed (615 library and 5 integration), one documentation test ignored, on final local source before the last commit.
• macOS library suite: 365 passed, 2 ignored, at dfd15de01f00b85d3685aba9039e5529ffcd5acc.
• Testkit library tests and AppKit regression compilation passed after the peer-connection helper was added.
• Native candidate replay was not certified: a private source-built daemon reported Accessibility=false and Screen Recording=false. It was stopped without modifying TCC or replacing the authorized installed application.
• The prototype's new missing-context refusal can precede Wayland background-unavailable refusal. This is an unresolved compatibility concern, not an accepted contract change.

See #3628 for its CI history and detailed validation record. Reuse its tests where appropriate; the prototype is not a shipped or certified fix. Preserve commit provenance and contributor credit when moving or materially adapting it.

Acceptance evidence

• Record maintainer selection of the screenshot-transform scope in [RFC]: simplify Cua Driver state and actions using pi-computer-use concepts #3473 / test(cua-driver): characterize snapshot invariants before refactoring #3616, or explicitly identify a separately approved scope.
• Demonstrate that A's coordinates cannot be interpreted using B's transform, including differently resized and native-sized captures of the same window.
• Cover independent windows, session retirement, and late capture completion.
• Define and test same-session replacement and stale-context behavior without silently changing the public contract.
• Publish, resolve, and retire interpretation data with its authoritative snapshot owner; remove replaced storage rather than retain parallel authorities.
• Preserve explicit platform limitations and test refusal ordering, especially on Wayland.
• Run focused platform coverage and the canonical desktop matrix at the exact stable candidate SHA before readiness. Unit arithmetic and installed-binary reproduction do not substitute for native candidate certification.

Workaround

Refresh immediately before acting and serialize competing captures as a temporary mitigation. Refresh alone is not a guarantee if another capture can interleave before the action.

Submission checks

• Searched duplicate issues and active pull requests.
• This issue describes one defect.
• No credentials, private UI content, sensitive screenshots, or vulnerability details are included.