bug(windows): foreground click on secondary monitor mis-routes to Progman (Desktop); element_token capability undeclared in tools/list

[wyuebei-cloud]

Summary

When running cua-driver (v0.23.2) on a Windows 11 multi-monitor workstation, two distinct issues occur during GUI automation:

1. Secondary-monitor foreground click mis-routing: Sending click(delivery_mode="foreground", coordinate=[X, Y]) to a target application maximized on a secondary monitor (Monitor 1, X > 1920) causes the synthetic click to land on the Primary Monitor's desktop (Progman, HWND 0x1022e). This steals focus and immediately fails the post-action foreground check with foreground_unavailable. Moving the same application to the Primary Monitor completely resolves this error.
2. element_token capability omitted in capability map: While cua-driver 0.23.2 strictly requires element_token or snapshot_id alongside element_index (rejecting bare indices with snapshot_id_required), the MCP tools/list capability manifest omits accessibility.element_tokens in its capability discovery map despite exposing it in inputSchema.properties. Client harnesses relying on capability checks rather than raw schema properties silently drop the token.

---

Environment & Test Rig Topology

• OS: Microsoft Windows 11 Pro [Version 10.0.26200.9278] (x86_64)
• Display Topology:
  • Monitor 0 (Primary, Flags=1): MonitorRect=(0, 0, 1536, 960), WorkRect=(0, 0, 1536, 912), DPI scale 125% (1920×1200 physical panel scaled to 1536×960 logical).
  • Monitor 1 (Secondary, Flags=0): MonitorRect=(1920, 0, 3840, 1080), WorkRect=(1920, 0, 3840, 1032), DPI scale 100%.
  • Virtual Desktop Metrics: SM_XVIRTUALSCREEN=0, SM_YVIRTUALSCREEN=0, SM_CXVIRTUALSCREEN=3840, SM_CYVIRTUALSCREEN=1080.
• Target Application: Bluebeam Revu 21 (x64) (WPF / Direct2D workstation application).
• Harness: Hermes Agent Desktop running on Windows 11 via MCP client.
• Model: Google Gemini 3.8 Flash (consuming native SOM screenshots).
• cua-driver Version: cua-driver 0.23.2 (release date 2026-08-31).

---

Workload / Test Scenario

Automating a PDF markup workflow in Bluebeam Revu:

1. Launch Bluebeam Revu with a multi-page document (sample-handbook.pdf).
2. Navigate across pages (ctrl+right / ctrl+left).
3. Activate the Rectangle markup tool (r), select blue line color (#FF0000FF), and draw a bounding box around the bottom-right page number on pages 3 and 5.

---

Observed Behavior & Failure Logs

Issue 1: Multi-Monitor Click Mis-Routing

When Revu is positioned on Monitor 1 (window rect: (1913, -7, 3847, 1039)):

{
  "action": "click",
  "app": "Revu.exe",
  "coordinate": [3228, 770],
  "delivery_mode": "foreground",
  "bring_to_front": true
}

Driver Result:

{
  "ok": false,
  "action": "click",
  "delivery_mode": "foreground",
  "message": "foreground_unavailable: exact target HWND 0x130cb4 or a verified same-process post-action window was not foreground after the click (actual foreground HWND 0x1022e)"
}

Querying HWND 0x1022e via Win32 confirms:

• Title: Program Manager
• Class: Progman
• Process: explorer.exe (Primary desktop surface on Monitor 0)

The target window (Revu) was foreground before the click, but the click physically struck Monitor 0's desktop, shifting system focus to Progman.

Issue 2: element_token Contract Desync

When calling element-index clicks:

{
  "action": "click",
  "element": 16
}

Driver Refusal:

click: bare element_index is not accepted; pass element_token, or snapshot_id together with element_index
code: snapshot_id_required

Checking _session.supports_capability("accessibility.element_tokens", tool="click") returns False, because the capability string is not advertised in driver capability metadata, even though element_token is present in click's inputSchema.properties.

---

Root Cause Analysis

1. SendInput Multi-Monitor Normalization:
   In Win32 SendInput with MOUSEEVENTF_ABSOLUTE:
   • Without MOUSEEVENTF_VIRTUALDESK, coordinates normalized to 0..65535 map strictly to the Primary Monitor (Monitor 0).
   • Coordinates referencing Monitor 1 (X: 1920~3840) either wrap around, clamp, or project onto Monitor 0 space.
   • When the synthetic click lands on Monitor 0's desktop, Progman gains focus, causing cua-driver's post-click foreground verification heuristic to detect a focus change and report foreground_unavailable.
2. Capability Advertising:
   cua-driver v0.23.2 tightened the element index validation to prevent stale snapshot clicks, but did not update the capability advertisement map to match the schema properties.

---

Proposed Fix Directions

1. Multi-Monitor Mouse Events:
   Ensure all SendInput mouse inputs on Windows:
   • Include MOUSEEVENTF_VIRTUALDESK when multi-monitor metrics indicate SM_CMONITORS > 1.
   • Normalize X/Y coordinates against the entire virtual desktop span:

   norm_x = ((x - SM_XVIRTUALSCREEN) * 65536) / SM_CXVIRTUALSCREEN
   norm_y = ((y - SM_YVIRTUALSCREEN) * 65536) / SM_CYVIRTUALSCREEN
   

2. Capability Map Parity:
   Register accessibility.element_tokens in the tool capabilities set whenever element_token is exposed in inputSchema.

---

中文说明

概述

在 Windows 11 双显示器物理工作站环境下运行 cua-driver (v0.23.2) 进行桌面自动化测试时，发现了两个底层实现问题：

1. 副屏前台点击焦点穿透：当目标应用窗口在副显示器（Monitor 1，X 轴范围 1920~3840）最大化运行时，调用 click(delivery_mode="foreground") 会导致物理点击偏移并击中主屏（Monitor 0）的桌面背景（Progman，HWND 0x1022e），导致目标窗口失焦，驱动报错 foreground_unavailable。将应用移至主屏后该问题立即消失。
2. element_token 协议能力广播遗漏：cua-driver 0.23.2 严格要求传递 element_token 或 snapshot_id，禁止裸 element_index；但在 MCP tools/list 的能力字典中未广播 accessibility.element_tokens，导致依据能力表判定的客户端（如 Hermes Agent）静默过滤了该参数。

硬件与软件环境

• 操作系统：Microsoft Windows 11 Pro [Version 10.0.26200.9278] (x86_64)
• 显示器物理拓扑：
  • Monitor 0（主屏）：MonitorRect=(0, 0, 1536, 960)，125% DPI 缩放（物理面板 1920×1200 逻辑映射为 1536×960）。
  • Monitor 1（副屏）：MonitorRect=(1920, 0, 3840, 1080)，100% DPI 缩放。
  • 虚拟屏幕跨度：SM_XVIRTUALSCREEN=0, SM_CXVIRTUALSCREEN=3840。
• 被测目标应用：Bluebeam Revu 21 (x64)（基于 WPF / Direct2D 渲染管线）。
• Agent 框架与底模：Hermes Agent Desktop (Windows) + Google Gemini 3.8 Flash（原生视觉多模态输入）。
• 驱动版本：cua-driver 0.23.2（2026-08-31 官方最新版本）。

修复思路建议

1. 多显示器 SendInput 坐标归一化：
   • 检查并显式加入 MOUSEEVENTF_VIRTUALDESK 标志位。
   • 采用全虚拟桌面宽度（GetSystemMetrics(SM_CXVIRTUALSCREEN)）而非主屏宽度作为 0~65535 归一化分母：

   norm_x = ((x - SM_XVIRTUALSCREEN) * 65536) / SM_CXVIRTUALSCREEN
   norm_y = ((y - SM_YVIRTUALSCREEN) * 65536) / SM_CYVIRTUALSCREEN
   

2. 能力清单同步：
   • 确保 tools/list 中的 capabilities 字典完整声明 accessibility.element_tokens。