Skip to content

Fix(uia): resolve selection retrieval failure with tabindex parent#14

Closed
sdp-ncd wants to merge 2 commits into
0xfullex:mainfrom
sdp-ncd:sdp-ncd-patch-1
Closed

Fix(uia): resolve selection retrieval failure with tabindex parent#14
sdp-ncd wants to merge 2 commits into
0xfullex:mainfrom
sdp-ncd:sdp-ncd-patch-1

Conversation

@sdp-ncd

@sdp-ncd sdp-ncd commented Feb 6, 2026

Copy link
Copy Markdown

[WIP] tested but not code reviewed

Description

This PR fixes an issue where text selection retrieval via UI Automation fails when the selected text resides within a parent element having tabindex="0".

Problem

In modern web browsers (Chrome/Edge), setting tabindex="0" on a container element (like a <div>) causes it to receive UI Automation focus instead of the specific text node or the document root. Since these container elements often do not implement the TextPattern interface (or delegate selection management to the document root), the previous implementation would fail to retrieve the selection, returning an empty result.

Additionally, the ElementFromPoint strategy would often target leaf nodes (like <span>) that also lack TextPattern support, missing the actual selection managed by ancestor nodes.

Solution

  • Implemented an ancestor traversal ("Walk Up") mechanism. When the initial targeted element (either from mouse position or focus) does not return a valid selection, the code now walks up the UI Automation tree (up to 10 levels) to find an ancestor (typically the Document element) that supports TextPattern and contains the active selection.
  • Refactored the text extraction logic into a reusable lambda helper (TryGetTextFromElement) to reduce code duplication and improve readability.
  • Prioritized the ElementFromPoint strategy with the new walk-up logic to better handle mouse-based selections.

Changes

  • Modified SelectionHook::GetTextViaUIAutomation to include ancestor traversal loops.
  • Added TryGetTextFromElement helper for consistent pattern checking.
  • Updated both mouse-based and focus-based strategies to use the new traversal logic.

Impact

This ensures reliable text selection retrieval in complex HTML structures involving tabindex and nested elements, significantly improving compatibility with web-based applications.

@sdp-ncd
sdp-ncd marked this pull request as draft February 6, 2026 10:23
@sdp-ncd

sdp-ncd commented Feb 11, 2026

Copy link
Copy Markdown
Author

@0xfullex I tested this fix, and it works correctly—it retrieves the content of child elements within the parent element that have tabindex=....

However, it introduced a new issue: it copies some icons from the webpage, like “\n”.

Do you think this should be handled at the C++ layer or the JavaScript layer? I believe handling it at the JavaScript layer would be more reasonable.

我这边尝试了一下,这个修复是正确的,能够获取父元素包含tabindex=...的子元素内容。

但是引发了一个新的问题:会把网页中的一些图标复制出来,类似于“\n”,因为这些元素返回给 UIA 的内容就是这个字符,表示这是一个"嵌入式对象"

你觉得是在c++层做处理呢?还是在js层做处理呢?在js层做处理可以由客户端自定义进行替换,例如替换成 [图标] 之类的内容。

但是对于系统原生的 ctrl+c 实际上只是简单地直接略过这个图标

@sdp-ncd

sdp-ncd commented Feb 11, 2026

Copy link
Copy Markdown
Author

最新的代码已经在c++原生层做处理了,最终的复制效果和直接 ctrl+c 一致,不会在文本中包含[obj]

@0xfullex 0xfullex left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR Review: Fix(uia): resolve selection retrieval failure with tabindex parent

Is this a real problem?

Yes. In modern browsers (Chrome/Edge), setting tabindex="0" on a container element causes it to receive UI Automation focus instead of the document root. Since these containers typically don't implement TextPattern, the original code fails to retrieve the selection. This is a well-known UIA focus mismatch issue.

Solution Assessment

The ancestor walk-up mechanism is the correct approach — walking up the UIA tree to find an ancestor (usually the Document element) that supports TextPattern and holds the active selection. The TryGetTextFromElement lambda and the ElementFromPoint-first strategy are solid improvements that reduce code duplication and improve accuracy for mouse-based selections. The RemoveObjectReplacementChars handling for U+FFFC is also correct and consistent with native Ctrl+C behavior.

Issues Found

1. SetTextRangeCoordinates return value ignored (Severity: High)

Before:

result = SetTextRangeCoordinates(pRange, selectionInfo); // failure = not successful

After:

SetTextRangeCoordinates(pRange, selectionInfo); // ignored, returns true regardless

This could return text with invalid coordinates to downstream consumers.

2. VT_ARRAY selection handling removed (Severity: Medium)

The original handled IAccessible::get_accSelection returning VT_ARRAY (multi-selection scenarios). This entire code path was removed without explanation. Even if uncommon, this is a functionality regression.

3. ExpandToEnclosingUnit fallback removed (Severity: Medium)

The original had a fallback that called pDocRange->ExpandToEnclosingUnit(TextUnit_Document) when the initial DocumentRange query didn't find an active selection. This fallback was removed, which could cause regressions with certain UIA providers that don't return the document range directly.

4. get_accName / get_accValue fallback behavior changed (Severity: Low)

Before: Checks SysStringLen(bstr) > 0 after get_accName; if empty, frees bstr and falls through to get_accValue.

After: Uses || short-circuit — if get_accName returns a non-null but empty/unhelpful BSTR, it won't attempt get_accValue.

5. uia_control_type semantics changed (Severity: Low)

Previously reflected the focused element's ControlType; now reflects the ancestor element that successfully returned text (e.g., Document instead of the originally focused element). Could affect downstream logic that depends on this value.

Summary

Aspect Verdict
Problem validity Real and meaningful
Core approach (Walk Up) Correct
ElementFromPoint priority Good improvement
RemoveObjectReplacementChars Correct
Code structure Improved (less duplication)
Feature completeness Regressions (issues 1-3)

The core fix is valuable. Consider restoring the SetTextRangeCoordinates return value check, the VT_ARRAY handling, and the ExpandToEnclosingUnit fallback before merging.

@ShirasawaSama

Copy link
Copy Markdown

I'll fix these issues.

@0xfullex

0xfullex commented Jul 10, 2026

Copy link
Copy Markdown
Owner

Thanks again for the investigation and this PR — the ancestor walk-up approach was exactly right, and it has now been adopted on main (fac4ef0 through a96d9ac, released in v2.0.2).

The fix was re-implemented on top of the current main so the existing retrieval logic is fully preserved (the items from the earlier review: SetTextRangeCoordinates result check, VT_ARRAY handling, ExpandToEnclosingUnit fallback, get_accName/get_accValue semantics, uia_control_type timing), with the walk-up added as a last resort inside the UIA path:

  • After the focused-element TextPattern and LegacyIAccessible attempts fail, walk up the ancestor chain (max 10 levels) retrying TextPattern
  • Ancestors may only succeed via GetSelection — the document-range fallbacks stay focused-element-only, so non-standard providers cannot return whole-document text as a false positive
  • Your RemoveObjectReplacementChars idea is included, extended to also strip the line breaks Chromium wraps around the marker (\n\n was the full pattern observed on-device), so inline icons no longer leave stray line breaks
  • macOS got a matching ancestor walk-up as well (fixes cross-row selections falling back to clipboard)

Closing as superseded by the main-branch fix. Thanks for your investigation and contribution — the walk-up idea directly led to the final fix!

@0xfullex 0xfullex closed this Jul 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants