Skip to content

Enhance verify-on-device skill with POS navigation and logcat verification#15576

Open
malinajirka wants to merge 2 commits intotrunkfrom
woomob-2552-iterate-on-verify-on-device-skill
Open

Enhance verify-on-device skill with POS navigation and logcat verification#15576
malinajirka wants to merge 2 commits intotrunkfrom
woomob-2552-iterate-on-verify-on-device-skill

Conversation

@malinajirka
Copy link
Copy Markdown
Contributor

@malinajirka malinajirka commented Mar 27, 2026

Fixes #woomob-2552

Summary

Restructures the verify-on-device skill for better POS coverage, lower token usage, and faster execution.

Changes

  • Split navigation into reference files — extracted inline navigation (194 lines) into references/main-app-navigation.md and references/pos-navigation.md with full feature trees
  • Added POS navigation coverage — screen identifiers, checkout/refund/bookings workflows, tracking events (previously absent)
  • Added logcat event verification — verify actions via analytics events instead of screenshots
  • Added context: fork — skill runs in isolated forked context, keeping main conversation clean
  • Added $ARGUMENTS — forked context receives the user's task
  • Eliminated routine screenshots — verification priority: logcat events > element lists > screenshots (last resort)
  • Added mobile-mcp + skill permissions to project settings so the skill runs without permission prompts

Benchmark Results (POS Checkout Flow)

Run Method Screenshots Tool Calls Tokens Duration
Before Agent (original skill) 5 27 43k 3.7 min
After context: fork (final) 0 18 n/a 1.4 min

33% fewer tool calls, 62% faster, zero screenshots, 5 logcat events verified.

Key Design Decisions

  • context: fork over Agent tool — declarative, always isolates regardless of how skill is invoked
  • Logcat-first verification — zero context cost vs screenshots which load images into context
  • Two reference files (main app + POS) — mirrors the real architectural boundary (different Activity, ViewModel patterns, navigation)
  • No milestone sub-agents — nested agent spawning isn't supported in forked contexts; direct navigation with logcat verification is efficient enough (18 tool calls / 85s for a full checkout)

Test plan

  • Ask claude to verify a certain UI flow on an emulator or a hw device.

- Expand description to clarify use cases (UI testing, POS flows, screenshots, etc.).
- Introduce logcat event verification navigation flows.
- Add references for Main App and POS navigation, detailing workflows, identifiers, and tracking events.
@dangermattic
Copy link
Copy Markdown
Collaborator

2 Errors
🚫 PR requires at least one label.
🚫 PR is not assigned to a milestone.
1 Warning
⚠️ This PR is larger than 300 lines of changes. Please consider splitting it into smaller PRs for easier and faster reviews.

Generated by 🚫 Danger

@wpmobilebot
Copy link
Copy Markdown
Collaborator

App Icon📲 You can test the changes from this Pull Request in WooCommerce Android by scanning the QR code below to install the corresponding build.

App NameWooCommerce Android
Platform📱 Mobile
FlavorJalapeno
Build TypeDebug
Build Number736
Version24.4-rc-1
Application IDcom.woocommerce.android.prealpha
Commit3966b0f
Installation URL78jhad7jea5v0
Automatticians: You can use our internal self-serve MC tool to give yourself access to those builds if needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants