nl-ui-automation

Project name: nl-ui-automation (GitHub/repo). Local clone folder may differ (e.g. ai_ui_automation).

AI-driven natural language → spec → run → heal → promote UI automation for Web (Playwright CLI) and Android App (uiautomator2).

Why Skills orchestration?

This repo is designed to run through Cursor Skills (.cursor/skills/), not ad-hoc prompts. You describe intent in chat; a small set of entry skills routes to the right pipeline, and stage skills (spec-gen → run → heal → promote) are injected automatically by orchestrators and scripts/orchestrator/prompts.py.

Benefit	What it means for you
One entry, full pipeline	Say “SauceDemo shop flow, promote when green” — no need to `@` spec-gen, heal, login, form-fill one by one
Same path as scripts	Draft specs, `run_case.py`, reports, and heal loops match what the Agent does — reproducible outside chat
Safe iteration	Failures stay in `specs/*/draft/`; promote only when green — production specs stay clean
Optional AI layers	Guardian L0 rules work without API keys; LLM heal/orchestration is opt-in via `config/local.env`
Web + App + PRD	Router skill picks Web, App, record, explore, or req-suite — one repo, consistent workflow

You can still run demos with Python only (no Cursor). Skills are the recommended way to build and maintain cases at scale.

Module	Cases	Run command	Orchestrator skill
Web	`specs/web/` + `examples/web/specs/`	`python scripts/run_case.py <spec.json>`	`playwright-ui-orchestrator-zh`
App	`specs/app/` + `examples/app/specs/`	`python scripts/run_app_case.py <spec.json>`	`app-ui-orchestrator-zh`

Configuration: config/README.md

Skills map

Details: .cursor/skills/<name>/SKILL.md. Users only need the 7 entry skills below.

flowchart TB
  subgraph L1["Layer 1 — users only need these (7)"]
    R[router]
    R --> W[Web orchestrator]
    R --> A[App orchestrator]
    R --> AR[App record orchestrator]
    R --> Q[PRD suite]
    R --> E[Web explore]
    R --> X[Export Excel]
  end

  subgraph L2["Layer 2 — orchestration stages (ignore)"]
    SG[spec-gen → draft]
    PF[preflight]
    RUN[nl2spec → run]
    H[heal on failure]
  end

  subgraph L3["Layer 3 — reference (rarely @)"]
    FF[form-fill]
    GD[guardian]
    LG[login]
    LA[launch]
    R2[req2excel]
  end

  W --> SG --> RUN --> H
  A --> SG --> PF --> RUN --> H
  AR --> RUN --> H
  Q -.-> R2
  SG -.-> FF
  H -.-> FF
  RUN -.-> LG
  RUN -.-> GD
  A -.-> LA

Entry skills (7)

Say what you want in Cursor — no need to @ skill names.

You want to…	Example prompt	Entry skill
Web regression: NL → run → heal → promote	“SauceDemo login, add to cart, promote when green”	`playwright-ui-orchestrator-zh`
Explore a module + HTML report	“Explore SauceDemo inventory and write a report”	`playwright-ui-explore-zh`
App regression on device	“Open Android Settings Wi-Fi”	`app-ui-orchestrator-zh`
Record device taps → spec	“Record on phone, then generate a test case”	`app-ui-record-orchestrator-zh`
PRD → Excel → batch run	“Turn this PRD into cases; run after I approve Excel”	`req-suite-orchestrator-zh`
Promoted JSON → review Excel	“Export demo login spec to xlsx”	`playwright-ui-spec-export-excel-zh`
Not sure	“Help me automate this UI”	`ui-automation-router-zh`

Do not confuse

Intent	Use	Not
Exploratory test + report	`playwright-ui-explore-zh`	`playwright-ui-orchestrator-zh`
Record device taps	`app-ui-record-orchestrator-zh`	`app-ui-orchestrator-zh`
PRD batch suite	`req-suite-orchestrator-zh`	`playwright-ui-spec-export-excel-zh`

Web / App NL pipeline: natural language → draft spec → run_case / run_app_case → heal → promote.

Stage skills (for Agent — no need to @)

Group	Skill	Role
Web draft	`playwright-ui-spec-gen-zh`	NL → `specs/web/draft/`
Web run	`playwright-ui-nl2spec-zh`	`run_case.py`
Web heal	`playwright-ui-heal-zh`	fix draft on failure
Web refs	`playwright-ui-form-fill-zh`, `playwright-ui-guardian-zh`, `playwright-ui-login-zh`	forms / recovery / login
App draft	`app-ui-spec-gen-zh`	NL → `specs/app/draft/`
App run	`app-ui-preflight-zh`, `app-ui-nl2spec-zh`	preflight, `run_app_case.py`
App heal	`app-ui-heal-zh`	fix draft on failure
App record	`app-ui-record-zh`, `app-ui-spec-from-record-zh`	record → draft
App refs	`app-ui-guardian-zh`, `app-ui-launch-zh`	recovery / launch config
PRD	`req2excel-zh`	PRD → Excel (via req-suite)

Sub-skills with disable-model-invocation are injected by scripts/orchestrator/prompts.py or entry orchestrators.

Quick start

1. Install

python -m venv .venv312          # Python 3.12+ recommended for orchestrator; any venv name is fine
.venv312\Scripts\activate        # Windows
# source .venv312/bin/activate   # macOS / Linux
pip install -r requirements.txt
npm i -g @playwright/cli
playwright-cli install-browser
python -m uiautomator2 init       # App only, once per device

2. Local config (optional for demos)

copy config\local.env.example config\local.env
# Edit CURSOR_API_KEY / GUARDIAN_LLM_* if using AI orchestration

3. Web — try the framework (recommended order)

No login — open any public site you can reach:

python scripts/run_case.py examples/web/specs/demo-open-example-com.json

SauceDemo full shop flow (sort, add 2 items, cart, remove — visible even with reused login state):

python scripts/run_case.py examples/web/specs/demo-saucedemo-shop-flow.json

First SauceDemo run or stale session? Add --login to refresh state/saucedemo-user-auth.json.

Other SauceDemo specs: demo-saucedemo-login.json (light assert only), demo-saucedemo-inventory.json (single add-to-cart).

4. Web — Captcha + state demo

# Terminal 1
cd examples\web\demo-site
python -m http.server 8765

# Terminal 2
python scripts/tools/run_login.py examples/web/logins/demo-captcha-user.example.json
python scripts/run_case.py examples/web/specs/demo-captcha-dashboard.json

Credentials: demo / demo123. Captcha OCR target is 1234 on the login page.

5. App — Android Settings

adb devices
python scripts/run_app_case.py examples/app/specs/demo-open-settings-wifi.json

Build your own Web automation

Three ways to go from natural language or any URL you can open in a browser to a runnable case.

Path 1 — Script only (no AI)

Copy a demo spec: examples/web/specs/demo-open-example-com.json → specs/web/my-first.json
Change the URL in step 1 to any site you can access (intranet, staging, public).
Add steps (打开 / 点击 / 填写 / 执行代码) or a small JS under scripts/run_code/.
Run:

python scripts/run_case.py specs/web/my-first.json

Report: reports/web/<runId>/report.html · screenshots: runs/web/<runId>/steps/

Path 2 — Cursor chat (natural language)

Open this repo in Cursor and describe what you want (skill playwright-ui-orchestrator-zh is picked automatically):

You say	Framework does
“Open https://example.com and check the h1 says Example Domain”	spec-gen → `specs/web/draft/` → run → heal
“On SauceDemo, sort by price low to high, add two products, open cart”	same pipeline
“Open https://playwright.dev and assert the page title contains Playwright”	same (no login if site is public)

Draft files land in specs/web/draft/; when green, ask to promote to specs/web/<name>.json.

Path 3 — CLI orchestrator (needs `CURSOR_API_KEY`)

python scripts/orchestrator/run_web.py "open example.com and assert Example Domain heading"
python scripts/orchestrator/run_web.py "saucedemo shop flow sort and add two items to cart"
python scripts/orchestrator/chat.py --platform web

Tips

Topic	Note
Public sites	No `登录配置` / `状态文件` — see `demo-open-example-com.json`
Login sites	Copy `examples/web/logins/*.example.json` → `config/web/logins/`, set `登录配置`
Reused login feels “empty”	Prefer multi-step demos like `demo-saucedemo-shop-flow.json`
Stale session	`python scripts/run_case.py <spec> --login` or delete `state/*.json`

Architecture

Pipeline: natural language → spec-gen → draft → run → heal (≤20 rounds) → promote → report

Layer	Path
Framework	`scripts/framework/`
Web executor	`scripts/web/` + playwright-cli
App executor	`scripts/app/` + uiautomator2 + adb
Artifacts	`runs/{web,app}/` → `reports/{web,app}/`

Web

Item	Location
Cases	`specs/web/*.json`, drafts `specs/web/draft/`
Demos	`examples/web/specs/`
Login	Demo: `examples/web/logins/`; yours: `config/web/logins/`; state `state/*.json`
Login engines	`web_login.py` → `web_simple_login.py`, `web_captcha_login.py`

App

Item	Location
Cases	`specs/app/*.json`, drafts `specs/app/draft/`
Demos	`examples/app/specs/`
Launch	`specs/app/launches/`, `examples/app/launches/`

Features

Capability	Web	App
JSON specs + step scripts	`specs/web/*.json`	`specs/app/*.json`
Login / launch	`simple` + `captcha` login specs, `state/`	`launches/*.json` + adb
Failure heal	`playwright-ui-heal-zh` (≤20 rounds)	`app-ui-heal-zh`
Guardian (L0 + optional LLM)	`specs/web/guardian/`	`specs/app/guardian/`
HTML reports	`reports/web/<runId>/`	`reports/app/<runId>/`
Cursor skills	`.cursor/skills/`	same

Scripts

Command	Purpose
`python scripts/run_case.py <spec.json>`	Web cases
`python scripts/run_app_case.py <spec.json>`	App cases
`python scripts/tools/run_login.py <login.json>`	Refresh Web login state
`python scripts/tools/probe_u2.py`	Check uiautomator2 device
`python scripts/tools/generate_run_report.py --platform web --latest`	Regenerate HTML report
`python scripts/tools/export_spec_to_excel.py <spec.json>`	Spec → review Excel → `exports/`

scripts/
  run_case.py / run_app_case.py
  framework/          # paths, reports, heal
  web/ app/           # platform executors
  orchestrator/       # Cursor SDK
  run_code/           # shared Web JS (draft/ for heal iterations)
  run_code_app/       # App Python steps
  tools/              # login, probes, PRD suite, Excel

Workflow modes

Mode	How	Uses AI tokens
A. Script only	`run_case` / `run_app_case`	No
B. CLI orchestrator	`run_web.py` / `run_app.py` / `chat.py`	Yes
C. Cursor editor	`@Skill` in chat	Yes

# Mode A
python scripts/run_case.py examples/web/specs/demo-saucedemo-login.json

# Mode B
python scripts/orchestrator/run_web.py "demo saucedemo login"
python scripts/orchestrator/chat.py --platform web

Case lifecycle

NL / PRD → spec-gen → draft → run → heal → promote → specs/web|app/

Promote checklist

Copy specs/*/draft/xxx.json → specs/{web,app}/
Move scripts from scripts/run_code/draft/ (or run_code_app/draft/) to production folders
Update @ paths (remove draft segment)

Draft vs final runs

	Draft (`specs/*/draft/`)	Final (`specs/*/` or `examples/`)
On failure	Auto-resume from `failure.json`	Full rerun from step 1
`--login`	Force re-login	Force re-login

Project layout

nl-ui-automation/
├── config/              # local.env, platform.json, web/logins/
├── examples/            # Runnable Web + App demos
├── specs/web|app/       # Your case JSON (empty template + draft/)
├── scripts/
│   ├── run_case.py      # Web entry
│   ├── run_app_case.py  # App entry
│   ├── run_code/        # Shared Web helpers (e.g. logout.js)
│   ├── framework/       # Reports, heal, paths
│   └── orchestrator/    # Cursor SDK automation
├── state/               # Web auth state (gitignored)
├── runs/ reports/       # Run artifacts
└── .cursor/skills/      # Agent skills

Where to put your cases

Asset	Location
Web cases	`specs/web/your-case.json`
Web step JS	`scripts/run_code/` or colocated under `examples/`
App cases	`specs/app/your-case.json`
App step Python	`scripts/run_code_app/`
Login configs	Copy `examples/web/logins/*.example.json` → `config/web/logins/` (gitignored)
Secrets	`config/local.env` only

Copy and adapt files from examples/ — do not commit real passwords or internal URLs.

Runnable demos

Demo	Spec
Public site (no login)	`examples/web/specs/demo-open-example-com.json`
SauceDemo shop flow (recommended)	`examples/web/specs/demo-saucedemo-shop-flow.json`
SauceDemo login assert only	`examples/web/specs/demo-saucedemo-login.json`
SauceDemo single add-to-cart	`examples/web/specs/demo-saucedemo-inventory.json`
Local captcha dashboard	`examples/web/specs/demo-captcha-dashboard.json`
Android Settings Wi-Fi	`examples/app/specs/demo-open-settings-wifi.json`

Examples layout

Path	Purpose
`examples/web/specs/`	Web demo case JSON
`examples/web/logins/`	Login templates
`examples/web/run_code/`	Web step scripts
`examples/web/demo-site/`	Static site for captcha demo
`examples/app/specs/`	App demo case JSON
`examples/app/launches/`	Package / Activity configs
`examples/app/run_code_app/`	App step scripts

Runtime folders: state/ (Web auth, gitignored) · exports/ (Excel from export_spec_to_excel.py) · assets/ (local upload fixtures, gitignored)

Screenshots & reports

Platform	Per-step screenshots	Click overlay
Web	`runs/web/<runId>/steps/`	Element highlight (`--no-show-click` to disable)
App	One per spec step	Red circles on same image (max 5)

Reports are auto-generated at reports/<platform>/<runId>/report.html. Overview: reports/index.html.

Regenerate from old runs:

python scripts/tools/generate_run_report.py --platform web --latest

AI orchestration (optional)

python scripts/orchestrator/run_web.py "demo saucedemo inventory flow"
python scripts/orchestrator/run_app.py "open android settings wifi"
python scripts/orchestrator/chat.py --platform web

Requires CURSOR_API_KEY in config/local.env. See Skills map at the top of this file.

Security / Git

Do not commit: config/local.env, config/web/logins/*.json (real creds), state/*.json, runs/.

git check-ignore -v config/local.env state/

License: MIT. Report security issues: SECURITY.md.

Requirements

Python 3.10+ (3.12+ recommended for orchestrator)
Node.js 18+ (@playwright/cli)
Android device + adb (App module)

FAQ

Issue	Fix
`playwright-cli` not found	`npm i -g @playwright/cli` or set `PLAYWRIGHT_CLI` in `local.env`
Web login / stale state	`python scripts/run_case.py <spec> --login` or delete `state/*.json`
Captcha demo unreachable	Start `examples/web/demo-site` on port 8765
App u2 fails	`python -m uiautomator2 init`, check `adb devices`; set `ADB` in `local.env` if needed
Guardian `missing_api_key`	Optional — set `GUARDIAN_LLM_API_KEY` in `local.env`; L0 rules still work
Settings demo Wi-Fi menu	Edit candidates in `examples/app/run_code_app/demo_tap_wifi.py`
Step 2 timeout on SauceDemo	Session expired — rerun with `--login`

Navigation Menu

Repository files navigation

nl-ui-automation

Why Skills orchestration?

Skills map

Entry skills (7)

Do not confuse

Stage skills (for Agent — no need to @)

Quick start

1. Install

2. Local config (optional for demos)

3. Web — try the framework (recommended order)

4. Web — Captcha + state demo

5. App — Android Settings

Build your own Web automation

Path 1 — Script only (no AI)

Path 2 — Cursor chat (natural language)

Path 3 — CLI orchestrator (needs `CURSOR_API_KEY`)

Tips

Architecture

Features

Scripts

Workflow modes

Case lifecycle

Project layout

Where to put your cases

Screenshots & reports

AI orchestration (optional)

Security / Git

Requirements

FAQ

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
Latest commit History 2 Commits
.cursor/skills		.cursor/skills
assets		assets
config		config
examples		examples
reports		reports
scripts		scripts
specs		specs
state		state
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
README.zh-CN.md		README.zh-CN.md
SECURITY.md		SECURITY.md
requirements.txt		requirements.txt

Folders and files

Latest commit

History

Repository files navigation

nl-ui-automation

Why Skills orchestration?

Skills map

Entry skills (7)

Do not confuse

Stage skills (for Agent — no need to @)

Quick start

1. Install

2. Local config (optional for demos)

3. Web — try the framework (recommended order)

4. Web — Captcha + state demo

5. App — Android Settings

Build your own Web automation

Path 1 — Script only (no AI)

Path 2 — Cursor chat (natural language)

Path 3 — CLI orchestrator (needs CURSOR_API_KEY)

Tips

Architecture

Features

Scripts

Workflow modes

Case lifecycle

Project layout

Where to put your cases

Screenshots & reports

AI orchestration (optional)

Security / Git

Requirements

FAQ

About

Topics

Resources

Security policy

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

Path 3 — CLI orchestrator (needs `CURSOR_API_KEY`)