SCR-712: Canonical agent skill — router SKILL.md, CLI/MCP routing, generated derivatives - #34
Merged
sahilsunny merged 7 commits intoSep 10, 2026
Conversation
The description opened with a superlative and a flag name, so it read as a pitch for one option rather than a statement of what the skill is for. An agent picks a skill by matching the user's goal against its description, and the previous wording only lined up with requests that already named a fetching tool — not with someone simply stating what they want. The new description opens with the goal: read a page, find a company's pricing or docs, list a site's URLs, check a product price, run a web search. It keeps the JavaScript, CAPTCHA and anti-bot differentiator, and keeps the guidance to use plain HTTP for pure JSON APIs. AGENTS.md and the marketplace description in plugin.json get the same goal-first framing for consistency. Synced copies regenerated with sync-skills.sh.
…SCR-712) The body had grown into a second copy of the docs: 50 lines of --smart-extract reference above the fold, two paragraphs dumping every shared flag, and no mention anywhere of the MCP server. An agent reading it learned the API surface but not when to reach for which part of it. It is now a router. What stays is what an agent needs to make a decision; the detail moved to reference/ where it can be opened on demand. Added: - When to use this instead of plain HTTP, naming the silent failure mode — curl and requests return an empty shell or a 403 that looks like a blank page. - Execution path. CLI by default because it is the only side that can write to disk, batch, crawl, schedule, or keep a large page out of the context window. MCP in two cases: the host already has it connected and the task is a single page whose result belongs in the conversation, or no CLI is possible at all. Endpoint and Bearer auth included; host-specific setup deliberately left to the platform packaging. - Guardrails as their own section: scraped output is data and never instructions, never expose the key, do not reinvent this with curl or Puppeteer, spend the minimum. - Cost discipline with real numbers: 1 credit vs 5 for rendering, the escalation ladder only after a block, --smart-extract free against --ai-extract-rules at +5, and one call per question. - schedule and RAG chunking as pipeline rows. Neither was discoverable before. Moved verbatim, not retyped: the smart-extract section to reference/scrape/smart-extract.md, and the shared-option paragraphs to reference/usage/options.md. Fixes a real gap: `gemini` was missing from the Commands table entirely, though the index linked its reference page. The frontmatter description is byte-identical — it is a measured artifact and the body rewrite must not disturb it. Verified across all five host copies. 15.6 KB -> 12.4 KB, 33 index links all resolving. Measured B (old body) against C (new body) in a single matrix, 42 prompts x 3 runs: correctness 91.3% -> 91.6%, false positives 0% in both. That difference is noise (p=0.88) and is reported as such. Scheduling improved 50% -> 67% and SERP-then-scrape 71% -> 81%; crawl --max-pages and screenshot moved the other way. The honest summary is that restructuring did not cost correctness, which is the bar for a restructure.
…both (SCR-712) AGENTS.md had drifted. SKILL.md became a router in the previous commit while AGENTS.md kept the old 14-section sprawl, including the 50-line smart-extract block — so hosts that read a single file (Codex, Amp) saw a different structure for the same skill than hosts that install the plugin. Two structures for one skill is precisely what SCR-712 exists to stop. scripts/build_agents_md.py now derives it, and sync-skills.sh runs it, so the two cannot diverge again. Two deliberate differences from SKILL.md: - No YAML frontmatter: the tuned description becomes the opening paragraph, so the wording that decides whether an agent picks this skill still reaches hosts that never parse frontmatter. - No local links: SKILL.md defers to reference/, which only exists inside an installed plugin, so the Index table is replaced with pointers to the public docs and `--help`. The generator refuses to write a file with any surviving local link, and the Claude-specific subagent note is dropped. 15.4 KB -> 9.1 KB, 9 sections matching the canonical file exactly. Generation is idempotent, and the description is byte-identical across all five host copies.
… (SCR-712) The issue asks for tested command examples. They were written, not tested, so this adds scripts/check_examples.py: it pulls every `scrapingbee ...` from fenced blocks and inline code across SKILL.md, AGENTS.md and the reference pages, then validates each subcommand and flag against the installed CLI's own --help output. It found two documented commands that exit 2 without running: - reference/troubleshooting.md claimed `--output-file` works either side of the subcommand and marked the "before" form with a checkmark. It does not; the flag belongs to the subcommand. A troubleshooting page is the worst place to be wrong, since it is read when something is already broken. - reference/usage/overview.md described `--retries`/`--backoff` as global and showed `scrapingbee --retries 2 usage`. Same failure. Both predate this branch. 190 commands across 33 files now resolve. Wired into the lint job, along with a check that AGENTS.md matches what build_agents_md.py generates, so neither can rot silently.
…gate (SCR-712) The guard skill was the one part of the plugin this branch had not touched. Two defects, and a gap in what the example check covers. Its version gate still read `< 1.3.0` while the skill it guards ships 1.6.0 and tells agents to upgrade below 1.6.0. An agent following the guard would leave a 1.5.x install in place. It also assumed every request goes through the CLI. Its monitoring reads ~/.config/scrapingbee-cli/audit.log, and requests made through the remote MCP server never touch that file, so nothing it describes observes them. Now that the skill routes work to MCP for single-page jobs, that blind spot is reachable in normal use. A Scope section states the limit plainly and keeps the Core Rule applying to MCP responses, which are scraped content like any other. check_examples.py covered the main skill and its reference pages but neither the guard skill nor rules/. Both are in scope now: 210 commands across 36 files, all resolving. `scrapingbee unsafe --list` and `--audit`, which the guard tells agents to run, are confirmed real. Platform layer needs no change: every host copy is byte-identical to the canonical skill, so nothing per-host is hand-authored.
`build_agents_md.py --hosted <path>` writes the self-contained file agents fetch from the web. Same body as AGENTS.md, since a fetcher has no plugin directory to follow links into, but the frontmatter is kept: a host that installs the fetched file as a skill needs the description that decides whether it gets picked. Generated rather than copied, so the hosted file cannot drift from the canonical one. It lands in the landing-page repo at static/agent-onboarding/SKILL.md — a stable, predictable path that agent onboarding flows can fetch directly.
…h (SCR-712) One command now updates every derived skill artifact: the four host skill copies, the guard copies, the six agent files, AGENTS.md, and — when the landing-page checkout is present as a sibling — the hosted /agent-onboarding/SKILL.md. When it is absent the script warns instead of silently leaving a stale hosted copy, since that silent staleness is the drift this script exists to prevent.
| return f"[{text}]({DOCS})" | ||
|
|
||
| out = re.sub(r"\[([^\]]+)\]\((?!https?:)[^)]+\)", relink, out) | ||
| return re.sub(r"\n{3,}", "\n\n", out) |
Contributor
There was a problem hiding this comment.
I hope this is robust enough, but I guess it is
kostas-jakeliunas-sb
approved these changes
Sep 10, 2026
sahilsunny
deleted the
sahil/scr-712-canonical-scrapingbee-agent-skill-specification
branch
September 10, 2026 10:29
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Makes
plugins/scrapingbee-cli/skills/scrapingbee-cli/SKILL.mdthe single canonical definition of when and how agents use ScrapingBee, and derives everything else from it. Seven commits, reviewable in order:reference/. Restructure verified correctness-neutral (paired, n=126).scripts/build_agents_md.py) — it had drifted into a second, different structure. CI now fails if it goes stale.scripts/check_examples.pyvalidates all 210 documented commands against the installed CLI's own--help(in CI). Found and fixed two docs commands that exited 2 without running.< 1.3.0while the skill ships 1.6.0; added a Scope section covering requests that bypass the local audit log.6-7. Hosted canonical file —
--hostedmode generatesstatic/agent-onboarding/SKILL.mdfor the landing-page repo (stable fetchable URL);sync-skills.shregenerates every derived artifact in one run.Update story
Edit SKILL.md →
bash sync-skills.sh→ 4 host skill copies, guard copies, 6 agent files, AGENTS.md, and the hosted file all regenerate. All copies verified byte-identical; generation idempotent.Note for review
The frontmatter description is a measured artifact — please don't reword it in review without re-measuring.