[SCR-577] API parity: youtube-subtitles command, google --pages and --search-type ads - #31
Merged
Merged
Conversation
… ads (SCR-577) API-parity sweep against scrapingbee.com/llms.txt found three gaps: - youtube-subtitles: new command for /api/v1/youtube/subtitles (video ID or URL, --language, --subtitle-origin auto-generated/uploader-provided, batch via --input-file, 5 credits). Registered in CLI, REPL Media group, --scraping-config router, and ESTIMATED_CREDITS. - google --pages: fetch up to 10 consecutive pages starting at --page in one combined response; validated via new _validate_pages() helper (1..10). - google --search-type ads: classic-result structure optimized for paid-ad visibility. Version bumped to 1.6.0 (unreleased section holds features: Auto-Mode + these additions). Docs updated: CHANGELOG, README, AGENTS.md, canonical .agents/skills tree (new reference/youtube/subtitles.md, google overview) synced to all mirrors. Verified live against the API: subtitles returned real captions, --pages 2 combined two SERP pages, ads returned 200 JSON; bounds rejected locally. 853 unit tests green (new client param-forwarding, validator, and help tests); ruff + ty clean. Installed via pipx and smoke-tested normal mode and REPL (help, completion, live call, session-default warnings) with no regressions.
The google --help variant matched on the raw PTY stream, but the REPL's differential renderer can repaint straight to the final frame without ever emitting the scrolled-off warning line into the stream, so the assertion raced on slow CI runners (failed on ubuntu/3.11 in PR CI and macos/3.10 on main — different cells, same mechanism). Restore the 1.5.1-era approach: usage --help keeps the output short so the warning stays on screen, and the transient pump checks every intermediate screen state.
sahilsunny
requested review from
NBens,
dbulbukas-sbee,
kirby81,
kostas-jakeliunas-sb and
marius-nemeiksis-sbee
August 3, 2026 11:29
This was referenced Aug 18, 2026
Contributor
|
Ok, but doc-side inconsistencies, @sahilsunny ideally should fix before merge:
|
kostas-jakeliunas-sb
approved these changes
Sep 4, 2026
Collaborator
Author
|
@kostas-jakeliunas-sb , will fix this in #32, as it requires changes |
sahilsunny
added a commit
that referenced
this pull request
Sep 4, 2026
- Custom -H headers are now Spb--prefixed on every method (idempotently),
not only GET. On POST/PUT raw headers were silently dropped by the API,
and after the Bearer migration a user Authorization header replaced the
session's API key (aiohttp per-request beats session headers), failing
auth with 400/401. Red->green verified live with the reviewer's repro;
fake-API regression tests added (3 fail without the fix).
- Stale --forward-headers help ('use Spb- prefix') removed — following it
produced Spb-Spb- double prefixes; prefixing is now automatic.
- google --pages billing corrected in CHANGELOG and references: flat 10/15
per request regardless of page count (measured: --pages 1/2/2-from-2 all
cost 10). Window semantics (--page start, N consecutive) re-verified by
URL-overlap comparison and were already documented correctly.
- youtube-subtitles: a language with no subtitles returns HTTP 200 with an
empty object (not 404 as the API docs claim) and still charges 5 credits
(verified live) — the CLI now warns instead of printing silent empty
JSON; docs corrected; unit tests added.
- youtube-subtitles added to the error-response, integration, and e2e test
registries.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds three missing API features to the CLI and bumps the unreleased version to 1.6.0 (the unreleased section holds features — Auto-Mode from #26 plus these — so semver calls for a minor bump).
Changes
youtube-subtitlescommand —/api/v1/youtube/subtitles(5 credits): accepts a video ID or full YouTube URL,--language,--subtitle-origin(auto-generated/uploader-provided),--tag, and batch via--input-file, mirroringyoutube-metadata.google --pages— fetch up to 10 consecutive result pages starting at--pagein one combined response; bounds validated client-side.google --search-type ads— classic-result structure optimized for paid-ad visibility.Verification
--pages 2combined two SERP pages;adsreturned 200 JSON; out-of-range--pagesrejected before any request.test_scroll_up_reaches_content_above_wrapped_line, a pre-existing flake that fails identically onmain.)