Update scrapingbee-cli package to support Header based Authorization

[sahilsunny]

Switches API auth from the deprecated api_key query param to the Authorization: Bearer header, so the key no longer shows up in URLs. The only exception is crawl, whose scrapy-scrapingbee middleware will be migrated separately.

Also adds two params that are missing from the docs but confirmed working against the live API (it rejects unknown params with 400, these return 200):

• google --nb-results
• amazon-product --autoselect-variant

Stays on 1.6.0 since it isn't published yet; release date set to 2026-08-24.

Tested live on a pipx install (usage, GET, PUT, both new flags) plus unit tests for the new behavior. Based on #31 — merge that first, with a merge commit (not squash).

[kostas-jakeliunas-sb]

Regression: -X post/-X put with a user Authorization header now fails API auth (400/401)

tl;dr — with header-based auth, any -H "Authorization: …" that the user wants forwarded to the target site now overwrites our own Authorization: Bearer <api key> on POST/PUT, so the API rejects the request. On main the api_key query param kept auth working. Fix is a 4-line change (always Spb--prefix custom headers, not only on GET) — verified live + with a regression test below.

Where

scrapingbee-cli/src/scrapingbee_cli/client.py

Lines 233 to 238 in 52c8403

	req_headers: dict[str, str] | None = None
	if custom_headers:
	if method_upper == "GET":
	req_headers = {f"Spb-{k}": v for k, v in custom_headers.items()}
	else:
	req_headers = dict(custom_headers)

# src/scrapingbee_cli/client.py @ 52c8403
233        req_headers: dict[str, str] | None = None
234        if custom_headers:
235            if method_upper == "GET":
236                req_headers = {f"Spb-{k}": v for k, v in custom_headers.items()}
237            else:
238                req_headers = dict(custom_headers)          # <-- raw, unprefixed

Those raw headers are passed straight to aiohttp as per-request headers:

# src/scrapingbee_cli/client.py @ 52c8403
121        req_kwargs: dict[str, Any] = {"params": params}
122        if headers:
123            req_kwargs["headers"] = headers
...
133        async with session.request(method, url, **req_kwargs) as resp:

and the session-level default now carries our key (this PR, line 48):

 48        session_headers = user_agent_headers() | {"Authorization": f"Bearer {self.api_key}"}

Why it breaks

aiohttp merges session default headers with per-request headers, and the per-request value wins on the same (case-insensitive) key. So on POST/PUT a user -H "Authorization: Basic …" replaces Authorization: Bearer <api key> before the request leaves the client. The GET branch is unaffected because it rewrites the key to Spb-Authorization. Before this PR params_clean["api_key"] = self.api_key (removed in d360217) meant the backend still found the key in the query string.

Reproduce (live API, 2026-09-04)

export SCRAPINGBEE_API_KEY=...   # any valid key
scrapingbee scrape https://httpbin.org/post -X post -d "x=1" \
  -H "Authorization: Basic Zm9vOmJhcg==" --forward-headers-pure true --verbose

main (1.5.2, 3874781):

HTTP Status: 200
Credit Cost: 1

this branch (52c8403):

Error: HTTP 400
{"errors": {"api_key": {"api_key": ["Missing data for required field."]}}}

Same with a user Bearer token for the target site (-H "Authorization: Bearer target-site-token"):

Error: HTTP 401
{"message": "Invalid api key: target-site-token"}

(the backend prefers the header over the query param, so that Bearer variant also fails on main — the Basic/Token/Digest case is the actual regression.)

Offline repro — pytest against a local fake API (fails on this branch: 2 failed, 1 passed)

# tests/unit/test_post_auth_header.py
import asyncio

from aiohttp import web

from scrapingbee_cli.client import Client


def _run_against_fake_api(method, custom_headers):
    """Return (headers, query) that the fake ScrapingBee endpoint received."""
    seen = {}

    async def endpoint(request):
        seen["headers"] = dict(request.headers)
        seen["query"] = dict(request.query)
        return web.json_response({"ok": True})

    async def run():
        app = web.Application()
        app.router.add_route("*", "/", endpoint)
        runner = web.AppRunner(app)
        await runner.setup()
        site = web.TCPSite(runner, "127.0.0.1", 0)
        await site.start()
        port = runner.addresses[0][1]
        try:
            async with Client("fake-key", base_url=f"http://127.0.0.1:{port}") as client:
                await client.scrape(
                    "https://example.com",
                    method=method,
                    body="x=1" if method != "get" else None,
                    custom_headers=custom_headers,
                    forward_headers_pure=True,
                    retries=0,
                )
        finally:
            await runner.cleanup()

    asyncio.run(run())
    return seen["headers"], seen["query"]


class TestUserAuthorizationHeaderDoesNotClobberApiAuth:
    def test_post_with_user_basic_auth_header_still_authenticates(self):
        headers, query = _run_against_fake_api("post", {"Authorization": "Basic Zm9vOmJhcg=="})
        assert headers.get("Authorization") == "Bearer fake-key", headers.get("Authorization")

    def test_put_with_user_token_header_still_authenticates(self):
        headers, query = _run_against_fake_api("put", {"Authorization": "Token abc"})
        assert headers.get("Authorization") == "Bearer fake-key", headers.get("Authorization")

    def test_get_with_user_auth_header_is_prefixed_and_keeps_bearer(self):
        headers, _ = _run_against_fake_api("get", {"Authorization": "Basic Zm9vOmJhcg=="})
        assert headers.get("Authorization") == "Bearer fake-key"
        assert headers.get("Spb-Authorization") == "Basic Zm9vOmJhcg=="

Output on this branch:

E   AssertionError: 'Basic Zm9vOmJhcg=='   (post)
E   AssertionError: 'Token abc'            (put)
2 failed, 1 passed

Suggested fix — prefix on every method, not only GET

--- a/src/scrapingbee_cli/client.py
+++ b/src/scrapingbee_cli/client.py
@@ -232,10 +232,10 @@ class Client:
         method_upper = (method or "GET").upper()
         req_headers: dict[str, str] | None = None
         if custom_headers:
-            if method_upper == "GET":
-                req_headers = {f"Spb-{k}": v for k, v in custom_headers.items()}
-            else:
-                req_headers = dict(custom_headers)
+            # Always Spb-prefix: the API only forwards prefixed headers (it
+            # strips the prefix in both forward modes), and a raw user
+            # Authorization header would replace the session Bearer key.
+            req_headers = {f"Spb-{k}": v for k, v in custom_headers.items()}

Why prefixing (rather than re-adding api_key to the query on POST) is the right call: the backend never forwarded raw custom headers on POST/PUT in the first place — only Spb- ones, and it strips the prefix in both --forward-headers and --forward-headers-pure mode. Measured on main against httpbin.org/post:

-X post with	--forward-headers true → target sees	--forward-headers-pure true → target sees
-H "X-Custom: 1"	(nothing)	(nothing)
-H "Spb-X-Custom: 1"	X-Custom: 1	X-Custom: 1
-H "Authorization: Basic …"	(nothing)	(nothing)
-H "Spb-Authorization: Basic …"	Authorization: Basic …	Authorization: Basic …

So the raw branch at line 238 was silently dropping every header anyway; prefixing fixes the auth clash and makes -H on POST/PUT actually work. With the patch applied: the three tests above pass, the existing tests/unit/test_client.py suite passes (74), and the live command returns HTTP 200 with httpbin showing "Authorization": "Basic Zm9vOmJhcg==" (i.e. the header now reaches the target; on main it never did).

Related, same code path: the --forward-headers help says "Use -H with Spb- prefix for GET", but the CLI already prefixes on GET, so a user who follows that hint gets Spb-Spb-X-Custom and the target receives Spb-X-Custom: 1 (verified on main). With the fix above the hint can simply go — or make the prefixing idempotent (if not k.lower().startswith("spb-")).

[kostas-jakeliunas-sb]

See comment, would be nice to fix this, apart from that otherwise looks good i think :)