Firecrawl MCP Gateway Production Kit

A local-first MCP web-access gateway kit for controlled crawl, search, and extract workflows.

Most web-access examples show how to call a provider. This repo focuses on the production handoff before agents can touch the web safely: local credentials, per-tool scopes, per-subject quotas, URL allow-lists, audit events, provider adapter boundaries, fallback decisions, and a quickstart that works without paid services.

Ships today:

• Local gateway dispatcher with crawl_url, search_web_fixture, and extract_page
• FastAPI endpoints for tool discovery, token creation, and tool calls
• HMAC-signed local demo tokens
• Per-tool scopes
• Per-subject quotas and local estimated cost units
• Default-deny URL policy with reviewed fixtures
• Local fixture provider and fallback decision records
• Structured audit events and JSONL fixture generation
• Pytest coverage for auth, URL policy, quotas, fallback, audit, and HTTP endpoints
• Public boundary scan
• Production docs for security, web policy, provider boundary, quotas, deployment, observability, and troubleshooting

Quickstart

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e '.[dev]'
pytest
python examples/local-only-demo/demo_client.py

Run the FastAPI app:

uvicorn prodkit_web_gateway.app:app --reload

List available tools:

curl -s http://127.0.0.1:8000/tools

Create a local token:

curl -s http://127.0.0.1:8000/demo/token \
  -H 'content-type: application/json' \
  -d '{"subject_id":"local-developer","scopes":["web:crawl","web:search","web:extract"]}'

Call a tool:

curl -s http://127.0.0.1:8000/tools/crawl_url \
  -H "authorization: Bearer $ACCESS_TOKEN" \
  -H 'content-type: application/json' \
  -d '{"arguments":{"url":"https://example.com/docs/gateway-safety"}}'

Generate audit fixtures:

python scripts/generate_audit_fixtures.py

Run the public boundary scan:

python scripts/scan_public_boundary.py

Tool Model

Tool	Scope	Estimated cost units	Purpose
crawl_url	web:crawl	3	Crawl one reviewed fixture URL
search_web_fixture	web:search	1	Search bundled fixture metadata
extract_page	web:extract	2	Extract text from one reviewed fixture URL

What This Is Not

This is not a hosted crawling platform, an arbitrary browsing tool, a bypass tool, or a replacement for legal review. It is a local reference kit for making web-access tool exposure easier to test and audit before remote deployment.

Production Guides

• Production docs map
• Security model
• Web-access policy
• Provider boundary
• Quotas and budget
• Audit events
• Deployment
• Observability
• Troubleshooting

License

MIT