Nice approach — local-first crawling with no API keys solves a real cost problem for agent research workflows.
One thing worth discussing for anyone hitting protected targets: sites behind Cloudflare, DataDome, or heavy fingerprint-based detection will block requests based on browser fingerprint consistency, not just rate or IP. Local-first crawling helps with cost but doesn't change that layer.
For that complementary piece, I maintain a curated resource on browser fingerprinting and anti-detection:
👉 https://github.com/pencil20388-eng/awesome-anti-detect
A setup that works for high-detection targets:
- wigolo for the search/fetch/crawl logic
- An antidetect browser (AdsPower, etc.) when a target needs full per-session fingerprint isolation
- Residential proxies matched to target geo when the target is geo-sensitive
Antidetect browsers expose a CDP endpoint per profile, so they can slot into browser-based crawling via Playwright/Puppeteer when a target can't be handled at the HTTP layer alone.
Sharing in case it's useful as the project grows into tougher targets.
Nice approach — local-first crawling with no API keys solves a real cost problem for agent research workflows.
One thing worth discussing for anyone hitting protected targets: sites behind Cloudflare, DataDome, or heavy fingerprint-based detection will block requests based on browser fingerprint consistency, not just rate or IP. Local-first crawling helps with cost but doesn't change that layer.
For that complementary piece, I maintain a curated resource on browser fingerprinting and anti-detection:
👉 https://github.com/pencil20388-eng/awesome-anti-detect
A setup that works for high-detection targets:
Antidetect browsers expose a CDP endpoint per profile, so they can slot into browser-based crawling via Playwright/Puppeteer when a target can't be handled at the HTTP layer alone.
Sharing in case it's useful as the project grows into tougher targets.