Skip to content

Harden untrusted page-fetch and redirect boundaries - #3

Open
yinli-systems wants to merge 1 commit into
mainfrom
agent/harden-page-fetch-boundary
Open

Harden untrusted page-fetch and redirect boundaries#3
yinli-systems wants to merge 1 commit into
mainfrom
agent/harden-page-fetch-boundary

Conversation

@yinli-systems

Copy link
Copy Markdown
Owner

What changed

  • validate page-fetch URLs before network access, allowing only HTTP(S) public-looking DNS names or globally routable IP literals
  • reject credentialed URLs, dotless/local hostnames, and loopback/private/link-local/reserved/multicast/unspecified IP literals
  • replace automatic redirects with a bounded five-hop loop that validates every target before the next request
  • preserve the existing snippet fallback when a target is unsafe or unreachable
  • add focused SSRF-boundary regressions, CPU CI, and a README section that states both the protection and its DNS-rebinding limitation

Why

Search result URLs are untrusted. The previous follow_redirects=True fetch could request a redirect target without applying any destination policy, including a direct private or loopback address.

Impact

Common direct and redirect-based local-network fetches now fail closed to the provider snippet without taking down retrieval. This is an application guard, not a replacement for a resolver-aware production egress proxy.

This branch intentionally does not duplicate open PR #1 (package discovery) or #2 (Tavily provider).

Checks

  • .venv/bin/python -m pytest -q tests — 83 passed, 1 dependency deprecation warning
  • focused URL/redirect/fallback tests — 3 passed
  • .venv/bin/python -m compileall -q fast_rag tests — passed
  • git diff --check — passed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant