Parallelize the document-scraping phase (bounded, per-host-polite) — embedding is already parallel

[swinney]

Objective

Parallelize archi's document-scraping phase (currently a fully sequential per-URL loop) with a bounded, per-host-polite thread pool, cutting re-ingest wall-clock. Embedding is already parallel — do not touch it; the target is the scrape/fetch loop only.

Context you need (all verified on origin/dev @ a8e06c79)

What's already parallel (leave alone): the embedding phase uses a thread pool — VectorStoreManager builds ThreadPoolExecutor(max_workers=self.parallel_workers) (src/data_manager/vectorstore/manager.py:578-581, logs "Processing files with up to 32 parallel workers"). parallel_workers comes from data_manager.parallel_workers (config default 32 — src/cli/templates/base-config.yaml:212; parsed at manager.py:148-160).

What's sequential (the bottleneck to fix): the scrape phase fetches one URL at a time. ScraperManager._collect_links_from_urls (src/data_manager/collectors/scrapers/scraper_manager.py:329) runs a plain for url in urls: loop (:346) calling _handle_standard_url per seed, each of which drives LinkScraper.crawl_iter (src/data_manager/collectors/scrapers/scraper.py:165) with blocking requests.get. Git collection (_collect_git_resources, scraper_manager.py:618) and the elog loop (:320) are likewise sequential.

Measured cost (this deployment, a clean re-ingest 2026-07-22):

• Scrape phase: Scraping documents onto filesystem 18:01:31 → Web scraping was completed successfully 18:21:05 = ~19.5 min, single-threaded, for 815 catalog docs (SELECT count(*) FROM documents WHERE is_deleted=false on postgres-dev, db archi-db, user archi).
• The source set is ~152 seed lines but expands (the KB sitemap- line → 212 pages; slurm.schedmd.com crawls to many). It is almost entirely network-bound I/O, so a bounded thread pool should give near-linear speedup up to per-host limits.

Host distribution matters (politeness constraint): the 815 docs concentrate on a few hosts — docs.rc.fas.harvard.edu (212 KB pages from one sitemap), slurm.schedmd.com (large crawl), github.com (git), en.wikipedia.org (already 403s us). Firing 32 concurrent requests at a single host risks rate-limiting/blocking (Wikipedia already returns 403). Concurrency must be bounded and per-host-aware, not a naive global fan-out.

Thread-safety surfaces to respect:

• Postgres writes go through a bounded pool — src/utils/connection_pool.py (logs "Connection pool initialized: min=5, max=20"). N scrape threads each upserting could exhaust 20 connections; size scrape concurrency against the pool, or raise the pool max in tandem.
• LinkScraper.crawl_iter resets its seen/pages_visited state per seed (confirmed in the implement sitemap- source-type v1 (runtime sitemap expansion) #133 work), so distinct seeds are independent — per-seed parallelism is the natural, race-free grain (do not parallelize within one crawl's link discovery).
• The Selenium/SSO authenticator (authenticator in _collect_links_from_urls, and collect_sso) is a shared, likely non-thread-safe browser session — keep the selenium/SSO path sequential (or one authenticator per worker); parallelize only the standard non-selenium link path.
• The sitemap-expansion dedup happens before this loop (collect_all_from_config, scraper_manager.py:125-147), so the loop receives an already-deduped list — parallelizing the loop does not affect dedup correctness.

Constraints (do not assume you read CLAUDE.md)

• Branch from origin/dev; gh pr create --repo fasrc/archi --base dev (target fasrc/archi:dev, never upstream/archi-physics). Never commit to dev directly.
• No Co-Authored-By / session trailers.
• TDD; gate before every commit: bash scripts/gate.sh (black 24.10.0 + isort 6.0.1, pytest tests/unit/, diff-cover --fail-under=80 vs origin/dev). Never --no-verify.
• Keep new logic in tested helper modules (large files like scraper.py/scraper_manager.py reflow under black — see the black-seam-scout guidance; touching them risks a diff-coverage churn trap).
• OpenSpec: non-trivial — run /opsx:propose first.
• Correctness & politeness over raw speed: default concurrency must be conservative and per-host-capped; a run that gets archi rate-limited/IP-blocked at docs.rc.fas.harvard.edu is a worse outcome than a slow run. Make the default safe and the ceiling configurable.
• Determinism: the total scraped/embedded document set and catalog contents must be identical to the sequential path (only wall-clock changes). No dropped or duplicated docs under concurrency.

Plan

Single behavior PR (optionally preceded by a tiny enabler PR if config plumbing is large):

1. Add a data_manager.scrape_workers config knob (default conservative, e.g. 8; document in base-config.yaml). Do not overload the embedding parallel_workers knob — scraping and embedding have different safe ceilings.
2. Replace the for url in urls: loop in _collect_links_from_urls with a bounded ThreadPoolExecutor(max_workers=scrape_workers) over seeds, with a per-host semaphore capping concurrent requests to any single host (e.g. ≤ 4/host). Route the standard non-selenium path through the pool; keep selenium/SSO sequential.
3. Ensure per-thread safety: confirm/each-thread Postgres access stays within the connection pool (raise pool max if needed), aggregate total_count via thread-safe accumulation, and preserve per-seed fail-open (one seed's exception must not abort the batch).
4. Log a one-line summary: seeds, workers, per-host cap, elapsed.

Commands

# Anchors
sed -n '329,360p' src/data_manager/collectors/scrapers/scraper_manager.py   # sequential loop
sed -n '575,600p' src/data_manager/vectorstore/manager.py                   # existing embed pool (pattern to mirror)
grep -n parallel_workers src/cli/templates/base-config.yaml                 # config knob precedent

# Measure sequential baseline vs parallel (dev deployment)
docker logs data-manager-dev 2>&1 | grep -E "Scraping documents onto filesystem|Web scraping was completed successfully"

# Gate
source ~/miniforge3/etc/profile.d/conda.sh && conda activate archi
bash scripts/gate.sh

Acceptance criteria (machine-checkable)

• A unit test proves seeds are fetched concurrently: with an injected fake fetch that sleeps, N independent seeds complete in ≈ (N/scrape_workers)×sleep, not N×sleep (assert wall-clock or assert the fake fetch's max observed concurrency > 1).
• A unit test proves the per-host cap: many seeds on one host never exceed the configured per-host concurrency (assert peak concurrent calls per host ≤ cap).
• A unit test proves the selenium/SSO path stays sequential (single shared authenticator is never used from two threads at once).
• A unit test proves determinism: the set of scraped resource URLs + total count is identical between sequential (workers=1) and parallel (workers=8) runs over the same fixture seeds.
• Per-seed fail-open preserved: a test where one seed raises still scrapes the others and returns their count.
• bash scripts/gate.sh exits 0 for every commit (black/isort clean, pytest green, diff-cover ≥80%).
• Default runtime behavior unchanged in correctness; only faster. A scrape_workers: 1 config reproduces the exact sequential path.

Start here

1. git fetch origin && git checkout -b feat/parallel-scrape origin/dev.
2. /opsx:propose to scope the change (config knob + per-host-capped seed pool), cross-referencing that embedding is already parallel and must not be modified.
3. Write the failing concurrency test (fake sleeping fetch, assert observed >1 concurrency for independent seeds) before implementing the pool.

Origin

Follow-up raised while validating the sitemap- KB migration (PRs #131/#133/#134) on dev — a clean re-ingest spent ~19.5 min in a single-threaded scrape loop while embedding was already 32-way parallel. Sibling efficiency issue: #135 (skip unchanged sources entirely via sitemap <lastmod> / git release SHAs) — these compose (skip fewer + fetch the rest in parallel).

[swinney]

🤖 Picked up by the nightly run on 2026-07-23. Will open a PR; a human reviews before merge.