Per-host download politeness, honest exits, and a bounded chroma FD budget

[lfnothias]

Three related fixes to the ingest path, each one measured rather than reasoned about.

The chroma FD and RSS leak

Root cause is not an application-level collection cache — there is none; every Collection in src/perspicacite is a local variable discarded at method exit. It is chromadb's rust HNSW segment cache, sized once at client construction as max_file_handles // 5 from RLIMIT_NOFILE and never re-read. With the host ulimit -n at 1,048,576 that budget is 209,715 resident indices — effectively unbounded. Observed on a live server: 8,207 FDs, 7,924 of them under chroma_db/ (1,981 collections x 4 files), ~11.5 GB RSS after 3h50m.

ChromaVectorStore now constructs through a bounded client, dropping the budget to 409 resident indices.

A second, separate bug: SharedSystemClient keys its System cache on the persist-directory string, unnormalised. bibtex_kb.py:301 passed a resolved absolute path and :452 a possibly-relative one, so one directory got two Systems and two HNSW caches. Normalising the path in __init__ collapses all four construction sites without touching bibtex_kb.py.

Lowering the soft FD limit naively is unsafe and this was found the hard way: if the process already holds more descriptors than the budget, chroma's rust binding does not merely bound its cache — it panics with EMFILE and the client cannot be constructed at all (reproduced at soft=100 and soft=300; soft=500 works). Since web/state.py and mcp/server.py construct at boot, that would mean the server refuses to start. The helper therefore never lowers below max(1024, open_fds + 256) and never raises an operator's own ulimit.

The plateau test nearly had no power: on this platform HNSW index files are memory-mapped, not held as descriptors, so /dev/fd shows zero and a "count FDs under the directory" assertion would have passed vacuously forever. lsof shows them, at exactly 4 entries per collection — matching the production signature. The test compares a budgeted client against an unbounded control in the same process: 222 vs 483 entries for 120 collections. Sabotaging the bound makes it fail (483 vs 483, perfectly linear).

bioRxiv 429s

There was no backoff, no per-host concurrency cap, and throttled responses were filed as permanent successes — a 17-byte body recorded as a completed fetch. A per-host politeness layer now spaces requests, retries only {429, 500, 502, 503, 504}, honours Retry-After, and raises RateLimited rather than handing a throttled body back as content.

The host policy is a data table with exactly one row (biorxiv.org, carrying the measured behaviour in a comment); the retry, spacing and cooldown machinery contains no host names at all. A URL with no parseable host gets an explicit unknown limiter rather than silently bypassing rate limiting.

Honest exits

content_type is a storage format, not a transient value: rag/dynamic_kb.py persists it into ChromaDB chunk metadata and selects the embedding model from it. The literal values are now named constants with is_abstract_only() / has_full_text() predicates, so callers stay correct without an equality test — and the values themselves are unchanged, deliberately.

Downstream consumers (search_to_kb, the MCP server, the KB router) now distinguish a retryable failure from a permanent one instead of collapsing both into "no full text".

Note on the PDF size floor

MIN_PLAUSIBLE_PDF_BYTES unifies checks that were previously duplicated literals. It is set to 1024, the floor pdf_cache has always enforced — not a tightened value. An earlier revision used 2048 and rejected legitimate small PDFs: the dropzone returned "failed to write PDF to cache" for a valid upload and a cached local PDF was reported as remote. Unification should not smuggle in a behaviour change.

Verification

tests/unit: 2789 passed, 8 skipped. One test is deselected — test_web_search_telemetry_sink.py::test_list_telemetry_sink_receives_events makes real SemanticScholar network calls and hangs. It was confirmed pre-existing by running it against unmodified HEAD in a separate worktree, where it hangs identically; note that it does transitively import retrieval.chroma_store, so source-level inspection alone was not sufficient to clear it.

Ruff findings on the touched files drop from 56 to 51; no new ones.