Skip to content

Land the ASBB P0 KB work, the SciLEx partial-collection fix, and green the CI - #37

Merged
lfnothias merged 22 commits into
mainfrom
chore/indicium-2.0-namespace
Aug 21, 2026
Merged

Land the ASBB P0 KB work, the SciLEx partial-collection fix, and green the CI#37
lfnothias merged 22 commits into
mainfrom
chore/indicium-2.0-namespace

Conversation

@lfnothias

Copy link
Copy Markdown
Collaborator

Land the ASBB P0 KB work, the SciLEx partial-collection fix, and green the CI

Consolidates every outstanding line of work into one releasable main. The repository is
already public, and its main has had a failing test job since 2026-07-10; this branch
closes that and brings the two stranded fix lines back together.

What it lands

Branch merges. origin/main (the zero-vector and session-database integrity fixes
#32/#33/#35/#36) and origin/asbb/p0-kb-doc (#34, plus the ASBB P0 KB work: symbol-level
code chunking, PMC-OA supplementary ingest, the supplementary endpoint). #34 had been
merged into a side branch only and was present on no other branch.

SciLEx partial collection. A single rate-limited backend (typically Semantic Scholar's
public tier) could outrun the aggregator's per-provider timeout, which cancelled the call
and discarded the results the fast backends had already collected — a query narrowed to
SciLEx sub-providers returned zero hits even when SciLEx had collected hundreds. The
adapter now takes a soft collection deadline, abandons stragglers, and returns partial
results with a structured partial_results_timeout warning.

Three defects found while hardening it, each with a regression test that fails without the
fix:

  • The deadline is budgeted per call, not per pass. search() retries title-like
    queries with a normalized title; giving each pass the full deadline took up to 1.6× the
    caller's timeout, so the outer cancel fired and discarded both passes — reintroducing the
    exact failure the deadline exists to prevent.
  • Abandoned backends keep writing into the scratch directory. Removing it inline raced them
    and raised Directory not empty, discarding collected results; suppressing that error
    instead leaked the directory permanently (measured: thousands of entries per abandoned
    search, unbounded in a long-running server). The directory is now reclaimed by a reaper
    once the abandoned backends actually stop.
  • The partial-collection warning is carried in a call-local report rather than on the
    adapter. A worker thread can outlive a cancelled call, so adapter state leaked into the
    next search — a clean query could report another query's "partial results" warning, and
    the reverse interleaving hid a genuinely partial one.

WRRF citation attribution. select_wrrf_merged_documents merged each chunk with its
neighbours in the score-sorted list, which are not neighbours in the source document, and
stamped the current chunk's citation on the result. Merged context could therefore carry one
paper's text under another paper's citation. Merging is now restricted to chunks from the
same paper.

Provenance deletion. Deleting a conversation left its provenance rows and its
data/provenance/<id>.jsonl sidecar — which holds verbatim prompts and responses — in
place. Both are now purged, including a guard against a conversation id that would escape
the sidecar directory.

export_astra without the private stack. The tool hard-required the private,
unpublishable indicium package, so the repository's only ASTRA feature errored for every
public user; its output was also not a valid ASTRA record (Insights lacked the required
created_at and evidence). It now uses a small vendored projection, emits a valid
Analysis, and reports claims it could not ground rather than emitting them without
evidence. Its test file is no longer excluded from CI.

Why CI goes green

main's failure was six tests importing indicium transitively with no guard, after
1c5c6736 added a module-scope from indicium import ASB_BASE to
indicium_layer/queries.py. Those tests now skip cleanly when the private package is
absent, exactly as the repository's existing convention does elsewhere.

Four other failures on this branch were stale tests, not product bugs, and were fixed on the
test side: an embedding stub that was not awaitable once the symbol-wise path began
embedding in the orchestrator; a tree-sitter probe assertion that still named only the
legacy parser package; composite-adapter tests importing a private package unguarded; and
CLI ingest-mode tests that failed on any machine holding a populated local session database.

Verification

  • Full unit suite: 2582 passed, 8 skipped, 0 failed.
  • CI simulation — the private indicium stack hidden and ci.yml's ignore/deselect list
    applied: 2462 passed, 8 skipped, 17 deselected, 0 failed.
  • uv lock resolves on a fresh clone (305 packages, no private dependency).
  • Reviewed by an adversarial panel before opening; both blocking findings it raised are
    fixed above and covered by tests.

Not included

Splitting _scilex_search_sync (353 lines) and _collect_all_backends into smaller units
is a real cleanup but predates this work and belongs in its own change.

lfnothias and others added 22 commits June 14, 2026 22:41
…h fallback

text-embedding-3-large hard-caps each input at 8192 tokens; one oversized
section-chunk (heading-aware DOI/PMC ingest) was 400-ing the whole batch ->
0 chunks -> silent no-grounding on large papers. _truncate_for_embedding()
(tiktoken cl100k_base, 8000-tok cap, char fallback) on every input, plus a
per-item fallback so one bad input can't drop the paper. Verified: a 196KB
paper went 0 -> 20 chunks.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When a DOI resolves to a PMC-OA paper with <supplementary-material> in its JATS XML, fetch those SI files and append their extracted text to the paper's full_text at ingest time, so SI grounds the KB alongside the article body. PMC-OA only (no-op otherwise -> OA-safe), fully fail-soft.

- PDFDownloadConfig: ingest_pmc_supplementary (default True) + file/char/byte caps
- kb.py: _augment_with_pmc_supplementary + _si_bytes_to_text (pdf->parser, xlsx->openpyxl, text->decode, media->skip), wired into the sync + async DOI paths and the /papers endpoint
- tests for the new config defaults

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…oint

GitHub repo ingest dumped every non-Python file whole (license headers, getters, UI), so Java/C++/R repos (mzmine, 833MB) flooded the KB with ~13k noise chunks. Now code files are symbol-chunked via the shared chunk_document/chunk_code engine (tree-sitter for compiled langs, regex for R), and large repos keep only the files most relevant to the tool.

- chunking_code.py: _chunk_treesitter compatibility shim (_v) — tree-sitter-language-pack exposes kind/child_count/start_position as zero-arg methods + parse() wants str; handles that and classic py-tree-sitter; per-method subchunks for big classes; never raises (falls back). New dep tree-sitter-language-pack.
- chunking_dispatch.py: added R/.c/.cxx to _EXT_TO_LANG so the local-docs path symbol-chunks R too (equivalence with the repo path).
- github_kb.py: _ingest_code_papers_symbolwise routes code files through chunk_document; _select_relevant_papers keeps docs/README + top-N code files by lexical overlap with a focus signal (opt-in focus/max_files; MCP tool + skill bundles unaffected). Python stays on the producer's docstrings path.
- kb.py: POST /api/kb/<kb>/github/async — job-based ingest (poll /api/jobs/<id>), fixes synchronous-MCP client timeouts on mega-repos.
- test_chunk_code_symbols.py: regression guard for the tree-sitter shim + R regex.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The github chunk producer kept Python as docstrings-only (a deliberate v1 choice predating symbol-aligned chunking). Now .py/.pyx keep raw source (content_kind github_code) and are AST symbol-chunked at ingest via chunk_code — one function/class per chunk, with bodies + docstrings — matching the local-docs path and the Java/R treatment. Removes the docstrings/bodies divergence between the two ingest paths.

- chunk_producer.py: _paper_from_python -> _paper_from_code (raw source); .py/.pyx routed there; dropped now-unused ast import.
- github_kb.py: .py/.pyx added to _CODE_EXT_TO_LANG.
- test_github_chunk_producer.py: updated to expect raw source + github_code.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…isting KB

Enriches a paper KB built before PMC-SI ingestion was default: fetches the paper's PMC-OA supplementary files and ingests their text as labeled documents WITHOUT re-ingesting the paper (chunks untouched). No-op when there's no PMC SI, so callers gate downstream work (LLM re-validation) on si_files>0. Idempotent via stable pmc-si:<doi>:<label> ids.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ing to embed (#34)

* fix(embeddings): raise instead of zero-filling a text that keeps failing to embed

* style(tests): drop an unused noqa directive
@lfnothias
lfnothias merged commit bafc47a into main Aug 21, 2026
3 checks passed
@lfnothias
lfnothias deleted the chore/indicium-2.0-namespace branch August 21, 2026 17:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant