Land the ASBB P0 KB work, the SciLEx partial-collection fix, and green the CI - #37
Merged
Conversation
…h fallback text-embedding-3-large hard-caps each input at 8192 tokens; one oversized section-chunk (heading-aware DOI/PMC ingest) was 400-ing the whole batch -> 0 chunks -> silent no-grounding on large papers. _truncate_for_embedding() (tiktoken cl100k_base, 8000-tok cap, char fallback) on every input, plus a per-item fallback so one bad input can't drop the paper. Verified: a 196KB paper went 0 -> 20 chunks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When a DOI resolves to a PMC-OA paper with <supplementary-material> in its JATS XML, fetch those SI files and append their extracted text to the paper's full_text at ingest time, so SI grounds the KB alongside the article body. PMC-OA only (no-op otherwise -> OA-safe), fully fail-soft. - PDFDownloadConfig: ingest_pmc_supplementary (default True) + file/char/byte caps - kb.py: _augment_with_pmc_supplementary + _si_bytes_to_text (pdf->parser, xlsx->openpyxl, text->decode, media->skip), wired into the sync + async DOI paths and the /papers endpoint - tests for the new config defaults Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…oint GitHub repo ingest dumped every non-Python file whole (license headers, getters, UI), so Java/C++/R repos (mzmine, 833MB) flooded the KB with ~13k noise chunks. Now code files are symbol-chunked via the shared chunk_document/chunk_code engine (tree-sitter for compiled langs, regex for R), and large repos keep only the files most relevant to the tool. - chunking_code.py: _chunk_treesitter compatibility shim (_v) — tree-sitter-language-pack exposes kind/child_count/start_position as zero-arg methods + parse() wants str; handles that and classic py-tree-sitter; per-method subchunks for big classes; never raises (falls back). New dep tree-sitter-language-pack. - chunking_dispatch.py: added R/.c/.cxx to _EXT_TO_LANG so the local-docs path symbol-chunks R too (equivalence with the repo path). - github_kb.py: _ingest_code_papers_symbolwise routes code files through chunk_document; _select_relevant_papers keeps docs/README + top-N code files by lexical overlap with a focus signal (opt-in focus/max_files; MCP tool + skill bundles unaffected). Python stays on the producer's docstrings path. - kb.py: POST /api/kb/<kb>/github/async — job-based ingest (poll /api/jobs/<id>), fixes synchronous-MCP client timeouts on mega-repos. - test_chunk_code_symbols.py: regression guard for the tree-sitter shim + R regex. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The github chunk producer kept Python as docstrings-only (a deliberate v1 choice predating symbol-aligned chunking). Now .py/.pyx keep raw source (content_kind github_code) and are AST symbol-chunked at ingest via chunk_code — one function/class per chunk, with bodies + docstrings — matching the local-docs path and the Java/R treatment. Removes the docstrings/bodies divergence between the two ingest paths. - chunk_producer.py: _paper_from_python -> _paper_from_code (raw source); .py/.pyx routed there; dropped now-unused ast import. - github_kb.py: .py/.pyx added to _CODE_EXT_TO_LANG. - test_github_chunk_producer.py: updated to expect raw source + github_code. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…isting KB Enriches a paper KB built before PMC-SI ingestion was default: fetches the paper's PMC-OA supplementary files and ingests their text as labeled documents WITHOUT re-ingesting the paper (chunks untouched). No-op when there's no PMC SI, so callers gate downstream work (LLM re-validation) on si_files>0. Idempotent via stable pmc-si:<doi>:<label> ids. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ing to embed (#34) * fix(embeddings): raise instead of zero-filling a text that keeps failing to embed * style(tests): drop an unused noqa directive
… when one backend is slow
…ium-2.0-namespace
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Land the ASBB P0 KB work, the SciLEx partial-collection fix, and green the CI
Consolidates every outstanding line of work into one releasable
main. The repository isalready public, and its
mainhas had a failing test job since 2026-07-10; this branchcloses that and brings the two stranded fix lines back together.
What it lands
Branch merges.
origin/main(the zero-vector and session-database integrity fixes#32/#33/#35/#36) and
origin/asbb/p0-kb-doc(#34, plus the ASBB P0 KB work: symbol-levelcode chunking, PMC-OA supplementary ingest, the supplementary endpoint). #34 had been
merged into a side branch only and was present on no other branch.
SciLEx partial collection. A single rate-limited backend (typically Semantic Scholar's
public tier) could outrun the aggregator's per-provider timeout, which cancelled the call
and discarded the results the fast backends had already collected — a query narrowed to
SciLEx sub-providers returned zero hits even when SciLEx had collected hundreds. The
adapter now takes a soft collection deadline, abandons stragglers, and returns partial
results with a structured
partial_results_timeoutwarning.Three defects found while hardening it, each with a regression test that fails without the
fix:
search()retries title-likequeries with a normalized title; giving each pass the full deadline took up to 1.6× the
caller's timeout, so the outer cancel fired and discarded both passes — reintroducing the
exact failure the deadline exists to prevent.
and raised
Directory not empty, discarding collected results; suppressing that errorinstead leaked the directory permanently (measured: thousands of entries per abandoned
search, unbounded in a long-running server). The directory is now reclaimed by a reaper
once the abandoned backends actually stop.
adapter. A worker thread can outlive a cancelled call, so adapter state leaked into the
next search — a clean query could report another query's "partial results" warning, and
the reverse interleaving hid a genuinely partial one.
WRRF citation attribution.
select_wrrf_merged_documentsmerged each chunk with itsneighbours in the score-sorted list, which are not neighbours in the source document, and
stamped the current chunk's citation on the result. Merged context could therefore carry one
paper's text under another paper's citation. Merging is now restricted to chunks from the
same paper.
Provenance deletion. Deleting a conversation left its provenance rows and its
data/provenance/<id>.jsonlsidecar — which holds verbatim prompts and responses — inplace. Both are now purged, including a guard against a conversation id that would escape
the sidecar directory.
export_astrawithout the private stack. The tool hard-required the private,unpublishable
indiciumpackage, so the repository's only ASTRA feature errored for everypublic user; its output was also not a valid ASTRA record (Insights lacked the required
created_atandevidence). It now uses a small vendored projection, emits a validAnalysis, and reports claims it could not ground rather than emitting them withoutevidence. Its test file is no longer excluded from CI.
Why CI goes green
main's failure was six tests importingindiciumtransitively with no guard, after1c5c6736added a module-scopefrom indicium import ASB_BASEtoindicium_layer/queries.py. Those tests now skip cleanly when the private package isabsent, exactly as the repository's existing convention does elsewhere.
Four other failures on this branch were stale tests, not product bugs, and were fixed on the
test side: an embedding stub that was not awaitable once the symbol-wise path began
embedding in the orchestrator; a tree-sitter probe assertion that still named only the
legacy parser package; composite-adapter tests importing a private package unguarded; and
CLI ingest-mode tests that failed on any machine holding a populated local session database.
Verification
indiciumstack hidden andci.yml's ignore/deselect listapplied: 2462 passed, 8 skipped, 17 deselected, 0 failed.
uv lockresolves on a fresh clone (305 packages, no private dependency).fixed above and covered by tests.
Not included
Splitting
_scilex_search_sync(353 lines) and_collect_all_backendsinto smaller unitsis a real cleanup but predates this work and belongs in its own change.