Concept: Local insights reports with model-backed enrichment

[tony]

Summary

• Add an agentgrep insights command tree that turns local agent history into a report — deterministic by default, with an opt-in ladder of HTML, classical ML, embeddings, a persistent hybrid index, and a local-LLM summary.
• Add a curated model registry that auto-downloads a sentence-embedding model through the same urllib + manifest artifact path used for local LLM models, into the platform cache.
• Add a pluggable Level-4 index — tantivy (BM25) + sqlite-vec (vectors) by default, with LanceDB as a drop-in single-store alternative behind one capability probe.
• Stream enrichment progress — phase lines, download bytes, and live LLM token deltas — to stderr, so long-running levels are visible without polluting machine-readable stdout.
• Keep the base install lean: importing agentgrep loads none of the optional backends, and every missing rung degrades to a precise install command instead of a traceback.

This is a concept branch demonstrating the ADR 0005 architecture end-to-end; it is not intended for direct merge.

Changes by area

Engine — src/agentgrep/insights/

• report.py: builds a report from a SearchRecord stream — resolves the effective level, attaches enrichment, and records diagnostics and grounded next actions.
• activity.py: deterministic level-0 analysis (per-agent/store counters, daily timeline, frequent terms, repeated instructions, an open-thread heuristic, coverage) emitting RecordRef drilldown handles.
• loader.py: lazy backend loading behind an injectable import_module seam; availability is probed with importlib.util.find_spec so a builtin report never imports a heavy backend just to list levels.
• models.py: the curated registry, an atomic urllib downloader, and a manifest sidecar; a torch-free model2vec model and the LLM artifacts share one cache layout.
• enrichers/: html (jinja2), ml (scikit-learn TF-IDF + KMeans), embeddings (sentence-transformers preferred, model2vec fallback), index (tantivy+sqlite-vec or LanceDB), llm (Ollama over httpx, grounded in compact facts).
• cache.py / model.py / progress.py: ADR-0005 cache-directory precedence, the typed report model with JSON payloads, and the progress protocol.

CLI — src/agentgrep/cli/

• parser.py: the insights subcommand tree (report, levels, doctor, setup, models, cache) with typed argument dataclasses.
• insights_render.py: dispatchers, the text/markdown/html/json/ndjson renderer, and the stderr console progress sink.
• __init__.py: main() dispatch for the new argument types.

Packaging & docs

• pyproject.toml: insights-html / -ml / -embeddings / -embeddings-st / -index / -index-lancedb / -llm / -all extras.
• docs/cli/insights.md: the CLI guide, registered in the CLI index toctree and card grid.

Design decisions

• Deterministic first, models opt-in: level 0 is always available and network-free; richer levels attach as enrichment only when their backend is installed and selected. best-installed never auto-selects the LLM level, since its runtime depends on an external daemon the import probe cannot see.
• One download pattern for every model: a sentence-embedding model provisions through the identical urllib + agentgrep-manifest.json path as a local LLM artifact, so "fetch an embedding model" and "fetch a local LLM" are the same mechanism with one cache layout.
• Pluggable index, default chosen for portability: tantivy + sqlite-vec is the default (tiny, ABI-stable wheels); LanceDB is offered as a single-store alternative rather than the default to keep the base index footprint small.
• Ollama is the only wired LLM runtime: LiteRT-LM and llama.cpp are fetch-only registry entries whose runtime returns a clear configuration error, rather than shipping half-working in-process bindings.
• Loaded backends are typed Any: an optionally-present third-party module cannot be statically typed, so EnricherContext.modules is dict[str, Any] — which also lets the tests inject plain fakes through the same seam.

Verification

No optional backend is imported at module top of the insights package (all loads are lazy):

rg -n "^import (torch|sklearn|tantivy|lancedb|httpx|jinja2|sentence_transformers|model2vec|numpy)" src/agentgrep/insights

The insights package itself is not imported by import agentgrep except function-locally inside the dispatchers:

rg -n "agentgrep.insights" src/agentgrep/__init__.py

Test plan

• Builtin report renders text/markdown/html/json/ndjson from the record stream
• Every enrichment level runs through injected fake backends — tests/test_insights_enrichers.py
• Model registry downloads via a monkeypatched urlopen, writes a manifest, and is a no-op when cached — tests/test_insights_models.py
• Level resolution, fallback diagnostics, and status classification — tests/test_insights_report.py
• Import-time guard: import agentgrep loads no optional backend or the insights package — tests/test_import_time.py
• Full gate: ruff check, ruff format, ty check, pytest, just build-docs
• Manual end-to-end: real model2vec download → semantic clustering, and a real tantivy + sqlite-vec index with working sample queries

[tony]

Auto-closed during a rebase onto master (a stray force-push briefly made the branch look merged). The same work, rebased onto current master, continues in #87.