RAG via sqlite-vec: Vector search for documentation Q&A

[raifdmueller]

Summary

Explore using sqlite-vec to enable RAG (Retrieval-Augmented Generation) for documentation Q&A. This builds on top of Issue #185 (SQLite Index) and the lessons from ADR-009 (abandoned iterative LLM ask approach).

Background

In #186 / ADR-009 we learned that iterating over all documentation files with an LLM is too slow for a built-in ask command. RAG offers a better approach: pre-compute vector embeddings for documentation sections, then retrieve only the most relevant chunks for a given question.

Concept

• sqlite-vec (pip install sqlite-vec) is a lightweight SQLite extension for vector similarity search (~30MB memory, no external server)
• Combines well with ADR-007: Persistent SQLite Index for CLI Performance and Fuzzy Search #185: one SQLite database for both structure index and vector embeddings
• Fits our "File System as Truth" principle (ADR-001) — SQLite serves as cache/index only

Proposed architecture (rough sketch)

1. Structure Index table — Section paths, titles, metadata (from ADR-007: Persistent SQLite Index for CLI Performance and Fuzzy Search #185)
2. Vector Embeddings table — One embedding per section/chunk for similarity search
3. Query flow: Question → embed → find top-k similar chunks → build context → LLM answers

Open research questions

• Which embedding model? Options: Anthropic Embeddings API (hosted), local models like sentence-transformers (offline, ~100MB). Trade-off: cost vs. offline capability
• Chunking strategy: Use existing sections as chunks, or split further? Sections vary widely in size
• sqlite-vec maturity: Currently v0.1.6 (pre-1.0). Evaluate stability for production use
• Embedding dimensions: What dimension works best for documentation search? (768, 1024, 1536?)
• Incremental updates: How to efficiently re-embed only changed sections when docs are updated?
• Should this be a separate dacli rag command, an MCP tool, or integrated into search?

References

• sqlite-vec: https://github.com/asg017/sqlite-vec
• sqlite-vec Python docs: https://alexgarcia.xyz/sqlite-vec/python.html
• ADR-009: Decision against iterative LLM ask feature
• Issue ADR-007: Persistent SQLite Index for CLI Performance and Fuzzy Search #185: SQLite-based index persistence
• Issue Feature: AI-powered 'ask' command with flexible LLM provider support #186: Original ask command (closed, won't implement)

Related issues

• ADR-007: Persistent SQLite Index for CLI Performance and Fuzzy Search #185 — SQLite Index (natural foundation for this feature)