Find your optimal RAG configuration — before you build your RAG application.
RAG parameter sweep experimentation tool — systematically evaluate embedding models, chunking strategies, and retrieval methods using MongoDB Atlas Vector Search (default) or Postgres/pgvector (STORAGE_BACKEND=postgres — local Docker or Supabase-hosted Postgres; dense, sparse, and hybrid). Supports Voyage AI (hosted), local sentence-transformers (no API key), and SIE open-source embeddings (remote gateway preferred; optional self-hosted Docker).
Most RAG projects start with a guess: pick an embedding model, pick a chunking method, a retrieval method (or a re-ranker), realise it's wrong, refactor. That loop is slow and expensive.
rag-params-finder inverts it.
Give it your data and your questions. It runs every combination — embedding model × chunking method × retrieval method — stores the retrieval scores and shows you exactly which configuration performs best. Before you write a single line of your RAG application.
Jump to: Quickstart | Who is this for? | Screenshots | Key Features | Documentation | Contributing
| What you avoid | What you get instead |
|---|---|
| No LLM calls | Embedding only — 10–100× cheaper |
| No eval framework setup | One YAML config, one CLI command |
| No deployed RAG app needed | Just your data, your questions, your credentials |
| No guessing | Actual retrieval scores across every config, side by side |
| No throwaway experiments | Results persist — compare runs across sessions |
- Embedding models: 13 Voyage models (voyage-4 series, domain, context, voyage-3 legacy) — see
server/core/model_registry.py - Chunking methods: Fixed · Recursive · Token · Sentence · Semantic
- Retrieval methods: Dense · Sparse · Hybrid
- Questions: Persona-organised — user provided or generated as part of golden master generation process
One YAML. N experiments. Evidence-based decision. Ship the right config first.
Pick the row that matches you — each links to a first step, not the whole README.
| Persona | Start here | What you will do |
|---|---|---|
| New user — cloud accounts | MongoDB Setup | Atlas + optional Voyage, then QUICKSTART |
| New user — Postgres (local or Supabase-hosted) | Postgres Setup | ./start-services.sh --postgres-local + configs/supabase/example-unified-retrievers.yaml |
| New user — first sweep | QUICKSTART | Install, run server + CLI, open dashboard |
| Operator — config & CLI | Configuration Reference | YAML sweeps, env vars, rag-params-finder commands |
| Operator — dashboard | Dashboard Guide | Live phases, Search Explorer, experiment controls |
| Operator — fixing errors | Troubleshooting | Indexes, Voyage limits, Docker, Postgres, storage quota |
| Contributor — system design | Architecture | Modules, data flow, ADRs |
| Contributor — dev setup | Development Guide | Quality gates, slices, Docker, hooks |
| Agent / slice worker | AGENTS.md · CLAUDE.md | PROGRESS → theme index → 0N-<theme>/SLICE-*.md |
All docs by topic: docs/README.md
| Screen | Description |
|---|---|
 |
Result-led experiment cards with lifecycle state, sweep outcome, and an explicit next action |
Runs entirely offline — no cloud account, no API key. Uses all-MiniLM-L6-v2 (384-dim) sentence-transformers via ./start-services.sh --mongodb-local.
| Screen | Description |
|---|---|
 |
Experiment detail — lifecycle summary, next step, configuration, and run results |
 |
Search Explorer — best-parameters card, ranked configs with score bars |
Uses SIE open-source embeddings — BGE-M3 and Stella-v5 (1024-dim dense) via remote gateway or self-hosted Docker.
| Screen | Description |
|---|---|
 |
Experiment detail — SIE lifecycle summary with BGE-M3 / Stella-v5 run results |
 |
Search Explorer — SIE sweep ranked configs |
See QUICKSTART.md for install, .env, server, dashboard, and first sweep commands (including optional Docker).
| I want to… | Start here |
|---|---|
| Run entirely offline (no Atlas cloud account) | MongoDB Setup → Path B · ./start-services.sh --mongodb-local |
| Run on Postgres/pgvector instead of Atlas | Postgres Setup · ./start-services.sh --postgres-local |
| Set up MongoDB Atlas or Voyage AI accounts | MongoDB Setup |
| Run my first experiment | Getting Started |
| Understand all config options | Configuration Reference |
| Learn all CLI commands | CLI Reference |
| Understand the dashboard | Dashboard Guide |
| Fix an error | Troubleshooting |
| Understand the system design | Architecture |
| Add a new model, chunker, or endpoint | Extending the System |
| Set up a development environment | Development Guide |
| Why these design choices? | ADR-001 · ADR-002 · ADR-003 |
- Weighted averaging (query-level fairness): Each query contributes equally, preventing queries with many results from dominating the average — configurable via
TIEBREAKER_METRICenv var (docs) - Tiebreaker explanation UI: When multiple configs achieve 100% max score, the dashboard shows amber alerts, explanation panels, visual badges (⭐ "Best by tiebreaker", 🔀 "Tied"), and contextual annotations explaining WHY each config is ranked
- Detailed Results ↔ Hyperparameters mapping: Chunk size/overlap badges, query text display, and explanatory headers help users map individual results back to aggregated configs
- Collapsible sweep dimensions panel: Shows unique values for all swept parameters + Cartesian product calculation (dashboard guide)
- 5 chunking methods: Fixed, Recursive, Token, Sentence, Semantic
- 3 retrieval methods: Dense (vector search), Sparse (BM25), Hybrid (Reciprocal Rank Fusion)
- Voyage AI models: all registered embeddings in
model_registry.py(voyage-4/3/domain/context) + rerankersrerank-2.5-lite,rerank-2.5, and legacy rerank APIs - Local models (no API key):
all-MiniLM-L6-v2+cross-encoder/ms-marco-MiniLM-L-6-v2 - Multi-format data loading: PDF, TXT, Markdown, CSV — files or directories
- Cartesian sweep: one YAML config → N models × M methods × P sizes × Q overlaps runs
- Live phase tracking: QUEUED → PARSING → CHUNKING → EMBEDDING → STORING → QUERYING → RERANKING → COMPLETE
- Experiment management: Pause/resume long sweeps, cancel running experiments, delete with cascade cleanup, boot orphan reconciliation
- Search index preflight: Validates required indexes before sweeps start and rejects with HTTP 422 — Atlas Search indexes plus cluster quota on MongoDB,
vectorextension plus HNSW/GIN catalog objects on Postgres - Index CLI:
indexes listreports Atlas quota usage or the Postgres catalog depending on the active backend;indexes resetis Atlas-only (M0 quota troubleshooting) - Result-led dashboard: Experiment cards lead with lifecycle and sweep outcome; detail connects identity and configuration to run results and the next valid action
- Vector DB stats: Cluster and per-experiment chunk/storage estimates; optional Atlas quota bar with tier, provider, and region when Admin API credentials are configured
- Progress feedback: Byte-level network loading, circular progress with elapsed time and ETA, background polling with "Syncing..." badges
- Scoped logging: Server and dashboard use
[rag-params-finder] [Scope] operation — detailsformat; setLOG_LEVEL=DEBUGfor verbose server output - Pagination: All list views paginated (10 items per page for experiments/runs, 5 for configs); collapsible experiment rows
Backend: FastAPI · Python 3.12 · Pydantic · PyMongo · LangChain text splitters · pypdf · Typer · Rich · sentence-transformers · NLTK · tiktoken
Frontend: React 19 · TypeScript 5.8 · Vite 6 · Tailwind CSS
AI/ML: Voyage AI · sentence-transformers · SIE (Superlinked Inference Engine) · MongoDB Atlas Vector Search · Postgres/pgvector (incl. Supabase-hosted)
Dev tools: uv · ruff · mypy · pytest · GitHub Actions
This project follows Semantic Versioning:
- MAJOR (x.0.0) — Breaking changes
- MINOR (0.x.0) — New features, completed slices (backward compatible)
- PATCH (0.0.x) — Bug fixes, polish, enhancements
Current version: v0.12.0 (CHANGELOG.md)
Release history: GitHub Releases from v0.0.1 through v0.12.0 (dual-backend Postgres, module theme separation, coverage floors, and more)
For contributors: See Release Process — cut release/vX.Y.Z + PR; never push a version bump directly to main.
Contributions welcome — please open an issue first to discuss the change.
Running experiments does not require any extra tooling — the user guide path (Atlas or Postgres/Supabase, CLI, optional dashboard) is enough.
Contributors use the Development Guide for setup (bash scripts/ci/install-git-hooks.sh — checks on commit and push), quality gates (./scripts/ci/quality-gates.sh), the slice workflow, and release cadence (release when a slice or feature is user-visible; see CHANGELOG Unreleased during development).
Optional AI-assisted development (Cursor / Claude Code with the code-review-graph knowledge graph) is documented in the development guide — it helps navigate this repo faster; it is not part of the RAG sweep runtime.
See How Codex and GPT-5.6 Were Used for the development history and model usage summary.
Priority areas: test suite with mock MongoDB fixtures, Search Explorer dashboard enhancements, SSE live updates.
Agent entry points: AGENTS.md · CLAUDE.md
MIT — see LICENSE
Inspired by pre-rag-explorer-dashboard.