Human-Guided Parametric-vs-Retrieval Gating

Human-Guided Parametric-vs-Retrieval Gating is a proof-of-concept system for controlling how a language model answers user queries. It decides, per query, whether to:

• Answer from the model’s parametric memory (fast but potentially stale)
• Retrieve and cite supporting documents
• Run computations
• Clarify the question
• Escalate / abstain when it cannot safely answer

The objective is to maximize truthfulness and citation quality while keeping latency and token costs within budget. This system is designed to be multilingual (English, Hindi, Urdu, and Spanish) and to train its decision policy from human-labelled data.

Repository Layout

app/            # Streamlit data labelling app and evaluation dashboard
data/           # Datasets and data cards
features/       # Feature extraction modules used by the gating policy
policy/         # Data loading and training code for the gating controller
orchestrator/   # Code that ties the policy, retrieval, and tools together
truth/          # Adapters for truthfulness and citation checking metrics
evals/          # Evaluation scripts (hallucination rate, citation precision, freshness SLA)
ci/             # Continuous integration configuration and quality gates
scripts/        # Helper scripts for data collection, training, and evaluation
tests/          # Unit tests

Quickstart

1. Install dependencies Built with Python 3. Requires numpy, pandas, fastapi, and optionally torch + transformers for model training and inference. Install with:

   pip install -r requirements.txt

2. Launch the data studio Start the Streamlit app for collecting labels:

   streamlit run app/data_studio.py

3. Train the gating policy After collecting labelled queries, create a dataset and train the policy:

   python scripts/train_policy.py --data=data/curated/labels.jsonl --out=policy/model

4. Run the orchestrator API Launch the FastAPI service to answer queries:

   uvicorn orchestrator.pipeline:app --reload

   Access the /answer endpoint to submit queries.

5. Evaluate performance Use scripts in evals/ to benchmark against baseline strategies. Reports include metrics such as:

   • Stale hallucination rate
   • Citation precision
   • Freshness SLA compliance

Development Notes

• Multilingual: Designed for English, Hindi, Urdu, and Spanish.
• Extensible: Policies and feature extractors are modular for experimentation.
• Evaluation-first: Emphasizes metrics that directly measure truthfulness, freshness, and cost trade-offs.
• CI Integration: Quality gates and workflows included under ci/.

Status

This project is under active development and not production-ready. Many components contain placeholders that should be replaced with stronger retrieval modules, advanced verification tools, and more robust training strategies. It serves as a framework to explore research ideas around freshness-aware retrieval gating.