███╗ ██╗ █████╗ ███████╗ █████╗ ██████╗ █████╗ ███████╗██╗ ██╗███████╗████████╗███████╗███╗ ███╗
████╗ ██║██╔══██╗██╔════╝██╔══██╗ ██╔═══██╗██╔══██╗ ██╔════╝╚██╗ ██╔╝██╔════╝╚══██╔══╝██╔════╝████╗ ████║
██╔██╗ ██║███████║███████╗███████║ ██║ ██║███████║ ███████╗ ╚████╔╝ ███████╗ ██║ █████╗ ██╔████╔██║
██║╚██╗██║██╔══██║╚════██║██╔══██║ ██║▄▄ ██║██╔══██║ ╚════██║ ╚██╔╝ ╚════██║ ██║ ██╔══╝ ██║╚██╔╝██║
██║ ╚████║██║ ██║███████║██║ ██║ ╚██████╔╝██║ ██║ ███████║ ██║ ███████║ ██║ ███████╗██║ ╚═╝ ██║
╚═╝ ╚═══╝╚═╝ ╚═╝╚══════╝╚═╝ ╚═╝ ╚══▀▀═╝ ╚═╝ ╚═╝ ╚══════╝ ╚═╝ ╚══════╝ ╚═╝ ╚══════╝╚═╝ ╚═╝
"I have a 270-page technical handbook with process diagrams, decision trees, tables, and heavy cross-referencing between chapters. I need a chatbot that can actually answer questions about it — not just find the paragraph with the right keyword."
— i2e Hireathon 2026, Problem Statement 2
A production-grade Retrieval-Augmented Generation (RAG) QA engine built specifically for the NASA Systems Engineering Handbook (SP-2016-6105 Rev2) — a 270-page, 17-chapter technical document dense with cross-references, multi-page tables, process diagrams, and deeply nested section hierarchies.
Standard RAG implementations break on technical manuals. This system was engineered from the ground up to handle those structural challenges.
| Feature | What It Does |
|---|---|
| 🧠 Multi-hop QA | Follows cross-references between chapters automatically (e.g. Ch 6.6 → Ch 6.8) |
| 📊 Table Extraction | pdfplumber parses multi-page tables as markdown — PDR/CDR criteria are queryable |
| 🖼️ Figure Awareness | Figure captions + surrounding context extracted as searchable chunks |
| 🔤 Acronym Resolution | 40+ NASA acronyms auto-expanded before embedding AND before LLM call |
| 📍 Citation Precision | Every answer cites Section, Page, Paragraph, and Parent Section |
| 📈 Real Confidence Score | Weighted across top-3 chunks with hop discount — never hardcoded |
| 🔒 Security | XSS prevention, input sanitisation, rate limiting, DevTools blocking |
| 🖥️ NASA Aesthetic | Mission control dark UI — amber accents, monospace terminal feel |
┌─────────────────────────────────────────────────────────────────┐
│ PDF (270 pages) │
└─────────────────────┬───────────────────────────────────────────┘
│
┌───────────▼───────────┐
│ ingestor.py │
│ ┌─────────────────┐ │
│ │ pypdf → text │ │ Section-boundary chunking
│ │ pdfplumber → │ │ NOT arbitrary token windows
│ │ tables (md) │ │
│ │ figure captions │ │
│ └────────┬────────┘ │
└───────────┼───────────┘
│ ~885 chunks
│ (text + table + figure)
┌───────────▼───────────┐
│ FAISS Index │ all-MiniLM-L6-v2 embeddings
│ index.faiss │ L2-normalised cosine similarity
│ chunks.pkl │ Runs entirely in-process (<5ms)
└───────────┬───────────┘
│
┌───────────▼───────────┐
│ retriever.py │
│ │
│ PASS 1: FAISS search │ expand_acronyms() → embed →
│ fetch 15 → dedup → 5 │ fetch-15 → deduplicate by
│ │ section_title → TOP_K=5
│ PASS 2: Cross-ref │
│ hop │ Scan chunks for "see Section X"
│ fetch 3 cross-refs │ → re-query → merge → dedup
└───────────┬───────────┘
│ up to 5 chunks
│ (hop=1 PRIMARY + hop=2 CROSS-REF)
┌───────────▼───────────┐
│ agent.py │
│ │
│ expand_acronyms() │ Pre-LLM expansion
│ build_context() │ Labels: [PRIMARY][CROSS-REFERENCE]
│ │ [TABLE][FIGURE]
│ Groq LLaMA-3.3-70B │ ~300 tok/s, grounded answer
│ calculate_confidence │ Weighted 60/25/15% + hop discount
└───────────┬───────────┘
│
┌───────────▼───────────┐
│ main.py │ FastAPI · uvicorn ASGI
│ /ask /health │ Returns answer + sources +
│ /ingest /shutdown │ confidence + expanded_query
└───────────┬───────────┘
│
┌───────────▼───────────┐
│ Frontend │ NASA Mission Control UI
│ index.html │ Dark navy + amber accents
│ style.css │ 3-panel responsive layout
│ app.js │ XSS-safe · rate-limited
└───────────────────────┘
Problem_2_Technical_Manual_QA/
│
├── backend/
│ ├── __init__.py
│ ├── main.py ← FastAPI app, all routes
│ ├── ingestor.py ← PDF → chunks → FAISS index
│ ├── retriever.py ← 2-pass multi-hop FAISS search
│ ├── agent.py ← Groq LLM call, context builder, confidence
│ ├── models.py ← Pydantic request/response schemas
│ └── utils.py ← NASA acronym resolver (40+ terms)
│
├── frontend/
│ ├── index.html ← Mission control UI
│ ├── style.css ← NASA dark aesthetic
│ └── app.js ← Fetch calls, chat logic, security
│
├── data/
│ └── nasa_handbook.pdf ← SP-2016-6105 Rev2 (public domain)
│
├── faiss_index/ ← Auto-created on first ingestion
│ ├── index.faiss
│ └── chunks.pkl
│
├── .env ← GROQ_API_KEY
├── requirements.txt
└── README.md
git clone https://github.com/Ajaysomala/i2e_Hireathon-Complex_Technical_Manual_QA_System
pip install -r requirements.txt# Create .env file
echo "GROQ_API_KEY=your_key_here" > .env
# Get your free key at: https://console.groq.commkdir -p data
# Download from NASA (public domain):
# https://www.nasa.gov/wp-content/uploads/2018/09/nasa_systems_engineering_handbook_0.pdf
# Save as: data/nasa_handbook.pdfpython -m backend.ingestor
# ~2-3 minutes on CPU
# Produces: faiss_index/index.faiss + faiss_index/chunks.pkl
# Expected: ~885 chunks (text + table + figure)uvicorn backend.main:app --reloadhttp://localhost:8000
fastapi
uvicorn
pypdf
sentence-transformers
faiss-cpu
python-dotenv
groq
pdfplumber| Method | Endpoint | Description |
|---|---|---|
GET |
/ |
Serves the frontend interface |
GET |
/health |
Index status, chunk count, model loaded |
POST |
/ask |
Main QA endpoint — body: { question, conversation_history } |
POST |
/ingest |
Triggers PDF re-ingestion and FAISS rebuild |
POST |
/shutdown |
Terminates the uvicorn server process |
// Request
{
"question": "What are the entry criteria for PDR?",
"conversation_history": []
}
// Response
{
"answer": "The entry criteria for Preliminary Design Review (PDR)...",
"sources": [
{
"section_title": "Table G-6 PDR Entry Criteria",
"page_number": 271,
"parent_section": "G",
"paragraph_hint": "table",
"relevance_score": 0.847,
"hop": 1,
"chunk_type": "table"
}
],
"confidence": 0.76,
"query_time_ms": 1241.0,
"expanded_query": "What are the entry criteria for Preliminary Design Review (PDR)?"
}Arbitrary 512-token windows split mid-sentence and mid-table. The NASA handbook's value is in its structured sections. A chunk that starts at Section 6.3.2.1 and ends at the next section boundary is always a semantically complete unit. This is the single most impactful decision in the system.
The evaluation requires local execution. FAISS runs entirely in-process with zero network latency. For 885 chunks, search completes in under 5ms. No API keys, no network RTT, no cost.
Direct TOP_K=5 can return 5 chunks from the same section (adjacent paragraphs embed similarly). Fetching 15 then deduplicating by section_title guarantees 5 topically distinct sections — dramatically improving cross-chapter answer quality.
Runs offline. Fast (batched ingestion < 2 min on CPU). 384-dim vectors suited to technical retrieval. No API key, no per-token cost, no latency.
Groq delivers ~300 tokens/second for a 70B model — comparable to GPT-4o latency at zero cost on the free tier. For technical domain reasoning across multiple sections, 70B significantly outperforms 7B/13B on instruction following and citation accuracy.
Pass 1 gets directly relevant chunks. Pass 2 scans those chunks for cross-references (see Section X.X, Table G-6, Appendix G) and re-queries those targets. This is how the system answers questions that span Chapter 6.6 and Chapter 6.8 without being explicitly told they're connected.
PDR and Preliminary Design Review embed as semantically distant vectors. Expanding at (1) pre-embedding in retriever.py fixes retrieval. Expanding at (2) pre-LLM in agent.py fixes answer quality. Both are necessary.
| Requirement | Implementation | Status |
|---|---|---|
| Ingest full PDF + searchable knowledge base | pypdf + pdfplumber + FAISS — 885 chunks | ✅ |
| Natural language questions with source references | FastAPI /ask → Groq LLaMA 3.3 70B |
✅ |
| Basic cross-reference resolution | TOP_K=5 retrieves across chapters | ✅ |
| Section-boundary chunking | Custom section-aware chunker in ingestor.py |
✅ |
| Stretch Goal | Implementation | Status |
|---|---|---|
| Image/diagram awareness | Figure captions + context extracted as chunk_type="figure" — searchable by concept |
|
| Table extraction | pdfplumber → markdown tables as chunk_type="table" — PDR/CDR criteria queryable |
✅ |
| Hierarchical chunking | parent_section + paragraph_index per chunk, passed to LLM context + shown in UI |
✅ |
| Acronym resolution | expand_acronyms() — 40+ NASA terms, applied pre-embed AND pre-LLM |
✅ |
| Citation precision | Section + Page + Paragraph + Parent + weighted confidence score | ✅ |
| Multi-hop QA | 2-pass retriever: cross-ref pattern detection → auto re-query → hop-2 labelling | ✅ |
5 of 6 stretch goals fully achieved. Image/diagram awareness is partial — figure captions are extracted and searchable, but visual diagram content (Vee Model logic) requires a vision model and is on the roadmap.
| Limitation | Why It Happens | Next Iteration Fix |
|---|---|---|
| Structured table queries sometimes miss rows | Tables ingested as text — row/column relationships partially lost | pdfplumber structured extraction → full markdown with row/col metadata |
| Diagram/Vee Model questions | Text extraction misses figures. Figure 2.1-1 is a diagram — no text to embed | pdf2image + vision model (GPT-4V / LLaVA) for page-level understanding |
| True multi-hop chaining | System retrieves from both chapters but doesn't trace explicit cross-ref links | Knowledge graph on section references + iterative retrieval agent loop |
| Short section queries | Low-density embeddings rank poorly against larger sections | Hybrid BM25 + dense retrieval (reciprocal rank fusion) |
| Hallucination on edge queries | When conf < 50%, LLM may extrapolate beyond context | Hard threshold: refuse to answer if confidence < 0.30 |
| Feature | Approach | Effort | Priority |
|---|---|---|---|
| pdfplumber full table extraction | Tables → structured markdown with row/col metadata | 2–3 hrs | HIGH |
| Explicit multi-hop chaining | Parse cross-ref text → section graph → BFS retrieval | 4–6 hrs | HIGH |
| Vision understanding (Vee Model) | pdf2image per page → GPT-4V description → embed | 3–4 hrs | MED |
| BM25 + dense hybrid retrieval | rank_bm25 + FAISS → Reciprocal Rank Fusion | 2 hrs | MED |
| Hard confidence floor | Refuse answer if weighted confidence < 0.30 | 30 min | LOW |
Somala Ajay Python Developer · AI/ML Engineer · Data Scientist
📍 Hyderabad, Telangana, India 🔗 GitHub: github.com/Ajaysomala 🌐 Portfolio: https://ajaysomala.github.io ✉️ ajaysomala@gmail.com
Built for i2e Hireathon 2026 · Problem Statement 2: Complex Technical Manual QA Final Round · Bangalore · March 28, 2026
NASA SE Handbook QA System · Built by Somala Ajay · i2e Hireathon 2026
"We are not looking for a polished product. We are looking for evidence that you can take a vague problem, decompose it into an architecture, build a working prototype, and articulate tradeoffs." — i2e Consulting AI Labs