⚡ NV-Agent

A production-ready, self-hosted RAG AI Agent that grounds every answer in your own documents — with source citations, real-time streaming, and a full browser UI.

Most AI chatbots give you generic answers. NV-Agent is an AI agent that retrieves relevant context from your knowledge base before generating a response — so every answer is grounded, cited, and honest about what it doesn't know.

---

📖 Table of Contents

• The Problem It Solves
• What is NV-Agent?
• Why NVIDIA NIM?
• Architecture
  • High-Level System Architecture
  • RAG Pipeline Flow
  • Data Flow Diagram
  • Streaming Architecture
  • Session Persistence
• Key Capabilities
• Project Structure
• Quick Start
  • Prerequisites
  • Install & Run
  • Quick API Test (No Server)
• How to Use
  • Adding Documents
  • Chatting with Your Knowledge Base
  • Managing Sessions
  • Choosing Models
• Configuration
• API Reference
  • Chat
  • Sessions
  • Knowledge Base
  • Health & Auth
• Docker
• Design Decisions
• Security
• Contributing
• License

---

🎯 The Problem It Solves

Problem	Why It Matters	How NV-Agent Solves It
Generic AI = generic answers	Off-the-shelf chatbots hallucinate about your domain because they don't know your data	RAG pipeline retrieves your documents before answering — every response is grounded in your knowledge base
RAG is hard to set up	Most RAG tutorials leave you with a notebook, not a production-ready app	One command starts a complete system: vector store, embeddings, LLM, session persistence, and browser UI
Knowledge trapped in files	PDFs, DOCX, and text docs sit in folders — unsearchable, unqueryable	Drop 12+ file formats into data/ and they're instantly indexed and queryable
Conversations survive refresh	Most demo chatbots lose everything when you refresh the browser	Session ID saved to sessionStorage — page refresh reconnects to the same session with full history
No infrastructure for AI	Running LLMs locally requires GPUs, drivers, and model management	NVIDIA NIM gives you hosted state-of-the-art models via a single API key — zero local GPU needed

---

🤖 What is NV-Agent?

NV-Agent is a complete, self-contained AI agent system — not a library, not a framework, not a notebook.

It is a RAG (Retrieval-Augmented Generation) agent that:

1. Retrieves relevant document chunks from a pluggable vector knowledge base (FAISS / Qdrant / ChromaDB)
2. Augments the LLM prompt with retrieved context and source citations
3. Generates grounded answers via NVIDIA NIM LLMs with real-time streaming

You run one command, open a browser, and chat over your own documents:

python main.py  # → http://localhost:8000

The agent follows the ReAct-inspired pattern: given a user query, it reasons about what context to retrieve, fetches the most relevant chunks from the knowledge base, then generates a cited answer — explicitly stating when the KB doesn't contain the answer.

---

🟢 Why NVIDIA NIM?

NVIDIA NIM (NVIDIA Inference Microservices) provides hosted, state-of-the-art open models via a single OpenAI-compatible API endpoint. By building on NVIDIA NIM, NV-Agent gives you:

Benefit	Details
Zero infrastructure	No local GPUs, no model downloads, no driver headaches — just an API key
Best-in-class models	Nemotron, Llama, DeepSeek, and 100+ models available instantly
OpenAI-compatible API	Same openai Python SDK you already know — just a different base_url
Enterprise-grade	NVIDIA-hosted inference with reliability, rate limiting, and monitoring
Free tier	1,000 credits/month at build.nvidia.com

---

🏗️ Architecture

High-Level System Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                          NV-Agent System                            │
│                                                                     │
│  ┌──────────┐    ┌──────────────┐    ┌───────────┐    ┌──────────┐ │
│  │          │    │              │    │           │    │          │ │
│  │  Chat    │◀──▶│  Chat API    │◀──▶│ RAG Agent │◀──▶│Knowledge │ │
│  │  UI      │    │  (FastAPI)   │    │ (LLM+RAG) │    │  Base    │ │
│  │          │    │              │    │           │    │(Vector   │ │
│  │ Browser  │    │ REST / SSE / │    │ Retrieve →│    │ Store)   │ │
│  │ WebSocket│    │ WebSocket    │    │ Augment → │    │          │ │
│  │          │    │              │    │ Generate  │    │          │ │
│  └──────────┘    └──────────────┘    └───────────┘    └──────────┘ │
│       ▲               ▲                   ▲               ▲       │
│       │               │                   │               │       │
│       │          ┌────┴─────┐        ┌────┴─────┐   ┌────┴────┐  │
│       │          │  Session │        │  NVIDIA   │   │ Vector  │  │
│       │          │  Store   │        │  NIM API  │   │ Store   │  │
│       │          │  (Disk)  │        │ (Cloud)   │   │(FAISS/  │  │
│       │          └──────────┘        └───────────┘   │Qdrant/  │  │
│       │                                               │ChromaDB)│  │
│  ┌────┴──────────────────────────────────────────────────────────┐  │
│  │                     config.py (.env)                          │  │
│  └──────────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────────┘

Four-layer design — each layer is a Python package with clear __init__.py and explicit __all__ exports:

Layer	Package	Responsibility
Presentation	chat/ui/	Browser UI — React 18 + TypeScript + Tailwind v4 (built from frontend/)
API	chat/	FastAPI routes, CORS, request/response models, streaming
Agent	agent/	RAG logic (retrieve → augment → generate), session management
Knowledge Base	kb/	Document ingestion, chunking, embedding, pluggable vector store (FAISS/Qdrant/ChromaDB)

---

RAG Pipeline Flow

This is the core of the AI agent — how a user question becomes a grounded, cited answer:

  User asks: "What does our deployment process look like?"
       │
       ▼
┌──────────────────────────────────────────────────────────────┐
│  1. EMBED QUERY                                              │
│     User question ──▶ nv-embed-v1 embedding ──▶ 4096-dim vector   │
│     (NVIDIA NIM embedding API)                               │
└──────────────┬───────────────────────────────────────────────┘
               │
               ▼
┌──────────────────────────────────────────────────────────────┐
│  2. RETRIEVE                                                 │
│     Query vector ──▶ Vector store search (FAISS/Qdrant/     │
│     ChromaDB) ──▶ top-5 most relevant chunks from KB         │
│     (Choice of backend via NV_AGENT_VECTOR_STORE)            │
└──────────────┬───────────────────────────────────────────────┘
               │
               ▼
┌──────────────────────────────────────────────────────────────┐
│  3. AUGMENT                                                  │
│     System prompt + retrieved chunks (with source citations) │
│     + conversation history + current user message            │
│     → one rich message list for the LLM                      │
└──────────────┬───────────────────────────────────────────────┘
               │
               ▼
┌──────────────────────────────────────────────────────────────┐
│  4. GENERATE                                                 │
│     NVIDIA NIM LLM (Nemotron / Llama / DeepSeek / etc.)     │
│     → Streams answer token-by-token via WebSocket/SSE        │
│     → Reasoning/thinking tokens shown in collapsible block    │
└──────────────┬───────────────────────────────────────────────┘
               │
               ▼
  Agent responds with citations:
  "Based on [Source: deploy.md (chunk 2)], the deployment
   process involves..." or honestly: "The knowledge base
   does not contain information about that."

Every answer cites its source file and chunk number. If the knowledge base doesn't contain the answer, the agent says so honestly — no hallucination.

---

Data Flow Diagram

                        ┌─────────────┐
                        │  Documents   │
                        │  (12 formats)│
                        └──────┬──────┘
                               │
                        ┌──────▼──────┐
                        │  Ingestion   │
                        │  Pipeline    │
                        └──────┬──────┘
                               │
                 ┌─────────────┼─────────────┐
                 │             │             │
          ┌──────▼──────┐ ┌───▼────┐ ┌──────▼──────┐
          │   Chunker   │ │ Embed  │ │  Vector     │
          │  (Boundary  │ │ Client │ │  Store      │
          │   Aware)    │ │(Batch) │ │(FAISS/Qdrant/│
          └──────┬──────┘ └───┬────┘ │  ChromaDB)  │
                 │            │      └──────┬──────┘
                 └────────────┼─────────────┘
                              │
                      ┌───────▼───────┐
                      │  Knowledge    │
                      │  Base Index   │
                      │ (kb/index/ or │
                      │  Docker vol)  │
                      └───────┬───────┘
                              │
              ┌───────────────┼───────────────┐
              │               │               │
       ┌──────▼──────┐ ┌─────▼──────┐ ┌──────▼──────┐
       │   REST API  │ │  SSE Stream│ │  WebSocket  │
       │  /api/chat  │ │  /api/chat │ │  /api/ws/   │
       │             │ │  /stream   │ │    chat      │
       └──────┬──────┘ └─────┬──────┘ └──────┬──────┘
              │               │               │
              └───────────────┼───────────────┘
                              │
                       ┌──────▼──────┐
                       │   Browser   │
                       │  Chat UI    │
                       └─────────────┘

---

Streaming Architecture

  Browser                    FastAPI                   Worker Thread
  ───────                    ───────                   ─────────────
     │                          │                          │
     │  WebSocket connect       │                          │
     │─────────────────────────▶│                          │
     │                          │                          │
     │  {"message": "hello"}    │                          │
     │─────────────────────────▶│                          │
     │                          │  run_in_executor()       │
     │                          │─────────────────────────▶│
     │                          │                          │
     │                          │               chat_stream() generator
     │                          │               ├─ yield {"type": "reasoning"}
     │                          │  queue.put("reasoning")  │
     │  {"type":"reasoning"}    │◀─────────────────────────│
     │◀─────────────────────────│                          │
     │                          │                          │
     │                          │               ├─ yield {"type": "text"}
     │                          │  queue.put("token")      │
     │  {"type":"token"}        │◀─────────────────────────│
     │◀─────────────────────────│                          │
     │                          │               ├─ generator ends
     │                          │  queue.put("done")       │
     │  {"type":"done"}         │◀─────────────────────────│
     │◀─────────────────────────│                          │
     │                          │                          │

Why thread+queue? The OpenAI SDK is synchronous. We run chat_stream() in a worker thread, pipe events through asyncio.Queue to the async FastAPI handler — avoiding the StopIteration-in-async bug.

---

Session Persistence

  ┌──────────┐         ┌───────────────┐         ┌──────────────┐
  │  RAGAgent│────────▶│ SessionStore  │────────▶│  data/       │
  │  (memory)│  save() │ (thread-safe) │  write() │  sessions/  │
  │          │◀────────│  mutex lock   │◀──────── │  {uuid}.json│
  └──────────┘  load() └───────────────┘  read()  └──────────────┘
       │                                          Atomic write:
       │  On startup: load_all()                  temp.json.tmp
       │  On chat: save() after each message        → rename()
       │  On delete: remove from memory + disk     (POSIX atomic)

Conversations survive server restarts. Every message is timestamped and saved to disk atomically.

---

✨ Key Capabilities

Capability	How It Works
Grounded answers	Every response is backed by vector search over your documents, with source citations
Real-time streaming	WebSocket + SSE streaming — see the answer (and the agent's reasoning) token by token
Thinking/reasoning display	Agent's reasoning tokens shown in a collapsible block — see how it thinks
Multi-format ingestion	Drop in .pdf, .docx, .txt, .md, .py, .json, .yaml, .yml, .csv, .html, .xml, .rst — auto-indexed on startup
Smart chunking	Paragraph-aware → sentence-aware → word-boundary splitting with overlap — no cutting mid-sentence
Session persistence	Conversations survive server restarts. Every message is timestamped and saved to disk atomically
Session continuity on refresh	Session ID saved to sessionStorage — page refresh reconnects to the same session with full history
Browser file upload	Upload PDFs and Word docs directly from the UI — no CLI needed
Dark-themed chat UI	Responsive single-page interface — sidebar, session management, KB status panel
Multi-model support	Any model on NVIDIA NIM (Nemotron, Llama, DeepSeek, etc.) — just change one config value
Production patterns	Custom exceptions, proper HTTP status codes, thread-safe state, filename sanitization, upload size limits
API key auth	Optional authentication middleware — protect your instance with NV_AGENT_AUTH_KEY
Rate limiting	Per-IP sliding window rate limiter — configurable via NV_AGENT_RATE_LIMIT
Pluggable vector stores	Choose FAISS (default, zero-infra), ChromaDB, or Qdrant (high-performance Rust DB)

---

📁 Project Structure

nv-agent/
│
├── main.py                      # 🚀 Entry point — startup sequence, init, run server
├── config.py                    # ⚙️ Central configuration (dataclasses, reads .env)
├── test-agent.py                # 🧪 Quick CLI smoke test (bypasses RAG pipeline)
├── requirements.txt             # 📦 Python dependencies
├── pyproject.toml               # 🔧 Ruff, mypy, pytest config
├── .env.example                 # 🔑 API key template (copy to .env)
├── .gitignore                   # 🛡️ Excludes .env, sessions, __pycache__, .venv
├── .pre-commit-config.yaml      # 🪝 Pre-commit hooks (ruff, mypy, checks)
├── build/                       # 🏗️ Build & Docker infrastructure
│   ├── Dockerfile               # 🐳 Multi-stage Docker build (Python 3.12-slim)
│   ├── .dockerignore            # 🐳 Docker build context exclusions
│   ├── docker-compose.yml       # 🐳 Compose: nv-agent + qdrant + chromadb
│   ├── docker-compose.prod.yml  # 🐳 Production compose (GHCR, Caddy, limits)
│   ├── Caddyfile                # 🌐 Reverse proxy config (auto HTTPS via Let's Encrypt)
│   └── deploy.sh                # 🚀 One-command remote deployment script
├── Makefile                     # ⚡ Build, test, lint, Docker, compose shortcuts
├── requirements-dev.txt     # 📦 Dev dependencies (pytest, ruff, mypy, pre-commit, flake8, pylint)
├── .pylintrc               # 🔧 Pylint configuration
├── LICENSE                  # 📄 MIT License
├── README.md                # 📖 You are here
├── AGENTS.md                # 🤖 AI agent guidelines and architecture
├── CLAUDE.md                # 🤖 Claude Code context and conventions
│
├── data/                    # 📂 Your documents — auto-indexed on startup
│   ├── sample.md            #    Example knowledge base doc
│   └── sessions/            #    Persisted conversation JSONs (auto-created)
│
├── kb/                      # 🧠 Knowledge Base Layer
│   ├── __init__.py          #    Exports: VectorStoreBase, Chunk, ingest_*, chunk_text
│   ├── chunker.py           #    Multi-level text splitting (paragraph→sentence→word)
│   ├── embed.py             #    NVIDIA embedding client (singleton, batched, error-handled)
│   ├── ingest.py            #    File ingestion + readers (PDF, DOCX, 10 text formats)
│   ├── vector_store_base.py     #    Abstract base class (VectorStoreBase)
│   ├── vector_store.py          #    ⚠️ Legacy FAISS store (tests only), superseded by factory pattern
│   ├── vector_store_faiss.py        #    FAISS vector store implementation
│   ├── vector_store_qdrant.py       #    Qdrant vector store implementation
│   ├── vector_store_chromadb.py     #    ChromaDB vector store implementation
│   ├── vector_store_factory.py      #    Factory for creating vector stores
│   └── index/               #    Saved FAISS index + chunks.json (auto-created)
│
├── agent/                   # 🤖 Agent Layer
│   ├── __init__.py          #    Exports: RAGAgent, Session, Message, exceptions, SessionStore
│   ├── rag_agent.py         #    RAG logic: retrieve → augment → generate + session mgmt
│   └── session_store.py     #    Thread-safe disk persistence (atomic JSON writes)
│
├── frontend/                # ⚛️ React Frontend (dev)
│   ├── src/
│   │   ├── components/      #    Chat, Sidebar, Modals, Shared UI
│   │   ├── context/         #    Auth, Chat, KB, Toast (React Context + useReducer)
│   │   ├── hooks/           #    useAutoResize, useClipboard, useDragDrop, useKeyboardShortcuts
│   │   ├── services/        #    api.ts (apiFetch), markdown.ts (marked + DOMPurify)
│   │   ├── types/           #    api.ts, chat.ts (TypeScript interfaces)
│   │   ├── utils/           #    cn.ts (clsx + tailwind-merge), constants.tsx
│   │   ├── App.tsx          #    Root component with provider composition
│   │   ├── main.tsx         #    React 18 entry point
│   │   └── index.css        #    Tailwind v4 + CSS variables + animations
│   ├── package.json         #    React, Vite, Tailwind, marked, DOMPurify, clsx, tailwind-merge
│   ├── vite.config.ts       #    Vite config (React, Tailwind, /api proxy to :8000)
│   └── tsconfig.json        #    Strict TypeScript config
│
├── chat/                    # 🌐 Chat API + UI Layer
│   ├── __init__.py          #    Exports: create_app, router
│   ├── app.py               #    FastAPI factory, middleware, CORS, static file mount
│   ├── auth.py              #    API key authentication middleware
│   ├── rate_limit.py        #    Per-IP sliding window rate limiter
│   ├── routes.py            #    All endpoints: REST, SSE, WebSocket, file upload, health, auth
│   └── ui/                  #    React production build (generated by `npm run build:deploy`)
│       ├── index.html       #    Entry point (loads /ui/assets/*.js, /ui/assets/*.css)
│       ├── assets/          #    Hashed JS/CSS bundles
│       └── favicon.svg      #    NVIDIA hexagon logo
│
├── tests/                   # 🧪 Pytest test suite
│   ├── conftest.py          #    Shared fixtures (mocks, temp dirs, test client)
│   ├── test_chunker.py      #    Chunking unit tests
│   ├── test_config.py       #    Configuration unit tests
│   ├── test_embed.py        #    Embedding client unit tests (mocked API)
│   ├── test_ingest.py       #    Ingestion pipeline unit tests
│   ├── test_session_store.py#    Session persistence unit tests
│   ├── test_rag_agent.py    #    RAG agent unit tests
│   └── test_api.py          #    FastAPI endpoint integration tests
│
├── .github/                 # 🔄 CI/CD
│   └── workflows/
│       └── ci.yml           #    GitHub Actions: lint + type check + test + Docker build
│
└── .claude/                 # 🔧 Claude Code settings
    └── settings.local.json  #    Local permissions

Design Decisions

Decision	Why
Build step for production	React/TypeScript compiles to chat/ui/ via npm run build:deploy; Docker serves static build. Dev server: npm run dev (HMR, proxies to backend)
Pluggable vector stores	Factory pattern with FAISS (default, zero-infra), Qdrant (high-perf Rust), ChromaDB (Python)
Singleton OpenAI client	Both embed.py and rag_agent.py lazily create one client — no connection churn
Thread+Queue for streaming	OpenAI SDK is synchronous — worker thread + asyncio.Queue bridges sync→async
Atomic file writes	Sessions and index use temp-file + rename() (POSIX atomic) — no corruption on crash
Skip dirs in ingestion	data/sessions/, .git, .venv are excluded — session JSONs don't pollute the KB
Custom exception hierarchy	RAGAgentError → SessionNotFoundError / LLMError — routes catch specific types, return proper HTTP codes
Filename sanitization	Upload endpoint strips path traversal (../../../etc/passwd → _.._.._.._etc_passwd)
OpenAI-compatible API	Using NVIDIA NIM's OpenAI-compatible endpoint means we use the standard openai SDK — no vendor lock-in

---

🚀 Quick Start

Prerequisites

• Python 3.11+
• NVIDIA NIM API key — create a free account at build.nvidia.com

NV-Agent accepts any of these environment variables for your API key:

• NVIDIA_NIM_API_KEY (preferred)
• NVIDIA_API_KEY
• NGC_API_KEY

Install & Run

# 1. Clone and enter
git clone <your-repo-url> nv-agent && cd nv-agent

# 2. Create virtual environment
python -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate

# 3. Install dependencies
pip install -r requirements.txt

# 4. Configure your API key (uses NVIDIA_NIM_API_KEY by default)
cp .env.example .env
# Edit .env — set NVIDIA_NIM_API_KEY="nvapi-your-key-here"
# Or export any of: NVIDIA_NIM_API_KEY, NVIDIA_API_KEY, NGC_API_KEY

# 5. Add your documents (optional)
cp ~/my-docs/*.pdf ~/my-docs/*.docx data/

# 6. Start the server
python main.py

Then open http://localhost:8000 in your browser and start chatting.

• 🖥️ Chat UI: http://localhost:8000
• 📖 Interactive API Docs: http://localhost:8000/docs

⚡ Makefile shortcuts (optional)

make install         # Install runtime dependencies
make install-dev     # Install runtime + dev dependencies
make run             # Run production server
make run-dev         # Run with uvicorn --reload
make lint            # Run flake8 + ruff + pylint
make format          # Auto-format with ruff
make test            # Run pytest -v
make test-cov        # Run pytest with coverage
make typecheck       # Run mypy
make ci              # Full CI pipeline (lint + test + typecheck)
make smoke           # Quick smoke: unit tests + Docker health check
make clean           # Remove caches, temp files, build artifacts

# Docker / Frontend
make frontend        # Build React frontend (npm ci + build:deploy)
make compose-build   # Build frontend + Docker image
make compose-up      # Build frontend + image, start all services (recommended)
make compose-down    # Stop all services
make compose-reset   # Stop, remove volumes, restart clean
make compose-logs    # Follow compose logs
make compose-ps      # List compose services

Quick API Test (No Server)

To verify your API key before launching the full server:

python test-agent.py

This runs a standalone CLI smoke test that calls the NVIDIA NIM API directly — no FAISS index, no document ingestion, no web server. It uses a hardcoded model (deepseek-ai/deepseek-v4-pro) for quick testing. If this works, your key is valid and the full server will too.

---

📖 How to Use

Adding Documents

NV-Agent supports three ways to add documents to your knowledge base:

Method	How	Best For
File drop	Copy files into data/ directory before startup	Bulk loading, initial setup
Browser upload	Click "Upload File" in the sidebar	Quick additions from the UI
API ingest	POST /api/kb/ingest with raw text, or POST /api/kb/ingest-file by path	Automation, scripts, CI/CD

Supported formats: .pdf, .docx, .txt, .md, .py, .json, .yaml, .yml, .csv, .html, .xml, .rst

Tip: After adding files to data/, hit "Refresh Status" in the sidebar, or call POST /api/kb/ingest-dir to re-index without restarting the server.

Chatting with Your Knowledge Base

1. Open http://localhost:8000
2. Click "+ New Chat" in the sidebar
3. Ask any question about your documents
4. The agent will:
   • Search the knowledge base for relevant chunks
   • Show its reasoning in a collapsible "💭 Thinking…" block
   • Stream the answer token-by-token with source citations
   • Honestly say when the knowledge base doesn't contain the answer

Example questions:

• "What does the deployment process look like?" (if you have deployment docs)
• "Summarize the key points from the quarterly report" (if you uploaded a PDF)
• "What are the API conventions used in this project?" (if you added code files)

Managing Sessions

• Create: Click "+ New Chat" or POST /api/sessions
• Switch: Click a session in the sidebar — full history loads from disk
• Delete: Click the × button on a session — removes from memory and disk
• Persist: Automatic — every message is saved to data/sessions/{uuid}.json
• Survive restarts: Sessions are loaded from disk on server startup
• Survive browser refresh: Session ID saved to sessionStorage — page refresh reconnects to the same session with full history

Choosing Models

NV-Agent works with any model on NVIDIA NIM. Change the chat model in config.py:

# In config.py, NVIDIAConfig class:
chat_model: str = "nvidia/nemotron-3-ultra-550b-a55b"  # Default
# chat_model: str = "meta/llama-3.1-8b-instruct"       # Fast, cheap
# chat_model: str = "deepseek-ai/deepseek-v4-pro"      # Strong reasoning

Or set the MODEL environment variable to override at runtime.

⚠️ If you change the embedding model: You MUST delete kb/index/ to reset the vector index — mismatched dimensions will cause search failures.

---

⚙️ Configuration

All settings are in config.py with sensible defaults. Override via environment variables.

Environment Variables

Variable	Required	Default	Description
NVIDIA_NIM_API_KEY	✅	—	Your NVIDIA NIM API key (also accepts NVIDIA_API_KEY, NGC_API_KEY)
MODEL	❌	nvidia/nemotron-3-ultra-550b-a55b	Override the chat model
NV_AGENT_AUTH_KEY	❌	—	API key for auth middleware (when set, all /api/* require X-API-Key)
NV_AGENT_RATE_LIMIT	❌	60/minute	Rate limit per IP (format: N/unit, e.g., 10/second, 100/hour)
NV_AGENT_VECTOR_STORE	❌	faiss	Vector store backend: faiss, chromadb, or qdrant

Vector Store Configuration

Backend	Required Env Vars	Optional Env Vars	Notes
FAISS (default)	—	—	Zero-infrastructure, index on disk, no external service
Qdrant	NV_AGENT_VECTOR_STORE=qdrant	NV_AGENT_QDRANT_COLLECTION, NV_AGENT_QDRANT_PATH (local mode), or NV_AGENT_QDRANT_HOST, NV_AGENT_QDRANT_PORT, NV_AGENT_QDRANT_API_KEY (server mode)	High-performance Rust vector DB
ChromaDB	NV_AGENT_VECTOR_STORE=chromadb	NV_AGENT_CHROMADB_COLLECTION, NV_AGENT_CHROMADB_PERSIST_DIR	Open-source embedding DB, local file-based

Install optional dependencies:

# ChromaDB
pip install chromadb

# Qdrant
pip install qdrant-client

Config Defaults (config.py)

Setting	Default	What It Controls
chat_model	nvidia/nemotron-3-ultra-550b-a55b	LLM model ID (any model on NVIDIA NIM)
embedding_model	nvidia/nv-embed-v1	Embedding model (must match embedding_dim)
embedding_dim	4096	Vector dimension — must match embedding model output
temperature	1.0	LLM sampling temperature
top_p	0.95	Nucleus sampling threshold
max_tokens	16384	Max response tokens
enable_thinking	True	Show reasoning/thinking tokens in streaming
reasoning_budget	16384	Token budget for reasoning
chunk_size	512	Characters per chunk
chunk_overlap	64	Overlap characters between chunks
top_k	5	Number of context chunks to retrieve per query
data_dir	./data	Directory for knowledge base documents
port	8000	Server port
cors_origins	["*"]	CORS allowed origins

---

📡 API Reference

All endpoints are under /api. Full interactive docs at /docs.

Chat

Method	Endpoint	Description
POST	/api/chat	Send a message, get a full response
POST	/api/chat/stream	Send a message, get a streaming SSE response
WS	/api/ws/chat	WebSocket chat with real-time token streaming

Request body (chat + stream):

{ "session_id": "uuid", "message": "What does the KB say about X?" }

SSE events:

data: {"token": "answer "}
data: {"reasoning": "thinking..."}
data: [DONE]

WebSocket messages:

→ {"message": "hello"}
← {"type": "token", "content": "answer "}
← {"type": "reasoning", "content": "thinking..."}
← {"type": "done", "full": "complete answer"}
← {"type": "error", "content": "error message"}

Sessions

Method	Endpoint	Description
POST	/api/sessions	Create a session (optional ?title=My+Chat)
GET	/api/sessions	List all sessions with titles
GET	/api/sessions/{id}/history	Get messages (?limit=50)
DELETE	/api/sessions/{id}	Delete a session

Knowledge Base

Method	Endpoint	Description
GET	/api/kb/status	Chunk count + index status
POST	/api/kb/ingest	Ingest raw text (JSON body: {text, source})
POST	/api/kb/ingest-dir	Re-scan data/ and index all files
POST	/api/kb/ingest-file	Ingest a file by server path
POST	/api/kb/upload	Upload file (multipart/form-data, 12 formats supported)
DELETE	/api/kb/reset	Clear the entire knowledge base

Health & Auth

Method	Endpoint	Description
GET	/api/health	Returns {"status": "ok", "kb_chunks": N}
GET	/api/health/detailed	Full system info: model, KB status, auth, rate limit, active sessions
GET	/api/auth/validate	Validate your NVIDIA NIM API key — returns {valid, model, error}

---

🐳 Docker

Run NV-Agent in a container — no Python installation needed on the host.

Option 1: FAISS (Default) — Zero External Dependencies

# Build frontend + run with docker compose
cp .env.example .env  # Set your NVIDIA API key
make compose-up       # Builds frontend (npm run build:deploy) + Docker image, then starts all services

# Or step-by-step:
cd frontend && npm run build:deploy  # Build React → chat/ui/
docker compose -f build/docker-compose.yml up -d --build  # Build image (copies chat/ui/) + start

# Or build manually
docker build -f build/Dockerfile -t nv-agent .
docker run -p 8000:8000 --env-file .env -v $(pwd)/data:/app/data nv-agent

Note: The Docker image copies the pre-built chat/ui/ directory. You must run npm run build:deploy (or make frontend) before building the Docker image — the Dockerfile does not run the frontend build. The Makefile compose-up target handles this automatically via compose-build.

Docker Compose persists your documents and FAISS index in named volumes, so they survive container restarts. Note: the nv-agent-sessions volume overrides the ./data:/app/data bind mount at the /app/data/sessions subpath — sessions live inside the Docker volume, not on the host filesystem at ./data/sessions/.

Option 2: With Qdrant Vector Database (Recommended for Production)

cp .env.example .env
# Add to .env:
# NV_AGENT_VECTOR_STORE=qdrant
# NV_AGENT_QDRANT_HOST=qdrant
# NV_AGENT_QDRANT_PORT=6333
docker compose up -d --build

This starts both NV-Agent and Qdrant containers, with Qdrant persisting data in a named volume.

Option 3: With ChromaDB

# Add to .env:
# NV_AGENT_VECTOR_STORE=chromadb
docker compose up -d --build

Custom Configuration

# With auth key and custom rate limit
docker run -p 8000:8000 \
  --env-file .env \
  -e NV_AGENT_AUTH_KEY=my-secret-key \
  -e NV_AGENT_RATE_LIMIT="30/minute" \
  nv-agent

Vector Store Comparison for Docker

Backend	Pros	Cons	Best For
FAISS	Zero config, fast, no extra container	Single-process only, no scaling	Dev, single-user, small KB
ChromaDB	Easy local setup, good features	Python-based, slower at scale	Medium KB, local dev with persistence
Qdrant	Rust performance, filtering, distributed, server mode	Extra container, more resources	Production, large KB, multi-user, filtering

---

Production Deployment (Remote Server)

For production on a remote VPS/server, use the production compose file with a reverse proxy:

# 1. Copy production files to server
scp build/docker-compose.prod.yml build/Caddyfile .env.prod.example user@your-server:~/

# 2. On server: setup
ssh user@your-server
mkdir -p /opt/nv-agent && cd /opt/nv-agent
cp ~/build/docker-compose.prod.yml ~/build/Caddyfile .
cp .env.prod.example .env.prod   # Edit with your API keys, domain

# 3. Configure Caddyfile: replace 'yourdomain.com' with your actual domain
# 4. Configure .env.prod: add NVIDIA_NIM_API_KEY, NV_AGENT_AUTH_KEY, etc.

# 5. Deploy (uses pre-built GHCR image)
docker compose -f build/docker-compose.prod.yml up -d

# Or use the deploy script from local machine
./build/deploy.sh --logs

Production compose (docker-compose.prod.yml) includes:

• Pre-built GHCR image (no build on server)
• Named volumes only (no host path dependencies)
• Resource limits (CPU/memory)
• Log rotation (10MB, 3 files)
• Caddy reverse proxy (automatic HTTPS via Let's Encrypt)
• Health checks with restart policies

Required environment (.env.prod):

NVIDIA_NIM_API_KEY=nvapi-your-key
NV_AGENT_AUTH_KEY=your-secure-random-key  # openssl rand -hex 32
NV_AGENT_RATE_LIMIT=60/minute
NV_AGENT_VECTOR_STORE=faiss  # or qdrant/chromadb
ACME_EMAIL=you@example.com  # for Let's Encrypt (optional)

CI/CD: Push to main or tag v* → GitHub Actions builds multi-arch image → pushes to GHCR. Server pulls via docker compose pull.

---

🔒 Security

Measure	Details
No API key in source control	.env is in .gitignore — never commit it
Optional API key auth	Set NV_AGENT_AUTH_KEY to require X-API-Key header on all /api/* routes
Per-IP rate limiting	Default: 60 req/min per IP. Override with NV_AGENT_RATE_LIMIT (e.g., "10/second")
Filename sanitization	Path traversal stripped, null bytes removed, special chars replaced with _
Upload size limit	50 MB max (MAX_UPLOAD_SIZE in routes.py)
Extension validation	Only SUPPORTED_EXTENSIONS allowed → 400 if unsupported
Generic error responses	Internal details logged server-side, clients get "Internal error"
Custom exception hierarchy	Specific HTTP codes: 401 (auth), 429 (rate limit), 404, 502, 500

---

🤝 Contributing

We welcome contributions! See AGENTS.md for the full architecture guidelines and coding standards.

Development Setup

git clone <your-repo-url> && cd nv-agent
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
pip install -r requirements-dev.txt  # Dev tools: pytest, ruff, mypy, pre-commit, flake8, pylint
cp .env.example .env        # Add your NVIDIA API key
pre-commit install           # Optional: set up git hooks

Contribution Workflow

1. Fork the repo and create a feature branch from main
2. Make your changes — follow the code style in AGENTS.md
3. Test manually — run python main.py and verify your change works end-to-end
4. Commit clearly — use descriptive messages (Add X support, Fix Y bug)
5. Open a Pull Request — describe what you changed and why

Adding a New File Format

1. Add a reader function in kb/ingest.py — follow the _read_pdf / _read_docx pattern (lazy import, per-page/per-section error handling)
2. Register it in the _READERS dict
3. Add the extension to SUPPORTED_EXTENSIONS in chat/routes.py
4. Update the <input accept="..."> attribute in frontend/src/components/sidebar/KBSection.tsx
5. Add the extension to SUPPORTED_EXTENSIONS in frontend/src/utils/constants.tsx
6. Add the dependency to requirements.txt
7. Test by uploading a file through the UI

Adding a New LLM Provider

NV-Agent uses the OpenAI-compatible API format via config.nvidia.base_url. To switch providers:

1. Update config.py — change base_url, chat_model, embedding_model, embedding_dim
2. Delete kb/index/ to reset the FAISS vector index (different embedding = different dimensions)
3. For Qdrant/ChromaDB backends, call DELETE /api/kb/reset to clear computed vectors
4. Set the appropriate API key env var

---

📄 License

This project is licensed under the MIT License.