Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -334,6 +334,7 @@ if __name__ == "__main__":
- Index backends are pluggable via factory:
- `faiss`
- `milvus`
- `qdrant`
- Web search backends are pluggable via factory:
- `exa`
- `tavily`
Expand Down
18 changes: 12 additions & 6 deletions docs/llms.txt
Original file line number Diff line number Diff line change
Expand Up @@ -1596,7 +1596,7 @@ async def retriever_init(
**Function**
- Initializes retrieval service.
- Embedding Backend (backend): Responsible for converting text/images into vectors (Infinity, SentenceTransformers, OpenAI, BM25).
- Index Backend (index_backend): Responsible for vector storage and retrieval (FAISS, Milvus).
- Index Backend (index_backend): Responsible for vector storage and retrieval (FAISS, Milvus, Qdrant).
- Demo Mode: If is_demo=True, forces OpenAI + Milvus configuration, ignoring some parameters.

---
Expand Down Expand Up @@ -1634,7 +1634,7 @@ async def retriever_index(
**Function**
- Builds retrieval index.
- FAISS: Reads embedding_path (.npy) to build local index file.
- Milvus / Demo: Reads corpus_path (.jsonl), generates vectors and inserts into specified collection_name.
- Milvus / Qdrant / Demo: Reads corpus_path (.jsonl), generates vectors and inserts into specified collection_name.

---

Expand All @@ -1652,7 +1652,7 @@ async def retriever_search(

**Function**
- Retrieves single or multiple queries.
- Automatically handles query vectorization (adds query_instruction) and finds Top-K in specified collection_name (for Milvus) or default index.
- Automatically handles query vectorization (adds query_instruction) and finds Top-K in specified collection_name (for Milvus or Qdrant) or default index.

**Output Format (JSON)**
```json
Expand Down Expand Up @@ -1820,7 +1820,7 @@ backend_configs:
save_path: index/bm25

# Index Backend Configuration
index_backend: faiss # options: faiss, milvus
index_backend: faiss # options: faiss, milvus, qdrant
index_backend_configs:
faiss:
index_use_gpu: True
Expand Down Expand Up @@ -1878,9 +1878,9 @@ Parameter Description:
| `model_name_or_path` | str | Retrieval model path or name (e.g., HuggingFace model ID) |
| `corpus_path` | str | Input corpus JSONL file path |
| `embedding_path` | str | Vector file save path (`.npy`) |
| `collection_name` | str | Milvus collection name |
| `collection_name` | str | Milvus or Qdrant collection name |
| `backend` | str | Select retrieval backend: `infinity`, `sentence_transformers`, `openai`, `bm25` |
| `index_backend` | str | Index backend: `faiss`, `milvus` |
| `index_backend` | str | Index backend: `faiss`, `milvus`, `qdrant` |
| `backend_configs` | dict | Parameter configuration for each backend (see table below) |
| `index_backend_configs` | dict | Parameter configuration for each index backend (see table below) |
| `websearch_backend` | str | Web search backend: `tavily`, `exa`, `zhipuai` |
Expand Down Expand Up @@ -1941,6 +1941,12 @@ Parameter Description:
||index_params| Dict| Index construction parameters (e.g., index_type: AUTOINDEX)|
||search_params |Dict |Retrieval parameters (e.g., nprobe etc.)|
||index_chunk_size| int| Batch size when inserting data|
|qdrant|url| str| Qdrant server URL (e.g., http://localhost:6333); leave null to use local path|
||path| str |Local persistent storage path; used when url is null|
||api_key| str |API key for authentication (if needed)|
||text_field_name| str |Payload field name for document text (default contents)|
||distance| str |Distance metric: Cosine, Dot, Euclid, Manhattan (default Cosine)|
||index_chunk_size| int |Batch size when upserting vectors|

================
File: pages/en/api/router.mdx
Expand Down
7 changes: 7 additions & 0 deletions examples/demos/parameter/AgentCPM-Report_parameter.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,13 @@ retriever:
metric_type: IP
params: {}
index_chunk_size: 1000
qdrant:
url: null
path: index/qdrant
api_key: null
text_field_name: contents
distance: Cosine
index_chunk_size: 50000
is_demo: false
collection_name: wiki
top_k: 5
Expand Down
7 changes: 7 additions & 0 deletions examples/demos/parameter/AgentCPM-Report_web_parameter.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,13 @@ retriever:
metric_type: IP
params: {}
index_chunk_size: 1000
qdrant:
url: null
path: index/qdrant
api_key: null
text_field_name: contents
distance: Cosine
index_chunk_size: 50000
is_demo: false
collection_name: wiki
top_k: 5
Expand Down
7 changes: 7 additions & 0 deletions examples/demos/parameter/LLM_memory_parameter.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,13 @@ retriever:
metric_type: IP
params: {}
index_chunk_size: 1000
qdrant:
url: null
path: index/qdrant
api_key: null
text_field_name: contents
distance: Cosine
index_chunk_size: 50000
is_demo: false
collection_name: wiki
top_k: 5
Expand Down
7 changes: 7 additions & 0 deletions examples/demos/parameter/LightResearch_parameter.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,13 @@ retriever:
token: null
uri: index/milvus_demo.db
vector_field_name: vector
qdrant:
url: null
path: index/qdrant
api_key: null
text_field_name: contents
distance: Cosine
index_chunk_size: 50000
is_demo: false
is_multimodal: false
model_name_or_path: openbmb/MiniCPM-Embedding-Light
Expand Down
7 changes: 7 additions & 0 deletions examples/demos/parameter/RAG_memory_parameter.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,13 @@ retriever:
metric_type: IP
params: {}
index_chunk_size: 1000
qdrant:
url: null
path: index/qdrant
api_key: null
text_field_name: contents
distance: Cosine
index_chunk_size: 50000
is_demo: false
collection_name: wiki
top_k: 5
Expand Down
7 changes: 7 additions & 0 deletions examples/demos/parameter/RAG_parameter.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,13 @@ retriever:
token: null
uri: index/milvus_demo.db
vector_field_name: vector
qdrant:
url: null
path: index/qdrant
api_key: null
text_field_name: contents
distance: Cosine
index_chunk_size: 50000
is_demo: false
is_multimodal: false
model_name_or_path: openbmb/MiniCPM-Embedding-Light
Expand Down
7 changes: 7 additions & 0 deletions examples/demos/parameter/RAG_web_parameter.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,13 @@ retriever:
metric_type: IP
params: {}
index_chunk_size: 1000
qdrant:
url: null
path: index/qdrant
api_key: null
text_field_name: contents
distance: Cosine
index_chunk_size: 50000
is_demo: false
collection_name: wiki
top_k: 5
Expand Down
7 changes: 7 additions & 0 deletions examples/demos/parameter/milvus_index_parameter.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,13 @@ retriever:
metric_type: IP
params: {}
index_chunk_size: 1000
qdrant:
url: null
path: index/qdrant
api_key: null
text_field_name: contents
distance: Cosine
index_chunk_size: 50000
is_demo: false
collection_name: wiki
embedding_path: embedding/embedding.npy
Expand Down
10 changes: 10 additions & 0 deletions examples/demos/qdrant_index.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# Qdrant Embedding and Indexing Demo for UltraRAG UI

# MCP Server
servers:
retriever: servers/retriever

# MCP Client Pipeline
pipeline:
- retriever.retriever_init
- retriever.retriever_index
10 changes: 10 additions & 0 deletions examples/experiments/qdrant_index.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# Qdrant Embedding and Indexing Demo for UltraRAG UI

# MCP Server
servers:
retriever: servers/retriever

# MCP Client Pipeline
pipeline:
- retriever.retriever_init
- retriever.retriever_index
4 changes: 3 additions & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -43,10 +43,12 @@ retriever = [
"sentence-transformers",
"openai",
"bm25s",
"faiss-gpu-cu12",
"faiss-gpu-cu12; sys_platform != 'darwin'",
"faiss-cpu; sys_platform == 'darwin'",
"exa_py",
"tavily-python",
"pymilvus",
"qdrant-client",
"numba",
"torch",
"fastapi",
Expand Down
9 changes: 9 additions & 0 deletions script/deploy_retriever_config.json
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,15 @@
"params": {}
},
"index_chunk_size": 50000
},
"qdrant": {
"url": null,
"path": "index/qdrant",
"api_key": null,
"collection_name": "ultrarag_embeddings",
"text_field_name": "contents",
"distance": "Cosine",
"index_chunk_size": 50000
}
},

Expand Down
9 changes: 8 additions & 1 deletion servers/retriever/parameter.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ backend_configs:
tokenizer: auto # options: auto, default, jieba, character; auto uses jieba for Chinese
save_path: index/bm25

index_backend: faiss # options: faiss, milvus
index_backend: faiss # options: faiss, milvus, qdrant
index_backend_configs:
faiss:
index_use_gpu: True
Expand All @@ -53,6 +53,13 @@ index_backend_configs:
metric_type: IP
params: {}
index_chunk_size: 1000
qdrant:
url: null # http://localhost:6333 for server, or leave null for local path
path: index/qdrant # local persistent storage; ignored when url is set
api_key: null
text_field_name: contents
distance: Cosine # options: Cosine, Dot, Euclid, Manhattan
index_chunk_size: 50000

websearch_backend: tavily # options: tavily, exa, zhipuai
websearch_backend_configs:
Expand Down
1 change: 1 addition & 0 deletions servers/retriever/src/index_backends/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
_INDEX_BACKENDS: Dict[str, str] = {
"faiss": ".faiss_backend.FaissIndexBackend",
"milvus": ".milvus_backend.MilvusIndexBackend",
"qdrant": ".qdrant_backend.QdrantIndexBackend",
}


Expand Down
Loading