Optimised RAG

[VyasGuru]

OPTIMISED RAG FOR FASTER CHUNKING, BATCH PROCESSING.
implemented hybrid search(testing out so i can tweak config)

Greptile Summary

This PR implements major optimizations to the RAG system with hybrid search capabilities. The rewrite introduces BM25 sparse keyword search alongside dense vector search, using Reciprocal Rank Fusion (RRF) to merge results. Document chunking has been completely rewritten with a recursive algorithm using hierarchical separators for better boundary detection. Batch embedding processing improves throughput significantly.

Key improvements:

• Hybrid search with BM25 + dense embeddings for better retrieval of exact keywords, acronyms, and rare terms
• Recursive chunking with configurable separators improves document splitting quality
• Batch embedding operations reduce overhead for multi-document ingestion
• i8 quantization in vector store reduces memory footprint
• SIMD tokenizer optimizations for ARM NEON architectures

Critical issue found:

• Compilation error in rac_rag_pipeline_create_standalone: undeclared variable embed_handle (line 124) will prevent the code from building

Confidence Score: 2/5

• Cannot merge due to compilation error that prevents building
• The undeclared embed_handle variable in rac_rag_pipeline.cpp line 124 is a critical syntax error that will cause compilation failure. While the architectural changes are sound and the hybrid search implementation is well-designed, the code cannot be built or tested in its current state.
• Pay close attention to sdk/runanywhere-commons/src/features/rag/rac_rag_pipeline.cpp which contains a critical compilation error

Important Files Changed

Filename	Overview
sdk/runanywhere-commons/src/features/rag/rac_rag_pipeline.cpp	Critical compilation error: undeclared embed_handle variable prevents build
sdk/runanywhere-commons/src/features/rag/bm25_index.cpp	Solid BM25 implementation with standard parameters (k1=1.2, b=0.75), thread-safe operations
sdk/runanywhere-commons/src/features/rag/rag_chunker.cpp	Recursive chunking implementation with hierarchical separators improves document splitting
sdk/runanywhere-commons/src/features/rag/rag_backend.cpp	Major refactor: hybrid search with RRF fusion, batch embedding support, improved architecture
sdk/runanywhere-commons/src/features/rag/vector_store_usearch.cpp	Optimized vector store with i8 quantization and batch operations for memory efficiency
sdk/runanywhere-commons/src/features/rag/onnx_embedding_provider.cpp	Tokenizer with SIMD optimizations for ARM NEON, LRU cache for token lookups

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[User Query] --> B[RAG Backend]
    B --> C[Embed Query Text]
    C --> D[Parallel Search]
    D --> E[Dense Vector Search<br/>USearch with i8 quantization]
    D --> F[BM25 Keyword Search<br/>Inverted index]
    E --> G[Dense Results<br/>Cosine similarity]
    F --> H[BM25 Results<br/>TF-IDF scoring]
    G --> I[Reciprocal Rank Fusion<br/>k=60]
    H --> I
    I --> J[Top-K Fused Results]
    J --> K[Build Context<br/>Token budget: 2048]
    K --> L[Format Prompt<br/>with context]
    L --> M[LLM Service]
    M --> N[Generated Answer]
    
    style B fill:#e1f5ff
    style I fill:#fff4e1
    style D fill:#f0f0f0

Loading

_{Last reviewed commit: 388aca0}

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[shubhammalhotra28]

@coderabbitai please do a detailed review. Thanks!

[shubhammalhotra28]

@greptile please do a detailed review. Thanks!

[coderabbitai]

@shubhammalhotra28: I'll conduct a detailed review of the PR focusing on the RAG optimizations, chunking improvements, batch processing implementation, and the hybrid search feature. Let me analyze the changes.

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[shubhammalhotra28]

https://github.com/RunanywhereAI/runanywhere-sdks/blob/main/Package.swift#L267-L379

We need to fix this as wel - need to clean up
@VyasGuru

[greptile-apps]

_{153 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

embed_handle not declared

Suggested change

	// Create embeddings service via registry
	rac_handle_t embed_handle = nullptr;
	rac_result_t result = rac_embeddings_create(config->embedding_model_path, &embed_handle);

Prompt To Fix With AI

This is a comment left during a code review.
Path: sdk/runanywhere-commons/src/features/rag/rac_rag_pipeline.cpp
Line: 124

Comment:
`embed_handle` not declared

```suggestion
        rac_handle_t embed_handle = nullptr;
        rac_result_t result = rac_embeddings_create(config->embedding_model_path, &embed_handle);
```

How can I resolve this? If you propose a fix, please make it concise.

[shubhammalhotra28]

this also needs to be removed

[shubhammalhotra28]

This doesn't seems to address the comment I added, so not sure if this was ever tested or addressed. @VyasGuru

[greptile-apps]

Too many files changed for review. (153 files found, 100 file limit)

[ellipsis-dev]

⚠️ This PR is too big for Ellipsis, but support for larger PRs is coming soon. If you want us to prioritize this feature, let us know at help@ellipsis.dev

---

Generated with ❤️ by ellipsis.dev

[shubhammalhotra28]

Merged this branch manually into shubham-rag-fix and resolved all merge conflicts. The changes are now in commit df98f9a0. Closing this PR — thanks @VyasGuru for the optimised RAG work.

[shubhammalhotra28]

If anything is remainng please lmk asap