feat: Created DocEmbedder class by patelchaitany · Pull Request #5973 · feast-dev/feast

patelchaitany · 2026-02-16T11:28:33Z

What this PR does / why we need it:

This PR adds a Document Embedder capability to Feast, allowing users to go from raw documents to embeddings stored in the online vector store in a single step. It handles chunking, embedding generation, and writing the results to the online store — providing an end-to-end ingestion pipeline for RAG workflows within Feast.

What changed:

sdk/python/feast/chunker.py

Defines the document chunking layer. Provides:

A BaseChunker abstract class that users can extend with custom chunking strategies
A built-in TextChunker that splits plain text by word count with configurable size, overlap, and thresholds
A chunk_dataframe() convenience method to chunk all documents in a DataFrame

Currently only basic text chunking is implemented. There is room for improvement — future iterations can support more advanced strategies like semantic chunking, sentence-aware splitting, or format-specific chunkers (PDF, HTML, etc.).

sdk/python/feast/embedder.py

Defines the embedding generation layer. Provides:

A BaseEmbedder abstract class with a modality-routing registry for extensibility
A MultiModalEmbedder with built-in support for text (via sentence-transformers) and image (via CLIP) embeddings
Lazy model loading and configurable batch processing

sdk/python/feast/doc_embedder.py

The high-level orchestrator that coordinates chunking, embedding, and storage. Provides:

A DocEmbedder class that runs the full pipeline: chunk -> embed -> logical layer -> write to online store
A user-defined logical layer function to map output to the FeatureView schema
Auto-generation of a default FeatureView if the user does not define their own (controlled by create_feature_view). This is a basic implementation intended as a starting point

sdk/python/feast/init.py

Updated to export DocEmbedder, LogicalLayerFn, BaseChunker, TextChunker, ChunkingConfig, BaseEmbedder, MultiModalEmbedder, and EmbeddingConfig as part of Feast's public API.

Which issue(s) this PR fixes:

Create DocEmbedder class along with RAGRetriever #5426

Misc

patelchaitany · 2026-02-16T11:53:20Z

@ntkathole @jyejare can You pls review this PR and let me know if any changes is needed.

jyejare

Great Addition @patelchaitany , this is a milestone for Feast in RAG. Glad to see multiple types of data are being supported by Embedder.

Few comments and we should be good to go.

jyejare · 2026-02-23T05:16:03Z

sdk/python/tests/unit/test_doc_embedder.py

+        chunker = TextChunker()
+        text = " ".join([f"word{i}" for i in range(200)])
+
+        chunks = chunker.load_parse_and_chunk(source=text, source_id="doc1")


I think chunker should decide the text in source to chunk the text, we should not need to manually feed that.

Yes, Chunker can decide the which field is text in the DataFrame We Just need to pass the column name in the chunk_dataframe function of the Chunker class.

This test only for testing that load_parse_and_chunk function return the match the required return type.

jyejare · 2026-02-23T05:33:27Z

sdk/python/tests/unit/test_doc_embedder.py

+    def test_supported_modalities(self):
+        """After init, supported_modalities returns text and image."""
+        embedder = MultiModalEmbedder()
+        modalities = embedder.supported_modalities()


Supported modalities can be set as a property

sdk/python/tests/unit/test_doc_embedder.py

jyejare · 2026-02-23T10:46:44Z

sdk/python/tests/unit/test_doc_embedder.py

+        assert embedder._image_model is None
+        assert embedder._image_processor is None
+
+    def test_custom_modality_registration(self):


Repeat test ?

This is for when we register new Modality then It will correctly route to the New Modality.

But I agree we can remove this test.

ntkathole · 2026-03-02T10:58:02Z

@patelchaitany filename typo - examples/rag-retriever/rag_feast_docebedder.ipynb should be rag_feast_docembedder.ipynb

sdk/python/feast/doc_embedder.py

sdk/python/feast/embedder.py

sdk/python/feast/chunker.py

ntkathole · 2026-03-02T11:18:17Z

sdk/python/feast/embedder.py

+from dataclasses import dataclass
+from typing import Any, Callable, List, Optional
+
+import numpy as np


Consider lazy-loading numpy too

we cannot do the lazy of numpy as it is required for the Type checking.

sdk/python/feast/embedder.py

…ng them into the FeatureView schema. - Added BaseChunker and TextChunker classes for document chunking. - Updated pyproject.toml to include sentence-transformers dependency. - Created a new Jupyter notebook example for using the RAG retriever with document embedding. Signed-off-by: Chaitany patel <patelchaitany93@gmail.com>

patelchaitany requested a review from a team as a code owner February 16, 2026 11:28

This comment was marked as resolved.

Sign in to view

patelchaitany changed the title ~~feat: Created DocEmbedder class #5972~~ feat: Created DocEmbedder class Feb 16, 2026

This comment was marked as resolved.

Sign in to view

jyejare reviewed Feb 23, 2026

View reviewed changes

patelchaitany requested a review from jyejare February 24, 2026 10:01

This comment was marked as resolved.

Sign in to view

ntkathole force-pushed the fet-DocEmbedder branch from 1ef9d8e to 083eadb Compare March 2, 2026 05:15

ntkathole added the ok-to-test label Mar 2, 2026

ntkathole force-pushed the fet-DocEmbedder branch from 083eadb to bb74079 Compare March 2, 2026 07:52