Introducing Community Extensions

[everettVT]

We're formalizing a pattern several of you have already started building: community extensions. These are reusable libraries that add domain-specific functionality on top of Daft without requiring every specialized function, datatype, or integration to live in core.

A few that already exist:

• daft-lance — Lance file compaction, scalar indexing, column merging, REST catalog ops
• daft-h3 — H3 geospatial indexing
• daft-html — HTML parsing, html_to_text, CSS selector extraction
• daft-geo — geospatial prototype with extension-backed Point2D / Point3D datatypes

Python UDF extensions

Package reusable logic with @daft.func, @daft.func.batch, @daft.cls, and @daft.method.batch. Good for orchestrating Python ecosystems, external services, ML models, GPUs, or domain-specific workflows. This is how daft-lance distributes Lance tasks, and how Daft's own daft.functions.ai module (prompt, embed_text, classify_image, etc.) is built.

Native ABI extensions

For lower-level vectorized performance, ship a pip-installable package that bundles a shared library implementing the Arrow C Data Interface. Register scalar functions, aggregate functions, expression wrappers, and extension-backed datatypes. Rust has the most ergonomic SDK today via daft-ext; C++ works via the raw ABI. Any Arrow implementation that can export a C ABI is feasible.

Examples:

• examples/hello — minimal Rust extension
• examples/dvector — pgvector-style vector distance functions
• examples/hello_cpp — pure C++ extension

Stewardship

We encourage contributors to own and maintain extensions in their own repositories and organizations. Most extensions don't need to live under daft-engine, and keeping them separate keeps iteration fast. Projects with significant adoption, active maintenance, and clear ecosystem alignment may move under daft-engine over time.

Build something and add it to the list!

• Python UDF docs
• Native extension authoring guide
• Reply here or open a discussion with your idea