Skip to content

GH-48986: [C++][Python] Add ORC predicate pushdown support #48984

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
cbb330 wants to merge 5 commits into
base: main
Choose a base branch
from
Draft

GH-48986: [C++][Python] Add ORC predicate pushdown support #48984

cbb330 wants to merge 5 commits into from
+2,939 −228

Conversation

@cbb330
Copy link

@cbb330 cbb330 commented Jan 26, 2026 •
edited by github-actions bot
Loading

Summary

  • Add comprehensive predicate pushdown support for ORC reader in C++ and Python
  • Implement OrcMultiStripeReader for efficient reading of filtered stripes
  • Add detailed design documentation for ORC predicate pushdown
  • Enhance Python bindings to expose stripe filtering capabilities
  • Add extensive test coverage for predicate pushdown functionality

Changes

  • C++ Core: New OrcMultiStripeReader class handles reading from multiple selected stripes after predicate filtering
  • Python Bindings: Expose stripe filtering through updated _orc.pyx interface
  • Dataset Integration: Enhanced file_orc.cc with predicate pushdown support
  • Documentation: Comprehensive design document explaining implementation approach
  • Testing: Added unit tests for stripe filtering and predicate evaluation

Test Plan

  • New C++ tests in file_orc_test.cc verify stripe filtering behavior
  • Python tests in test_orc.py validate end-to-end predicate pushdown
  • Standalone test program cpp/test_orc_pushdown.cc for integration testing

cbb330 and others added 3 commits January 24, 2026 13:40
@cbb330
[C++][Python] Add predicate pushdown support for ORC reader
da2d57c 
Add `filters` parameter to ORC reader that enables stripe skipping
for integer columns using liborc's SearchArgument mechanism.

C++ changes:
- Add ExpressionToSearchArgument() to convert Arrow Expression to
  liborc SearchArgument
- Add Read() overload accepting compute::Expression filter
- Support comparison operators (=, !=, <, <=, >, >=), is_in, is_null,
  and logical operators (AND, OR, NOT)

Python changes:
- Add filters parameter to ORCFile.read() and read_table()
- Support both DNF format and Arrow Expression filters
- Apply row-level filtering after stripe-level pushdown

Tests:
- Add TestORCPredicatePushdown with 11 tests covering correctness,
  safety, operators, null handling, and Expression API
@cbb330
[Docs] Add ORC predicate pushdown design document
02b8f90 
Add comprehensive design document for ORC predicate pushdown feature:
- Architecture overview and data flow
- API design with examples
- Implementation plan across multiple PRs
- Technical decisions and rationale
- Type mapping and operator support tables
- Testing strategy
- Future considerations

This document serves as a reference for community review and tracks
progress across the phased implementation.
@cbb330 @claude
[C++][Python] Enhance ORC predicate pushdown with multi-stripe reader
f46d666 
Add OrcMultiStripeReader class to handle reading from multiple selected stripes
after predicate pushdown filtering. Update Python bindings to expose stripe
filtering capabilities and enhance test coverage for ORC predicate pushdown.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@github-actions
Copy link

github-actions bot commented Jan 26, 2026

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

@cbb330 cbb330 changed the title [C++][Python] Add ORC predicate pushdown support GH-48986: [C++][Python] Add ORC predicate pushdown support Jan 26, 2026
@github-actions
Copy link

github-actions bot commented Jan 26, 2026

⚠️ GitHub issue #48986 has been automatically assigned in GitHub to PR creator.

cbb330 and others added 2 commits January 26, 2026 15:39
@cbb330 @claude
[C++][Python] Fix memory safety issues in ORC multi-stripe reader
d6efeae 
Fix critical lifetime management issues in OrcMultiStripeReader where the
underlying liborc::Reader could be destroyed while stripe readers were still
in use. The reader now holds a shared_ptr to the liborc::Reader, ensuring
proper lifetime management and preventing use-after-free errors.

Changes:
- OrcMultiStripeReader now holds shared_ptr to liborc::Reader
- Add lifetime management documentation
- Ensure reader remains valid for all stripe operations
- Fix memory safety issues in multi-stripe predicate pushdown

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@cbb330 @claude
[C++][Python] Improve error handling in ORC predicate pushdown
f011d77 
Change GetStripeStatsAsExpression to return std::optional<Expression> instead
of Result<Expression> for better handling of missing statistics. Return nullopt
when statistics are unavailable rather than returning literal(true) or propagating
errors, allowing stripe filtering to gracefully skip fields without statistics.

Add comprehensive dataset API integration tests covering stripe skipping with
filters, column projection, batch size handling, and edge cases.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@AlenkaF AlenkaF Awaiting requested review from AlenkaF AlenkaF will be requested when the pull request is marked ready for review AlenkaF is a code owner

@raulcd raulcd Awaiting requested review from raulcd raulcd will be requested when the pull request is marked ready for review raulcd is a code owner

@rok rok Awaiting requested review from rok rok will be requested when the pull request is marked ready for review rok is a code owner

@wgtmac wgtmac Awaiting requested review from wgtmac wgtmac will be requested when the pull request is marked ready for review wgtmac is a code owner

Assignees

No one assigned

Labels

awaiting review Awaiting review Component: C++ Component: Documentation Component: Python

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@cbb330