Background & Motivation
I'd like to contribute MATCH_RECOGNIZE (event sequence pattern matching) support to Arroyo. This is a powerful streaming feature for detecting trends, anomalies, and complex event patterns (e.g., "price goes up then down" stock patterns).
However, implementing this requires DataFusion's RelationPlanner extension API, which was introduced in apache/datafusion#17843 and first released in DataFusion v51.0.0.
Arroyo currently depends on DataFusion v48.0.0, which lacks this critical extension point.
Proposal
Upgrade Arroyo's DataFusion dependency to v51.0.0 or later.
Benefits
Unlocks RelationPlanner: The foundation for implementing MATCH_RECOGNIZE – intercepting the SQL syntax at parse time and converting it to a custom logical plan.
Stay current with upstream: DataFusion v48 → v51 includes numerous performance improvements, bug fixes, and new features that Arroyo would benefit from.
Paves the way for future extensions: RelationPlanner can also support other SQL extensions like TABLESAMPLE, PIVOT/UNPIVOT, not just MATCH_RECOGNIZE.
Upgrade Impact Assessment
DataFusion maintains good API stability across releases, though there may be some deprecated API changes between v48 and v51. I'm willing to take on the following responsibilities:
Evaluate compatibility between DataFusion v51.0.0 and Arroyo's existing codebase.
Fix any API breaking changes that arise during the upgrade.
Ensure all existing tests pass.
Roadmap
Once the upgrade is complete, I plan to implement MATCH_RECOGNIZE in two phases:
Logical planning: Following the official DataFusion example match_recognize.rs, implement a MatchRecognizePlanner (the RelationPlanner trait) to convert SQL into a custom logical plan node.
Physical execution: Implement the corresponding ExecutionPlan that ports Flink's NFA-based matching logic to actually execute pattern matching over streams.
Questions
Are there any specific concerns about upgrading DataFusion (stability, regression risk, etc.) from the team's side?
Are there any known compatibility issues with v51.0.0 that I should be aware of?
I'm looking forward to your feedback. If the upgrade is feasible, I'm fully committed to driving the subsequent development.
Background & Motivation
I'd like to contribute MATCH_RECOGNIZE (event sequence pattern matching) support to Arroyo. This is a powerful streaming feature for detecting trends, anomalies, and complex event patterns (e.g., "price goes up then down" stock patterns).
However, implementing this requires DataFusion's RelationPlanner extension API, which was introduced in apache/datafusion#17843 and first released in DataFusion v51.0.0.
Arroyo currently depends on DataFusion v48.0.0, which lacks this critical extension point.
Proposal
Upgrade Arroyo's DataFusion dependency to v51.0.0 or later.
Benefits
Unlocks RelationPlanner: The foundation for implementing MATCH_RECOGNIZE – intercepting the SQL syntax at parse time and converting it to a custom logical plan.
Stay current with upstream: DataFusion v48 → v51 includes numerous performance improvements, bug fixes, and new features that Arroyo would benefit from.
Paves the way for future extensions: RelationPlanner can also support other SQL extensions like TABLESAMPLE, PIVOT/UNPIVOT, not just MATCH_RECOGNIZE.
Upgrade Impact Assessment
DataFusion maintains good API stability across releases, though there may be some deprecated API changes between v48 and v51. I'm willing to take on the following responsibilities:
Evaluate compatibility between DataFusion v51.0.0 and Arroyo's existing codebase.
Fix any API breaking changes that arise during the upgrade.
Ensure all existing tests pass.
Roadmap
Once the upgrade is complete, I plan to implement MATCH_RECOGNIZE in two phases:
Logical planning: Following the official DataFusion example match_recognize.rs, implement a MatchRecognizePlanner (the RelationPlanner trait) to convert SQL into a custom logical plan node.
Physical execution: Implement the corresponding ExecutionPlan that ports Flink's NFA-based matching logic to actually execute pattern matching over streams.
Questions
Are there any specific concerns about upgrading DataFusion (stability, regression risk, etc.) from the team's side?
Are there any known compatibility issues with v51.0.0 that I should be aware of?
I'm looking forward to your feedback. If the upgrade is feasible, I'm fully committed to driving the subsequent development.