Skip to content

feat: add a custom ExprPlanner to support colon operator#57

Open
Samyak2 wants to merge 4 commits intodatafusion-contrib:mainfrom
Samyak2:colon-operator
Open

feat: add a custom ExprPlanner to support colon operator#57
Samyak2 wants to merge 4 commits intodatafusion-contrib:mainfrom
Samyak2:colon-operator

Conversation

@Samyak2
Copy link
Copy Markdown

@Samyak2 Samyak2 commented Mar 30, 2026

Closes #26

  • Add a custom ExprPlanner that handles : as an operator and re-writes it to variant_get.

Closes datafusion-contrib#26

- Add a custom `ExprPlanner` that handles `:` as an operator and
  re-writes it to `variant_get`.
@Samyak2
Copy link
Copy Markdown
Author

Samyak2 commented Mar 30, 2026

TODO:

  • This handles the 2-arg version. The 3-arg version (with cast) is yet to be converted
  • Tests

Copy link
Copy Markdown
Member

@friendlymatthew friendlymatthew left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Samyak2

Can we add some tests as well please

Comment on lines +17 to +18
#[derive(Debug)]
pub struct VariantExprPlanner;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add some comments here that explain what this does

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Please take a look

@Samyak2
Copy link
Copy Markdown
Author

Samyak2 commented Mar 31, 2026

There seems to a bug in the sql parser. These are failing:

select variant_pretty(json_to_variant(json_str):[1]) from json_data;

and

select variant_pretty(json_to_variant(json_str):['name']) from json_data;

fails with:

SQL error: ParserError("Expected: variant object key name, found: [ at Line: 1, Column: 49")

@Samyak2
Copy link
Copy Markdown
Author

Samyak2 commented Mar 31, 2026

Another problem. Cast does not get parsed as a BinaryOperator. It's a separate variant of Expr: https://docs.rs/sqlparser/0.61.0/sqlparser/ast/enum.Expr.html#variant.Cast

ExprPlanner currently does not support custom parsing for cast.

So we won't be able to support 3-arg version like variant_col:name::string, etc.

@Samyak2 Samyak2 marked this pull request as ready for review March 31, 2026 08:47
@sdf-jkl
Copy link
Copy Markdown
Contributor

sdf-jkl commented Apr 3, 2026

There seems to a bug in the sql parser. These are failing:

select variant_pretty(json_to_variant(json_str):[1]) from json_data;

and

select variant_pretty(json_to_variant(json_str):['name']) from json_data;

fails with:

SQL error: ParserError("Expected: variant object key name, found: [ at Line: 1, Column: 49")

Is this a bug though? It seems like a Databricks dialect specifics.

Another problem. Cast does not get parsed as a BinaryOperator. It's a separate variant of Expr: https://docs.rs/sqlparser/0.61.0/sqlparser/ast/enum.Expr.html#variant.Cast

ExprPlanner currently does not support custom parsing for cast.

So we won't be able to support 3-arg version like variant_col:name::string, etc.

Another issue to work on in sqlparser-rs?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

colon : operator - initial implementation

4 participants