fix: Preserve casts for horizontal ops with untyped literals#27011
Open
abhidotsh wants to merge 2 commits intopola-rs:mainfrom
Open
fix: Preserve casts for horizontal ops with untyped literals#27011abhidotsh wants to merge 2 commits intopola-rs:mainfrom
abhidotsh wants to merge 2 commits intopola-rs:mainfrom
Conversation
6e5ae19 to
3b67ecd
Compare
3b67ecd to
6e91964
Compare
abhidotsh
commented
Mar 24, 2026
py-polars/tests/conftest.py
Outdated
| lambda: ( | ||
| lf.remote(plan_type="plain") | ||
| .distributed() | ||
| .execute() |
Author
There was a problem hiding this comment.
This was failing due to the lint error in https://github.com/pola-rs/polars/actions/runs/23468773183/job/68286715771?pr=27011 - currently investigating why this is the case. mypy was locally green for me after doing this:
Author
There was a problem hiding this comment.
I see it is failing here for quite a few PRs: https://github.com/pola-rs/polars/actions/workflows/lint-python.yml
If there is a preference to fix this in a separate PR, please let me know.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #27011 +/- ##
==========================================
+ Coverage 80.92% 81.47% +0.54%
==========================================
Files 1809 1809
Lines 249500 249507 +7
Branches 3140 3140
==========================================
+ Hits 201919 203280 +1361
+ Misses 46776 45422 -1354
Partials 805 805 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR aims to solve for #26723, and is a follow up based on the review from: #26767
As per the https://docs.pola.rs/development/contributing/#pull-requests, here are the screenshots of


make testlocally passing:In this PR, AI was used to assist with additional root cause investigation, identifying the fix approach given reviewer feedback on #26767, and initial test generation. However, I confirm the final implementation and test updates were written by me.
@nameexhaustion following up on the last comments regarding
args_to_supertype, my understanding of the suggestion was, "can we fix this bug by only changingargs_to_supertype?" If that is the case, I could not find a correct fix by only changingargs_to_supertype, because the bug is not in the supertype computation. It is that horizontal ops were not opting into the shared coercion pass.To support the above with some more details:
args_to_supertypeappears to compute a type from a list of dtypes based on this: https://github.com/abhidotsh/polars/blob/e58a0c3bed13720c2e99725865339dab13eede0b/crates/polars-plan/src/plans/aexpr/function_expr/schema.rs#L809-L832 It does not control whether the function's inputs actually get cast to that type prior to execution. The casting appears to happen in the shared coercion pass right here: https://github.com/abhidotsh/polars/blob/e58a0c3bed13720c2e99725865339dab13eede0b/crates/polars-plan/src/plans/conversion/type_coercion/mod.rs#L639-L704materialize_smallest_dyn_intwhen combining a concrete numeric type with an unknown integer literal to pick the smallest fitting integer type. For0, that leads toInt8viamaterialize_smallest_dyn_int(https://github.com/abhidotsh/polars/blob/e58a0c3bed13720c2e99725865339dab13eede0b/crates/polars-core/src/utils/supertype.rs#L652-L663)materialize_dyn_intwhich turns small ints into Int32, this can be seen here: https://github.com/abhidotsh/polars/blob/e58a0c3bed13720c2e99725865339dab13eede0b/crates/polars-core/src/utils/supertype.rs#L611-L618 and which appears to be the mismatch.To add on to the above points,
args_to_supertypeis also shared by other functions viamap_to_supertype, includingFillNullandCoalesce:This means that if we tried to fix this via
args_to_supertype, we would be changing a shared schema helper instead of horizontal-specific behavior.If the above makes sense, the proper fix should be to not change the supertype computation, but to instead make horizontal ops opt into the shared coercion pass so the literal gets cast to the agreed supertype before the function runs. As part of the fix, the supertyping uses
!ALLOW_PRIMITIVE_TO_STRINGto preventmin_horizontal(int_col, str_col)from silently coercing to string comparison instead of erroring. This changes the existing error from a runtimeComputeErrorto a planning-timeInvalidOperationErrorwhich we can see is reflected intest_raise_invalid_types_21835.Please let me know if there are any questions/comments/suggestions/concerns, and correct me if my understanding is incorrect on any of the above, given my new (and limited) understanding of the Polars codebase. If there is a specific fix already in mind or any general suggestions on the current PR, please let me know and I am happy to adjust.