Skip to content

refactor: Add sinked paths callback#26995

Draft
nameexhaustion wants to merge 1 commit intomainfrom
nxs/sink-paths-callback
Draft

refactor: Add sinked paths callback#26995
nameexhaustion wants to merge 1 commit intomainfrom
nxs/sink-paths-callback

Conversation

@nameexhaustion
Copy link
Collaborator

@nameexhaustion nameexhaustion commented Mar 23, 2026

For sink_iceberg / polars cloud.

pl.LazyFrame({"a": [0, 1, 2, 3, 4]}).sink_parquet(
    pl.PartitionBy(
        ".env/_data_out/partitioned",
        key="a",
    ),
    _sinked_paths_callback=pprint,
)

# SinkedPathsCallbackArgs(paths=['.env/_data_out/partitioned/a=2/00000000.parquet',
#                                '.env/_data_out/partitioned/a=3/00000000.parquet',
#                                '.env/_data_out/partitioned/a=0/00000000.parquet',
#                                '.env/_data_out/partitioned/a=4/00000000.parquet',
#                                '.env/_data_out/partitioned/a=1/00000000.parquet'])

@github-actions github-actions bot added internal An internal refactor or improvement python Related to Python Polars rust Related to Rust Polars changes-dsl Do not merge if this label is present and red. labels Mar 23, 2026
@codecov
Copy link

codecov bot commented Mar 23, 2026

Codecov Report

❌ Patch coverage is 95.76271% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.78%. Comparing base (cd9de5d) to head (22009cd).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
crates/polars-plan/src/dsl/options/sink.rs 90.47% 2 Missing ⚠️
...es/io_sinks/pipeline_initialization/single_file.rs 89.47% 2 Missing ⚠️
...s/io_sinks/pipeline_initialization/partition_by.rs 93.75% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #26995      +/-   ##
==========================================
- Coverage   81.78%   81.78%   -0.01%     
==========================================
  Files        1809     1810       +1     
  Lines      249417   249518     +101     
  Branches     3141     3141              
==========================================
+ Hits       203980   204057      +77     
- Misses      44632    44656      +24     
  Partials      805      805              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@nameexhaustion nameexhaustion marked this pull request as ready for review March 23, 2026 07:47
file_path_provider=lambda _: io.BytesIO(),
max_rows_per_file=1,
),
_sinked_paths_callback=lambda _: None,
Copy link
Member

@ritchie46 ritchie46 Mar 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thinking aloud here, but can we make this callback only exists in IR? I don't think it should live in DSL.

@nameexhaustion nameexhaustion marked this pull request as draft March 23, 2026 10:04
@nameexhaustion nameexhaustion force-pushed the nxs/sink-paths-callback branch from 80700f9 to 22009cd Compare March 23, 2026 10:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changes-dsl Do not merge if this label is present and red. internal An internal refactor or improvement python Related to Python Polars rust Related to Rust Polars

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants