chore: update DataFusion dependencies to 53.0.0#1442
Draft
Conversation
…anges (partial) Co-authored-by: linhr <5601366+linhr@users.noreply.github.com>
…SimplifyContext, PartitionedFile, ScalarValue) Co-authored-by: linhr <5601366+linhr@users.noreply.github.com>
Copilot stopped work on behalf of
linhr due to an error
February 28, 2026 04:12
…physical-plan properties, CoalesceBatches removal Co-authored-by: linhr <5601366+linhr@users.noreply.github.com>
Copilot
AI
changed the title
[WIP] Update DataFusion dependencies to git revision 5d8249f
chore: update DataFusion dependencies to git rev Feb 28, 2026
5d8249f
linhr
added
run spark tests
Spark 3.5.7 Test ReportCommit Information
Test Summary
Test DetailsError CountsPassed Tests Diff--- before.txt 2026-03-25 11:25:06.692763681 +0000
+++ after.txt 2026-03-25 11:25:06.821763680 +0000
@@ -138 +137,0 @@
-pyspark/sql/functions.py::pyspark.sql.functions.array_compact
@@ -143 +141,0 @@
-pyspark/sql/functions.py::pyspark.sql.functions.array_intersect
@@ -150,0 +149 @@
+pyspark/sql/functions.py::pyspark.sql.functions.array_unionFailed Tests |
Spark 4.1.1 Test ReportCommit Information
Test Summary
Test DetailsError CountsPassed Tests Diff--- before.txt 2026-03-25 11:27:07.897123861 +0000
+++ after.txt 2026-03-25 11:27:08.346126989 +0000
@@ -146 +145,0 @@
-pyspark/sql/functions/builtin.py::pyspark.sql.functions.builtin.array_compactFailed Tests(truncated) |
Ibis Test ReportCommit Information
Test Summary
Test DetailsError CountsPassed Tests Diff--- before.txt 2026-03-25 11:26:56.225576034 +0000
+++ after.txt 2026-03-25 11:26:56.421577040 +0000
@@ -0,0 +1,2 @@
+ibis/backends/pyspark/tests/test_aggregation.py::test_aggregation_float_nulls[count]
+ibis/backends/pyspark/tests/test_aggregation.py::test_aggregation_float_nulls[sum]
@@ -724 +725,0 @@
-ibis/backends/tests/test_generic.py::test_uncorrelated_subquery[pyspark]
@@ -1408,0 +1410 @@
+ibis/backends/tests/test_temporal.py::test_temporal_binop[pyspark-date-subtract-date]
@@ -1478,0 +1481 @@
+ibis/backends/tests/test_udf.py::test_vectorized_udf[pyspark-add_one_pandas]Failed Tests |
Contributor
|
@copilot Please investigate and fix the issues with Rust tests and Python tests. |
…oid push_down_leaf_projections optimizer issue Co-authored-by: linhr <5601366+linhr@users.noreply.github.com>
Contributor
Author
Investigated both Rust and Python test failures. Here's a summary of what was fixed: Rust test failures:
Python test failures (including Root cause: The new DataFusion optimizer rule Fix (commit |
26 tasks
Contributor
|
@copilot Please change DataFusion |
…ect_store to 0.13.1, fix breaking changes Co-authored-by: linhr <5601366+linhr@users.noreply.github.com>
…for arrow 58 upgrade Co-authored-by: linhr <5601366+linhr@users.noreply.github.com>
Contributor
Author
Updated DataFusion to rev Key changes made:
Note: Warning Firewall rules blocked me from connecting to one or more addresses (expand for details)I tried to connect to the following addresses, but was blocked by firewall rules:
If you need me to access, download, or install something from one of these locations, you can either:
|
Copilot
AI
changed the title
chore: update DataFusion dependencies to git rev
chore: update DataFusion dependencies to git rev Feb 28, 2026
5d8249f73fbd48
Contributor
Author
Investigation complete. Here's the root cause: Root cause: 4 MERGE tests fail with data loss when run after Mechanism: In
Under some execution conditions (specifically when the session has executed prior MERGEs), the NOT MATCHED BY SOURCE rows in Candidate fix: Wrap |
Contributor
|
@copilot Please continue the investigation and implement the fix. |
…loss in MERGE Co-authored-by: linhr <5601366+linhr@users.noreply.github.com>
Contributor
Author
Fixed in commit Root cause:
…NOT MATCHED BY SOURCE rows sharing a physical file with MATCHED rows were silently dropped under certain execution conditions (specifically after a prior MERGE had established multi-partition execution state in the session). Fix: Added |
Contributor
|
@copilot Please merge from |
Co-authored-by: linhr <5601366+linhr@users.noreply.github.com>
…rmat Co-authored-by: linhr <5601366+linhr@users.noreply.github.com>
Contributor
Author
Merged from 3 MERGE tests remain failing ( |
Contributor
|
@copilot Please merge from |
…sion-dependencies
Contributor
Author
Merged from
Build passes ✓ |
Contributor
|
@copilot Please merge from main and resolve merge conflicts. Then bump datafusion version to 53.0.0. |
Co-authored-by: lonless9 <167735979+lonless9@users.noreply.github.com> Agent-Logs-Url: https://github.com/lakehq/sail/sessions/448d1f16-215f-4d33-8697-3063dfcede45
Co-authored-by: lonless9 <167735979+lonless9@users.noreply.github.com> Agent-Logs-Url: https://github.com/lakehq/sail/sessions/448d1f16-215f-4d33-8697-3063dfcede45
Copilot
AI
changed the title
chore: update DataFusion dependencies to git rev
chore: update DataFusion dependencies to 53.0.0
Mar 25, 2026
73fbd48
Gold Data ReportNotes
Commit Information
Summary
DetailsGold Data Metrics
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
73fbd48, Arrow/Parquet to 58.0.0, object_store to 0.13.1sail-data-sourcesail-object-store,sail-iceberghdfs-native-object-storeto 0.16.0cargo clippy --all-targets --all-features -- -D warningspasses ✓explodeworkaround; restore originalget_field-based approach inExplodeRewriterpush_down_leaf_projectionsoptimizer rule incrates/sail-session/src/optimizer.rsSAIL_UPDATE_GOLD_DATA=1 cargo nextest run)sail-iceberg physical_plan::action_schema::tests::encode_add_and_commit_meta_roundtripmainbranch (which removed the delta-kernel-rs dependency via the "internalization for Delta Lake" refactor)CoalescePartitionsExecbeforeprojectedinfinalize_mergemainagain (YAML plan snapshots, updated snapshot file locations)mainagain (resolved add/add conflicts indelete.yamlandmerge.yaml)mainagain (in-commit timestamps, time travel, write mode refactoring, checksum, checkpoint features)ObjectStoreExtimports in newdelta_log/module and Iceberg files)finalize_mergetargeted-rewrite path)🔒 GitHub Advanced Security automatically protects Copilot coding agent pull requests. You can protect all pull requests by enabling Advanced Security for your repositories. Learn more about Advanced Security.