Skip to content

deps: DataFusion 52.0.0 migration (SchemaAdapter changes, etc.) #3536

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
comphead wants to merge 32 commits into
base: main
Choose a base branch
from
Open

deps: DataFusion 52.0.0 migration (SchemaAdapter changes, etc.) #3536

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
3e1dbd0
Move DF52 work to shared branch (#3469)
comphead Feb 10, 2026
88caabb
fix: use Spark date_add/date_sub UDFs for Date32 +/- Int arithmetic (…
andygrove Feb 10, 2026
7983091
Fix fuzz shuffle tests
comphead Feb 10, 2026
36dcbcd
chore: [df52] fix index out of bounds for `native_datafusion` scan (#…
comphead Feb 11, 2026
3a73345
fix: handle complex nested type casts in schema adapter for DF52 (#3475)
andygrove Feb 11, 2026
d410fcf
fix: [df52] make relabel_array recursive for nested type mismatches (…
andygrove Feb 11, 2026
c564aaa
fix: [df52] route timestamp timezone mismatches through spark_parquet…
andygrove Feb 12, 2026
71c9e56
fix: [df52] handle case-insensitive column matching in PhysicalExprAd…
andygrove Feb 12, 2026
a7d9d43
fix: update dictionary unpack tests for DF52 stream polling changes (…
andygrove Feb 12, 2026
ca89797
clippy
andygrove Feb 12, 2026
86b3c16
fix: [df52] timestamp nanos precision loss with nanosAsLong (#3502)
andygrove Feb 12, 2026
8c08c27
Df52 migration - ignore `width_bucket` (#3501)
comphead Feb 12, 2026
c420c10
fix: [df52] schema pruning crash on complex nested types (#3500)
andygrove Feb 12, 2026
788c90c
fix: [df52] skip ParquetVariantShreddingSuite for Spark 4.0 (#3503)
andygrove Feb 12, 2026
ae5ec18
Df52 migration - get failed tests
comphead Feb 12, 2026
1f9b9ff
[df52] miri
comphead Feb 12, 2026
1497da4
[df52] miri
comphead Feb 12, 2026
1f6a539
[df52] miri
comphead Feb 12, 2026
3e1eaee
[df52] miri
comphead Feb 12, 2026
adfaf62
[df52] tests ignore
comphead Feb 13, 2026
08344e1
[df52] tests ignore
comphead Feb 13, 2026
cc5220d
DataFusion 52 migration
comphead Feb 13, 2026
4a60a8d
[df52] tests ignore
comphead Feb 13, 2026
32501ec
DataFusion 52 migration
comphead Feb 14, 2026
6fd3bb7
[df52] tests ignore
comphead Feb 14, 2026
694faab
DataFusion 52 migration
comphead Feb 14, 2026
329a42b
DataFusion 52 migration
comphead Feb 14, 2026
749a407
DataFusion 52 migration
comphead Feb 16, 2026
a63b9bf
DataFusion 52 migration
comphead Feb 16, 2026
7d2b842
DataFusion 52 migration
comphead Feb 18, 2026
5bbd4f0
DataFusion 52 migration
comphead Feb 18, 2026
105ccec
DataFusion 52 migration
comphead Feb 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion dev/diffs/3.5.8.diff
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2795,7 +2795,7 @@ index d675503a8ba..f220892396e 100644
+ }
Copy link
Contributor

@mbutrovich mbutrovich Feb 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this file just has a whitespace change, can you get the version of the file from main to reduce the diff?

assert(bucketedScan.length == expectedNumBucketedScan)
}

diff --git a/sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSinkSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSinkSuite.scala
index 7f6fa2a123e..c778b4e2c48 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSinkSuite.scala
Expand Down
24 changes: 24 additions & 0 deletions dev/diffs/4.0.1.diff
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2960,6 +2960,30 @@ index 09ed6955a51..236a4e99824 100644
)
}
test(s"parquet widening conversion $fromType -> $toType") {
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetVariantShreddingSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetVariantShreddingSuite.scala
index 458b5dfc0f4..d209f3c85bc 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetVariantShreddingSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetVariantShreddingSuite.scala
@@ -26,7 +26,7 @@ import org.apache.parquet.hadoop.util.HadoopInputFile
import org.apache.parquet.schema.{LogicalTypeAnnotation, PrimitiveType}
import org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName

-import org.apache.spark.sql.{QueryTest, Row}
+import org.apache.spark.sql.{IgnoreCometSuite, QueryTest, Row}
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.internal.SQLConf.ParquetOutputTimestampType
import org.apache.spark.sql.test.SharedSparkSession
@@ -35,7 +35,9 @@ import org.apache.spark.unsafe.types.VariantVal
/**
* Test shredding Variant values in the Parquet reader/writer.
*/
-class ParquetVariantShreddingSuite extends QueryTest with ParquetTest with SharedSparkSession {
+class ParquetVariantShreddingSuite extends QueryTest with ParquetTest with SharedSparkSession
+ // TODO enable tests once https://github.com/apache/datafusion-comet/issues/2209 is fixed
+ with IgnoreCometSuite {

private def testWithTempDir(name: String)(block: File => Unit): Unit = test(name) {
withTempDir { dir =>
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/debug/DebuggingSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/debug/DebuggingSuite.scala
index b8f3ea3c6f3..bbd44221288 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/execution/debug/DebuggingSuite.scala
Expand Down
Loading
Loading