Skip to content

feat: Support int to timestamp casts#3541

Open
coderfender wants to merge 7 commits intoapache:mainfrom
coderfender:support_int_to_timestamp_casts
Open

feat: Support int to timestamp casts#3541
coderfender wants to merge 7 commits intoapache:mainfrom
coderfender:support_int_to_timestamp_casts

Conversation

@coderfender
Copy link
Contributor

Which issue does this PR close?

Closes #.

Rationale for this change

What changes are included in this PR?

How are these changes tested?

@coderfender coderfender force-pushed the support_int_to_timestamp_casts branch from 6bec627 to 11ef1b6 Compare February 17, 2026 01:03
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current testing framework which uses collect() to fetch data into a list . However, for long timestamps this errors out .

Steps to reproduce :

spark.sql("SELECT cast(-9223372036854775808 as timestamp)").collect()

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could read the data from a table instead?

scala> val data = Seq(1L, 10324234242L, 234234234234L, -9223372036854775808L).toDF("a")

scala> data.createTempView("t1")

scala> spark.sql("select a, cast(a as timestamp) from t1").show(false)
+--------------------+-----------------------------+
|a                   |a                            |
+--------------------+-----------------------------+
|1                   |1969-12-31 17:00:01          |
|10324234242         |2297-02-28 03:50:42          |
|234234234234        |9392-08-02 03:03:54          |
|-9223372036854775808|-290308-12-21 12:59:09.224192|
+--------------------+-----------------------------+

Copy link
Contributor Author

@coderfender coderfender Feb 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the suggestion . The issue is rather on the castTest framework in which we have a collect() statement. This collect would convert the spark's timestamp (microseconds) to java's timestamp (which is basically a date) causing error / overflow. In order to avoid that I have had to implement assertDataFrameEquals (inspired from spark-testing-base) to make sure we have data and schema parity .
Please let me know if you think we have easier options to do this and I would love to change the code

Copy link
Member

@andygrove andygrove Feb 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another option to look at:

scala> spark.sql("select a, cast(cast(a as timestamp) as string) from t1").collect()
res3: Array[org.apache.spark.sql.Row] = Array(
  [1,1969-12-31 17:00:01], 
  [10324234242,2297-02-28 03:50:42], 
  [234234234234,9392-08-02 03:03:54], 
  [-9223372036854775808,-290308-12-21 12:59:09.224192]
)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the comment @andygrove . I tried double casting to string but that didnt work out given the existing limitation of String -> Date . However , casting it back to Long simplified things

@coderfender
Copy link
Contributor Author

I believe this is ready for your review @andygrove

@coderfender coderfender force-pushed the support_int_to_timestamp_casts branch from 7859888 to 464cce2 Compare February 18, 2026 03:21
@coderfender
Copy link
Contributor Author

Thank you for merging main and the suggestions re testing long to timestamp @andygrove , @mbutrovich

@coderfender coderfender force-pushed the support_int_to_timestamp_casts branch from 427d3b6 to fbacef0 Compare February 18, 2026 18:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

Comments