Skip to content

Feat: Safely convert mappings (dictionaries) to JSON serializable objects - #493

Merged
mtcarlone merged 8 commits into
mainfrom
feat/safe-walk-dict
Dec 15, 2025
Merged

Feat: Safely convert mappings (dictionaries) to JSON serializable objects#493
mtcarlone merged 8 commits into
mainfrom
feat/safe-walk-dict

Conversation

@mtcarlone

Copy link
Copy Markdown
Contributor

Overview

There have been a few errors due to dbt's microbatch models. Namely, these models introduce datetime objects to different context variables available to us. These changes are not explicitly documented, and have caused a number of failures for users of this package.

In order to better safeguard ourselves against future versions, this PR introduces a set of macros that convert all values in a mapping (dictionary) object to something that is JSON serializable. Right now that is a datetime object, but it can be updated easily to handle for more later on.

This macro has been deployed to the two areas that were impacted by microbatch models and CLI arguments (models and invocations). The deployment pattern can be followed for other uploads.

Update type - breaking / non-breaking

  • Minor bug fix
  • Documentation improvements
  • Quality of Life improvements
  • New features (non-breaking change)
  • New features (breaking change)
  • Other (non-breaking change)
  • Other (breaking change)
  • Release preparation

What does this solve?

This solves a number of issues caused by microbatch models. It should also future-proof us a bit ... hopefully.

Outstanding questions

What databases have you tested with?

  • Snowflake
  • Google BigQuery
  • Databricks
  • Spark
  • N/A

@mtcarlone
mtcarlone temporarily deployed to Approve Integration Tests April 10, 2025 21:59 — with GitHub Actions Inactive
@mtcarlone
mtcarlone had a problem deploying to Approve Integration Tests April 10, 2025 21:59 — with GitHub Actions Failure
@mtcarlone
mtcarlone had a problem deploying to Approve Integration Tests April 10, 2025 21:59 — with GitHub Actions Failure
@mtcarlone
mtcarlone had a problem deploying to Approve Integration Tests April 10, 2025 21:59 — with GitHub Actions Failure
@mtcarlone
mtcarlone temporarily deployed to Approve Integration Tests April 10, 2025 21:59 — with GitHub Actions Inactive
@mtcarlone
mtcarlone had a problem deploying to Approve Integration Tests April 10, 2025 21:59 — with GitHub Actions Failure
@mtcarlone
mtcarlone had a problem deploying to Approve Integration Tests April 10, 2025 21:59 — with GitHub Actions Failure
@mtcarlone
mtcarlone had a problem deploying to Approve Integration Tests April 10, 2025 21:59 — with GitHub Actions Failure
@mtcarlone
mtcarlone temporarily deployed to Approve Integration Tests April 10, 2025 21:59 — with GitHub Actions Inactive

@llifoawing llifoawing left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! (as a side note, the microbatch model configs dont work in databricks but since we dont have an integration test for databricks can tackle that in another PR)

@mtcarlone
mtcarlone had a problem deploying to Approve Integration Tests June 19, 2025 19:02 — with GitHub Actions Failure
@mtcarlone
mtcarlone temporarily deployed to Approve Integration Tests June 19, 2025 19:02 — with GitHub Actions Inactive
@mtcarlone
mtcarlone had a problem deploying to Approve Integration Tests June 19, 2025 19:02 — with GitHub Actions Failure
@mtcarlone
mtcarlone temporarily deployed to Approve Integration Tests June 19, 2025 19:02 — with GitHub Actions Inactive
@mtcarlone
mtcarlone had a problem deploying to Approve Integration Tests June 19, 2025 19:02 — with GitHub Actions Failure
@mtcarlone
mtcarlone temporarily deployed to Approve Integration Tests June 19, 2025 19:02 — with GitHub Actions Inactive
@mtcarlone
mtcarlone had a problem deploying to Approve Integration Tests June 19, 2025 19:02 — with GitHub Actions Failure
@mtcarlone
mtcarlone had a problem deploying to Approve Integration Tests June 19, 2025 19:02 — with GitHub Actions Failure
@mtcarlone
mtcarlone had a problem deploying to Approve Integration Tests June 19, 2025 19:02 — with GitHub Actions Failure
@mtcarlone
mtcarlone had a problem deploying to Approve Integration Tests June 19, 2025 19:02 — with GitHub Actions Failure
@llifoawing
llifoawing temporarily deployed to Approve Integration Tests June 20, 2025 22:22 — with GitHub Actions Inactive
@llifoawing
llifoawing had a problem deploying to Approve Integration Tests June 20, 2025 22:22 — with GitHub Actions Failure
@llifoawing
llifoawing had a problem deploying to Approve Integration Tests June 20, 2025 22:22 — with GitHub Actions Failure
@llifoawing
llifoawing had a problem deploying to Approve Integration Tests June 20, 2025 22:22 — with GitHub Actions Failure
@llifoawing
llifoawing had a problem deploying to Approve Integration Tests June 20, 2025 22:22 — with GitHub Actions Failure
@llifoawing
llifoawing had a problem deploying to Approve Integration Tests June 20, 2025 22:22 — with GitHub Actions Failure
@llifoawing
llifoawing had a problem deploying to Approve Integration Tests June 20, 2025 22:22 — with GitHub Actions Failure
@llifoawing
llifoawing temporarily deployed to Approve Integration Tests June 20, 2025 22:22 — with GitHub Actions Inactive
@mtcarlone

Copy link
Copy Markdown
Contributor Author

Damn this breaks Trino 🤦 will get a fix up for that ...

@mtcarlone
mtcarlone had a problem deploying to Approve Integration Tests July 11, 2025 18:28 — with GitHub Actions Error
@mtcarlone
mtcarlone had a problem deploying to Approve Integration Tests July 11, 2025 18:28 — with GitHub Actions Error
@mtcarlone
mtcarlone had a problem deploying to Approve Integration Tests July 11, 2025 18:28 — with GitHub Actions Error
@mtcarlone
mtcarlone had a problem deploying to Approve Integration Tests July 11, 2025 18:28 — with GitHub Actions Error
@mtcarlone
mtcarlone had a problem deploying to Approve Integration Tests July 11, 2025 18:28 — with GitHub Actions Error
@mtcarlone
mtcarlone had a problem deploying to Approve Integration Tests July 11, 2025 18:28 — with GitHub Actions Error
@mtcarlone
mtcarlone had a problem deploying to Approve Integration Tests July 11, 2025 18:28 — with GitHub Actions Error
@mtcarlone
mtcarlone had a problem deploying to Approve Integration Tests July 11, 2025 18:28 — with GitHub Actions Error
@mtcarlone
mtcarlone had a problem deploying to Approve Integration Tests July 11, 2025 18:28 — with GitHub Actions Error
@mtcarlone
mtcarlone had a problem deploying to Approve Integration Tests July 11, 2025 18:28 — with GitHub Actions Error
@mtcarlone
mtcarlone temporarily deployed to Approve Integration Tests July 11, 2025 19:01 — with GitHub Actions Inactive
@mtcarlone
mtcarlone temporarily deployed to Approve Integration Tests July 11, 2025 19:01 — with GitHub Actions Inactive
@mtcarlone
mtcarlone temporarily deployed to Approve Integration Tests July 11, 2025 19:01 — with GitHub Actions Inactive
@mtcarlone
mtcarlone temporarily deployed to Approve Integration Tests July 11, 2025 19:01 — with GitHub Actions Inactive
@mtcarlone
mtcarlone temporarily deployed to Approve Integration Tests July 11, 2025 19:01 — with GitHub Actions Inactive
@mtcarlone
mtcarlone temporarily deployed to Approve Integration Tests July 11, 2025 19:01 — with GitHub Actions Inactive
@mtcarlone
mtcarlone had a problem deploying to Approve Integration Tests July 11, 2025 19:01 — with GitHub Actions Failure
@mtcarlone
mtcarlone temporarily deployed to Approve Integration Tests July 11, 2025 19:01 — with GitHub Actions Inactive
@mtcarlone
mtcarlone had a problem deploying to Approve Integration Tests July 11, 2025 19:01 — with GitHub Actions Failure
@mtcarlone
mtcarlone temporarily deployed to Approve Integration Tests July 11, 2025 19:01 — with GitHub Actions Inactive
@mtcarlone mtcarlone self-assigned this Dec 15, 2025
@mtcarlone
mtcarlone merged commit 3091fc7 into main Dec 15, 2025
@mtcarlone
mtcarlone deleted the feat/safe-walk-dict branch December 15, 2025 14:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants