feat: add overwrite_src_cols parameter to derived_columns#474
Open
pkumar-data wants to merge 5 commits into
Open
feat: add overwrite_src_cols parameter to derived_columns#474pkumar-data wants to merge 5 commits into
pkumar-data wants to merge 5 commits into
Conversation
pkumar-data
force-pushed
the
fix/328-derived-columns-overwrite-src-cols
branch
from
66403e7 to
ca5a1e8
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Adds support for a new per-derived-column boolean parameter overwrite_src_cols that suppresses specified source columns (from src_cols_required) after the derived_columns CTE, enabling true “rename/replace” behavior and preventing duplicated original+alias columns downstream.
Changes:
- Introduces
extract_overwrite_columns()helper macro to collectsrc_cols_requiredentries for derived columns marked withoverwrite_src_cols: true. - Updates adapter-specific
stage.sqlmacros to (a) extract overwrite columns before datatype detection and (b) remove those source columns fromfinal_columns_to_selectin thederived_columnsCTE.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| macros/staging/stage_processing_macros.sql | Adds extract_overwrite_columns() helper macro used by staging to compute which source columns to drop. |
| macros/staging/bigquery/stage.sql | Extracts overwrite columns and excludes them from derived_columns CTE output. |
| macros/staging/databricks/stage.sql | Extracts overwrite columns and excludes them from derived_columns CTE output. |
| macros/staging/exasol/stage.sql | Extracts overwrite columns and excludes them from derived_columns CTE output. |
| macros/staging/fabric/stage.sql | Extracts overwrite columns and excludes them from derived_columns CTE output. |
| macros/staging/oracle/stage.sql | Extracts overwrite columns and excludes them from derived_columns CTE output. |
| macros/staging/postgres/stage.sql | Extracts overwrite columns and excludes them from derived_columns CTE output. |
| macros/staging/redshift/stage.sql | Extracts overwrite columns and excludes them from derived_columns CTE output. |
| macros/staging/snowflake/stage.sql | Extracts overwrite columns and excludes them from derived_columns CTE output. |
| macros/staging/sqlserver/stage.sql | Extracts overwrite columns and excludes them from derived_columns CTE output. |
| macros/staging/synapse/stage.sql | Extracts overwrite columns and excludes them from derived_columns CTE output. |
| macros/staging/trino/stage.sql | Extracts overwrite columns and excludes them from derived_columns CTE output. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
tkiehn
requested changes
Jun 22, 2026
Adds a new per-derived-column boolean parameter overwrite_src_cols (default: false). When true, the columns listed in src_cols_required are suppressed from the derived_columns CTE SELECT onward, so the source column is replaced by the alias rather than duplicated alongside it. Fixes the case where renaming a column with special characters (e.g. German umlauts on Databricks) resulted in both the original and the alias appearing in all downstream CTEs. Implemented via a new extract_overwrite_columns() helper macro in stage_processing_macros.sql. The helper is called before derived_columns_datatypes() to avoid the in-place dict mutation that strips custom keys from the columns dict.
…exclusion Databricks normalizes final_columns_to_select to lowercase in the first process_columns_to_select call, so overwrite_src_columns must also be lowercased to ensure case-insensitive matching works correctly.
Initialize overwrite_src_columns = [] before the if block rather than in the else clause. This eliminates the redundant else block, makes the intent clearer (variable is always needed), and follows DRY principle. Applies to all staging macros: bigquery, snowflake, postgres, redshift, oracle, exasol, trino, databricks, sqlserver, synapse, fabric.
pkumar-data
force-pushed
the
fix/328-derived-columns-overwrite-src-cols
branch
from
78ede82 to
c429a44
Compare
tkiehn
requested changes
Jul 6, 2026
tkiehn
left a comment
tkiehn
left a comment
Collaborator
There was a problem hiding this comment.
Looks good in general, nice and easy implementation!
Lastly please also add documentation for the new parameter in the top-level macro macros/staging/stage.sql and the docs-markdowns.
Documents the new per-column boolean parameter in macros/staging/stage.sql (derived_columns_description) and in the derived-columns markdown guide, including a umlaut-rename example and an OVERWRITE SOURCE COLUMNS section.
pkumar-data
force-pushed
the
fix/328-derived-columns-overwrite-src-cols
branch
from
ac2bfa3 to
212f037
Compare
tkiehn
requested changes
Jul 13, 2026
tkiehn
added
the
waiting-on-response
- Merge 'defaults to false' note into main parameter description - Remove redundant second sentence; example already shows the umlaut use case - Rename example column from account_name_clean/Kontonäme to domain/Domäne for clarity and consistency across docs and stage.sql
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a new per-derived-column boolean parameter
overwrite_src_cols(default:false). Whentrue, the columns listed insrc_cols_requiredare suppressed from thederived_columnsCTE SELECT onward, so the source column is replaced by the alias rather than duplicated alongside it. Fixes the case where renaming a column with special characters (e.g. German umlauts on Databricks) resulted in both the original and the alias appearing in all downstream CTEs.Implemented via a new
extract_overwrite_columns()helper macro instage_processing_macros.sql. The helper is called beforederived_columns_datatypes()to avoid the in-place dict mutation that strips custom keys from the columns dict.Fixes ScalefreeCOM/product_development_milestones#328
Type of change
How Has This Been Tested?
Tested on BigQuery using a stage model with a source column renamed via
overwrite_src_cols: true:dbt compile— verified thederived_columnsCTE no longer includes the original source column alongside the alias.dbt run— view created successfully with the correct output schema (original column absent, alias present).Test Configuration:
Checklist: