Skip to content

Commit ac2bfa3

Browse files
pkumar-dataclaude
andcommitted
docs: add overwrite_src_cols documentation for derived_columns
Documents the new per-column boolean parameter in macros/staging/stage.sql (derived_columns_description) and in the derived-columns markdown guide, including a umlaut-rename example and an OVERWRITE SOURCE COLUMNS section. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent c429a44 commit ac2bfa3

2 files changed

Lines changed: 35 additions & 3 deletions

File tree

docs/01_macro-instructions/02_staging/04_derived-columns.md

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,11 @@ derived_columns:
2424
src_cols_required:
2525
- <src_col_2>
2626
- <src_col_3>
27+
<col_alias_3>:
28+
value: <expression_3>
29+
datatype: <datatype_3>
30+
src_cols_required: <src_col_3>
31+
overwrite_src_cols: true # optional — drops src_col_3 from the output
2732
```
2833

2934
Depending on how `col_alias` and `src_col` are called, two different behaviors can be achieved:
@@ -51,3 +56,20 @@ It must only be set for SQL expressions. For static strings, it will be set to S
5156
The parameter `src_cols_required` is only required when the Stage model is configured to not include the source columns by setting the parameter `include_source_columns` to false.
5257

5358
If this is the case, you have to list all columns used within the SQL expressions under the parameter `src_cols_required`. This information is required to properly generate the model SQL.
59+
60+
## OVERWRITE SOURCE COLUMNS
61+
62+
The optional boolean parameter `overwrite_src_cols` can be set per derived column. When set to `true`, the source columns listed under `src_cols_required` are **excluded from the staging CTE output**, so only the derived alias column appears — preventing duplicate columns.
63+
64+
This is particularly useful when **renaming columns whose original name cannot be referenced downstream**, for example columns containing special characters (e.g. German umlauts like `ä`, `ö`, `ü`) on platforms such as Databricks. Without `overwrite_src_cols: true`, both the original and the renamed column would appear side by side in the output.
65+
66+
```jinja
67+
derived_columns:
68+
account_name_clean:
69+
value: Kontonäme
70+
datatype: STRING
71+
src_cols_required: Kontonäme
72+
overwrite_src_cols: true
73+
```
74+
75+
In this example, `Kontonäme` is excluded from the staging CTE and only `account_name_clean` appears in the output. The parameter defaults to `false` when omitted.

macros/staging/stage.sql

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -53,9 +53,14 @@
5353
" %}
5454

5555
{% set derived_columns_description = "
56-
derived_columns::dictionary Defines values and datatypes for derived ('added' or 'calculated') columns. The values of this dictionary are the desired column names,
57-
the value is another dictionary with the keys 'value' (holding a column name, a SQL expression, or a static string beginning with '!') and
58-
'datatype' (holding a valid SQL datatype for the target database).
56+
derived_columns::dictionary Defines values and datatypes for derived ('added' or 'calculated') columns. The key of each entry is the desired output column name.
57+
The value is a dictionary that supports the following keys:
58+
'value' (required) A column name, a SQL expression, or a static string beginning with '!'.
59+
'datatype' (required) A valid SQL datatype for the target database.
60+
'src_cols_required' (optional) A column name or list of column names used by the expression. Required when include_source_columns=false.
61+
'overwrite_src_cols' (optional, default false) When true, the columns listed in 'src_cols_required' are excluded from the staging CTE
62+
output, so only the aliased column appears. Useful for renaming columns whose original name cannot be referenced
63+
downstream (e.g. columns with special characters such as umlauts on Databricks).
5964
6065
Examples:
6166
{'conversion_duration': {'value': 'TIMESTAMP_DIFF(conversion_date, created_date, DAY)', Creates three derived columns. The column 'conversion_duration' calculates
@@ -64,6 +69,11 @@
6469
'datatype': 'STRING'}, The column 'account_name' duplicates an already existing column and gives
6570
'account_name': {'value': 'name', it another name. More derived columns can be added as additional keys of
6671
'datatype': 'String'}} the dictionary.
72+
73+
{'account_name_clean': {'value': 'Kontonäme', Renames a source column whose name contains a special character. Setting
74+
'datatype': 'STRING', 'overwrite_src_cols: true' ensures the original column is dropped from the
75+
'src_cols_required': 'Kontonäme', CTE output, so only 'account_name_clean' appears and the special-character
76+
'overwrite_src_cols': true}} column name does not propagate downstream.
6777
" %}
6878

6979
{% set sequence_description = "

0 commit comments

Comments
 (0)