Add a new operator EXPLODE in the PyDough DSL syntax emulating the LATERAL FLATTEN and SPLIT_TO_ARRAY functionalities from dialects such as Snowflake. Like #521, this functionality will only be allowed for certain dialects.
Suppose we have a collection tbl that looks like this:
A B C
0 | 1 A [1]
1 | 2 B []
2 | 3 A [2, 3]
3 | 4 A [4, 5]
4 | 5 B [6, 7, 8]
5 | 6 A [9]
The PyDough code would look something like:
tbl.EXPLODE(C, index_name="C1", value_name="C2", version="array", keep=True, filtering=True, is_distinct=True)
What the various (sometimes optional) arguments mean:
- Explode
tbl using column C
index_name="C1": create a new column C1 in the output, indicating the index within each list that the elements belonged to
value_name="C2": create a new column C2 in the output, which has the elements from C
version="array": Assume column C is an array (alternative is a string that gets split on a delimiteter)
keep=True: Keep column C in the output answer
filtering=True: Not every row in tbl will be kept in the final answer
is_distinct=True: every element inside each list is unique
The resulting output collection:
A B C C1 C2
0 | 1 A [1] 0 1
2 | 3 A [2, 3] 0 2
2 | 3 A [2, 3] 1 3
3 | 4 A [4, 5] 0 4
3 | 4 A [4, 5] 1 5
4 | 5 B [6, 7, 8] 0 6
4 | 5 B [6, 7, 8] 1 7
4 | 5 B [6, 7, 8] 2 8
5 | 6 A [9] 0 9
If the original uniqueness information about tbl was ["A"], then the new uniqueness would be: [["A", "C1"], ["A", "C2"]].
For each unique value of A, it will get duplicated but the duplicates are uniquely identifiable by the index within the list (C1)
Alternatively, can uniquely identify them by the elements IN the list since we know is_distinct=True
What the SQL for this looks like (only valid in certain dialects, this example is Snowflake; postgres/trino should also work):
SELECT T.A, T.B, T.C, L.index AS C1, L.value AS C2
FROM T
LATERAL FLATTEN(T.C) AS L
The main variant is the string-based version (version="string", and a new argument delimeter=...), where each row of C is something like "I love you" getting split on " " to form 3 rows "I", "love", and "you".
The exact argument rules:
- Positional argument: required, this is the column that will be exploded
index_name: required, unless is_distinct=True. If not provided, the enumeration column will not be included in the answer
value_name: required
version: optional, must either be "array" or "string" to indicate which version is used. Default is "array".
keep: optional, default is True, keeps the positional argument in the output answer.
filtering: optional, default is False, indicates whether every original row of tbl is preserved at least once in the final answer
is_distinct: optional, default is False, indicates whether every element within the lists/strings is unique, meaning that the enumeration column is not required to uniquely identify every row in the final output.
delimiter: required if version="string", prohibited otherwise
Note: relies on #521 for ease of testing.
Add a new operator EXPLODE in the PyDough DSL syntax emulating the
LATERAL FLATTENandSPLIT_TO_ARRAYfunctionalities from dialects such as Snowflake. Like #521, this functionality will only be allowed for certain dialects.Suppose we have a collection
tblthat looks like this:The PyDough code would look something like:
What the various (sometimes optional) arguments mean:
tblusing columnCindex_name="C1": create a new column C1 in the output, indicating the index within each list that the elements belonged tovalue_name="C2": create a new column C2 in the output, which has the elements from Cversion="array": Assume column C is an array (alternative is a string that gets split on a delimiteter)keep=True: Keep column C in the output answerfiltering=True: Not every row in tbl will be kept in the final answeris_distinct=True: every element inside each list is uniqueThe resulting output collection:
If the original uniqueness information about
tblwas["A"], then the new uniqueness would be:[["A", "C1"], ["A", "C2"]].For each unique value of
A, it will get duplicated but the duplicates are uniquely identifiable by the index within the list (C1)Alternatively, can uniquely identify them by the elements IN the list since we know
is_distinct=TrueWhat the SQL for this looks like (only valid in certain dialects, this example is Snowflake; postgres/trino should also work):
The main variant is the string-based version (version="string", and a new argument
delimeter=...), where each row ofCis something like"I love you"getting split on" "to form 3 rows"I","love", and"you".The exact argument rules:
index_name: required, unlessis_distinct=True. If not provided, the enumeration column will not be included in the answervalue_name: requiredversion: optional, must either be"array"or"string"to indicate which version is used. Default is"array".keep: optional, default isTrue, keeps the positional argument in the output answer.filtering: optional, default isFalse, indicates whether every original row oftblis preserved at least once in the final answeris_distinct: optional, default isFalse, indicates whether every element within the lists/strings is unique, meaning that the enumeration column is not required to uniquely identify every row in the final output.delimiter: required ifversion="string", prohibited otherwiseNote: relies on #521 for ease of testing.