Skip to content

Add EXPLODE operator to PyDough #522

Description

@knassre-bodo

Add a new operator EXPLODE in the PyDough DSL syntax emulating the LATERAL FLATTEN and SPLIT_TO_ARRAY functionalities from dialects such as Snowflake. Like #521, this functionality will only be allowed for certain dialects.

Suppose we have a collection tbl that looks like this:

    A  B  C
0 | 1  A  [1] 
1 | 2  B  []
2 | 3  A  [2, 3]
3 | 4  A  [4, 5]
4 | 5  B  [6, 7, 8]
5 | 6  A  [9]

The PyDough code would look something like:

tbl.EXPLODE(C, index_name="C1", value_name="C2", version="array", keep=True, filtering=True, is_distinct=True)

What the various (sometimes optional) arguments mean:

  • Explode tbl using column C
  • index_name="C1": create a new column C1 in the output, indicating the index within each list that the elements belonged to
  • value_name="C2": create a new column C2 in the output, which has the elements from C
  • version="array": Assume column C is an array (alternative is a string that gets split on a delimiteter)
  • keep=True: Keep column C in the output answer
  • filtering=True: Not every row in tbl will be kept in the final answer
  • is_distinct=True: every element inside each list is unique

The resulting output collection:

    A  B  C          C1  C2
0 | 1  A  [1]         0   1
2 | 3  A  [2, 3]      0   2
2 | 3  A  [2, 3]      1   3  
3 | 4  A  [4, 5]      0   4  
3 | 4  A  [4, 5]      1   5 
4 | 5  B  [6, 7, 8]   0   6 
4 | 5  B  [6, 7, 8]   1   7  
4 | 5  B  [6, 7, 8]   2   8  
5 | 6  A  [9]         0   9  

If the original uniqueness information about tbl was ["A"], then the new uniqueness would be: [["A", "C1"], ["A", "C2"]].

For each unique value of A, it will get duplicated but the duplicates are uniquely identifiable by the index within the list (C1)
Alternatively, can uniquely identify them by the elements IN the list since we know is_distinct=True

What the SQL for this looks like (only valid in certain dialects, this example is Snowflake; postgres/trino should also work):

SELECT T.A, T.B, T.C, L.index AS C1, L.value AS C2
FROM T
LATERAL FLATTEN(T.C) AS L

The main variant is the string-based version (version="string", and a new argument delimeter=...), where each row of C is something like "I love you" getting split on " " to form 3 rows "I", "love", and "you".

The exact argument rules:

  • Positional argument: required, this is the column that will be exploded
  • index_name: required, unless is_distinct=True. If not provided, the enumeration column will not be included in the answer
  • value_name: required
  • version: optional, must either be "array" or "string" to indicate which version is used. Default is "array".
  • keep: optional, default is True, keeps the positional argument in the output answer.
  • filtering: optional, default is False, indicates whether every original row of tbl is preserved at least once in the final answer
  • is_distinct: optional, default is False, indicates whether every element within the lists/strings is unique, meaning that the enumeration column is not required to uniquely identify every row in the final output.
  • delimiter: required if version="string", prohibited otherwise

Note: relies on #521 for ease of testing.

Metadata

Metadata

Assignees

No one assigned

    Labels

    effort - highmajor issue that will require multiple steps or complex designenhancementNew feature or requestextensibilityIncreasing situations in which PyDough workstestingAlters the testing/CI process for PyDoughuser featureAdding a new user-facing feature/functionality

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions