pyflink.dataframe.DataFrame.explode#

DataFrame.explode(column: Union[str, pyflink.table.expression.Expression], output_column: Optional[Union[str, List[str]]] = None, ignore_empty_and_null: bool = False) → pyflink.dataframe.dataframe.DataFrame[source]#

Expand one ARRAY, MAP, or MULTISET column into rows.

The exploded input column is removed from the result. If output_column is omitted, the result reuses the exploded column name. Omitting output_column is only valid when the operation produces one output field. Pass a string for one output field, or a list of strings for multiple fields, such as for MAP or ARRAY<ROW<…>>.

ARRAY values are exploded into one row per array element. MULTISET values are exploded into one row per element occurrence, according to each element’s multiplicity. MAP values are exploded into one row per entry, producing key and value output fields.

When ignore_empty_and_null is False, rows whose input emits no rows are preserved with null output fields. When it is True, those rows are dropped.

Parameters

column – Column to explode, as a column name or Expression.
output_column – Optional output field name, or output field names for multi-field output.
ignore_empty_and_null – Whether to drop rows whose input is empty or null.

Returns

A new DataFrame with the exploded output fields replacing the input column.

Example:

>>> import pyflink.dataframe as pf
>>> df = pf.from_dict({
...     "id": [1, 2],
...     "tags": [["a", "b"], []],
... })
>>> df.explode("tags")
# Produces:
# (1, "a")
# (1, "b")
# (2, NULL)

>>> df.explode("tags", "tag")
# Produces:
# (1, "a")
# (1, "b")
# (2, NULL)
# with columns: id, tag

>>> attrs = pf.from_dict({
...     "id": [1],
...     "props": [{"color": "red", "size": "M"}],
... })
>>> attrs.explode("props", ["prop_key", "prop_value"])
# Produces one row per map entry with key and value columns,
# for example:
# (1, "color", "red")
# (1, "size", "M")

>>> df.explode("tags", "tag", ignore_empty_and_null=True)
# Produces:
# (1, "a")
# (1, "b")
# The row with id 2 is dropped because tags is empty.