pyflink.dataframe.DataFrame.drop_duplicates#

DataFrame.drop_duplicates(subset: Optional[Union[str, List[str], Tuple[str, ...]]] = None, *, keep: str = 'first', order_by: Optional[Union[str, pyflink.table.expression.Expression, List[Union[str, pyflink.table.expression.Expression]]]] = None, nulls_first: Optional[Union[bool, List[bool]]] = None) → pyflink.dataframe.dataframe.DataFrame[source]#

Drop duplicate rows, keeping one row per group of subset columns.

Parameters

subset – Column name, or list/tuple of column names, identifying a duplicate group. If None, all columns are used.
order_by – Column name, Expression, or list of those values, that decide which row in each group is kept. If None, processing time is used; in streaming this means arrival order, while in batch the result is non-deterministic.
keep – "first" keeps the row that sorts first by order_by (ascending); "last" keeps the row that sorts last (descending).
nulls_first – Bool or list of bool values aligned with order_by that control whether NULLs sort first. If None, Flink’s default NULL ordering applies. nulls_first requires explicit order_by.

Returns

A new DataFrame with duplicates removed.

Note

If order_by is omitted, processing time is used. In streaming mode, keep="first"/keep="last" keep the first/last row to arrive for each duplicate group. In batch mode there is no stable arrival order, so the surviving row is non-deterministic.

In streaming mode, Flink can use its efficient streaming deduplicate operator only when the input is insert-only and the order is a single time attribute: the default PROCTIME() order, a proctime column, or a rowtime/event-time column. Both keep values use this path; keep changes the sort direction, and nulls_first does not affect the fast-path check. Other orders, including ordinary columns or multiple order keys, use a top-1 rank. Batch mode also uses rank.

Example::

>>> df.drop_duplicates(["k"])
>>> df.drop_duplicates(["k"], keep="last")
>>> df.drop_duplicates(["k"], order_by="ts")
>>> df.drop_duplicates(["k"], order_by=["ts", "seq"], nulls_first=[True, False])
>>> df.drop_duplicates()