DataFrame

DataFrame

A DataFrame is the core abstraction of the PyFlink DataFrame API. It wraps a PyFlink Table and provides a pandas-like interface for data transformations. All transformation methods return a new DataFrame.

Example:

>>> import pyflink.dataframe as pf
>>> df = pf.from_dict({"id": [1, 2], "name": ["a", "b"]})
>>> result = df.select("id", "name") \
...            .with_column("id_doubled", pf.col("id") * 2) \
...            .filter(pf.col("id") > 0) \
...            .rename({"id": "identifier"})

Selection and Projection

Create, reshape, rename, and redistribute DataFrame columns.

DataFrame.select(*columns, **projections)	Select columns from the DataFrame.
DataFrame.with_column(name, expr)	Add a new column or replace an existing column.
DataFrame.with_columns(*exprs, **named_exprs)	Add multiple columns or replace existing columns.
DataFrame.drop_columns(*columns[, strict])	Drop columns from the DataFrame.
DataFrame.drop(*columns[, strict])	Drop columns from the DataFrame.
DataFrame.rename_columns(*args[, mapping])	Rename columns.
DataFrame.rename(*args[, mapping])	Rename columns.
DataFrame.rebalance()	Explicitly redistribute rows across downstream parallel subtasks in round-robin fashion.

Filtering and Missing Values

Filter rows and handle duplicate, null, or NaN values.

DataFrame.filter(predicate, **constraints)	Filter rows based on a predicate.
DataFrame.where(predicate, **constraints)	Filter rows based on a predicate.
DataFrame.drop_duplicates([subset, keep, ...])	Drop duplicate rows, keeping one row per group of subset columns.
DataFrame.drop_null([subset])	Drop rows containing null values.
DataFrame.drop_nan([subset])	Drop rows containing NaN values in float columns.
DataFrame.fill_null(value[, subset])	Replace null values with a given value.
DataFrame.fill_nan(value[, subset])	Replace NaN values with a given value in float columns.

Aggregation

Aggregate the entire DataFrame or aggregate rows after grouping.

DataFrame.group_by(*columns)	Group the DataFrame by columns.
DataFrame.agg(*aggs)	Apply aggregation expressions to the entire DataFrame.

Join and Set Operations

Combine DataFrames by keys, lateral table functions, or set algebra.

DataFrame.join(other[, on, how, left_on, ...])	Join with another DataFrame.
DataFrame.join_asof(other[, how, on, by, ...])	ASOF join with a dimension table at a time attribute.
DataFrame.join_lateral(table_function_call, *)	Join with an element-wise UDTF call.
DataFrame.union(other)	Union with another DataFrame, removing duplicate rows.
DataFrame.union_all(other)	Union with another DataFrame, preserving duplicate rows.
DataFrame.minus(other)	Return rows from this DataFrame that do not exist in another DataFrame.
DataFrame.minus_all(other)	Return rows from this DataFrame that do not exist in another DataFrame.
DataFrame.intersect(other)	Intersect with another DataFrame, removing duplicate rows.
DataFrame.intersect_all(other)	Intersect with another DataFrame, preserving duplicate rows.

Row, Batch, and Function Operations

Apply Python functions, batch functions, or table functions to DataFrame rows.

DataFrame.explode(column[, output_column, ...])	Expand one ARRAY, MAP, or MULTISET column into rows.
DataFrame.map(func, *[, return_dtype, ...])	Apply a function to each row, producing a new DataFrame.
DataFrame.map_batches(func, *[, ...])	Apply a function to batches of rows, producing a new DataFrame.
DataFrame.flat_map(func, *[, return_dtype, ...])	Apply a row-based UDTF to each row, producing zero or more rows.
DataFrame.pipe(func, *args, **kwargs)	Apply a chainable function to the DataFrame.

Conversion and Collection

Convert DataFrames to other objects or collect bounded results locally.

DataFrame.to_table()	Convert to a PyFlink Table.
DataFrame.to_pandas()	Convert to a Pandas DataFrame.
DataFrame.collect()	Collect the DataFrame results to a list.
DataFrame.limit(n)	Limit the DataFrame to the first n rows.
DataFrame.offset(n)	Skip the first n rows of the DataFrame.
DataFrame.head(n)	Return the first n rows as a new DataFrame.
DataFrame.iter_rows()	Return an iterator over rows of the DataFrame.
DataFrame.iter_batches(*[, batch_size])	Return an iterator of pandas DataFrames, each containing up to batch_size rows.
DataFrame.explain(*[, show_estimated_cost, ...])	Print the AST and execution plan of this DataFrame.

Properties

Inspect DataFrame metadata and access AI helper methods.

DataFrame.llm	Access LLM / AI functions.
DataFrame.schema	Get the schema of the DataFrame.
DataFrame.columns	Get the column names.

GroupedDataFrame

A DataFrame that has been grouped on a set of grouping keys.

GroupedDataFrame.agg(*aggs)

Apply aggregation expressions.

Expressions

Helper functions to create column references, literal values, and time markers.

col(name)	Create a column reference expression.
lit(value[, data_type])	Create a literal expression.
proctime()	Return a processing-time marker for DataFrame.join_asof.