pyflink.dataframe.DataFrame

class DataFrame(table: pyflink.table.table.Table)

A modern DataFrame API for PyFlink.

DataFrame provides a Pythonic interface for data transformations, built on top of the PyFlink Table API. It supports fluent chaining of operations and provides a familiar DataFrame-style API.

Example::

>>> import pyflink.dataframe as pf
>>> df = pf.from_dict({"id": [1, 2], "name": ["a", "b"]})
>>> result = df.select("id", "name") \
...              .with_column("id_doubled", pf.col("id") * 2) \
...              .filter(pf.col("id") > 0) \
...              .rename({"id": "identifier"})

Methods

agg(*aggs, **named_aggs)	Apply aggregation expressions to the entire DataFrame.
collect()	Collect the DataFrame results to a list.
drop(*columns[, strict])	Drop columns from the DataFrame.
drop_columns(*columns[, strict])	Drop columns from the DataFrame.
drop_duplicates([subset, keep, order_by, ...])	Drop duplicate rows, keeping one row per group of subset columns.
drop_nan([subset])	Drop rows containing NaN values in float columns.
drop_null([subset])	Drop rows containing null values.
explain(*[, show_estimated_cost, ...])	Print the AST and execution plan of this DataFrame.
explode(column[, output_column, ...])	Expand one ARRAY, MAP, or MULTISET column into rows.
fill_nan(value[, subset])	Replace NaN values with a given value in float columns.
fill_null(value[, subset])	Replace null values with a given value.
filter(*predicate, **constraints)	Filter rows based on one or more predicates.
flat_map(func, *[, return_dtype, ...])	Apply a row-based UDTF to each row, producing zero or more rows.
group_by(*columns)	Group the DataFrame by columns.
head(n)	Return the first n rows as a new DataFrame.
intersect(other)	Intersect with another DataFrame, removing duplicate rows.
intersect_all(other)	Intersect with another DataFrame, preserving duplicate rows.
iter_batches()	Return an iterator of pandas DataFrames or PyArrow Tables.
iter_rows(*[, include_row_kind, row_kind_field])	Return an iterator over rows of the DataFrame.
join(other, *[, on, how, left_on, right_on])	Join with another DataFrame.
join_asof(other, *[, how, on, by, left_by, ...])	ASOF join with a dimension table at a time attribute.
join_lateral(table_function_call, *[, on, ...])	Join with an element-wise UDTF call.
limit(n)	Limit the DataFrame to the first n rows.
map(func, *[, return_dtype, concurrency, ...])	Apply a function to each row, producing a new DataFrame.
map_batches(func, *[, return_dtype, ...])	Apply a function to batches of rows, producing a new DataFrame.
minus(other)	Return rows from this DataFrame that do not exist in another DataFrame.
minus_all(other)	Return rows from this DataFrame that do not exist in another DataFrame.
offset(n)	Skip the first n rows of the DataFrame.
pipe(func, *args, **kwargs)	Apply a chainable function to the DataFrame.
rebalance()	Explicitly redistribute rows across downstream parallel subtasks in round-robin fashion.
rename(*args[, mapping])	Rename columns.
rename_columns(*args[, mapping])	Rename columns.
select(*columns, **projections)	Select columns from the DataFrame.
to_pandas()	Convert to a Pandas DataFrame.
to_table()	Convert to a PyFlink Table.
union(other)	Union with another DataFrame, removing duplicate rows.
union_all(other)	Union with another DataFrame, preserving duplicate rows.
where(*predicate, **constraints)	Filter rows based on one or more predicates.
with_column(name, expr)	Add a new column or replace an existing column.
with_columns(*exprs, **named_exprs)	Add multiple columns or replace existing columns.
write_catalog_table(path, *[, overwrite, ...])	Write the DataFrame to a catalog table.
write_generic(connector, *[, primary_key, ...])	Write the DataFrame using a generic connector.
write_hologres(endpoint, *, db_name, ...[, ...])	Write the DataFrame to a Hologres table.
write_json(path, *[, mode, ...])	Write the DataFrame to JSON file(s) at the given path.
write_kafka(bootstrap_servers, *[, ...])	Write the DataFrame to a Kafka topic.
write_milvus(endpoint, *, username, ...[, ...])	Write the DataFrame to a Milvus collection.
write_odps(endpoint, *[, tunnel_endpoint, ...])	Write the DataFrame to a MaxCompute (ODPS) table.
write_paimon([path, primary_key, ...])	Write the DataFrame to a Paimon table.
write_parquet(path, *[, mode, compression, ...])	Write the DataFrame to Parquet file(s) at the given path.
write_sls(endpoint, *, project, logstore[, ...])	Write the DataFrame to an SLS (Aliyun Log Service) logstore.

Attributes

columns	Get the column names.
llm	Access LLM / AI functions.
schema	Get the schema of the DataFrame.