pyflink.dataframe.DataFrame.iter_batches#

DataFrame.iter_batches(*, batch_size: int = '1000', batch_format: Literal['pandas'] = "'pandas'") → Iterator['pd.DataFrame'][source]#

DataFrame.iter_batches(*, batch_size: int = '1000', batch_format: Literal['arrow']) → Iterator['pa.Table']

Return an iterator of pandas DataFrames or PyArrow Tables.

This is a terminal operation that triggers execution. It uses Arrow-based serialization for efficient data transfer. In streaming mode, the DataFrame must produce append-only rows.

Parameters

batch_size – The maximum number of rows per batch. Defaults to 1000.
batch_format – Batch result format, either “pandas” or “arrow”.

Returns

A closeable iterator that yields pandas DataFrames or PyArrow Tables.

Example::

>>> df = pf.from_records([(i,) for i in range(2500)], schema=["a"])
>>> for batch in df.iter_batches(batch_size=1000):
...     print(len(batch))
1000
1000
500