pyflink.multimodal.operators.audio_asr_whisper#

audio_asr_whisper(*columns, language=None, task='transcribe', model='openai/whisper-tiny', model_sharing=None, concurrency=None, batch_size=None, num_gpus=None, gpu_type=None)[source]#

Run Whisper automatic speech recognition over audio waveforms.

The input contract is intentionally narrow: each row must be a waveform row with 16 kHz sample rate and one channel. Use audio_decode and audio_standardize before calling this operator. Rows that do not match the Whisper contract fail early instead of being implicitly resampled inside the model operator.

The output is a row with these fields:

asr_result: full transcription text.
timestamps: ARRAY<ROW<start_ms BIGINT, end_ms BIGINT>>.
segments: ARRAY<ROW<text STRING, start_ms BIGINT, end_ms BIGINT>>.

The operator uses HuggingFace Whisper-compatible checkpoints through the PyFlink model runtime. The selected model is resolved by the PyFlink model store. Depending on the model store policy, the model may be loaded from a local path/cache or resolved by model id.

Parameters

*columns – Optional audio columns. Supported input is a 16 kHz mono waveform row.
language – Optional source language hint passed to Whisper. None lets the model infer the language.
task – "transcribe" or "translate".
model – HuggingFace Whisper or Whisper-compatible model id. The default is "openai/whisper-tiny". Other checkpoints can be used when they are available in the Python worker environment or model cache.
model_sharing – Model handle sharing mode understood by the PyFlink model runtime.
concurrency – UDF concurrency. None uses the framework default.
batch_size – Pandas UDF batch size.
num_gpus – GPU resource amount requested by the UDF runtime.
gpu_type – GPU type requested for model inference. The DataFrame UDF runtime requires this when num_gpus is set.

Returns

A pandas UDF producing ROW<asr_result STRING, timestamps ARRAY<ROW<start_ms BIGINT, end_ms BIGINT>>, segments ARRAY<ROW<text STRING, start_ms BIGINT, end_ms BIGINT>>>. In SQL, downstream code can access fields with dot notation, for example asr.asr_result or asr.segments after aliasing.

Raises

ValueError – If a row is not a 16 kHz mono waveform row or if task is unsupported.

Notes

In SQL, positional configuration arguments follow the same order: audio_asr_whisper(audio, language, task, model). model is last because most jobs keep the checkpoint fixed and tune language/task more often.

Examples::

>>> decode = audio_decode()
>>> standardize = audio_standardize(sample_rate=16000, channels=1)
>>> speech_ready = standardize(decode(col("audio_bytes")))
>>> # Usage 1: create a reusable UDF and apply it to a column.
>>> asr = audio_asr_whisper()
>>> df = df.with_column(
...     "asr",
...     asr(speech_ready),
... )
>>>
>>> # Usage 2: pass the column directly when building the expression.
>>> df = df.with_column(
...     "asr",
...     audio_asr_whisper(speech_ready),
... )