pyflink.multimodal.operators.audio_detect_language#

audio_detect_language(*columns, top_k=1, model='openai/whisper-tiny', model_sharing=None, concurrency=None, batch_size=None, num_gpus=None, gpu_type=None)[source]#

Detect likely spoken languages in an audio waveform.

The operator uses the selected Whisper-compatible checkpoint to rank likely languages for the whole audio row. It returns the top top_k language candidates; it does not split code-switched audio into time ranges.

The input contract is the same as audio_asr_whisper: a 16 kHz mono waveform row.

Each result item contains these fields:

language_tag: Model language tag, such as en or zh. Some checkpoints may return longer tags such as yue. The value is named language_tag because it comes from the model vocabulary and is not guaranteed to be a normalized BCP 47 language tag.
language_name: English display name for the language.
confidence: Model confidence for the candidate language.

Parameters

*columns – Optional audio columns. Supported input is a 16 kHz mono waveform row.
top_k – Number of language candidates to return per audio row.
model – HuggingFace Whisper or Whisper-compatible model id. The default is "openai/whisper-tiny".
model_sharing – Model handle sharing mode understood by the PyFlink model runtime.
concurrency – UDF concurrency. None uses the framework default.
batch_size – Pandas UDF batch size.
num_gpus – GPU resource amount requested by the UDF runtime.
gpu_type – GPU type requested for model inference. The DataFrame UDF runtime requires this when num_gpus is set.

Returns

A pandas UDF producing ARRAY<ROW<language_tag STRING, language_name STRING, confidence DOUBLE>> sorted by confidence. In SQL, unnest or index the array before consuming item fields.

Raises

ValueError – If a row is not a 16 kHz mono waveform row.

Notes

In SQL, positional configuration arguments follow the same order: audio_detect_language(audio, top_k, model).

Examples::

>>> decode = audio_decode()
>>> standardize = audio_standardize(sample_rate=16000, channels=1)
>>> speech_ready = standardize(decode(col("audio_bytes")))
>>> # Usage 1: create a reusable UDF and apply it to a column.
>>> detect_language = audio_detect_language(top_k=3)
>>> df = df.with_column(
...     "language",
...     detect_language(speech_ready),
... )
>>>
>>> # Usage 2: pass the column directly when building the expression.
>>> df = df.with_column(
...     "language",
...     audio_detect_language(speech_ready, top_k=3),
... )