pyflink.multimodal.operators.audio_detect_speech#

audio_detect_speech(*columns, aggressiveness=0, frame_ms=30, min_speech_ms=0, merge_gap_ms=0, max_segments=1024, concurrency=None)[source]#

Detect speech activity ranges in a mono waveform with WebRTC VAD.

This operator is the speech-range producer for audio_split_by_speech. It uses the third-party WebRTC VAD implementation, not an energy threshold or an ASR model. The input must be mono and have a WebRTC-supported sample rate: 8000, 16000, 32000, or 48000 Hz. For most ASR pipelines, call audio_standardize(sample_rate=16000, channels=1) first.

Parameters

*columns – Optional audio columns. Supported input is a row with fields data TENSOR('float32'), sample_rate INT, channels INT, frames BIGINT, sample_format STRING, layout STRING, duration_ms BIGINT, source_uri STRING, start_time_ms BIGINT, end_time_ms BIGINT.
aggressiveness – WebRTC VAD aggressiveness mode, from 0 (least aggressive) to 3 (most aggressive). The default 0 follows WebRTC VAD’s least aggressive mode rather than a PyFlink threshold.
frame_ms – WebRTC frame size in milliseconds. Must be 10, 20, or 30.
min_speech_ms – Drop detected speech ranges shorter than this duration. The default 0 keeps raw WebRTC VAD ranges.
merge_gap_ms – Merge speech ranges separated by this gap or less. The default 0 only merges adjacent speech frames.
max_segments – Maximum number of ranges to return.
concurrency – UDF concurrency. None uses the framework default.

Returns

A UDF producing ARRAY<ROW<start_ms BIGINT, end_ms BIGINT, duration_ms BIGINT>> sorted by time. Pass this array as the second argument to audio_split_by_speech. For waveform rows decoded from AUDIO_CLIP_REF segments, start_ms and end_ms are reported in the source audio timeline so they can be fed back into split operators.

Raises

ValueError – If the input is not a supported mono waveform, the sample rate is unsupported by WebRTC VAD, or parameters are invalid.

Examples::

>>> # Prepare a mono waveform at a WebRTC-supported sample rate.
>>> decode = audio_decode()
>>> standardize = audio_standardize(sample_rate=16000, channels=1)
>>> waveform = standardize(decode(col("audio_bytes")))
>>> # Usage 1: create a reusable UDF and apply it to a column.
>>> detect_speech = audio_detect_speech()
>>> df = df.with_column("speech_ranges", detect_speech(waveform))
>>>
>>> # Usage 2: pass the column directly when building the expression.
>>> df = df.with_column(
...     "speech_ranges",
...     audio_detect_speech(waveform),
... )