pyflink.multimodal.operators.audio_split_by_speech#

audio_split_by_speech(*columns, segment_type='audio', pre_padding_ms=0, post_padding_ms=0, merge_gap_ms=0, min_segment_ms=0, max_segment_ms=None, max_segments=1024, concurrency=None)[source]#

Split audio by speech activity ranges into one output row per segment.

The second input column must contain speech ranges with start_ms and end_ms fields, such as output from a VAD or upstream speech activity model. This operator does not detect speech by itself; it only applies padding, merging, filtering, and splitting to caller-provided ranges. Ranges outside the actual audio duration are clipped, and fully out-of-range ranges are dropped. SQL module calls put segment_type before the optional tuning arguments. Use audio_split_by_speech(uri, ranges, 'ref') for default lazy references, or provide all tuning literals as audio_split_by_speech(uri, ranges, 'ref', 200, 200, 30, 30, 1000, 1024).

Parameters

*columns – Audio input followed by speech activity ranges. segment_type="audio" supports waveform rows. segment_type="ref" supports URI STRING and reference rows.
segment_type – "audio" for materialized waveform rows or "ref" for reference rows.
pre_padding_ms – Milliseconds to extend before each speech range.
post_padding_ms – Milliseconds to extend after each speech range.
merge_gap_ms – Merge adjacent ranges separated by this gap or less.
min_segment_ms – Drop segments shorter than this duration after padding and merging.
max_segment_ms – Optionally split long speech ranges into chunks no longer than this value.
max_segments – Maximum allowed number of returned segments.
concurrency – UDF concurrency. None uses the framework default.

Returns

A UDTF producing one segment column per emitted segment. For segment_type="audio", segment is a waveform row with data TENSOR('float32'), sample_rate INT, channels INT, frames BIGINT, sample_format STRING, layout STRING, duration_ms BIGINT, source_uri STRING, start_time_ms BIGINT, end_time_ms BIGINT. For segment_type="ref", segment has uri STRING, start_time_ms BIGINT, end_time_ms BIGINT.

Raises

ValueError – If the input does not match segment_type, if speech ranges or split parameters are invalid, or if the split would produce more than max_segments items.

Examples::

>>> # Usage 1: create a reusable UDTF and join it laterally.
>>> split = audio_split_by_speech(
...     pre_padding_ms=200,
...     post_padding_ms=200,
... )
>>> segments = df.join_lateral(
...     split(col("waveform"), col("speech_ranges")).alias("segment")
... )
>>>
>>> # Usage 2: pass columns directly when joining laterally.
>>> segments = df.join_lateral(
...     audio_split_by_speech(
...         col("waveform"),
...         col("speech_ranges"),
...         pre_padding_ms=200,
...         post_padding_ms=200,
...     ).alias("segment")
... )