Image#

Image Operations#

Transform#

Image transformation operators.

image_decode() accepts encoded image bytes and returns a decoded image. Pixel transforms accept decoded images and return decoded images so chained operators can avoid repeated JPEG/PNG encode-decode cycles. image_encode() accepts decoded images and returns encoded bytes; image_compress() and image_convert_format() accept encoded bytes and return encoded bytes.

Typical pipeline:

from pyflink.multimodal.operators.image_transform import (
    image_decode, image_encode, image_resize, image_blur,
)

decode = image_decode()
resize = image_resize(width=512, height=512)
blur = image_blur(radius=2)
encode = image_encode(format="JPEG")

df = df.with_columns("img", decode(col("raw_bytes")))
df = df.with_columns("img", resize(col("img")))
df = df.with_columns("img", blur(col("img")))
df = df.with_columns("jpg", encode(col("img")))

image_decode(*columns[, on_error, mode, ...])

Decode encoded image bytes into a decoded image.

image_encode(*columns[, format, quality, ...])

Encode a decoded image into compressed bytes or a base64 data URL.

image_compress(*columns[, quality, format, ...])

Compress encoded image bytes with the specified format and quality.

image_convert_format(*columns, format[, ...])

Convert encoded image bytes to a different encoded format.

image_to_tensor(*columns, width, height[, ...])

Convert an image to a fixed-shape float32 tensor.

image_convert_mode(*columns, mode[, concurrency])

Convert an image to a target mode.

image_resize(*columns, width, height[, ...])

Resize an image to fixed output dimensions.

image_rescale(*columns, scale[, method, ...])

Rescale an image by a multiplicative factor, preserving aspect ratio.

image_crop(*columns[, crop_coords, ...])

Crop an image by coordinates, center ratio, or center size.

image_crop_black_border(*columns[, ...])

Detect and remove black borders from an image.

image_flip(*columns[, mode, concurrency])

Flip an image horizontally, vertically, or rotate 180 degrees.

image_blur(*columns[, radius, blur_type, ...])

Apply blur to an image.

image_adjust_color(*columns[, brightness, ...])

Adjust image brightness, contrast, and saturation.

image_remove_background(*columns[, ...])

Remove the background from an image using rembg (U2-Net based).

Information#

Image information extraction operators.

Most operators in this module take decoded Image values and return numeric values, strings, or booleans - they do not modify the image. Lightweight metadata and filter operators may also inspect encoded bytes when documented.

Without business params (factory function, no args):

from pyflink.multimodal.operators.image_info import image_aspect_ratio
df.with_column("ratio", image_aspect_ratio()(col("img")))

With business params (factory function):

from pyflink.multimodal.operators.image_info import image_hash
phash = image_hash(method="phash")
df.with_column("hash", phash(col("img")))
Runtime args:

concurrency is forwarded to the DataFrame UDF runtime.

image_metadata(*columns[, concurrency])

Extract metadata (width, height, channels, mode) from an image.

image_aspect_ratio(*columns[, concurrency])

Compute the aspect ratio (width / height) of an image.

image_sharpness(*columns[, max_edge, ...])

Compute a raw sharpness metric using Laplacian variance.

image_hash(*columns[, method, concurrency])

Compute a perceptual hash of an image.

is_valid_image(*columns[, mode, ...])

Check if an image passes lightweight validity checks.

image_size_filter(*columns[, min_w, min_h, ...])

Filter images by pixel dimensions.

image_shape_filter(*columns[, min_ratio, ...])

Filter images by aspect ratio (width / height).

image_file_size_filter(*columns[, ...])

Filter images by encoded file size in bytes.

Detection#

Detection and recognition operators.

Operators in this module detect or recognize structured content within images: objects, text, segments, and subplots.

Model-based operators (ImageDetection, ImageSegment, ImageOCR) are registered as pandas batch UDFs, so Flink batches rows automatically. image_detect_objects returns axis-aligned boxes x/y/w/h; image_ocr returns four-point text polygons; image_segment returns indexed PNG mask bytes; image_detect_subplot returns subplot metadata.

Example:

>>> from pyflink.multimodal.operators.image_detect import (
...     image_detect_objects, image_ocr, image_segment, image_detect_subplot,
... )
>>> from pyflink.dataframe import col

>>> detect = image_detect_objects(model="yolov8n", confidence=0.05)
>>> df = df.with_column("objects", detect(col("img")))

>>> ocr = image_ocr(lang=["en", "ch_sim"])
>>> df = df.with_column("text", ocr(col("img")))
Runtime args:

Model-based pandas factories forward keyword-only concurrency, batch_size, num_gpus, and gpu_type to the DataFrame UDF runtime. image_detect_subplot forwards concurrency.

image_detect_objects(*columns[, model, ...])

Create an object detection UDF (YOLO-based).

image_segment(*columns[, model, confidence, ...])

Create a semantic segmentation UDF (FastSAM-based).

image_ocr(*columns[, lang, model_sharing, ...])

Create an OCR text extraction UDF (EasyOCR-based).

image_detect_subplot(*columns[, threshold, ...])

Create a subplot / mosaic detection UDF (OpenCV Hough-based).

Face#

Face processing operators.

Operators in this module detect, count, or blur faces in image inputs with OpenCV Haar cascade classifiers.

Requires: pip install opencv-python-headless. image_face_blur also requires pip install Pillow.

Runtime args:

concurrency is forwarded to the DataFrame UDF runtime.

Usage:

>>> from pyflink.multimodal.operators.image_face import (
...     image_face_detect, image_face_count, image_face_blur,
... )
>>> from pyflink.dataframe import col
>>> detect = image_face_detect(min_neighbors=3)
>>> df = df.with_column("faces", detect(col("img")))
>>> count = image_face_count(min_neighbors=3)
>>> df = df.with_column("face_count", count(col("img")))
>>> blur = image_face_blur(radius=15, min_neighbors=3)
>>> df = df.with_column("blurred", blur(col("img")))

image_face_detect(*columns[, cv_classifier, ...])

Create a face detection UDF (OpenCV Haar Cascade-based).

image_face_count(*columns[, cv_classifier, ...])

Create a face count UDF (OpenCV Haar Cascade-based).

image_face_blur(*columns[, cv_classifier, ...])

Create a face blurring UDF (OpenCV Haar Cascade + PIL-based).

Quality#

Image quality and safety assessment operators.

Operators in this module score images based on quality, safety, and watermark detection. All operators return float scores.

Example:

>>> from pyflink.multimodal.operators.image_quality import image_quality_score
>>> quality = image_quality_score()
>>> df = df.with_column("quality", quality(col("img")))

>>> from pyflink.multimodal.operators.image_quality import image_nsfw_score
>>> nsfw = image_nsfw_score()
>>> df = df.with_column("nsfw", nsfw(col("img")))
Runtime args:

Pandas batch UDFs accept concurrency, batch_size, num_gpus, and gpu_type. Row-level UDFs accept concurrency.

Input errors:

Null image inputs produce None outputs. These operators accept decoded images; use decode_image(on_error="null") when processing untrusted raw bytes. is_valid_image can be used as a lightweight pre-filter but does not guarantee full pixel materialization.

image_nsfw_score(*columns[, hf_nsfw_model, ...])

Create an NSFW scoring UDF (HuggingFace image classification-based).

image_aesthetic_score(*columns[, ...])

Create an aesthetic quality scoring UDF (HuggingFace aesthetics predictor-based).

image_quality_score(*columns[, concurrency])

Create an image quality scoring UDF (Laplacian + contrast + colorfulness-based).

image_watermark_score(*columns[, ...])

Create a watermark scoring UDF (HuggingFace image classification-based).

Embedding#

Image embedding and similarity operators.

Operators in this module generate vector embeddings from images or compute image-text similarity scores using CLIP.

All operators in this module use pandas batch UDFs and run CLIP inference once per input batch.

Example:

>>> from pyflink.multimodal.operators.image_embed import (
...     image_embedding, image_text_similarity,
... )

>>> embed = image_embedding(model="ViT-B/32")
>>> df = df.with_column("vector", embed(col("img")))

>>> # Fixed text mode — compare all images against one prompt
>>> sim = image_text_similarity(text="a photo of a cat")
>>> df = df.with_column("score", sim(col("img")))

>>> # Per-row text mode — each row has its own text
>>> sim = image_text_similarity()
>>> df = df.with_column("score", sim(col("img"), col("text")))
Runtime args:

concurrency, batch_size, num_gpus, and gpu_type are forwarded to the DataFrame UDF runtime.

Requires: pip install open_clip_torch torch pillow

image_embedding(*columns[, model, ...])

Create an image embedding UDF (CLIP / open_clip-based).

image_text_similarity(*columns[, text, ...])

Create an image-text similarity scoring UDF (CLIP / open_clip-based).

Image Codec#

Codec and format conversion helpers for multimodal data.

Decoded Image encode/decode conversion helpers for multimodal operators.

image_mode_channels(mode)

Return the channel count used by a decoded Image mode.

image_to_ndarray(image[, mode])

Return decoded Image pixels as an ndarray for operator internals.

ndarray_to_image(pixel_array[, mode])

Convert an ndarray to a decoded Image.

pil_to_image(pil_image[, mode])

Convert a PIL image to a decoded Image.

convert_image_array_to_mode(pixel_array, mode)

Convert an image ndarray to one of the built-in image modes.

image_array_to_pil(pixel_array[, mode])

Convert ndarray pixels to a PIL Image, converting mode only when requested.

image_array_to_pil_compatible(pixel_array)

Convert ndarray pixels to a PIL Image, normalizing only when required.

decode_image(image_bytes[, mode, max_pixels])

Decode encoded image bytes into a decoded Image.

encode_image_input(image_input[, ...])

Encode a decoded Image input to compressed image bytes.