pyflink.multimodal.operators.image_text_similarity#

image_text_similarity(*columns, text=None, model='ViT-B/32', pretrained='openai', model_sharing=None, concurrency=None, batch_size=None, num_gpus=None, gpu_type=None)[source]#

Create an image-text similarity scoring UDF (CLIP / open_clip-based).

When text is provided, computes similarity of all images against that fixed text prompt (single-column UDF). When text is None, computes per-row similarity between an image column and a text column (two-column UDF).

Requires pip install open_clip_torch torch Pillow. This is a pandas batch UDF that supports GPU acceleration via num_gpus / gpu_type.

Parameters

  • *columns – Optional image column(s). When provided, the UDF is applied directly instead of returning a factory.

  • text – Fixed text prompt to compare against all images. None (default) enables per-row mode where a text column is passed at call time.

  • model – CLIP model architecture name. Default "ViT-B/32".

  • pretrained – Pretrained weights checkpoint. Default "openai".

  • model_sharing – Model sharing mode across parallel subtasks. None uses per-process caching.

  • concurrency – UDF concurrency. None uses the framework default.

  • batch_size – Pandas batch size. None uses the framework default.

  • num_gpus – Fractional GPU count per subtask, e.g. 0.5. None runs on CPU.

  • gpu_type – Required GPU type, e.g. "A10". None accepts any available GPU.

Returns

A UDF that returns a cosine similarity score in [-1, 1], or None for null image inputs. Per-row mode also returns None for null text inputs.

Example::
>>> # As a reusable variable (fixed text mode)
>>> sim = image_text_similarity(text="a photo of a cat")
>>> df = df.with_column("score", sim(col("img")))
>>>
>>> # As a reusable variable (per-row text mode)
>>> sim = image_text_similarity()
>>> df = df.with_column("score", sim(col("img"), col("text")))
>>>
>>> # Inline
>>> df = df.with_column(
...     "score", image_text_similarity(col("img"), col("text"))
... )