Phrase Transformers: Multimodal Revolution and Reranking

Understanding Multimodal Models

Traditional embedding models focus on converting text into fixed-size vectors. However, multimodal embedding models go further by integrating inputs from various modalities such as text, images, audio, or video into a common embedding space. This approach allows for comparing a text query with image documents or vice versa, using already known similarity functions.

Traditional reranking models, often referred to as Cross Encoders, calculate relevance scores between pairs of texts. In contrast, multimodal rerankers can evaluate pairs where one or both elements are images, combined text-image documents, or other modalities. This opens up possibilities such as comparing a text query with image documents, searching for video clips that match a description, or building RAG pipelines that operate across modalities.

To use these multimodal models, some additional dependencies are required. For example, for images, you can install the necessary extras with pip install -U "sentence-transformers[image]". Similarly, for audio and video modalities, similar commands are available. VLM-based models, such as Qwen3-VL-2B, require a GPU with at least 8 GB of VRAM. For more advanced variants, like the 8B models, around 20 GB of VRAM is needed. If you do not have a local GPU, it is advisable to use a cloud GPU service or Google Colab. On a CPU, these models will be extremely slow, and it is better to use purely textual models or CLIP for inference on CPU.

How Multimodal Embedding Models Work

Loading a multimodal embedding model is similar to loading a text-only model. For example, with the sentence-transformers library, you can load the model "Qwen/Qwen3-VL-Embedding-2B". Some models may require a revision argument if integration requests for the model are still pending. Once they are merged, you will be able to load them without specifying a revision.

The model automatically detects the modalities it supports, so no additional configuration is necessary. You can check the processor and model parameters if you want to control elements like image resolution or model accuracy. With a loaded multimodal model, model.encode() accepts images in addition to text. Images can be provided as URLs, local file paths, or PIL Image objects. Here’s an example of usage:

from sentence_transformers import SentenceTransformer
model = SentenceTransformer("Qwen/Qwen3-VL-Embedding-2B")
img_embeddings = model.encode([
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg",
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg",
])
print(img_embeddings.shape)

Calculating Cross-Modal Similarity

It is possible to calculate similarities between text and image embeddings, as the model maps both into the same space. For example:

from sentence_transformers import SentenceTransformer
model = SentenceTransformer("Qwen/Qwen3-VL-Embedding-2B")
img_embeddings = model.encode([
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg",
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg",
])
text_embeddings = model.encode([
    "A green car parked in front of a yellow building",
    "A red car driving on a highway",
    "A bee on a pink flower",
    "A wasp on a wooden table",
])
similarities = model.similarity(text_embeddings, img_embeddings)
print(similarities)

In this example, "A green car parked in front of a yellow building" is most similar to the image of the car (0.51), and "A bee on a pink flower" is most similar to the image of the bee (0.67). Hard negatives, such as "A red car driving on a highway" and "A wasp on a wooden table," correctly receive lower scores.

It is worth noting that even the best matching scores (0.51, 0.67) are not very close to 1.0. This is due to the modality gap: embeddings from different modalities tend to cluster in separate regions of space. Cross-modal similarities are generally weaker than those within the same modality (e.g., text-text), but the relative order is preserved, ensuring effective retrieval.

Encoding Queries and Documents

For retrieval tasks, the methods encode_query() and encode_document() are recommended. Many retrieval models prefix different instructions based on whether the input is a query or a document, similar to how chat models might apply different system prompts based on the objective. Model authors can specify their prompts in the model configuration, and encode_query() / encode_document() automatically load and apply the correct one:

• encode_query() uses the model's "query" prompt (if available) and sets task="query".
• encode_document() uses the first available prompt among "document," "passage," or "corpus," and sets task="document".

Behind the scenes, both are lightweight wrappers around encode(), simply managing prompt selection for you. Here’s what cross-modal retrieval looks like:

from sentence_transformers import SentenceTransformer
model = SentenceTransformer("Qwen/Qwen3-VL-Embedding-2B")
query_embeddings = model.encode_query([
    "Find me a photo of a vehicle parked near a building",
    "Show me an image of a pollinating insect",
])
doc_embeddings = model.encode_document([
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg",
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg",
])
similarities = model.similarity(query_embeddings, doc_embeddings)
print(similarities)

These methods accept the same input types as encode() (images, URLs, multimodal dictionaries, etc.) and pass the same parameters. For models without specialized prompts for queries/documents, they behave identically to encode().

Multimodal Reranking Models

Multimodal reranking models (CrossEncoders) evaluate relevance between pairs of inputs, where each element can be text, an image, audio, video, or a combination. They tend to outperform embedding models in terms of quality but are slower as they process each pair individually. Currently available pre-trained multimodal rerankers focus on text and image inputs, but the architecture supports any modality that the underlying model can handle.

Ranking Mixed Modality Documents

The rank() method evaluates and ranks a list of documents against a query, supporting mixed modalities:

from sentence_transformers import CrossEncoder
model = CrossEncoder("Qwen/Qwen3-VL-Reranker-2B")
query = "A green car parked in front of a yellow building"
documents = [
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg",
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg",
    "A vintage Volkswagen Beetle painted in bright green sits in a driveway.",
    {"text": "A car in a European city", "image": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg"},
]
rankings = model.rank(query, documents)
for rank in rankings:
    print(f"{rank['score']:.4f}\t(document {rank['corpus_id']})")

The reranker correctly identifies the image of the car (document 0) as the most relevant result, followed by the combined text+image document about a car in a European city (document 3). The image of the bee (document 1) receives the lowest score.

Keep in mind that the modality gap can influence absolute scores: text-image pair scores may occupy a different range than text-text or image-image pairs.

You can also check which modalities a reranker supports using modalities and supports(), just like with embedding models:

print(model.modalities)
print(model.supports("image"))
print(model.supports(("image", "text")))

Predicting Pair Scores

You can also use predict() to get raw relevance scores for specific input pairs:

from sentence_transformers import CrossEncoder
model = CrossEncoder("jinaai/jina-reranker-m0", trust_remote_code=True)
scores = model.predict([
    ("A green car", "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg"),
    ("A bee on a flower", "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"),
    ("A green car", "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"),
])

Retrieve and Rerank

A common scheme is to use an embedding model for a quick initial retrieval, then refine the best results with a reranker:

from sentence_transformers import SentenceTransformer, CrossEncoder
embedder = SentenceTransformer("Qwen/Qwen3-VL-Embedding-2B")
query = "revenue growth chart"
query_embedding = embedder.encode_query(query)

This process optimizes the quality of search results by combining the speed of embedding models with the precision of rerankers.