Sentence Transformers: Key Advancement in Multimodal Embedding

Sentence Transformers: A Breakthrough in Multimodal Embedding

The Sentence Transformers library is an essential tool for developers looking to utilize and train embedding and reranking models. These models are crucial for applications such as retrieval-augmented generation and semantic search. This article details how these models can be fine-tuned on specific data to enhance their performance.

Practical Example: Fine-tuning for Visual Document Retrieval

A concrete application example is the fine-tuning of the Qwen/Qwen3-VL-Embedding-2B model for Visual Document Retrieval (VDR). This task involves retrieving relevant document pages in image form while preserving their original layout in response to a textual query. The fine-tuned model, named tomaarsen/Qwen3-VL-Embedding-2B-vdr, has demonstrated significant improvements. On evaluation data, it achieved an NDCG@10 of 0.947, surpassing the baseline model, which had a score of 0.888. This result is noteworthy as it outperforms all existing tested VDR models, even those with sizes up to four times larger.

Essential Components of Training

Training multimodal models with Sentence Transformers relies on the same components as for textual models. These components include the model to be fine-tuned, the dataset, the loss function, and optionally training arguments and an evaluator. The training process is orchestrated by the SentenceTransformerTrainer, which automatically manages the preprocessing of images and other modalities.

The multimodal model to be trained or fine-tuned is central to this process. The dataset used for training and evaluation must be carefully selected to match the specific task. The loss function quantifies the model's performance and guides the optimization process. Optional training arguments can influence training performance and facilitate tracking and debugging. An evaluator can be used to assess the model before, during, or after training.

Dataset for Visual Document Retrieval

To illustrate this fine-tuning, the dataset tomaarsen/llamaindex-vdr-en-train-preprocessed was used. This preprocessed English subset comes from llamaindex/vdr-multilingual-train. The original dataset, published by LlamaIndex, contains approximately 500k samples of multilingual query-image pairs derived from public PDFs. The queries were synthetically generated using VLMs such as gemini-1.5-pro and Qwen2-VL-72B.

In the preprocessed version, 53,512 English samples were selected. This version resolves 4 of the 16 difficult negatives based on the ID per sample in real images of document screenshots, allowing for direct use in training without further preprocessing.

The dataset format must be compatible with the chosen loss function. If a label is required, a "label" or "score" column must be present. Other columns are considered inputs, which may include images, audio, video, or multimodal dictionaries. The data collator uses model.preprocess() to automatically detect and process each modality, thereby eliminating the need for tokenization or manual image processing.