Gemini Embedding 2: Google's AI Revolutionizes Integration

Gemini Embedding 2: A Breakthrough in Multimodal Integration

Traditional embedding systems were often limited to text processing, requiring separate pipelines to handle other types of content such as images or audio. Gemini Embedding 2, developed by Google, disrupts this approach by integrating various data formats into a unified vector space, thus facilitating their simultaneous processing.

Google designed Gemini Embedding 2 to handle multiple types of content:

• Text with an input capacity of up to 8192 tokens.
• Images, supporting up to 6 images per query, in PNG and JPEG formats.
• Videos with a maximum duration of 120 seconds, compatible with mp4 and mov formats.
• Audio, without the need for prior transcription.
• PDF documents containing up to 6 pages.

This model also allows for multimodal inputs, combining, for example, an image and text in a single query, enriching the understanding of relationships between different types of data.

Another notable feature is the dimensional flexibility offered by Matryoshka Representation Learning. By default, the output is in 3072 dimensions, but it can be adjusted to 1536 or 768 dimensions, allowing developers to optimize between quality, storage space, and processing speed according to their specific needs.

Development of an Image Matching System with Gemini Embedding 2

To illustrate the application of Gemini Embedding 2, a project was set up to create an image matching system. This project uses three folders within a data directory:

• dataset/nitika/vasu/janvi/

Each folder contains several images of the same person. The process is structured as follows:

• Reading all images from the dataset.
• Generating an embedding for each image using Gemini Embedding 2.
• Storing the embeddings in memory and caching them locally.
• Taking a query image.
• Generating its embedding.
• Comparing this embedding with those stored, using cosine similarity.
• Returning the most similar images and predicting the person's name.

This project demonstrates how Gemini Embedding 2 can be used for image retrieval and lightweight classification, without requiring extensive training in deep learning. There is no need for custom CNNs, fine-tuning, or heavy annotation processes, significantly speeding up development.

Thanks to the multimodal nature of Gemini Embedding 2, this approach can be extended to other types of content, such as:

• Associating an audio clip with a person's profile.
• Finding a relevant PDF from an image.
• Retrieving a video segment from a text query.
• Comparing mixed image and text descriptions in a single embedding space.

Thus, this project serves as a starting point towards a broader multimodal retrieval architecture.

Using the Gemini Embedding 2 API

Google offers Gemini Embedding 2 through the Gemini API and Vertex AI. The embed_content method is used to call the embedding.

A multimodal example provided by Google might look like this:

from google import genai
from google.genai import types

client = genai.Client()

with open("example.png", "rb") as f:
    image_bytes = f.read()

with open("sample.mp3", "rb") as f:
    audio_bytes = f.read()

result = client.models.embed_content(
    model="gemini-embedding-2-preview",
    "What is the meaning of life?",
    types.Part.from_bytes(
        data=image_bytes,
        mime_type="image/png",
        types.Part.from_bytes(
            data=audio_bytes,
            mime_type="audio/mpeg",
        )
    )
)

print(result.embeddings)

For my project, I only needed the image part of this workflow. Instead of sending text, an image, and audio together, I used a single image per query to generate its embedding.

Implementing the Project

The project begins by loading the Gemini API key from a .env file and creating a client:

from dotenv import load_dotenv
from google import genai

GEMINI_API_KEY = os.getenv("GEMINI_API_KEY")
client = genai.Client(api_key=GEMINI_API_KEY)

Next, helper functions are defined to validate images, detect MIME types, normalize data, calculate cosine similarity, and display images.

The main embedding function reads the bytes of the image and sends them to Gemini Embedding 2:

def embed_image(image_path):
    image_path = Path(image_path)
    mime_type = guess_mime_type(image_path)
    with open(image_path, "rb") as f:
        image_bytes = f.read()
    result = client.models.embed_content(
        model="gemini-embedding-2-preview",
        types.Part.from_bytes(
            data=image_bytes,
            mime_type=mime_type,
            config=types.EmbedContentConfig(
                output_dimensionality=3072
            )
        )
    )
    emb = np.array(result.embeddings[0].values, dtype=np.float32)
    return normalize(emb)

This function is the heart of the pipeline, transforming each image into a 3072 dimensional vector representation.

Building the Dataset Embeddings Database

The next step is to traverse the dataset folder, read all images for each person, and embed them individually.

Each embedded image is stored as a dictionary containing:

• the person's label
• the embedding vector

To avoid recalculating the embeddings each time, they are cached in a local pickle file:

def build_embeddings_db(dataset, cache_file="image_embeddings_cache.pkl", force_rebuild=False):
    cache_path = Path(cache_file)
    if cache_path.exists() and not force_rebuild:
        with open(cache_path, "rb") as f:
            embeddings_db = pickle.load(f)
        return embeddings_db

    embeddings_db = []
    for item in dataset:
        emb = embed_image(item["path"])
        embeddings_db.append({
            "label": item["label"],
            "path": item["path"],
            "embedding": emb
        })
    
    with open(cache_path, "wb") as f:
        pickle.dump(embeddings_db, f)
    
    return embeddings_db

This makes the process much more efficient, as embeddings are generated only once unless the dataset changes.

Matching a Query Image

Once the dataset embeddings are ready, the next step is to test the system with a new query image.

The query image is embedded using the same function, and its embedding is compared to all stored embeddings via cosine similarity.

def find_best_matches(query_image_path, top_k=5):
    query_emb = embed_image(query_image_path)
    results = []
    
    for item in embeddings_db:
        score = cosine_similarity(query_emb, item["embedding"])
        results.append({
            "label": item["label"],
            "path": item["path"],
            "score": score
        })
    
    results.sort(key=lambda x: x["score"], reverse=True)
    return results[:top_k]

This function returns the best matching images from the dataset.

To predict the final label of the person, a vote from the top-k is used:

def predict_person(query_image_path, top_k=5):
    matches = find_best_matches(query_image_path, top_k=top_k)
    labels = [m["label"] for m in matches]
    predicted_label = Counter(labels).most_common(1)[0][0]
    return predicted_label, matches

This method is more reliable than relying on a single closest image.

Testing the Project

As part of the project, tests were conducted with query images such as:

query_image = "Nitika_Test_Image.jpeg"
predicted_person, matches = predict_person(query_image, top_k=2)
print("\nQuery Image:")
show_image(query_image, title="Query Image")
print("Predicted Person:", predicted_person)
print("\nBest Matches:")
for i, match in enumerate(matches, 1):
    print(f"{i}. {match['label']} | score={match['score']:.4f} | path={match['path']}")
    show_image(match["path"], title=f"Rank {i} | {match['label']} | score={match['score']:.4f}")

This allows for visualizing the results and verifying the accuracy of the image matching system.