Gemini Revolutionizes File Search with Multimodal RAG API

The Gemini API: A Breakthrough in File Search

A New Approach to Accessing Data

The Gemini API offers an innovative solution for accessing and utilizing information from various data sources. Whether it's reports, research documents, code, or private knowledge bases, this technology enables simplified and efficient data management. When a file is uploaded, Gemini segments it into smaller units, called "chunks," and generates embeddings for each of these segments. These embeddings are numerical representations that capture the meaning of the content, allowing Gemini to understand the context in depth. This data is then stored in a File Search Store, facilitating later retrieval.

When you ask a question to Gemini, it searches for the most relevant embeddings among those stored and uses them to generate answers. This process is at the heart of Retrieval Augmented Generation (RAG), a method that enriches content generation by relying on pre-existing data.

Multimodal Search for Enhanced Understanding

Gemini's file search is not limited to text. It also incorporates multimodal RAG, allowing for the indexing and searching of both textual and visual information. This means you can extract information from PDF files, images, graphs, screenshots, and much more using queries formulated in natural language. For multimodal tasks, Gemini uses gemini-embedding-2 for images and multimodal embeddings, while gemini-embedding-001 is dedicated to textual embeddings. However, it is important to note that audio and video formats are not yet supported.

Detailed Operation of File Search

Gemini's file search relies on semantic vector search. Unlike traditional search methods that depend on exact word matching, this approach finds information by analyzing meaning and context. This allows Gemini to provide relevant information even if the phrasing of the query differs from that of the source documents.

Here’s an overview of the process:

• File Upload: The file is divided into smaller sections, called "chunks."
• Embedding Generation: Each chunk is transformed into a numerical vector that represents its meaning.
• Storage: These embeddings are stored in a File Search Store, a dedicated space for retrieval.
• Query: When a user asks a question, it is transformed into an embedding.
• Retrieval: The question embedding is compared to the stored embeddings to identify the most similar chunks.
• Anchoring: The relevant chunks are integrated into the prompt of the Gemini model, ensuring that the response is anchored in the factual data from the documents.

This process is entirely managed by the Gemini API, relieving developers from the need to handle additional infrastructure or databases.

Requirements for Using the Tool

To take advantage of the file search tool, developers need to have certain essential elements. It is necessary to have Python 3.9 or a later version, the google-genai client library, and a valid Gemini API key with access to gemini-2.5-pro or gemini-2.5-flash.

To install the client library, run the following command:

pip install google-genai -U

Then, set up your environment variable for the API key:

export GOOGLE_API_KEY="your_api_key_here"

Creating a File Search Store

A File Search Store is where Gemini stores and indexes the embeddings created from uploaded files. Once a file is uploaded and indexed, the data remains available for retrieval until it is manually deleted.

For text-only RAG, you can create a standard File Search Store. For multimodal RAG, where you want to upload and search both documents and images, create the store with models/gemini-embedding-2.

from google import genai
from google.genai import types
from pathlib import Path
import os

# Do not hardcode your API key in the notebook.
# Instead, set it as an environment variable.
os.environ["GOOGLE_API_KEY"] = "enter_your_api_key"
client = genai.Client(api_key=os.environ["GOOGLE_API_KEY"])
file_search_store = client.file_search_stores.create(
    display_name="my_multimodal_rag_store",
    embedding_model="models/gemini-embedding-2"
)
print("File Search Store created:", file_search_store.name)

This update is crucial as the official documentation specifies the use of embedding_model: models/gemini-embedding-2 when creating a File Search Store for multimodal use.

Uploading and Indexing Files

Once the File Search Store is created, you can upload files to it. When a file is uploaded, Gemini's file search automatically segments the content, generates embeddings, and indexes it for quick retrieval.

For text-based RAG, the file search supports documents such as PDF, DOCX, TXT, JSON, as well as programming files like .py and .js.

For multimodal RAG, the file search also supports image files. This means you can upload both documents and images in the same File Search Store and ask questions that require both textual and visual context. For example, you can upload a research paper, a product image, and a chart, then ask Gemini to summarize the paper and explain the associated visual information.

For image uploads, ensure that the File Search Store is created with models/gemini-embedding-2. According to the official documentation, supported image formats are PNG and JPEG. Image files must be a maximum of 4K x 4K pixels.

Uploading a Document File

# Upload and import a document into the File Search Store.
# The display name will be visible in citations.
operation = client.file_search_stores.upload_to_file_search_store(
    file="/content/Paper2Agent.pdf",
    file_search_store_name=file_search_store.name,
    display_name="Paper2Agent.pdf",
)

# Wait for the upload to complete
while not operation.done:
    operation = client.operations.get(operation)

print("Document successfully uploaded and indexed.")

After this step, the document is divided into chunks, embedded, indexed, and ready for retrieval.

Uploading an Image File for Multimodal Retrieval

You can also upload an image file in the same File Search Store. This is useful when your application needs to retrieve information from product images, screenshots, graphs, diagrams, or other visual content.

# Upload an image file for multimodal retrieval.
operation = client.file_search_stores.upload_to_file_search_store(
    file="/content/product_image.jpg",
    file_search_store_name=file_search_store.name,
    display_name="product_image.jpg",
)

# Wait for the upload to complete
while not operation.done:
    operation = client.operations.get(operation)

print("Image successfully uploaded and indexed.")

Once the image is indexed, Gemini can retrieve it during file searches when the user's query is relevant to the image.

Uploading Multiple Documents and Images

In real-world applications, you may want to upload multiple files at once. These files can include both text documents and images.

from pathlib import Path

files_to_upload = [
    "/content/Paper2Agent.pdf",
    "/content/product_image.jpg",
    "/content/sales_chart.png"
]

for file_path in files_to_upload:
    operation = client.file_search_stores.upload_to_file_search_store(
        file_search_store_name=file_search_store.name,
        display_name=Path(file_path).name,
    )
    
    while not operation.done:
        operation = client.operations.get(operation)
    
    print(f"Uploaded and indexed: {file_path}")

After the upload step, all files are divided into chunks, embedded, indexed, and ready for retrieval. If the File Search Store contains both documents and images, Gemini can retrieve relevant context from both sources while answering user questions.

Asking Questions About the File

Once your files are indexed, Gemini can answer questions using the uploaded documents and images as context. It searches the File Search Store, retrieves the most relevant chunks, and uses them to generate a fact-based response.

For a purely text-based use case, you can ask a question about the uploaded PDF:

response = client.models.generate_content(
    model="gemini-2.5-flash",
    contents="Summarize what is there in the research paper.",
    config=types.GenerateContentConfig(
        file_search_store_name=file_search_store.name,
    )
)

For multimodal queries, a request can include a maximum of 6 images.