Gemma 4 and Ollama: Revolutionizing Local Multimodal Analysis

Applications of LLMs

Building Multimodal Workflows with a Local LLM

The use of Gemma 4 and Ollama to integrate image inputs and structured outputs is particularly appealing when it comes to handling private data or developing workflows on local machines. These workflows are no longer limited to text, paving the way for richer and more diverse applications.

This article explores how to build a workflow using Gemma 4 and Ollama. It demonstrates how Gemma 4's multimodal capability can be integrated into a broader process, allowing subsequent steps to consume its structured output.

The author of the study recently traveled to Finland and took numerous photos during this trip. This context serves to illustrate how this workflow can be used to analyze these images and feed a smaller application.

1. The Multimodal Workflow

The concept of a "multimodal workflow" is based on two main ideas.

• Multimodal: This means that the LLM does not only work with text but also receives inputs of other types, such as images. In this article, the local LLM from Google's Gemma 4 family is used, capable of processing both images and text.

• Workflow: This means that the LLM does not autonomously decide the next step, as in an agentic loop. Instead, it operates in a predefined sequence, where each step receives an input and produces an output for the next step. The LLM serves here as a function to perform semantic transformations.

For the case study, the goal is to transform a folder of travel photos into organized memory records. This involves analyzing each photo and translating its content into a structured and coherent record. Once these records are available, they can be combined to gain an overall understanding of the collection.

Thus, a three-step workflow is established:

photos = prepare_photos("Finland_trip")
photo_memories = [
    image=photo.image,
    metadata=photo.metadata,
    for photo in photos
]
trip_memory = synthesize_trip(photo_memories)

First, Python prepares the individual images and extracts useful metadata such as capture times and GPS coordinates.

Next, the local LLM Gemma 4 analyzes each photo and returns a structured record of the visual content of the photo.

Finally, the individual records are passed back to Gemma 4 to produce a structured summary of the entire collection.

In the following sections, the individual steps will be constructed.

2. Building the Workflow with Gemma 4

2.1 Running Gemma 4 Locally

To begin, it is necessary to make Gemma 4 available locally. Ollama is used for this, providing a local execution environment and an interface to interact with the model.

In this article, the E4B variant of Gemma is used, which is one of the models suited for devices in the family.

On Windows and macOS, the installer can be downloaded and executed from the Ollama website. On Linux, installation is done via the terminal:

curl -fsSL https://ollama.com/install.sh | sh

Once Ollama is installed, the Gemma model can be retrieved:

ollama pull gemma4:e4b

The necessary Python packages for the workflow must also be installed:

pip install ollama pillow pydantic

Here, ollama is needed to connect the Python code to the local LLM, pillow handles image processing, and pydantic is used for structured output.

2.2 From Photo to Structured Record

For each photo, a structured record describing what the model sees is desired.

Before the images reach Gemma 4, they must be preprocessed. In the workflow, a deterministic logic for resizing the image and extracting available EXIF metadata is implemented, encapsulated in prepare_photo().

From the workflow's perspective, only the outputs are needed:

from pathlib import Path

image, metadata = prepare_photo(
    Path("Finland_trip/photo.jpg")
)

Here, image contains the bytes of the prepared image, while metadata is a dictionary containing the information extracted from the file.

Complete implementation details can be found in the repository linked at the end of the article.

Next, the structured record that Gemma 4 should return is defined:

from pydantic import BaseModel, Field

class PhotoMemoryAnalysis(BaseModel):
    scene_summary: str
    memory_caption: str
    visible_activities: list[str]
    visible_objects: list[str]
    inferred_interest_signals: list[str]
    uncertainty_notes: list[str]
    confidence: float = Field(ge=0, le=1)

This is what is called a structured output. Indeed, this schema is predefined, and the LLM is asked to produce an output according to this form. In this way, downstream code can access the result via typed attributes or convert it into a dictionary with model_dump().

Then, the instruction and prompt are constructed:

PHOTO_ANALYSIS_INSTRUCTION = """
Analyze the supplied travel photo and its metadata.
Return a structured record grounded in the provided inputs.
"""

def build_photo_prompt(metadata: dict) -> str:
    return f"""Photo metadata:
{json.dumps(metadata, indent=2)}

Everything can now be assembled:

MODEL = "gemma4:e4b"

def analyze_photo() -> PhotoMemoryAnalysis:
    response = ollama.chat(
        "role": "system",
        "content": PHOTO_ANALYSIS_INSTRUCTION,
        "content": build_photo_prompt(metadata),
        "images": [image],
        format=PhotoMemoryAnalysis.model_json_schema(),
        options={"temperature": 0},
    )
    return PhotoMemoryAnalysis.model_validate_json(
        response.message.content
    )

Note that the images field is used to provide the visual input, and format is used to request Ollama to follow the defined schema. The response can then be parsed into a classic Python object:

photo_analysis = analyze_photo(metadata=metadata)

In this way, the routine for transforming a photo into a structured memory record is established.

2.3 A Compatibility Issue with Image Input

One note to mention: with the current setup for Ollama (0.32.5) on a Windows machine with gemma4:e4b, it seems that Ollama accepted the multimodal request, but the model failed to utilize its visual content.

A simple solution adopted is to load Gemma 4 using two separate files from the Unsloth repository of Gemma 4 E4B GGUF:

• mmproj-BF16.gguf, which contains the multimodal projector.
• gemma-4-E4B-it-UD-Q4_K_XL.gguf, which contains the quantized model.

The projector is the component that allows the model to consume visual information.

After placing both files in the same folder, a Modelfile is created as follows:

FROM ./gemma-4-E4B-it-UD-Q4_K_XL.gguf
FROM ./mmproj-BF16.gguf

The model is then imported into Ollama:

ollama create gemma4-e4b-split-test -f Modelfile

Finally, the model name is updated in Python:

MODEL = "gemma4-e4b-split-test"

With this configuration, the Gemma model can correctly consume the image. Of course, if gemma4:e4b already responds correctly to image inputs on a machine, this workaround is not necessary.

2.4 From Photo Records to Trip Memory

At this stage, each photo can be understood independently.

For the final step, the goal is to understand the trip as a whole. For this, these records can be passed to Gemma 4 once again and asked to link the individual moments into a trip memory.

First, analyze_photo() is applied to the entire folder:

from pathlib import Path

photo_memories = []
for image_path in Path("Finland_trip").glob("*.jpg"):
    image, metadata = prepare_photo(image_path)
    analysis = analyze_photo(metadata=metadata)
    photo_memories.append({
        "photo_id": image_path.name,
        "analysis": analysis.model_dump(),
        "metadata": metadata,
    })

Note that each photo memory now combines two sources of information: the metadata extracted from the file and the semantic interpretation of the image provided by Gemma 4.

Another schema for the final output is defined:

class MemorableMoment(BaseModel):
    description: str
    evidence_photo_ids: list[str]

class TripMemorySynthesis(BaseModel):
    narrative_summary: str
    inferred_interests: list[str]
    recurring_themes: list[str]
    memorable_moments: list[MemorableMoment]
    uncertainty_notes: list[str]

The instruction and prompt are then defined:

TRIP_SYNTHESIS_INSTRUCTION = """
Synthesize the supplied photo records into a structured trip memory.
Use only the information contained in those records.
"""

def build_trip_prompt(photo_memories: list[dict]) -> str:
    return f"""Photo memory records:
{json.dumps(photo_memories, indent=2)}

The final call to the model follows the same schema as before. Note that this call is purely textual:

def synthesize_trip(photo_memories: list[dict]) -> TripMemorySynthesis:
    response = ollama.chat(
        "role": "system",
        "content": TRIP_SYNTHESIS_INSTRUCTION,
        "content": build_trip_prompt(photo_memories),
        format=TripMemorySynthesis.model_json_schema(),
        options={"temperature": 0},
    )
    return TripMemorySynthesis.model_validate_json(
        response.message.content
    )

The workflow can now be completed:

trip_memory = synthesize_trip(photo_memories)

For this case study, the author selected 7 photos from their trip to Finland and executed the workflow on them.

Let’s first look at two of the photo records.

Figure 1. A Ferris wheel in Helsinki. (Image by the author)

Here is what Gemma 4 returned for the first image:

"scene_summary": (
    "A large Ferris wheel dominates the frame against "
    "a clear sky at dusk."
),
"memory_caption": (
    "Evening views from the Ferris wheel ride."
),
"visible_objects": [
    "Signage",
    "Guardrail/front platform",
],
"mood": "Festive",
"uncertainty_notes": [
    "The specific location is not "
    "provided, only the name of the attraction."
]

The model recognized the main attraction. It also read the visible text and incorporated the evening setting into the record.

Here’s another photo and the model's output:

Figure 2. An Arctic owl at the Ranua Wildlife Park in Finnish Lapland. (Image by the author)

"scene_summary": (
    "A large owl or raptor stands near a wooden structure "
    "and a wire fence in a lush environment."
),
"memory_caption": (
    "A wildlife encounter: A majestic bird "
    "observing its surroundings amidst dense greenery."
),
"visible_objects": [
    "Wooden structure",
    "Wire fence",
    "Dense foliage",
],
"mood": "Impressive, Peaceful",
"uncertainty_notes": [
    "The precise species of the bird cannot be "
    "defined solely from the image.",
    "The specific function of the wooden structure "
    "is ambiguous."
]

After analyzing the seven photos, the workflow passed the results to the next step.