Gemma 4 from Google DeepMind: Multimodal AI on Hugging Face

Gemma 4 from Google DeepMind: Multimodal AI on Hugging Face

Google DeepMind, in collaboration with Hugging Face, has launched the Gemma 4 family of multimodal models. These models are now available on the Hugging Face platform, offering cutting-edge artificial intelligence with open Apache 2 licenses. They are designed for use on devices, incorporating significant advancements over previous versions.

The Gemma 4 models are truly open and high-quality, boasting impressive scores in the Pareto arena. They are multimodal, including audio, and available in sizes suitable for device use. During testing with preliminary checkpoints, the capabilities of the models were so impressive that it was challenging to find examples of fine-tuning, as they perform exceptionally well right out of the box.

The development of Gemma 4 involved close collaboration with Google and the community, making these models available on various platforms and libraries such as transformers, llama.cpp, MLX, WebGPU, and Rust. This openness allows developers to build with their preferred tools and share their experiences.

What’s New with Gemma 4?

Just like its predecessor Gemma-3n, Gemma 4 supports image, text, and audio inputs, generating text responses. The text decoder is based on the Gemma model with support for long contexts. The image encoder, similar to that of Gemma 3, has been enhanced with variable aspect ratios and a configurable number of image token inputs, allowing for the right balance between speed, memory, and quality.

The Gemma 4 models support images (or videos) and text inputs, while the smaller variants, E2B and E4B, also support audio. They come in four sizes, all fine-tuned for base and instruction fine-tuning:

• 2.3B effective, 5.1B with embeddings
• 4.5B effective, 8B with embeddings
• A mixture of experts model with 4B activated out of a total of 26B parameters

Overview of Capabilities and Architecture

Gemma 4 builds on several architectural components from previous versions and other open models, while eliminating complex or inconclusive features like Altup. This blend is designed to be highly compatible across libraries and devices, capable of efficiently supporting long contexts and agentic use cases, while being ideal for quantization.

Benchmarks show that this mix of features, combined with the training data and recipe, allows the dense model of 31B to achieve an estimated LMArena score of 1452, while the 26B MoE reaches 1441 with only 4B of active parameters. The multimodal operation is as effective as text generation, at least in informal and subjective tests.

The main architectural features of Gemma 4 include alternating attention layers with local sliding windows and full global context. Smaller dense models use sliding windows of 512 tokens, while larger models use 1024 tokens. The dual RoPE configurations, with standard RoPE for sliding layers and proportional RoPE for global layers, allow for longer context.

Per-Layer Embeddings (PLE)

One of the distinctive features of the smaller Gemma 4 models is the Per-Layer Embeddings (PLE), previously introduced in Gemma-3n. In a standard transformer, each token receives a single embedding vector at the input, and the same initial representation is built across all layers. PLE adds a parallel conditioning path of lower dimension alongside the main residual flow. For each token, it produces a small dedicated vector for each layer, combining a token identity component and a context-aware component.

Each decoder layer then uses its corresponding vector to modulate the hidden states via a lightweight residual block after attention and feed-forward. This allows each layer to receive token-specific information only when it becomes relevant, rather than requiring everything to be packed into a single initial embedding. Since the PLE dimension is much smaller than the main hidden size, this adds significant specialization per layer at a modest parameter cost.

For multimodal inputs (images, audio, video), PLE is computed before soft tokens are merged into the embedding sequence. The multimodal positions use the padding token ID, effectively receiving neutral layer signals.

The shared KV cache is an efficiency optimization that reduces both computation and memory during inference. The last layers of the model do not compute their own key and value projections but reuse the K and V tensors from the last non-shared layer of the same attention type.

Multimodal Capabilities

Tests have shown that Gemma 4 supports full multimodal capabilities right out of the box. While the exact training mix is not known, the model has successfully accomplished tasks such as OCR, speech recognition, object detection, and pointing. It also supports purely text-based and multimodal function calls, reasoning, code completion, and correction.

Object Detection and Pointing

In tests for detecting graphical user interface elements and pointing, Gemma 4 was tested with the following image and text prompt: "What is the frame for the 'view recipe' element in the image?" The model responded natively in JSON format with the detected frames, without the need for specific instructions or grammar-constrained generation. The coordinates refer to an image size of 1000x1000, relative to the input dimensions.

Object Detection

The models were also tested for detecting everyday objects, such as a bicycle, by comparing the different outputs from the models. The frames were analyzed from the JSON and translated into image coordinates.

Multimodal Thinking and Function Calling

Gemma 4 was used to write HTML code to reconstruct a page created with Gemini 3. The code was generated by activating reflection and asking each model to generate up to 4000 new tokens, ensuring a flawless process.

Video Understanding

The smaller Gemma 4 models can process videos with audio, while the larger ones can handle videos without audio. Although the models are not explicitly post-trained on videos, they can understand videos with and without audio. The model performs particularly well on audio.

The analysis results of a concert video revealed an outdoor scene with several musicians, a singer/guitarist in a blue shirt and white pants actively playing, and an energetic and dramatic atmosphere characterized by bright stage lights.