Gemma 3 Technical Report (2503.19786v1)

Abstract: We introduce Gemma 3, a multimodal addition to the Gemma family of lightweight open models, ranging in scale from 1 to 27 billion parameters. This version introduces vision understanding abilities, a wider coverage of languages and longer context - at least 128K tokens. We also change the architecture of the model to reduce the KV-cache memory that tends to explode with long context. This is achieved by increasing the ratio of local to global attention layers, and keeping the span on local attention short. The Gemma 3 models are trained with distillation and achieve superior performance to Gemma 2 for both pre-trained and instruction finetuned versions. In particular, our novel post-training recipe significantly improves the math, chat, instruction-following and multilingual abilities, making Gemma3-4B-IT competitive with Gemma2-27B-IT and Gemma3-27B-IT comparable to Gemini-1.5-Pro across benchmarks. We release all our models to the community.

Gemma 3 represents an evolution of the Gemma family of open models, introducing multimodal capabilities, expanded language coverage, and significantly extended context length support, while optimizing for reduced memory usage during inference (Team et al., 25 Mar 2025 ). The family includes models ranging from 1 billion to 27 billion parameters.

Architectural Enhancements

Gemma 3 incorporates several architectural modifications compared to its predecessor, Gemma 2, primarily aimed at enabling vision understanding and efficiently handling long sequence lengths.

Vision Integration: The models are now multimodal, capable of processing visual input alongside text. While the specific vision encoder architecture isn't detailed in the abstract, it typically involves integrating representations from a pre-trained vision model (e.g., a Vision Transformer) with the LLM's embeddings, often via cross-attention mechanisms or by projecting visual features into the LLM's input space.

KV-Cache Optimization for Long Context: A significant challenge with transformer models operating on long sequences (Gemma 3 supports at least 128K tokens) is the quadratic growth of the Key-Value (KV) cache size associated with standard self-attention mechanisms. To mitigate this, Gemma 3 employs a modified attention strategy:

• Increased Ratio of Local to Global Attention: The architecture increases the proportion of attention layers that operate locally (e.g., using sliding window attention) relative to those performing global attention across the entire sequence. This drastically reduces the number of key-value pairs that need to be stored for most layers.
• Short Span Local Attention: The local attention mechanism utilizes a constrained window or span. This means each token only attends to a limited number of preceding (and potentially succeeding) tokens, rather than the full context.

This hybrid attention approach aims to balance computational/memory efficiency with the model's ability to capture long-range dependencies. The layers with global attention ensure that information can still propagate across the entire sequence length, albeit less frequently than in a standard transformer. The reduction in KV-cache size is crucial for deploying these models in memory-constrained environments or for applications requiring very long context processing without prohibitive hardware costs. For a sequence length $L$ and hidden dimension $d$, the standard attention KV-cache is $O(L \times d)$ per layer per head. By using predominantly local attention with span $w << L$, the cache size per layer is significantly reduced, closer to $O(w \times d)$, plus the cache for the fewer global layers.

Training and Fine-tuning Strategy

Gemma 3's training involves both pre-training and post-training phases, leveraging distillation and a refined instruction-tuning recipe.

Pre-training with Distillation: The models are pre-trained using distillation, likely transferring knowledge from a larger, more capable teacher model (potentially a member of the Gemini family). Distillation can involve matching output distributions (token-level) or intermediate representations between the teacher and student (Gemma 3) models. This approach allows the smaller Gemma 3 models to benefit from the knowledge learned by the larger teacher during its own extensive pre-training, often leading to improved performance compared to training from scratch on the same data budget. The pre-training data incorporates multimodal and multilingual sources to equip the base models with the foundational capabilities.

Instruction Fine-tuning: A key contribution highlighted is a "novel post-training recipe" designed to significantly enhance specific capabilities. This instruction-tuning phase focuses on improving performance in:

• Mathematics: Enhancing logical reasoning and numerical computation abilities.
• Chat: Improving conversational flow, coherence, and adherence to user instructions in multi-turn dialogues.
• Instruction Following: Increasing the model's reliability in executing complex instructions accurately.
• Multilingualism: Broadening and deepening the model's understanding and generation capabilities across various languages.

This fine-tuning process likely involves curated datasets tailored to these specific domains and capabilities, potentially using techniques like Supervised Fine-Tuning (SFT) on high-quality instruction-response pairs, possibly augmented with reinforcement learning methods (e.g., RLHF or DPO) to further align model outputs with desired characteristics like helpfulness and safety.

Performance Evaluation

The technical report emphasizes significant performance gains achieved by Gemma 3 over Gemma 2, particularly for the instruction-tuned variants.

• Gemma3-4B-IT vs. Gemma2-27B-IT: The 4 billion parameter instruction-tuned Gemma 3 model is reported to be competitive with the much larger 27 billion parameter instruction-tuned Gemma 2 model across various benchmarks. This demonstrates the effectiveness of the architectural changes, distillation, and the enhanced post-training recipe in improving parameter efficiency.
• Gemma3-27B-IT vs. Gemini-1.5-Pro: The largest instruction-tuned model, Gemma3-27B-IT, is claimed to achieve performance comparable to Gemini 1.5 Pro on evaluated benchmarks. This positions Gemma 3 27B as a strong open model alternative to leading proprietary models, particularly considering its capabilities in long context, multimodality, and multilingualism.
• Specific Capabilities: The improvements are noted across math, chat, instruction-following, and multilingual tasks, reflecting the success of the targeted post-training strategy. Evaluation would typically involve standard academic benchmarks like MMLU, GSM8K, HumanEval, MT-Bench, and multilingual benchmarks like TyDi QA or Flores. Vision capabilities would be assessed using benchmarks like VQAv2, GQA, or TextVQA.

Implementation Considerations

The architectural choices in Gemma 3 have direct implications for practical deployment:

• Reduced Inference Memory: The primary benefit of the local/global attention mix is reduced GPU memory consumption during inference, especially critical when processing sequences close to the 128K token limit. This makes deploying Gemma 3 feasible on hardware with more modest VRAM capacities compared to models using standard attention over similar context lengths.
• Hardware Requirements: While memory is reduced compared to standard attention, running inference with 128K context still requires substantial compute and memory resources, particularly for the 27B parameter variant. Quantization techniques (e.g., 4-bit or 8-bit) would likely be necessary for deployment on consumer-grade hardware or edge devices, even for the smaller models.
• Trade-offs of Local Attention: The reliance on local attention might introduce limitations in tasks requiring fine-grained understanding of very long-range dependencies scattered across the entire context, although the inclusion of global attention layers aims to mitigate this. Performance on "needle-in-a-haystack" retrieval tasks over the full context length would be an important evaluation point.
• Model Availability: The release includes models at various scales (1B, 4B, 27B). This allows practitioners to choose a model size that balances performance requirements with available computational resources. The smaller models are potentially suitable for on-device or edge applications, while the larger models target more demanding tasks requiring higher capacity.
• Multimodality: Implementing multimodal applications requires handling both image preprocessing (feeding inputs to the vision encoder) and integrating the combined text/image processing pipeline. Inference endpoints need to support multimodal inputs.

Conclusion

Gemma 3 introduces significant advancements over Gemma 2, incorporating multimodality and substantially longer context handling (128K tokens) while mitigating the associated memory costs through architectural changes focused on hybrid local/global attention. The use of distillation and a refined instruction-tuning process yields notable performance improvements, making the smaller Gemma3-4B-IT competitive with the previous generation's largest model, and positioning the Gemma3-27B-IT as comparable to leading closed models like Gemini 1.5 Pro on several benchmarks. The release of these models provides powerful open-source tools for applications requiring advanced reasoning, multilingual support, vision understanding, and long-context processing.