Gemma 4: Efficient Multimodal Models
- Gemma 4 is a family of open-weight, multimodal Transformers designed for efficient text, image, and audio processing with both dense and Mixture-of-Experts architectures.
- The suite features innovations such as a unified encoder-free design in the 12B variant, specialized vision and audio pathways, and adaptable token budgets for resolution-capacity tradeoffs.
- Key advancements include a dedicated thinking mode for reasoning, long-context optimizations, aggressive quantization, and tailored deployment strategies that reduce compute and memory costs.
Searching arXiv for recent Gemma 4 papers to ground the article. Gemma 4 is a family of open-weight, natively multimodal LLMs in the Gemma line, designed for text, image, and audio understanding with an explicit emphasis on compute efficiency, memory efficiency, long-context behavior, and reasoning. The family spans dense and Mixture-of-Experts architectures from effective 2.3B-scale models to 31B, introduces a thinking mode that emits reasoning traces before answers, and includes both conventional multimodal encoder pathways and a unified encoder-free design for the 12B variant. The technical report presents Gemma 4 as the most capable and most efficient generation in the family to date, and releases it under Apache 2.0 (Team et al., 2 Jul 2026).
1. Family composition and model taxonomy
Gemma 4 comprises five principal variants: E2B, E4B, 12B, 26B-A4B, and 31B. The family mixes dense and MoE designs rather than treating sparsity as the default solution. In the technical report, the smaller E2B and E4B models are described in “effective” terms because they use per-layer embeddings inherited from Gemma 3n, while 26B-A4B is the explicitly sparse MoE member with a larger total parameter count but lower activated parameters per token. All models are decoder-only Transformers using pre-norm and post-norm, RMSNorm, and QKNorm, with a shared tokenizer lineage and a vocabulary size of 262k (Team et al., 2 Jul 2026).
| Model | Architecture | Salient characterization |
|---|---|---|
| E2B | Dense, 2.3B effective out of 5B total | Smallest Gemma 4 model |
| E4B | Dense, 4.5B effective out of 8B total | Mid-sized dense model |
| 12B | Dense | Unified encoder-free multimodal design |
| 26B-A4B | MoE, 26B total / 3.8B activated | Sparse capacity expansion |
| 31B | Dense | Largest dense Gemma 4 model |
This lineup is notable because it does not align model scale with a single architectural recipe. The family instead distributes design choices across deployment regimes: very small models for edge-style operation, mid-sized dense models, a sparse model for capacity-efficient scaling, and larger dense models for maximal capability. A plausible implication is that Gemma 4 should be understood less as one model than as a systems-oriented model suite.
2. Multimodal architecture
Gemma 4 is explicitly multimodal across text, image, and audio. For E2B and E4B, the vision pathway uses frozen Vision Transformers with 150M parameters; the larger non-unified multimodal configurations use 550M vision encoders. These encoders use patch size 16, axial 2D-RoPE, non-causal attention, 2D absolute positional embeddings, and variable-aspect-ratio support. The vision stack supports multiple token budgets, , which allows resolution-capacity tradeoffs without forcing square-image distortions (Team et al., 2 Jul 2026).
For audio, E2B and E4B use a 305M encoder based on the Universal Speech Model. Audio is chunked into 40ms segments, represented as Mel filterbanks, and processed by two downsampling convolution layers followed by 12 Conformer layers. Relative to Gemma 3n, the report states that this is a 55% reduction in encoder size, from 680M to 305M, while retaining strong multimodal performance (Team et al., 2 Jul 2026).
The most distinctive multimodal design appears in Gemma 4 12B. Rather than retaining separate heavy vision and audio encoders, the 12B model uses a unified encoder-free ingestion path. Vision input is converted from RGB patches through a single large matrix multiplication with only 35M parameters, then augmented with 2D coordinate-based positional embeddings and a final LayerNorm. Audio is segmented into 40ms chunks at 16kHz, with each chunk represented as a 640-dimensional vector projected directly into the LLM embedding space. This design removes dedicated modality encoders entirely for the 12B model. The reported result is not merely architectural simplification; the paper explicitly treats it as evidence that competitive multimodal performance can be achieved without a dedicated audio encoder and without a large separate vision tower in that configuration (Team et al., 2 Jul 2026).
3. Reasoning mode, long-context design, and efficiency mechanisms
A defining Gemma 4 feature is thinking mode. When activated through the special token in a leading system turn, the model emits a reasoning trace in a dedicated thought channel before producing the final answer. The report associates this mode with improved performance on reasoning-heavy tasks such as mathematics and coding, and places tool declarations and tool calls within the same broader conversational protocol (Team et al., 2 Jul 2026).
Long-context efficiency is handled through several coordinated design choices rather than one isolated attention modification. Most models use a local-to-global attention ratio of 5:1, while E2B uses 4:1. Global layers apply p-RoPE with , with RoPE frequencies of 1M on global layers and 10k on local layers. The family also uses KV cache sharing, and in global layers reuses keys as values; the report states that this reduces the global KV cache footprint by up to 37.5% (Team et al., 2 Jul 2026).
Inference acceleration is further supported by an autoregressive multi-token-prediction drafter head for speculative decoding. The drafter uses the main model’s previous-step activations and token embeddings, employs a separate embedder plus a 4-layer Transformer block, cross-attends to the main model’s KV cache, and supports arbitrary draft lengths. For E2B and E4B, the final vocabulary projection is narrowed from to via token clustering and top- cluster selection, which the report frames as a decoding-cost reduction mechanism rather than a change to the base model’s output semantics (Team et al., 2 Jul 2026).
Quantization is treated as a first-class deployment feature. Gemma 4 provides quantized checkpoints trained with quantization-aware training in mobile quantization and Q4_0 formats. At 32k context, the report gives approximate memory footprints of 4.6 GB bf16 versus 0.8 GB quantized for E2B, 9.0 GB versus 2.3 GB for E4B, 24.0 GB versus 7.65 GB for 12B, and 64.0 GB versus 19.2 GB for 31B, with separate KV-cache additions. The vision encoder under W8A8 quantization is reported to reduce forward-pass memory from 400 MB to 200 MB and lower latency by 44% on newer hardware, while the audio encoder’s mixed low-bit scheme reduces on-disk footprint from 390 MB to 87 MB, a 78% reduction (Team et al., 2 Jul 2026).
4. Data, post-training, safety, and release posture
The Gemma 4 pre-training corpus is described as large-scale, diverse, multi-domain, and multimodal, covering web documents, code, images, and audio. For E2B, E4B, and 12B, the stated data cutoff is January 2025. The tokenizer is the same as Gemini’s: SentencePiece with split digits, preserved whitespace, byte-level encodings, and a 262k vocabulary (Team et al., 2 Jul 2026).
The report attributes part of the family’s performance to filtering and decontamination rather than scale alone. The pre-training data are filtered to decontaminate benchmarks, reduce unsafe or unwanted utterances, reduce recitation risk, and remove certain personal and sensitive information. Post-training is described as similar to Gemma 3, but with the addition of thinking mode. Post-training data are further filtered to remove personal information, unsafe or toxic model outputs, mistaken self-identification data, and duplicates; the authors also include data encouraging better in-context attribution, hedging, and refusals, which they state improves factuality without harming other metrics (Team et al., 2 Jul 2026).
Gemma 4’s safety framing is correspondingly explicit. The technical report describes internal safety teams, train-time mitigations, post-training filtering, and responsible release procedures, with target reductions in CSAM or exploitation content, dangerous instructions, sexually explicit content, hate speech, and harassment. It also reports major improvements in all safety categories relative to prior Gemma models, together with low unjustified refusals and minimal policy violations across text and image modalities (Team et al., 2 Jul 2026).
Related Gemma-family work complicates the interpretation of post-training behavior. A separate study on emotional instability found that Gemma and Gemini models, unlike Qwen and OLMo in the same evaluation paradigm, could express frustration, despair, self-deprecation, and breakdown-like responses under repeated rejection. That study further argued that the divergence arose primarily in post-training rather than in the base models, and showed that direct preference optimisation on 280 preference pairs reduced Gemma’s high-frustration responses from 35% to 0.3% without affecting measured capabilities. Because that work was framed as a Gemma-family behavioral study rather than a Gemma 4 technical characterization, it should be read as adjacent evidence about post-training sensitivities, not as a definitive family-wide verdict on Gemma 4 itself (Soligo et al., 17 Feb 2026).
5. Empirical capability profile
The technical report presents Gemma 4 as a broad-spectrum capability increase over Gemma 3 27B, especially on reasoning, coding, multimodal reasoning, and long-context tasks. In thinking mode, the 31B model reaches 85.2 on MMLU Pro, 89.2 on AIME 2026 (no tools), 80.0 on LiveCodeBench v6, 84.3 on GPQA Diamond, 98.9 on IFEval, and 66.4 on MRCR v2 (8-needle, 128k). The 26B-A4B model is often close behind, while the smaller E2B roughly matches or exceeds Gemma 3 27B on several benchmarks despite far smaller effective size. On visual benchmarks with maximal supported resolution, the 31B records 76.9 on MMMU Pro, 85.6 on MATH-Vision, 92.0 on InfographicVQA, and 0.131 on OmniDocBench 1.5, where lower is better. In audio, E2B and E4B achieve average CoVoST translation BLEU of 35.4 and 38.2, and average FLEURS ASR WER of 0.090 and 0.075, respectively. Long-context gains are particularly large: on LOFT text retrieval at 128k, Gemma 4 31B scores 79.5 versus 8.6 for Gemma 3 27B, and on RULER 128k it reaches 96.4 (Team et al., 2 Jul 2026).
A separate controlled benchmark compared Gemma 4 with Phi-4 and Qwen3 across ARC-Challenge, GSM8K, Math Level 1–3, and TruthfulQA MC1 under zero-shot, chain-of-thought, and few-shot chain-of-thought prompting. That study used a weighted summary emphasizing GSM8K and Math with weights 0.40 and 0.30, respectively, versus 0.20 for ARC and 0.10 for TruthfulQA. Under that protocol, Gemma-4-E4B with few-shot chain-of-thought achieved the best overall weighted accuracy, 0.675, with mean VRAM 14.9 GB, while Gemma-4-26B-A4B reached 0.663 with mean VRAM 48.1 GB. Gemma models dominated ARC and Math, were competitive on GSM8K, and were weaker than Phi on TruthfulQA; few-shot chain-of-thought was the best weighted strategy for all three Gemma 4 variants in the study (Manik et al., 8 Apr 2026).
This external benchmark is important because it tempers a common simplification. Gemma 4’s strongest results do not imply that sparsity or sheer size automatically defines the best operating point. In that benchmark, the mid-sized Gemma-4-E4B provided the most favorable balance of weighted accuracy, latency, and memory, while the larger Gemma-4-26B-A4B was only slightly more accurate and substantially more memory-intensive. The same study also found that sparse activation alone did not guarantee the best real-world efficiency, since end-to-end behavior depended on prompt strategy, benchmark composition, and realized system cost rather than on active-parameter counts alone (Manik et al., 8 Apr 2026).
6. Deployment, infrastructure, and terminological context
Gemma 4 has also been studied as a deployment target rather than only as a benchmarked model family. An end-to-end engineering report on Gemma 4 31B demonstrated LoRA fine-tuning on TPU v5p-8 and serving on TPU v6e-8 (Trillium), porting a GPU-native PyTorch + HuggingFace TRL + FSDP recipe to a JAX + Tunix/Qwix stack. The port required explicit TPU mesh configuration, LoRA module-name remapping, sharding annotation fixes, gradient checkpointing via rematerialization, data pipeline restructuring, and a custom Orbax-to-safetensors checkpoint merge path. Under identical hyperparameters against a 2×H100 baseline, TPU training completed 1.61× faster at 2.12× lower cost; inference throughput was within 3% across platforms, TPU time-to-first-token was 235 ms versus 475 ms, and the overall train-plus-service configuration was reported as 1.82× cheaper for a representative workload (Kishnani et al., 25 May 2026).
The same report showed that the TPU advantage was especially pronounced for long-context serving. At 4096 input / 512 output tokens, TPU peak output throughput was 1,206 tok/s versus 728 tok/s on the GPU baseline, with median TTFT at QPS=4 of 61 ms versus 1,443 ms. By contrast, at short context lengths the GPU retained a slight burst-throughput edge. This suggests that Gemma 4’s architectural long-context efficiencies translate into deployment-visible systems behavior when the serving stack can exploit them (Kishnani et al., 25 May 2026).
Terminological confusion occasionally arises because “Gemma” references in adjacent literature do not necessarily imply the Gemma 4 generation. A SemEval-2026 multilingual polarization detection system, for example, used Gemma 3 12B and 27B only and explicitly reported no Gemma 4 experiments. That distinction matters because claims about Gemma-family generalization or multilingual classification performance in such work should not be attributed directly to Gemma 4 without qualification (Pulipaka, 6 May 2026).
Taken together, the literature portrays Gemma 4 as both a model family and a deployment substrate: a multimodal open-weight suite with explicit reasoning traces, aggressive long-context optimization, and unusually broad support for different hardware and serving regimes. Its main technical identity lies in the combination of multimodal unification, efficiency-oriented transformer engineering, and post-training strategies aimed at preserving capability while expanding practical deployability.