Gemma-3-4B: Multimodal Transformer Model

• Gemma-3-4B is a mid-sized, open-source multimodal language model featuring a 4.3B parameter decoder-only Transformer architecture that integrates vision encoding for image-text reasoning.
• It employs innovative local and global attention mechanisms with efficient memory management to handle up to 128K-token contexts using a two-phase training strategy.
• Optimized with parameter-efficient fine-tuning (LoRA & QLoRA) and post-training objectives, it achieves competitive benchmarks in domains such as natural language understanding, medical imaging, and low-resource language tasks.

Gemma-3-4B is a mid-sized, open-source multimodal LLM in the Gemma 3 family. It integrates architectural innovations for efficient long-context processing, provides a vision encoder for image-text reasoning, and is optimized for broad multilingual coverage and practical deployment on consumer hardware. Developed from Google’s Gemini project, Gemma-3-4B is intended for both research and applied use, supporting up to 128,000-token contexts and advanced instruction-following via a two-phase post-training scheme with knowledge distillation and reinforcement learning. Its flexible quantization and memory-efficient attention configuration enable scalable inference and fine-tuning across resource-constrained environments, with strong benchmarks in natural language understanding, mathematical reasoning, and domain-specific applications such as medical imaging and low-resource language tasks (Team et al., 25 Mar 2025, Sellergren et al., 7 Jul 2025, Islam et al., 19 Oct 2025).

1. Model Architecture and Multimodal Extensions

Gemma-3-4B employs a decoder-only, autoregressive Transformer backbone at approximately 4.3 billion parameters. The configuration is as follows:

• 32 transformer layers ($L=32$) with hidden dimension $d=2560$ and feed-forward dimension $d_{ff}=10240$.
• 32 attention heads ($h=32$), $d_h = 80$ per head.
• Rotational positional encoding (RoPE), SwiGLU feed-forward activations, and RMSNorm are utilized throughout (Sellergren et al., 7 Jul 2025, Team et al., 25 Mar 2025).

A distinctive aspect is its multimodal capability. An integrated SigLIP vision encoder (400M parameters, or, in MedGemma variants, MedSigLIP) processes images into 256 “soft” tokens, which are then concatenated into the text stream. Input images can be up to $896 \times 896$ pixels, and the architecture enables arbitrary interleaving of image and text tokens. Visual and textual modalities can be jointly modeled for tasks like medical visual question answering, classification, and report generation (Sellergren et al., 7 Jul 2025).

Architectural innovations include grouped-query attention (GQA) with explicit QK-norm, and a 5:1 pattern of local to global self-attention layers. Local “sliding-window” attention spans 1024 tokens; global layers process the full context. Only global layers store extended KV caches, minimizing memory growth with context length (Team et al., 25 Mar 2025).

2. Context Length Scaling and Memory Efficiency

Gemma-3-4B is engineered for genuine long-context capacity. The model is trained on 32K-token sequences with RoPE at a 10K base frequency. At the conclusion of pre-training, rotary encodings on the global attention layers are rescaled by a factor of 8 (10K → 1M), enabling 128K-token inference without architectural change. Local attention layers persist at the original 10K frequency. This strategy, combined with selective KV cache management, allows Gemma-3-4B to operate within 8–12 GB of memory even for maximal context (Team et al., 25 Mar 2025).

The KV cache memory for context length $T$ is:

$$M \approx G \times H \times d \times T + L \times H \times d \times W$$

where $G$ is the number of global layers, $L$ is local, $H$ is heads, $d$ is hidden dimension per head, and $W=1024$ is the local span. This partitioning keeps memory growth essentially sublinear beyond the local context, sustaining tractability for both training and inference (Team et al., 25 Mar 2025).

3. Pretraining, Distillation, and Post-Training Objectives

Core training is conducted on 4 trillion tokens across 50+ languages (balanced monolingual and parallel corpora) and image–text pairs. Image tokens are injected in the text stream at various ratios, domain-adaptive for applications such as medical imaging in MedGemma. The primary objective is next-token prediction. Distillation from a larger, instruction-tuned multimodal teacher model is leveraged: 256 soft-target logits per token are stratified, with non-selected logits masked, and cross-entropy minimized over the sampled subset (Team et al., 25 Mar 2025, Sellergren et al., 7 Jul 2025).

After pre-training, Gemma-3-4B undergoes a two-phase post-training strategy:

1. Supervised fine-tuning (SFT) with filtered data and teacher distillation; hazardous or noisy samples are excised with additional data hygiene for sensitive domains.
2. Reinforcement learning with blended reward heads—helpfulness, best-of-N policy (BOND), WARP-style weighted averaging, programmatic code execution, and math ground truth—adapting for domains including math, QA, and medical reasoning (Team et al., 25 Mar 2025, Sellergren et al., 7 Jul 2025).

4. Parameter-Efficient Fine-Tuning (PEFT): LoRA & QLoRA

For efficient adaptation to downstream tasks, Gemma-3-4B supports PEFT via LoRA and QLoRA:

• LoRA injects low-rank matrices ($$A \in \mathbb{R}^{r \times d}, B \in \mathbb{R}^{d \times r}$$, $r=16$) into each transformer layer, updating only adapter weights and freezing the original $W$. The update rule:

$$W' = W + \Delta W, \quad \Delta W = \alpha r^{-1} B A, \quad \alpha=16$$

• QLoRA applies 4-bit NF4 quantization to frozen $W$, backpropagating gradients solely into $A$ and $B$.
• Hyperparameters (for PEFT on BD-SHS): learning rate $2 \times 10^{-4}$, batch size 32, max sequence length 2048, 3 epochs, gradient accumulation (Islam et al., 19 Oct 2025).

Parameter-efficient adaptation enables updating $\leq 1\%$ of weights (29.8M of 4.24B, or 0.69% for Gemma-3-4B), allowing experiments on a single consumer GPU.

5. Benchmark Evaluation and Comparative Analysis

Quantitatively, Gemma-3-4B demonstrates strong performance for its scale across both general and domain-specific benchmarks:

• On instruction-tuned natural language tasks, Gemma-3-4B achieves: MMLU-Pro zero-shot 43.6%, MATH 75.6% (4-shot CoT), GPQA 30.8%, FACTS 70.1%, LiveCodeBench 12.6%, and MMMU 48.8%. In many areas, it surpasses Gemma 2’s 9B or even 27B predecessors, despite its much smaller footprint (Team et al., 25 Mar 2025).
• MedGemma-4B (Gemma-3-4B + MedSigLIP + medical post-training) achieves substantial absolute and relative gains over baseline Gemma 3 4B and other generalist models on medical QA (e.g., MedQA 64.4%), chest X-ray classification (MIMIC-CXR macro F1 88.9%), VQA, and electronic health record retrieval (Sellergren et al., 7 Jul 2025).

In low-resource language moderation (Bengali hate speech detection, BD-SHS dataset), Gemma-3-4B with QLoRA+LoRA PEFT achieves weighted F1 of 80.25% on the test set, updating only 0.69% of parameters, and requiring 15.6GB of VRAM at peak—offering substantial efficiency versus full fine-tuning (Islam et al., 19 Oct 2025).

<table> <thead> <tr> <th>Model</th> <th>F1 on BD-SHS</th> <th>Trainable Param. (%)</th> </tr> </thead> <tbody> <tr> <td>Gemma-3-4B</td> <td\>80.25%</td> <td\>0.69%</td> </tr> <tr> <td>Llama-3.2-3B</td> <td\>92.23%</td> <td\>0.75%</td> </tr> <tr> <td>Mistral-7B</td> <td\>88.94%</td> <td\>0.58%</td> </tr> </tbody> </table>

On medical and technical tasks, Gemma-3-4B exhibits competitive accuracy, approaching or exceeding prior state-of-the-art for comparable parameter and compute budgets (Sellergren et al., 7 Jul 2025).

6. Deployment Characteristics and Limitations

Gemma-3-4B is architected for efficient deployment:

• Quantization and hardware compatibility: INT4 quantized weights reduce the footprint to 2.6–2.9GB, enabling full 128K context on 8–12GB VRAM, even with extended KV caches. Single-token inference latency is below 100ms on commodity GPUs or TPUv4/v5 (Team et al., 25 Mar 2025, Sellergren et al., 7 Jul 2025).
• PEFT and sub-day fine-tuning: Supports practical, low-cost adaptation for applications in languages and domains with constrained resources.
• Integration in regulated domains: The model can be frozen for clinical or safety-critical settings, or adapted further by SFT/RL for institutional or site-specific style (Sellergren et al., 7 Jul 2025).

Limitations include:

• The 1B Gemma variant supports only 32K context.
• Perplexity increases substantially beyond 128K context, marking this as a practical upper bound.
• Despite robust pre- and post-training safety filters, output risk remains nonzero; application-level controls are necessary (Team et al., 25 Mar 2025, Sellergren et al., 7 Jul 2025).
• For languages with complex code-mixing or non-standard orthography, Gemma-3-4B may display lower recall or robustness relative to more multilingual-optimized models (Islam et al., 19 Oct 2025).

7. Domain-Specific Adaptation and Impact

Gemma-3-4B supports extensive domain adaptation via fine-tuning or PEFT:

• In medical applications (MedGemma), post-training on curated QA and imaging datasets yields double-digit accuracy improvements in QA and classification, with fine-tuning further reducing diagnostic error (e.g., EHRQA error halved after RL-based tuning) (Sellergren et al., 7 Jul 2025).
• In moderation and low-resource applications (e.g., Bengali hate speech), PEFT adaptation with LoRA+QLoRA demonstrates practical and replicable improvement, even when annotation resources and compute are limited (Islam et al., 19 Oct 2025).

A plausible implication is that Gemma-3-4B’s scalable memory footprint and fine-tuning efficiency catalyze development of NLP, vision-language, and moderation systems in domains and languages previously underserved by large models. The model’s architectural choices—particularly local/global attention partitioning and quantization-ready design—provide a template for balanced performance, compute, and memory efficiency in future model architecture development.

---

References:

• "Parameter-Efficient Fine-Tuning for Low-Resource Languages: A Comparative Study of LLMs for Bengali Hate Speech Detection" (Islam et al., 19 Oct 2025)
• "Gemma 3 Technical Report" (Team et al., 25 Mar 2025)
• "MedGemma Technical Report" (Sellergren et al., 7 Jul 2025)