---
title: 'Gemma 3n LLM: Efficient, Multilingual Transformer'
url: https://www.emergentmind.com/topics/gemma-3n-llm
type: topic
---

# Gemma 3n LLM: Efficient, Multilingual Transformer

Gemma 3n refers to the Gemma 3 series of large language models (LLMs), notably including the Gemma 3–4B and Gemma 3–27B variants, characterized by efficient open-source decoder‐only Transformer architectures, multilingual and multimodal capabilities, and innovations in memory, alignment, and training protocols. These models have demonstrated state-of-the-art performance in both English and low-resource dialects such as Moroccan Arabic (Darija), while enabling scalable and energy-efficient tuning for domain-specific tasks [2505.17082] [2503.19786].

## 1. Architectural Foundations

Gemma 3 models maintain a decoder‐only Transformer backbone with Grouped‐Query Attention and RMSNorm. A distinguishing innovation is the interleaved local/global attention layer scheme: for every global self-attention layer, five local self-attention layers are interposed, with local layers attending to a sliding window of $S=1024$ tokens. Global attention layers compute full-sequence attention over length $N$, while local layers restrict attention to the most recent $S$ tokens,
\[
\mathrm{Attn}_\mathrm{local}(Q,K,V)_i =\sum_{j=i-S+1}^i \mathrm{softmax}\!\bigl(Q_i K_j^\top/\sqrt{d}\bigr)\,V_j\,.
\]
Consequently, the total KV-cache memory per token reduces from $M_\text{global only}\propto N\times d\times L$ to $M_\text{Gemma3}\propto N d L_g + S d L_\ell$, with $L_g=L/6$ global and $L_\ell=5L/6$ local layers. Empirical analysis shows KV-cache overhead at $N=32$k drops from $\sim$60% (Gemma 2) to under 15% in Gemma 3, with negligible impact on perplexity [2503.19786].

## 2. Model Variant Specifications

Gemma 3 is released in several parameter scales: 1 B, 4 B, 12 B, and 27 B, supporting context windows up to 128,000 tokens via RoPE positional-embedding rescaling (except 1 B, limited to 32 K). The sequence rescaling on global layers applies a factor $f=4$ so tokens beyond 32 K receive valid positional signals. Both Gemma 3–4B and Gemma 3–27B variants adopt BF16 precision and a SentencePiece tokenizer with 262,000 entries. These models display strong zero-shot capabilities across languages, mathematics, scientific, and commonsense reasoning [2503.19786].

## 3. Training and Instruction Tuning Protocols

Pretraining follows a distilled student protocol: mixing text and images (2 T–14 T tokens, increasing with size), quality re-weighting, and safety filtering. Distillation samples $m=256$ logits per token from the teacher distribution $p_T$, zeros unsampled logits, renormalizes, and minimizes the cross-entropy loss,
\[
\mathcal{L}_\text{distill} = -\sum_{\ell\in S} p_T(\ell)\,\log p_S(\ell)\,.
\]
Post-training involves instruction tuning ("Gemma3-IT") via supervised distillation (best-of-$N$ method, BOND) and RL fine-tuning with reward models (WARM, WARP) focused on factuality, helpfulness, code execution, mathematical correctness, multilinguality, and safety,
\[
\mathcal{L}_\mathrm{RL} =-\mathbb{E}_{y\sim\pi_\theta(\cdot|x)}[R(x,y)] \approx -\frac{1}{K}\sum_{k=1}^K R(x,y_k)\log\pi_\theta(y_k|x).
\]
Quantization-aware training (QAT) yields per-channel int4 and per-block int4 (block=32) checkpoints, with switched fp8 supported after 5,000 QAT steps [2503.19786].

## 4. Data Alignment and Darija Instruction Tuning

A quality-over-quantity alignment strategy surfaces latent proficiency in Darija, a marginalized Moroccan Arabic dialect. The pipeline translates and filters three prominent instruction suites—LIMA 1K, DEITA 6K, and TULU 50K—into Arabic-script Darija using the Gemini 2.0 Flash API, prompt engineering, and code block/LaTeX preservation. For cross-lingual robustness and to prevent catastrophic forgetting, 20% of each suite's data is retained in English.

| Suite     | Total Samples | Darija (≈ %) | English (≈ %) |
|-----------|--------------|--------------|---------------|
| LIMA 1K   | 1,000        | 700 (70%)    | 300 (30%)     |
| DEITA 6K  | 5,000        | 3,700 (74%)  | 1,300 (26%)   |
| TULU 50K  | 46,000       | 33,000 (72%) | 13,000 (28%)  |

Adopting parameter-efficient LoRA adapters, the 4B model employs LoRA rank $r=32$, $\alpha=64$, while the 27B model uses $r=16$, $\alpha=32$. Training proceeds on mixed Darija/English data, with 15 epochs (LIMA), 6 (DEITA), 3 (TULU), and respective learning rates ($4\times10^{-4}$ for LIMA/DEITA, $1\times10^{-4}$ for TULU). Compute remains minimal: $<100$ GPU·h and $<\$100$ cloud cost [2505.17082].

## 5. Quantitative Performance and Scaling Effects

Gemma 3 models demonstrate substantial gains in Darija proficiency post LoRA tuning:

| Model/Benchmark                  | Size | DarijaMMLU | HellaSwag | GSM8K@5 | MMLU |
|-----------------------------------|------|------------|-----------|---------|------|
| Gemma 3–4B (untuned)              | 4B   | 32.8%      | 36.3%     | 74.8%   | 51.1% |
| +LoRA LIMA 1K                     | 4B   | 34.9%      | 39.3%     | 51.2%   | 29.3% |
| +LoRA DEITA 6K                    | 4B   | 42.7%      | 44.3%     | 53.2%   | 51.4% |
| +LoRA TULU 50K                    | 4B   | 47.5%      | 47.1%     | 56.0%   | 54.1% |

| Model/Benchmark                  | Size | DarijaMMLU | HellaSwag | GSM8K@5 | MMLU |
|-----------------------------------|------|------------|-----------|---------|------|
| Atlas-Chat-27B                    | 27B  | 61.9%      | 48.4%     | 82.0%   | 72.1% |
| GemMaroc-27B (LoRA TULU 50K)      | 27B  | 61.6%      | 60.5%     | 84.2%   | 73.6% |

Post-alignment, the 4B model achieves a $\Delta$DarijaMMLU of $+14.73$ pp and $\Delta$HellaSwag of $+10.83$ pp; the 27B model matches or exceeds prior state-of-the-art on DarijaMMLU and shows $+12.1$ pp over Atlas-Chat-27B on Darija HellaSwag. Cross-lingual and mathematical benchmark retention is also achieved (GSM8K moves from 82.0% to 84.2% on 27B). Performance scaling from 4B to 27B delivers a 1.3$\times$ gain on DarijaMMLU and 1.28$\times$ on HellaSwag, with negligible English regression [2505.17082].

## 6. Multimodal and Multilingual Capabilities

Gemma 3 integrates vision-language support through a frozen 400 M-parameter SigLIP Vision Transformer, encoding images into 16$\times$16 patch embeddings, average-pooled to 256 soft tokens. Inference uses Pan-and-Scan tiling for artifact-free high-resolution image handling, yielding up to +17 points on document VQA tasks, and achieving 85.6 CIDEr (COCO Caption) and 59.4 ANLS (InfoVQA) for the 4B model. Multilingual performance is enhanced by UniMax-inspired data mixing, with the 27B model attaining 75.7% GMMLU, 76.8 F1 XQuAD, and outperforming Gemma 2 [2503.19786].

## 7. Energy Efficiency and Green AI Implications

Fine-tuning Gemma 3 with LoRA adapters and curated Darija instruction suites requires only 58 GPU·h (10 GPU·h on A100 for 4B, 48 GPU·h on H100 for 27B), resulting in total energy consumption of 32 kWh ($\sim$13 kg CO$_2$e). This contrasts sharply with the Atlas-Chat-27B full fine-tune at 1.4 MWh ($>$610 kg CO$_2$e), representing over a 48$\times$ reduction in energy and 98% lower emissions. The recipe demonstrates a Green AI pathway: inclusive, sustainable, low-resource dialect tuning without sacrificing performance [2505.17082].

## 8. Practical Deployment and Model Release

All bfloat16 and quantized checkpoints are released under an open license. Quantized weights and KV-cache at 32 K context reduce the memory footprint to 1.4–7.3 GB. Code, model cards, and formatting scripts are available to facilitate further research, educational, public service, and everyday digital applications centered on dialect inclusivity and computational efficiency [2503.19786][2505.17082].

Source: https://www.emergentmind.com/topics/gemma-3n-llm