---
title: 'Gemma: Multi-Domain Transformer Models'
url: https://www.emergentmind.com/topics/gemma
type: topic
---

# Gemma: Multi-Domain Transformer Models

Gemma is a family of open-weight transformer-based models released by Google and DeepMind that serve as foundation models across natural language, multimodal, and scientific domains. The name “Gemma” refers to a lineage of state-of-the-art decoder-only (and, more recently, encoder-decoder–adapted) architectures co-developed with and inspired by the proprietary Gemini series. The Gemma models are designed for high efficiency, modern safety practices, and extensibility, featuring pre-trained and instruction-tuned variants at parameter scales from 2 billion to 27 billion, with Gemma 3 introducing vision, extended context, and new architectural innovations.

## 1. Model Family, Architecture, and Training

The Gemma architecture evolves over three generations:

- **Gemma 1**: Transformer decoder-only, with dense global self-attention; text-only; maximum context length 8,192 tokens; sizes 2B and 7B parameters [2403.08295].
- **Gemma 2**: Integrates interleaved local-global attention (sliding window w=4,096, global span L=8,192), grouped-query attention (GQA, reducing KV head redundancy), and knowledge distillation for 2B and 9B (27B trained with standard next-token objective) [2408.00118]. Key hyperparameters per size:

    | Model      | Layers | $d_{model}$ | FFN dim | Heads / KV | Params  | Vocab          |
    |------------|--------|-------------|---------|------------|---------|----------------|
    | 2B         | 26     | 2304        | 18432   | 8/4        | ~2B     | 256K           |
    | 9B         | 42     | 3584        | 28672   | 16/8       | ~9B     | 256K           |
    | 27B        | 46     | 4608        | 73728   | 32/16      | ~27B    | 256K           |

  The 2B and 9B use distillation from internal Gemma teachers, improving perplexity and robustness for their size.

- **Gemma 3**: Adds vision (via SigLIP), extends maximum context to 128K (by interpolated RoPE and a 5:1 local-to-global attention schedule), and upweights multilingual pre-training [2503.19786]. All models remain decoder-only, except later adaptations as encoder-decoder.

  Distinctive elements for efficiency and long-context support in Gemma 3 include:

    - Sliding-window local attention (w=1,024 for 5/6 layers) plus sparse global full-attention (every 6th layer).
    - RoPE scaling, with $f_{local}=10^4$, $f_{global}=10^6$.
    - Knowledge distillation using sampled teacher softmax output, followed by SFT and RLHF with BOND, WARM, WARP, code and math reward signals.

- Instruction-tuned checkpoints (supervised and RLHF) are provided for all major parameter scales.

## 2. Benchmark Performance and Empirical Results

Gemma models demonstrate strong empirical performance in open and instruction-tuned settings:

- **Academic benchmarks** (text, reasoning, code): Gemma 2 9B achieves average 70.2% (vs. 61.0% for Mistral 7B and 61.9% for LLaMA 3 8B) across tasks such as MMLU, ARC, GSM8K, BBH, HellaSwag [2408.00118].
- **Zero-/few-shot code completion**: CodeGemma, built atop Gemma 2B/7B, matches or exceeds StarCoder2 and DeepSeek in HumanEval/MBPP and completes code in 1.8–3× less time at similar quality for infill tasks [2406.11409].
- **Fine-tuned sentiment and text-to-SQL**: Gemma-7B fine-tuned on FinancialPhraseBank attains F1=0.876 (vs. 0.872/0.861 for Llama/Phi-3) [2406.13626]. GEMMA-SQL (2B) achieves TS=66.8% and EM=63.3% on SPIDER, outperforming IRNet and matching CodeX-DaVinci on exact match [2511.04710].
- **Multimodal**: LLaVA-Gemma2B, with DINOv2 vision, achieves VQAv2 accuracy 71.4 and GQA 0.587 (on par with Phi-2B, though LLaVA-Llama2-7B outperforms at 78.5/0.62) [2404.01331]. Gemma 3-27B-IT, in LMSYS Arena, scores Elo 1338, competitive with Gemini-1.5-Pro.

## 3. Adaptation, Fine-Tuning, and Extensibility

Gemma supports diverse downstream adaptation pipelines:

- **Parameter-efficient fine-tuning (PEFT)**: LoRA (rank=16) adapters in self-attention/FFN matrices permit rapid, low-resource updates, especially for domain/language adaptation. Merging at 4-bit weights with full precision delta is preferred to minimize rounding loss [2503.13988].
- **Multilingual adaptation**: Branch-and-Merge continual pretraining of Gemma-2-9B/27B on large Bulgarian-English corpora (BgGPT) raised Bulgarian task performance to 61.3%/64.7% (9B/27B), with no drop in English [2412.10893].
- **Modular reuse**: Frozen mid-layers from Gemma 3, coupled with custom input adapters, yield efficient predictors for out-of-domain tasks (e.g., tabular wildfire prediction; Gemma internal blocks provide a robust “internal world” for environmental forecasting, with high recall and interpretable attention maps) [2504.18562].
- **Encoder-decoder adaptation**: Systematic conversion (via PrefixLM, UL2) of decoder-only Gemma 2B/9B yields encoder-decoder LLMs matching original pretrain quality and outperforming on downstream tuning (e.g., +7% instruction-tune, +12.6 points SuperGLUE at 2B scale) [2504.06225].

## 4. Safety, Hallucination, and Interpretability

Gemma’s safety, hallucination, and interpretability profile is well studied:

- **Safety**: On red-teaming datasets for factuality, bias, and toxicity, Gemma-2B/7B IT models behave conservatively—refusing potentially unsafe tasks more often than Llama2, but trailing in instruction adherence and multi-turn robustness (toxicity drops to 0.03–0.14 in 4-turn dialogs) [2404.09785].
- **Hallucination**: Systematic symbolic input triggers (modifiers, named entities, numbers) elicit high hallucination rates (>80%) in Gemma-2 2B, dropping only to ~64% at 27B scale—a fundamental encoding limitation not yet solved by scaling alone [2509.09715].
- **Interpretability tools**: Gemma Scope provides >2,000 JumpReLU sparse autoencoders (SAEs) covering every layer and sub-layer of Gemma 2B/9B and parts of 27B, supporting supervised and unsupervised circuit analysis and feature steering [2408.05147]. These tools are fully open source and compatible with research libraries such as HuggingFace Transformers and TransformerLens.

## 5. Multimodal and Academic Writing Applications

Gemma’s deployment includes:

- **Vision-language**: Gemma 3 inherits SigLIP vision backbone and can handle ≥128K-token context multimodal prompts, with instruction-tuned models matching previous SOTA on math, code, chat, and vision.
- **Academic writing**: Gemma 27B yields mid-range output lengths, high semantic fidelity (~92–98% overlap with source), and low paraphrase plagiarism (11%), but is flagged by AI detectors (≥95%) and has dense, low-readability prose—manipulable by further style adaptation [2503.04765].
- **Prompt recovery**: Fused models (Gemma-2B-IT + Phi2) excel at reconstructing prompts in text rewriting tasks (SCS=0.61 > all baselines), leveraging dual-stage pretraining and contextual fusion to bolster semantic alignment [2407.05233].

## 6. Scientific and Physical Experiments: GEMMA and Neutrino Physics

The acronym GEMMA also denotes the Germanium Experiment for the Measurement of the Magnetic Moment of Antineutrino, based at Kalinin NPP, which sets world-leading direct laboratory bounds on neutrino electromagnetic properties [1005.2736, 1411.2279]:

- Using a 1.5 kg HPGe detector and low threshold ($T_\text{min} \approx 2.8$ keV), GEMMA obtains upper bounds:
    - Neutrino anomalous magnetic moment: $\mu_\nu < 3.2\times 10^{-11}\mu_B$ (without atomic ionization), improved to $5.0\times 10^{-12}\mu_B$ (including atomic ionization).
    - Direct neutrino electric millicharge: $|q_\nu| < 2.7 \times 10^{-12}\,e_0$ (90% CL)—tightest reactor-based limit to date [1411.2279].
- Future GEMMA-II/III upgrades aspire to reach $|q_\nu| \sim 5.5 \times 10^{-13}\,e_0$ and $\mu_\nu \lesssim 1\times 10^{-12}\mu_B$, placing further constraints on beyond-Standard-Model physics.

## 7. Scientific Applications: Asteroseismology

In astrophysics, “Gemma” refers to the young subgiant KIC 11026764, whose oscillation modes measured by Kepler underwrite benchmark models for stellar evolution [2502.19900]:

- 45 oscillation modes, including g-dominated mixed modes, are fit using the EGGMiMoSA tool to resolve core chemical gradients.
- Best-fit models for Gemma require convective overshooting parameter $\alpha_{\text{ov}}=0.17$, and achieve mass $M=1.20^{+0.07}_{-0.04}M_\odot$, age $5.75$ Gyr, radius $2.08 R_\odot$.
- Gemma thus serves as a reference point for calibrating convective core mixing in solar-like subgiants.

---

Gemma models combine architectural efficiency, broad open-source release, safety-aware pre-training, adaptability to language and domain specialization, and coverage of both scientific and engineering benchmarks. In both machine learning and physical sciences, “Gemma” denotes state-of-the-art instrumentation and methodology, with ongoing extensions toward better multimodal understanding, efficient adaptation, enhanced interpretability, and deeper participation in foundational physics and astrophysical measurement.

Source: https://www.emergentmind.com/topics/gemma