Gemma: Multi-Domain Transformer Models

Updated 26 February 2026

Gemma is a family of transformer-based foundation models designed for diverse applications in natural language processing, multimodal tasks, and scientific experiments.
The architecture evolves over three generations with innovations such as interleaved local-global attention, vision integration via SigLIP, and extended context support up to 128K tokens.
Gemma models demonstrate strong benchmark performance across academic tasks, code completion, and physical experiments, while emphasizing safety, interpretability, and efficient adaptation.

Gemma is a family of open-weight transformer-based models released by Google and DeepMind that serve as foundation models across natural language, multimodal, and scientific domains. The name “Gemma” refers to a lineage of state-of-the-art decoder-only (and, more recently, encoder-decoder–adapted) architectures co-developed with and inspired by the proprietary Gemini series. The Gemma models are designed for high efficiency, modern safety practices, and extensibility, featuring pre-trained and instruction-tuned variants at parameter scales from 2 billion to 27 billion, with Gemma 3 introducing vision, extended context, and new architectural innovations.

1. Model Family, Architecture, and Training

The Gemma architecture evolves over three generations:

Gemma 1: Transformer decoder-only, with dense global self-attention; text-only; maximum context length 8,192 tokens; sizes 2B and 7B parameters (Team et al., 2024).
Gemma 2: Integrates interleaved local-global attention (sliding window w=4,096, global span L=8,192), grouped-query attention (GQA, reducing KV head redundancy), and knowledge distillation for 2B and 9B (27B trained with standard next-token objective) (Team et al., 2024). Key hyperparameters per size:

Model Layers $d_{model}$ FFN dim Heads / KV Params Vocab

2B 26 2304 18432 8/4 ~2B 256K

9B 42 3584 28672 16/8 ~9B 256K

27B 46 4608 73728 32/16 ~27B 256K

Model	Layers	$d_{model}$	FFN dim	Heads / KV	Params	Vocab
2B	26	2304	18432	8/4	~2B	256K
9B	42	3584	28672	16/8	~9B	256K
27B	46	4608	73728	32/16	~27B	256K

The 2B and 9B use distillation from internal Gemma teachers, improving perplexity and robustness for their size.

Gemma 3: Adds vision (via SigLIP), extends maximum context to 128K (by interpolated RoPE and a 5:1 local-to-global attention schedule), and upweights multilingual pre-training (Team et al., 25 Mar 2025). All models remain decoder-only, except later adaptations as encoder-decoder.

Distinctive elements for efficiency and long-context support in Gemma 3 include:

- Sliding-window local attention (w=1,024 for 5/6 layers) plus sparse global full-attention (every 6th layer). - RoPE scaling, with $f_{local}=10^4$ , $f_{global}=10^6$ . - Knowledge distillation using sampled teacher softmax output, followed by SFT and RLHF with BOND, WARM, WARP, code and math reward signals.

Instruction-tuned checkpoints (supervised and RLHF) are provided for all major parameter scales.

2. Benchmark Performance and Empirical Results

Gemma models demonstrate strong empirical performance in open and instruction-tuned settings:

Academic benchmarks (text, reasoning, code): Gemma 2 9B achieves average 70.2% (vs. 61.0% for Mistral 7B and 61.9% for LLaMA 3 8B) across tasks such as MMLU, ARC, GSM8K, BBH, HellaSwag (Team et al., 2024).
Zero-/few-shot code completion: CodeGemma, built atop Gemma 2B/7B, matches or exceeds StarCoder2 and DeepSeek in HumanEval/MBPP and completes code in 1.8–3× less time at similar quality for infill tasks (Team et al., 2024).
Fine-tuned sentiment and text-to-SQL: Gemma-7B fine-tuned on FinancialPhraseBank attains F1=0.876 (vs. 0.872/0.861 for Llama/Phi-3) (Mo et al., 2024). GEMMA-SQL (2B) achieves TS=66.8% and EM=63.3% on SPIDER, outperforming IRNet and matching CodeX-DaVinci on exact match (Pandey et al., 5 Nov 2025).
Multimodal: LLaVA-Gemma2B, with DINOv2 vision, achieves VQAv2 accuracy 71.4 and GQA 0.587 (on par with Phi-2B, though LLaVA-Llama2-7B outperforms at 78.5/0.62) (Hinck et al., 2024). Gemma 3-27B-IT, in LMSYS Arena, scores Elo 1338, competitive with Gemini-1.5-Pro.

3. Adaptation, Fine-Tuning, and Extensibility

Gemma supports diverse downstream adaptation pipelines:

Parameter-efficient fine-tuning (PEFT): LoRA (rank=16) adapters in self-attention/FFN matrices permit rapid, low-resource updates, especially for domain/language adaptation. Merging at 4-bit weights with full precision delta is preferred to minimize rounding loss (Syromiatnikov et al., 18 Mar 2025).
Multilingual adaptation: Branch-and-Merge continual pretraining of Gemma-2-9B/27B on large Bulgarian-English corpora (BgGPT) raised Bulgarian task performance to 61.3%/64.7% (9B/27B), with no drop in English (Alexandrov et al., 2024).
Modular reuse: Frozen mid-layers from Gemma 3, coupled with custom input adapters, yield efficient predictors for out-of-domain tasks (e.g., tabular wildfire prediction; Gemma internal blocks provide a robust “internal world” for environmental forecasting, with high recall and interpretable attention maps) (Jadouli et al., 20 Apr 2025).
Encoder-decoder adaptation: Systematic conversion (via PrefixLM, UL2) of decoder-only Gemma 2B/9B yields encoder-decoder LLMs matching original pretrain quality and outperforming on downstream tuning (e.g., +7% instruction-tune, +12.6 points SuperGLUE at 2B scale) (Zhang et al., 8 Apr 2025).

4. Safety, Hallucination, and Interpretability

Gemma’s safety, hallucination, and interpretability profile is well studied:

Safety: On red-teaming datasets for factuality, bias, and toxicity, Gemma-2B/7B IT models behave conservatively—refusing potentially unsafe tasks more often than Llama2, but trailing in instruction adherence and multi-turn robustness (toxicity drops to 0.03–0.14 in 4-turn dialogs) (Nadeau et al., 2024).
Hallucination: Systematic symbolic input triggers (modifiers, named entities, numbers) elicit high hallucination rates (>80%) in Gemma-2 2B, dropping only to ~64% at 27B scale—a fundamental encoding limitation not yet solved by scaling alone (Lamba et al., 9 Sep 2025).
Interpretability tools: Gemma Scope provides >2,000 JumpReLU sparse autoencoders (SAEs) covering every layer and sub-layer of Gemma 2B/9B and parts of 27B, supporting supervised and unsupervised circuit analysis and feature steering (Lieberum et al., 2024). These tools are fully open source and compatible with research libraries such as HuggingFace Transformers and TransformerLens.

5. Multimodal and Academic Writing Applications

Gemma’s deployment includes:

Vision-language: Gemma 3 inherits SigLIP vision backbone and can handle ≥128K-token context multimodal prompts, with instruction-tuned models matching previous SOTA on math, code, chat, and vision.
Academic writing: Gemma 27B yields mid-range output lengths, high semantic fidelity (~92–98% overlap with source), and low paraphrase plagiarism (11%), but is flagged by AI detectors (≥95%) and has dense, low-readability prose—manipulable by further style adaptation (Aydin et al., 11 Feb 2025).
Prompt recovery: Fused models (Gemma-2B-IT + Phi2) excel at reconstructing prompts in text rewriting tasks (SCS=0.61 > all baselines), leveraging dual-stage pretraining and contextual fusion to bolster semantic alignment (Chen et al., 2024).

6. Scientific and Physical Experiments: GEMMA and Neutrino Physics

The acronym GEMMA also denotes the Germanium Experiment for the Measurement of the Magnetic Moment of Antineutrino, based at Kalinin NPP, which sets world-leading direct laboratory bounds on neutrino electromagnetic properties (Beda et al., 2010, Brudanin et al., 2014):

Using a 1.5 kg HPGe detector and low threshold ( $T_\text{min} \approx 2.8$ keV), GEMMA obtains upper bounds:
- Neutrino anomalous magnetic moment: $\mu_\nu < 3.2\times 10^{-11}\mu_B$ (without atomic ionization), improved to $5.0\times 10^{-12}\mu_B$ (including atomic ionization).
- Direct neutrino electric millicharge: $|q_\nu| < 2.7 \times 10^{-12}\,e_0$ (90% CL)—tightest reactor-based limit to date (Brudanin et al., 2014).
Future GEMMA-II/III upgrades aspire to reach $|q_\nu| \sim 5.5 \times 10^{-13}\,e_0$ and $\mu_\nu \lesssim 1\times 10^{-12}\mu_B$ , placing further constraints on beyond-Standard-Model physics.

7. Scientific Applications: Asteroseismology

In astrophysics, “Gemma” refers to the young subgiant KIC 11026764, whose oscillation modes measured by Kepler underwrite benchmark models for stellar evolution (Farnir et al., 27 Feb 2025):

45 oscillation modes, including g-dominated mixed modes, are fit using the EGGMiMoSA tool to resolve core chemical gradients.
Best-fit models for Gemma require convective overshooting parameter $\alpha_{\text{ov}}=0.17$ , and achieve mass $M=1.20^{+0.07}_{-0.04}M_\odot$ , age $5.75$ Gyr, radius $2.08 R_\odot$ .
Gemma thus serves as a reference point for calibrating convective core mixing in solar-like subgiants.

Gemma models combine architectural efficiency, broad open-source release, safety-aware pre-training, adaptability to language and domain specialization, and coverage of both scientific and engineering benchmarks. In both machine learning and physical sciences, “Gemma” denotes state-of-the-art instrumentation and methodology, with ongoing extensions toward better multimodal understanding, efficient adaptation, enhanced interpretability, and deeper participation in foundational physics and astrophysical measurement.