---
title: 'DiffusionGemma 26B: Diffusion Language Model'
url: https://www.emergentmind.com/topics/diffusiongemma-26b
type: topic
---

# DiffusionGemma 26B: Diffusion Language Model

to=arxiv_search  北京赛车前json
{"query":"DiffusionGemma 26B", "max_results": 10}
to=arxiv_search  久赢json
{"query":"\"DiffusionGemma-26B\" OR \"DiffusionGemma 26B\" OR DiffusionGemma", "max_results": 10, "sort_by":"relevance"}
DiffusionGemma 26B is a Gemma-4-based diffusion language model studied in 2026 as a masked discrete-diffusion, mixture-of-experts checkpoint that generates text by iteratively denoising a fixed token canvas rather than emitting tokens left to right. Across the main analyses now available, it is treated as a model with **25.2B total parameters and 3.8B active parameters**, using **8-of-128 expert routing**, a **256-token canvas**, and an **entropy-bounded sampler**; one paper adapts it as a multimodal medical foundation model with a SigLIP-lineage vision encoder, another measures how it actually commits tokens during decoding, and a third studies the transparency of its intermediate denoising states relative to autoregressive Gemma 4 [2607.01436] [2606.14620] [2606.20560].

## 1. Identity, naming, and architectural basis

The model discussed in the literature is typically called **DiffusionGemma-26B**, but the papers are careful about what that label means. In the decoding-order study, the checkpoint is described as a public **masked discrete-diffusion language model** built on **Gemma 4**, marketed as **“26B-A4B”** but counted by the authors as **25.2B total parameters** with **3.8B active parameters**, using **8-of-128 expert routing** [2606.14620]. The medical adaptation paper uses the same quantitative description and repeatedly notes that the finetuned model is referred to as **“26B”** while clarifying that it is **“3.8B active”** at inference; in that paper, “3.8B active” means that although the full MoE contains **25.2B parameters**, only about **3.8B parameters are activated per token/forward pass** [2607.01436].

The matched comparison target throughout the medical study is **Gemma-4-26B**, its same-family autoregressive sibling. The comparison is explicitly controlled so that the two models have the **same family, size, vision tower, LoRA targets, and training data**, and differ primarily in the **generative paradigm**. Both are treated as multimodal MoE models, but DiffusionGemma-26B is the diffusion decoder and Gemma-4-26B is the standard left-to-right autoregressive decoder [2607.01436].

A related point of identification is that the transparency paper centers its concrete analyses on **DiffusionGemma 26B A4B**, compared against **Gemma 4 26B A4B**. That paper therefore anchors the term “DiffusionGemma” not to an abstract model class, but to a specific Gemma-4-derived 26B A4B checkpoint with iterative denoising over a token canvas and an inter-step self-conditioning state [2606.20560].

## 2. Generative mechanism and conditioning structure

DiffusionGemma 26B differs from autoregressive Gemma-4-26B at the level of factorization and sampling. The autoregressive sibling follows the left-to-right decomposition
$$
p_\theta(\mathbf{x}\mid c)=\prod_i p_\theta(x_i\mid x_{<i},c),
$$
whereas DiffusionGemma generates by **denoising a token canvas bidirectionally instead of emitting tokens left to right** [2607.01436]. In the medical paper, the diffusion training objective is the model’s standard **uniform-state dLLM objective**: a random fraction of target tokens is replaced with uniform draws from the vocabulary, and the model is trained to recover them. In the transparency paper, inference is described as a setup stage followed by **\(T\)** denoising iterations, each comprising a forward pass through the transformer and a sampling or renoising step [2607.01436] [2606.20560].

The core state carried across denoising steps consists of the current token canvas \(\mathbf{o}^t\) and a self-conditioning matrix \(S^t \in \mathbb{R}^{C \times d_{\text{model}}}\). At setup, the prompt is processed once to produce reusable key-value activations, the canvas is initialized with random tokens, and \(S^0 = 0\). At each denoising step, the model passes \(S^{t-1}\) through a gated MLP, adds it to token embeddings, and applies **bidirectional attention**, yielding logits \(\boldsymbol{\ell}^t \in \mathbb{R}^{C \times |V|}\). The transparency paper gives the self-conditioning update as
$$
\mathbf{S}^t = \mathrm{softmax}(\hat{\boldsymbol{\ell}}^t)\,\mathbf{W}_E \in \mathbb{R}^{C \times d_{\text{model}}},
$$
after temperature shaping, and candidate tokens are sampled independently before **Entropy-Bounded (EB) sampling** selects a low-entropy subset to retain [2606.20560].

The decoding-order paper makes the operational role of the sampler especially explicit. It studies a shipped **`EntropyBoundSampler`** with `entropy_bound = 0.1` nats, running on a fixed **256-token canvas** for **up to 48 denoising steps**. The sampler’s method
$$
\texttt{accept\_canvas(current\_canvas, denoiser\_canvas, logits, cur\_step)}
$$
is treated as the concrete commit mechanism, because this step decides which positions become accepted or finalized on a given accept-call [2606.14620].

This conditioning structure enables **any-order infill**, which the medical paper treats as the model’s most distinctive practical affordance. The relevant conditional is
$$
\mathbf{x}_{\bar F}\sim p_\theta\!\big(\mathbf{x}_{\bar F}\mid \mathbf{x}_F{=}\mathbf{a},\,c\big),
$$
where \(F\) are fixed positions, \(\mathbf{a}\) are user-specified tokens at those positions, and \(\bar F\) are the positions to be filled. Because denoising is bidirectional over the full canvas, the model can condition on arbitrary fixed spans anywhere in the sequence, not only on a prefix. The implementation requires **no retraining**: at each denoising step, fixed tokens are clamped both before the update, so the model conditions on them, and after the update, so they survive the step’s re-randomization [2607.01436].

## 3. Medical multimodal adaptation and report drafting

In the medical adaptation study, DiffusionGemma-26B is not treated as a generic text-only model, but as a **native multimodal diffusion language model** with a **SigLIP-lineage vision encoder** producing roughly **280 image tokens**. The adaptation is a vision-language medical-domain adaptation in which the model is conditioned on medical images and asked to generate either short VQA answers or radiology report text. The authors state the setup directly: **“We condition on the image and diffuse the text target; the image is never generated.”** A full report is constrained to fit inside a **256-token canvas**, because that allows **intra-report attention** to be bidirectional end to end, which any-order infill requires [2607.01436].

The LoRA recipe is deliberately matched between diffusion and autoregressive variants. Both DiffusionGemma-26B and Gemma-4-26B are adapted with **rank-64 LoRA**, **\(\alpha=128\)**, targeting the **attention projections** and the **shared MLP projections**, written as
$$
\text{LoRA} = r64/\alpha128,\ \{q,k,v,o,\text{mlp}\}.
$$
The **128 experts**, the **router**, and the **vision tower** are **frozen**. Training runs are in **bf16 on a single H100**. For DiffusionGemma-26B, the optimizer is **AdamW** with \(\beta=(0.9,0.95)\), **learning rate \(1\times 10^{-4}\)**, **cosine schedule**, and **100 warmup steps**; the diffusion-specific settings listed are **\(\epsilon=0.1\)** and **canvas length \(L=256\)** [2607.01436].

For medical VQA, the authors perform **per-dataset LoRA finetuning** on **VQA-RAD**, **SLAKE**, and **VQA-Med-2019**. Appendix dataset sizes are **1,817 train / 431 test** for VQA-RAD, **4,919 / 1,053** for SLAKE, and approximately **12.8k / 2.0k** for VQA-Med-2019; evaluation uses a fixed random subset of **350 held-out test questions per dataset**. Scoring is not exact match but a **verbosity-robust LLM judge**, specifically **Claude Sonnet 4.6**, which returns a binary semantic-equivalence verdict for each **(question, reference answer, model answer)** triple [2607.01436].

Under this matched setup, diffusion equals or exceeds autoregressive Gemma-4-26B on all three VQA datasets. On **VQA-RAD**, base diffusion scores **0.614** versus base AR **0.523**, and finetuned diffusion **0.649** versus finetuned AR **0.649**. On **SLAKE**, base diffusion is **0.700** versus **0.674**, and finetuned diffusion rises to **0.863** versus AR **0.817**. On **VQA-Med-2019**, base diffusion is **0.629** versus **0.614**, and finetuned diffusion **0.666** versus **0.631**. The base VQA-RAD gain of **+0.091** is reported as significant with **\(p<0.001\)**, and the finetuned SLAKE gain of **+0.046** is significant by McNemar test with **\(p=0.026\)**; the remaining four diffusion-versus-AR comparisons are not statistically significant. The paper further notes that the advantage is concentrated on **closed yes/no questions**; for **VQA-RAD finetuned closed questions**, diffusion achieves **0.825** versus AR **0.757** [2607.01436].

Against frontier zero-shot VLMs, the finetuned diffusion model is competitive rather than uniformly dominant. On **VQA-RAD**, **Gemini-3.5-Flash** is best at **0.777**, while diffusion reaches **0.649**, **GPT-4.1-mini** **0.571**, and **Sonnet-4.6** **0.654**. On **SLAKE**, finetuned diffusion is the **best overall model** at **0.863**, ahead of Gemini **0.751**, GPT **0.646**, and Sonnet **0.703**. On **VQA-Med-2019**, Gemini leads with **0.683**, while diffusion gets **0.666**, Sonnet **0.654**, and GPT **0.586** [2607.01436].

The same paper also emphasizes drafting speed. On **one H100 in bf16** for a **\(\sim256\)-token generation**, **Gemma-4 (AR), greedy** takes **6.43 s** with **24.6 tok/s**. DiffusionGemma at **16 steps** takes **1.46 s**, **175.3 tok/s**, for a **4.4× speedup**; at **32 steps**, **1.74 s**, **147.4 tok/s**, **3.7×**; at **48 steps**, **1.84 s**, **139.4 tok/s**, **3.5×**. The authors explicitly tie this claim to a **256-token drafting workload** and note that the timing setup is generous to AR because AR is timed on its **natural, shorter output** while diffusion fills the full canvas [2607.01436].

The report-drafting and infill experiment on **MIMIC-CXR** is the clearest demonstration of a capability that the authors regard as structurally inherent to diffusion. They remove one complete sentence near the middle of each held-out report, with **\(n=249\)** reports, and compare **bidirectional** infill, where fragments on both sides of the gap are clamped, against **left-only** infill. DiffusionGemma-26B improves from **0.211** to **0.320** on token-F1, a gain of **+0.109**, with **\(p<10^{-10}\)** and **95% CI \([+0.077,+0.141]\)**; judge accuracy improves from **0.157** to **0.285**, a gain of **+0.129**, with **\(p=2\times10^{-5}\)**. By contrast, AR moves only from **0.244** to **0.275** on token-F1, **+0.031**, not significant at **\(p=0.08\)**, and judge accuracy shifts from **0.273** to **0.257**, **\(-0.016\)**, also not significant. The model-by-context interaction is significant on both metrics: **+0.078 token-F1** with **\(p=2\times10^{-4}\)** and **+0.145 judge** with **\(p=3\times10^{-4}\)**. The paper summarizes this as diffusion benefiting about **3.5× more** from bidirectional context [2607.01436].

## 4. Decoding dynamics and token commitment

A separate empirical question is how DiffusionGemma 26B actually commits tokens during decoding. The decoding-order study instruments the sampler’s accept step and defines, for each position \(p\), \(\mathrm{commit}(p)\) as **the first accept-call index that commits that position**. It also records the **entropy-at-commit**, computed as the Shannon entropy of the logits at the committing call over the **262,144**-item vocabulary. Because raw padded canvases contain trailing EOS and related special tokens that would distort order statistics, all main analyses restrict attention to **content positions**, defined as final positions whose token is not in
$$
\texttt{all\_special\_ids} \cup \{0,1,2,\langle \text{end\_of\_turn}\rangle\}.
$$
The main order statistic is content-only tie-aware Kendall correlation,
$$
\tau_b\big(\mathrm{commit}(p),\,p\big),
$$
which treats same-call commits as simultaneous rather than force-ordering them [2606.14620].

The probe suite contains **686 prompts** across six regimes: **R1 math (GSM8K)** with **200** prompts, **R2 code completion (HumanEval)** with **164**, **R3 code synthesis (MBPP)** with **200**, **R4 short factual recall** with **52**, **R5 open-ended instructions** with **40**, and **R6 constrained JSON / key-value output** with **30**. The confirmatory strengthening run uses **5 seeds** \(\{11,22,33,44,55\}\), **20 prompts per regime**, and thus **100 traces per regime** and **600 generations total**, with **95% prompt-clustered bootstrap intervals** [2606.14620].

The headline result is that decoding is **neither parallel nor sequential**. Across prose, code, math, and factual regimes, token-level order shows a real but only moderate positive left-to-right bias; at the same time, commit batches are large and many token pairs tie on the same accept-call. Structured JSON is the outlier, with approximately order-independent commitment.

| Regime | Token-level \(\tau_b\) | Tokens per accept-call / same-call fraction |
|---|---:|---:|
| R1 GSM8K | **0.512** \([0.456, 0.569]\) | **25.5 / 0.28** |
| R2 HumanEval | **0.430** \([0.318, 0.527]\) | **23.0 / 0.49** |
| R3 MBPP | **0.604** \([0.561, 0.647]\) | **19.8 / 0.19** |
| R4 factual | **0.460** \([0.422, 0.496]\) | **5.8 / 0.50** |
| R5 open-ended | **0.502** \([0.463, 0.542]\) | **12.6 / 0.11** |
| R6 JSON | **-0.044** \([ -0.086, -0.00]\) | **8.5 / 0.72** |

These values are incompatible with a cleanly autoregressive token-by-token process, but they are also far from order-independence in most regimes. The paper’s interpretation is that DiffusionGemma 26B has a **partial left-to-right commit bias**: earlier positions tend to be accepted somewhat earlier, yet later tokens can commit before earlier ones, and many tokens commit simultaneously [2606.14620].

A further finding is that apparent “block-sequentiality” depends strongly on the granularity of measurement. For **R1**, block-\(\tau_b\) rises smoothly from
$$
0.59 \to 0.64 \to 0.70 \to 0.79 \to 0.91
$$
as bin size grows
$$
4 \to 8 \to 16 \to 32 \to 64.
$$
The paper argues that this smooth increase means there is **no special jump at 16**; what had looked like a 16-token block process is instead a consequence of coarsening the ruler. Its formulation is that **16 is an analyst bin, not a privileged architectural block** [2606.14620].

The study also shows that commitment is **aggressive** rather than leisurely. Despite a nominal **48-step** denoising budget, generations finish after only **3.3 to 17.1 accept-calls on average**, and entropy-at-commit lies in the range **0.002 to 0.012 nats**, far below the sampler threshold of **0.1** nats. This indicates that accepted positions are typically committed with substantially stronger confidence than the threshold minimally requires. Confidence is not uniformly meaningful across regimes, however: in **R1 GSM8K**, \(-\)entropy predicts correctness with **AUROC 0.749** \([0.602, 0.879]\), whereas in **R4 factual recall** the corresponding AUROC is **0.471** \([0.383, 0.544]\), which the paper interprets as null signal for factual recall [2606.14620].

A methodological caveat is that commitment is **non-monotonic**. Across the **600** generations in the strengthening run, the authors observe **4524 un-accept events**, with mean **7.5 per generation**, median **2**, and only **220/600** generations fully monotone. For that reason, the paper defines order in terms of **first acceptance**, not permanent freezing, and then verifies that the main conclusions are similar on the strictly monotone subset [2606.14620].

## 5. Transparency, interpretability, and monitorability

The transparency study decomposes reasoning transparency into **variable transparency** and **algorithmic transparency**. Variable transparency asks whether intermediate computational states are interpretable; algorithmic transparency asks whether those interpretable states are enough to reconstruct the process by which the model arrived at its outputs. DiffusionGemma is challenging on both axes because much of its computation occurs between denoising steps, and because every token in the canvas can change at every step, permitting distributed and non-chronological computation [2606.20560].

The paper formalizes part of this issue through **opaque serial depth**, defined as the length of the longest serial computation that occurs without passing through an interpretable bottleneck. For sequence length **256k**, the reported empirical upper bounds are **21,235** for **Gemma 4 26B A4B**, **608,016** for **DiffusionGemma 26B A4B** when the inter-step bottleneck is treated as uninterpretable, and **23,571** for DiffusionGemma when that bottleneck is treated as interpretable. This yields the headline ratios:
$$
608{,}016 / 21{,}235 = 28.6\text{X}
$$
and
$$
23{,}571 / 21{,}235 = 1.1\text{X}.
$$
The claim is therefore not that diffusion is intrinsically almost as transparent as autoregression, but that for this particular model the apparent **28.6X** gap collapses to about **1.1X** if the information flowing between denoising steps can be routed through an interpretable token bottleneck [2606.20560].

That bottleneck claim is tested directly. Because the self-conditioning state is constructed as a soft projection of logits back into embedding space, the authors intervene on the logits before projection and keep only either the **top-8** tokens per position or those above probability **\(p=0.03\)**, while setting discarded logits to a constant chosen so that the retained token probabilities remain unchanged after softmax. On **Natural2Code**, **LiveCodeBench**, **AMC/AIME/IMO variants**, and **GPQA**, with **dynamic thinking** enabled, **\(k=8\)** or **\(p=0.03\)** is reported as sufficient to match baseline capability. The paper takes this as evidence that the load-bearing inter-step information can be expressed through a small number of token hypotheses per position [2606.20560].

The intermediate states themselves are then examined on **800 WildChat prompts**, up to **1024 generated tokens** or **4 canvases**, with **no early stopping** and **\(T=48\)**. Tokens are categorized as **Final Token**, **Adjacent Token**, **Similar to Final**, **Similar to Adjacent**, or **Other**. Under the probability-threshold intervention **\(>0.03\)**, the paper reports that even the earliest denoising steps have **more than 85%** of tokens categorized as final, adjacent, or similar. This is presented as support for variable transparency: the inter-step state is often readable as guesses about the current token or nearby tokens, rather than as an entirely opaque latent representation [2606.20560].

Algorithmic transparency remains more difficult, and the paper’s case studies emphasize specifically diffusion-like behaviors. One example shows **early response-length prediction**: for the prompt “Explain how photosynthesis works, in 2 sentences,” the model predicts padding or EOS with near-100% probability beyond around position **55** after the first denoising step, even though the actual content tokens remain unsettled. Another shows **retroactive self-correction**: on “How many square numbers are there between 400 and 800? State your answer first, then give your reasoning,” the leading answer token is **9** at step **4** and later changes to **8** with higher confidence. The paper also describes recurring **skeleton-first** code generation, **token smearing** over adjacent positions, **sequence smearing** across alternative whole continuations, and **intermediate-context reasoning** in a recurrence task where the model first generates the digit **3**, uses it internally to continue the sequence, and only later replaces it with the token **“Gold”** before final output [2606.20560].

The paper additionally evaluates **monitorability**, treating it as a downstream application of transparency. On the tested benchmarks, which include intervention, process, and outcome-property tasks and are run over **multiple canvases**, DiffusionGemma and Gemma 4 are reported as **similarly monitorable**. The authors do not, however, treat this as evidence that single-canvas monitorability is fully comparable, and they explicitly note that the evaluation spans multiple canvases, where DiffusionGemma behaves more autoregressively [2606.20560].

## 6. Scope, limitations, and common misconceptions

Several limitations recur across the literature. The medical adaptation paper does not report **LoRA dropout**, **number of epochs**, **total number of optimization steps**, **gradient clipping**, **weight decay**, or a detailed **data mixture** beyond stating that matched runs use the same image-text data and that per-dataset LoRA finetuning is performed for VQA. Its report-drafting experiments are restricted to **single-sentence infill** in one **256-token canvas**, and the paper does not claim deployment readiness or recommend unsupervised clinical use. Its framing is consistently **interactive human-in-the-loop drafting**, with a clinician fixing report fragments and regenerating drafts as needed [2607.01436].

The decoding-order paper, for its part, is explicit that many order results can be manufactured by poor measurement. It identifies five distortion sources: **trailing-EOS padding artifacts**, **within-regime confounding / Simpson’s paradox**, **commit non-monotonicity**, **block-size sensitivity**, and **large commit-batch ties**. A plausible implication is that coarse summaries of diffusion decoding can be misleading in either direction: they can make the model look spuriously anti-sequential, or conversely more block-autoregressive than the actual commitment process warrants [2606.14620].

The transparency paper is similarly cautious. Its positive result depends in part on a specific architectural fact: the self-conditioning state is formed by projecting soft token distributions back into embedding space. The paper does not establish that future latent reasoning models, or even future diffusion variants, will preserve equally token-aligned intermediates. It also does not establish that all reasoning is faithfully exposed by the token bottleneck, nor that **single-canvas** monitorability matches the multi-canvas results [2606.20560].

A common naming confusion concerns papers that analyze **Gemma 4 26B A4B** but not DiffusionGemma 26B. The security-agent paper **“Measuring Safety Alignment Effects in Autonomous Security Agents”** studies **Google DeepMind’s `gemma-4-26B-A4B-it`** and **TrevorJS `gemma-4-26B-A4B-it-uncensored`**, along with other families, under an autonomous security-agent harness; it explicitly does **not** mention the exact term **“DiffusionGemma 26B.”** The closest mapping it offers is therefore to **Gemma 4 26B A4B** and its **uncensored derivative**, not to the diffusion model itself. That distinction matters because results about authorized security-agent behavior in Gemma 4 26B A4B cannot be treated as direct evidence about DiffusionGemma 26B [2605.19722].

Taken together, the available evidence supports a precise but bounded characterization. DiffusionGemma 26B is a Gemma-4-derived diffusion model whose denoising decoder enables bidirectional conditioning and any-order infill, whose matched medical adaptation is at least as strong as autoregressive Gemma-4-26B on the reported VQA tasks while being **3.5–4.4× faster** on a **256-token** drafting workload, whose actual token commitment is governed by large multi-token accept batches with only a partial left-to-right bias, and whose intermediate denoising states appear substantially more interpretable than naive latent-space arguments would suggest. At the same time, the literature does not claim universal superiority over autoregressive models, deployment-ready clinical autonomy, or exhaustive transparency of the denoising process [2607.01436] [2606.14620] [2606.20560].

Source: https://www.emergentmind.com/topics/diffusiongemma-26b