DiffusionGemma 26B: Diffusion Language Model

• DiffusionGemma 26B is a diffusion language model built on Gemma-4 that uses masked discrete-diffusion to iteratively denoise a fixed 256-token canvas.
• It activates only 3.8B parameters per token via an 8-of-128 expert routing mechanism, enabling efficient and bidirectional any-order infill.
• Its multimodal medical adaptation and accelerated decoding (up to 4.4× speedup) demonstrate competitive performance compared to autoregressive counterparts.

to=arxiv_search 北京赛车前json {"query":"DiffusionGemma 26B", "max_results": 10} to=arxiv_search 久赢json {"query":"\"DiffusionGemma-26B\" OR \"DiffusionGemma 26B\" OR DiffusionGemma", "max_results": 10, "sort_by":"relevance"} DiffusionGemma 26B is a Gemma-4-based diffusion LLM studied in 2026 as a masked discrete-diffusion, mixture-of-experts checkpoint that generates text by iteratively denoising a fixed token canvas rather than emitting tokens left to right. Across the main analyses now available, it is treated as a model with 25.2B total parameters and 3.8B active parameters, using 8-of-128 expert routing, a 256-token canvas, and an entropy-bounded sampler; one paper adapts it as a multimodal medical foundation model with a SigLIP-lineage vision encoder, another measures how it actually commits tokens during decoding, and a third studies the transparency of its intermediate denoising states relative to autoregressive Gemma 4 (Puyvelde et al., 1 Jul 2026, Asaria et al., 12 Jun 2026, Engels et al., 18 Jun 2026).

1. Identity, naming, and architectural basis

The model discussed in the literature is typically called DiffusionGemma-26B, but the papers are careful about what that label means. In the decoding-order study, the checkpoint is described as a public masked discrete-diffusion LLM built on Gemma 4, marketed as “26B-A4B” but counted by the authors as 25.2B total parameters with 3.8B active parameters, using 8-of-128 expert routing (Asaria et al., 12 Jun 2026). The medical adaptation paper uses the same quantitative description and repeatedly notes that the finetuned model is referred to as “26B” while clarifying that it is “3.8B active” at inference; in that paper, “3.8B active” means that although the full MoE contains 25.2B parameters, only about 3.8B parameters are activated per token/forward pass (Puyvelde et al., 1 Jul 2026).

The matched comparison target throughout the medical study is Gemma-4-26B, its same-family autoregressive sibling. The comparison is explicitly controlled so that the two models have the same family, size, vision tower, LoRA targets, and training data, and differ primarily in the generative paradigm. Both are treated as multimodal MoE models, but DiffusionGemma-26B is the diffusion decoder and Gemma-4-26B is the standard left-to-right autoregressive decoder (Puyvelde et al., 1 Jul 2026).

A related point of identification is that the transparency paper centers its concrete analyses on DiffusionGemma 26B A4B, compared against Gemma 4 26B A4B. That paper therefore anchors the term “DiffusionGemma” not to an abstract model class, but to a specific Gemma-4-derived 26B A4B checkpoint with iterative denoising over a token canvas and an inter-step self-conditioning state (Engels et al., 18 Jun 2026).

2. Generative mechanism and conditioning structure

DiffusionGemma 26B differs from autoregressive Gemma-4-26B at the level of factorization and sampling. The autoregressive sibling follows the left-to-right decomposition

$$p_\theta(\mathbf{x}\mid c)=\prod_i p_\theta(x_i\mid x_{<i},c),$$

whereas DiffusionGemma generates by denoising a token canvas bidirectionally instead of emitting tokens left to right (Puyvelde et al., 1 Jul 2026). In the medical paper, the diffusion training objective is the model’s standard uniform-state dLLM objective: a random fraction of target tokens is replaced with uniform draws from the vocabulary, and the model is trained to recover them. In the transparency paper, inference is described as a setup stage followed by $T$ denoising iterations, each comprising a forward pass through the transformer and a sampling or renoising step (Puyvelde et al., 1 Jul 2026, Engels et al., 18 Jun 2026).

The core state carried across denoising steps consists of the current token canvas $\mathbf{o}^t$ and a self-conditioning matrix $S^t \in \mathbb{R}^{C \times d_{\text{model}}}$. At setup, the prompt is processed once to produce reusable key-value activations, the canvas is initialized with random tokens, and $S^0 = 0$. At each denoising step, the model passes $S^{t-1}$ through a gated MLP, adds it to token embeddings, and applies bidirectional attention, yielding logits $\boldsymbol{\ell}^t \in \mathbb{R}^{C \times |V|}$. The transparency paper gives the self-conditioning update as

$$\mathbf{S}^t = \mathrm{softmax}(\hat{\boldsymbol{\ell}}^t)\,\mathbf{W}_E \in \mathbb{R}^{C \times d_{\text{model}}},$$

after temperature shaping, and candidate tokens are sampled independently before Entropy-Bounded (EB) sampling selects a low-entropy subset to retain (Engels et al., 18 Jun 2026).

The decoding-order paper makes the operational role of the sampler especially explicit. It studies a shipped EntropyBoundSampler with entropy_bound = 0.1 nats, running on a fixed 256-token canvas for up to 48 denoising steps. The sampler’s method

$$\texttt{accept\_canvas(current\_canvas, denoiser\_canvas, logits, cur\_step)}$$

is treated as the concrete commit mechanism, because this step decides which positions become accepted or finalized on a given accept-call (Asaria et al., 12 Jun 2026).

This conditioning structure enables any-order infill, which the medical paper treats as the model’s most distinctive practical affordance. The relevant conditional is

$$\mathbf{x}_{\bar F}\sim p_\theta\!\big(\mathbf{x}_{\bar F}\mid \mathbf{x}_F{=}\mathbf{a},\,c\big),$$

where $T$0 are fixed positions, $T$1 are user-specified tokens at those positions, and $T$2 are the positions to be filled. Because denoising is bidirectional over the full canvas, the model can condition on arbitrary fixed spans anywhere in the sequence, not only on a prefix. The implementation requires no retraining: at each denoising step, fixed tokens are clamped both before the update, so the model conditions on them, and after the update, so they survive the step’s re-randomization (Puyvelde et al., 1 Jul 2026).

3. Medical multimodal adaptation and report drafting

In the medical adaptation study, DiffusionGemma-26B is not treated as a generic text-only model, but as a native multimodal diffusion LLM with a SigLIP-lineage vision encoder producing roughly 280 image tokens. The adaptation is a vision-language medical-domain adaptation in which the model is conditioned on medical images and asked to generate either short VQA answers or radiology report text. The authors state the setup directly: “We condition on the image and diffuse the text target; the image is never generated.” A full report is constrained to fit inside a 256-token canvas, because that allows intra-report attention to be bidirectional end to end, which any-order infill requires (Puyvelde et al., 1 Jul 2026).

The LoRA recipe is deliberately matched between diffusion and autoregressive variants. Both DiffusionGemma-26B and Gemma-4-26B are adapted with rank-64 LoRA, $T$3, targeting the attention projections and the shared MLP projections, written as

$T$4

The 128 experts, the router, and the vision tower are frozen. Training runs are in bf16 on a single H100. For DiffusionGemma-26B, the optimizer is AdamW with $T$5, learning rate $T$6, cosine schedule, and 100 warmup steps; the diffusion-specific settings listed are $T$7 and canvas length $T$8 (Puyvelde et al., 1 Jul 2026).

For medical VQA, the authors perform per-dataset LoRA finetuning on VQA-RAD, SLAKE, and VQA-Med-2019. Appendix dataset sizes are 1,817 train / 431 test for VQA-RAD, 4,919 / 1,053 for SLAKE, and approximately 12.8k / 2.0k for VQA-Med-2019; evaluation uses a fixed random subset of 350 held-out test questions per dataset. Scoring is not exact match but a verbosity-robust LLM judge, specifically Claude Sonnet 4.6, which returns a binary semantic-equivalence verdict for each (question, reference answer, model answer) triple (Puyvelde et al., 1 Jul 2026).

Under this matched setup, diffusion equals or exceeds autoregressive Gemma-4-26B on all three VQA datasets. On VQA-RAD, base diffusion scores 0.614 versus base AR 0.523, and finetuned diffusion 0.649 versus finetuned AR 0.649. On SLAKE, base diffusion is 0.700 versus 0.674, and finetuned diffusion rises to 0.863 versus AR 0.817. On VQA-Med-2019, base diffusion is 0.629 versus 0.614, and finetuned diffusion 0.666 versus 0.631. The base VQA-RAD gain of +0.091 is reported as significant with $T$9, and the finetuned SLAKE gain of +0.046 is significant by McNemar test with $\mathbf{o}^t$0; the remaining four diffusion-versus-AR comparisons are not statistically significant. The paper further notes that the advantage is concentrated on closed yes/no questions; for VQA-RAD finetuned closed questions, diffusion achieves 0.825 versus AR 0.757 (Puyvelde et al., 1 Jul 2026).

Against frontier zero-shot VLMs, the finetuned diffusion model is competitive rather than uniformly dominant. On VQA-RAD, Gemini-3.5-Flash is best at 0.777, while diffusion reaches 0.649, GPT-4.1-mini 0.571, and Sonnet-4.6 0.654. On SLAKE, finetuned diffusion is the best overall model at 0.863, ahead of Gemini 0.751, GPT 0.646, and Sonnet 0.703. On VQA-Med-2019, Gemini leads with 0.683, while diffusion gets 0.666, Sonnet 0.654, and GPT 0.586 (Puyvelde et al., 1 Jul 2026).

The same paper also emphasizes drafting speed. On one H100 in bf16 for a $\mathbf{o}^t$1-token generation, Gemma-4 (AR), greedy takes 6.43 s with 24.6 tok/s. DiffusionGemma at 16 steps takes 1.46 s, 175.3 tok/s, for a 4.4× speedup; at 32 steps, 1.74 s, 147.4 tok/s, 3.7×; at 48 steps, 1.84 s, 139.4 tok/s, 3.5×. The authors explicitly tie this claim to a 256-token drafting workload and note that the timing setup is generous to AR because AR is timed on its natural, shorter output while diffusion fills the full canvas (Puyvelde et al., 1 Jul 2026).

The report-drafting and infill experiment on MIMIC-CXR is the clearest demonstration of a capability that the authors regard as structurally inherent to diffusion. They remove one complete sentence near the middle of each held-out report, with $\mathbf{o}^t$2 reports, and compare bidirectional infill, where fragments on both sides of the gap are clamped, against left-only infill. DiffusionGemma-26B improves from 0.211 to 0.320 on token-F1, a gain of +0.109, with $\mathbf{o}^t$3 and 95% CI $\mathbf{o}^t$4; judge accuracy improves from 0.157 to 0.285, a gain of +0.129, with $\mathbf{o}^t$5. By contrast, AR moves only from 0.244 to 0.275 on token-F1, +0.031, not significant at $\mathbf{o}^t$6, and judge accuracy shifts from 0.273 to 0.257, $\mathbf{o}^t$7, also not significant. The model-by-context interaction is significant on both metrics: +0.078 token-F1 with $\mathbf{o}^t$8 and +0.145 judge with $\mathbf{o}^t$9. The paper summarizes this as diffusion benefiting about 3.5× more from bidirectional context (Puyvelde et al., 1 Jul 2026).

4. Decoding dynamics and token commitment

A separate empirical question is how DiffusionGemma 26B actually commits tokens during decoding. The decoding-order study instruments the sampler’s accept step and defines, for each position $S^t \in \mathbb{R}^{C \times d_{\text{model}}}$0, $S^t \in \mathbb{R}^{C \times d_{\text{model}}}$1 as the first accept-call index that commits that position. It also records the entropy-at-commit, computed as the Shannon entropy of the logits at the committing call over the 262,144-item vocabulary. Because raw padded canvases contain trailing EOS and related special tokens that would distort order statistics, all main analyses restrict attention to content positions, defined as final positions whose token is not in

$S^t \in \mathbb{R}^{C \times d_{\text{model}}}$2

The main order statistic is content-only tie-aware Kendall correlation,

$S^t \in \mathbb{R}^{C \times d_{\text{model}}}$3

which treats same-call commits as simultaneous rather than force-ordering them (Asaria et al., 12 Jun 2026).

The probe suite contains 686 prompts across six regimes: R1 math (GSM8K) with 200 prompts, R2 code completion (HumanEval) with 164, R3 code synthesis (MBPP) with 200, R4 short factual recall with 52, R5 open-ended instructions with 40, and R6 constrained JSON / key-value output with 30. The confirmatory strengthening run uses 5 seeds $S^t \in \mathbb{R}^{C \times d_{\text{model}}}$4, 20 prompts per regime, and thus 100 traces per regime and 600 generations total, with 95% prompt-clustered bootstrap intervals (Asaria et al., 12 Jun 2026).

The headline result is that decoding is neither parallel nor sequential. Across prose, code, math, and factual regimes, token-level order shows a real but only moderate positive left-to-right bias; at the same time, commit batches are large and many token pairs tie on the same accept-call. Structured JSON is the outlier, with approximately order-independent commitment.

Regime	Token-level $S^t \in \mathbb{R}^{C \times d_{\text{model}}}$5	Tokens per accept-call / same-call fraction
R1 GSM8K	0.512 $S^t \in \mathbb{R}^{C \times d_{\text{model}}}$6	25.5 / 0.28
R2 HumanEval	0.430 $S^t \in \mathbb{R}^{C \times d_{\text{model}}}$7	23.0 / 0.49
R3 MBPP	0.604 $S^t \in \mathbb{R}^{C \times d_{\text{model}}}$8	19.8 / 0.19
R4 factual	0.460 $S^t \in \mathbb{R}^{C \times d_{\text{model}}}$9	5.8 / 0.50
R5 open-ended	0.502 $S^0 = 0$0	12.6 / 0.11
R6 JSON	-0.044 $S^0 = 0$1	8.5 / 0.72

These values are incompatible with a cleanly autoregressive token-by-token process, but they are also far from order-independence in most regimes. The paper’s interpretation is that DiffusionGemma 26B has a partial left-to-right commit bias: earlier positions tend to be accepted somewhat earlier, yet later tokens can commit before earlier ones, and many tokens commit simultaneously (Asaria et al., 12 Jun 2026).

A further finding is that apparent “block-sequentiality” depends strongly on the granularity of measurement. For R1, block-$S^0 = 0$2 rises smoothly from

$S^0 = 0$3

as bin size grows

$S^0 = 0$4

The paper argues that this smooth increase means there is no special jump at 16; what had looked like a 16-token block process is instead a consequence of coarsening the ruler. Its formulation is that 16 is an analyst bin, not a privileged architectural block (Asaria et al., 12 Jun 2026).

The study also shows that commitment is aggressive rather than leisurely. Despite a nominal 48-step denoising budget, generations finish after only 3.3 to 17.1 accept-calls on average, and entropy-at-commit lies in the range 0.002 to 0.012 nats, far below the sampler threshold of 0.1 nats. This indicates that accepted positions are typically committed with substantially stronger confidence than the threshold minimally requires. Confidence is not uniformly meaningful across regimes, however: in R1 GSM8K, $S^0 = 0$5entropy predicts correctness with AUROC 0.749 $S^0 = 0$6, whereas in R4 factual recall the corresponding AUROC is 0.471 $S^0 = 0$7, which the paper interprets as null signal for factual recall (Asaria et al., 12 Jun 2026).

A methodological caveat is that commitment is non-monotonic. Across the 600 generations in the strengthening run, the authors observe 4524 un-accept events, with mean 7.5 per generation, median 2, and only 220/600 generations fully monotone. For that reason, the paper defines order in terms of first acceptance, not permanent freezing, and then verifies that the main conclusions are similar on the strictly monotone subset (Asaria et al., 12 Jun 2026).

5. Transparency, interpretability, and monitorability

The transparency study decomposes reasoning transparency into variable transparency and algorithmic transparency. Variable transparency asks whether intermediate computational states are interpretable; algorithmic transparency asks whether those interpretable states are enough to reconstruct the process by which the model arrived at its outputs. DiffusionGemma is challenging on both axes because much of its computation occurs between denoising steps, and because every token in the canvas can change at every step, permitting distributed and non-chronological computation (Engels et al., 18 Jun 2026).

The paper formalizes part of this issue through opaque serial depth, defined as the length of the longest serial computation that occurs without passing through an interpretable bottleneck. For sequence length 256k, the reported empirical upper bounds are 21,235 for Gemma 4 26B A4B, 608,016 for DiffusionGemma 26B A4B when the inter-step bottleneck is treated as uninterpretable, and 23,571 for DiffusionGemma when that bottleneck is treated as interpretable. This yields the headline ratios:

$S^0 = 0$8

and

$S^0 = 0$9

The claim is therefore not that diffusion is intrinsically almost as transparent as autoregression, but that for this particular model the apparent 28.6X gap collapses to about 1.1X if the information flowing between denoising steps can be routed through an interpretable token bottleneck (Engels et al., 18 Jun 2026).

That bottleneck claim is tested directly. Because the self-conditioning state is constructed as a soft projection of logits back into embedding space, the authors intervene on the logits before projection and keep only either the top-8 tokens per position or those above probability $S^{t-1}$0, while setting discarded logits to a constant chosen so that the retained token probabilities remain unchanged after softmax. On Natural2Code, LiveCodeBench, AMC/AIME/IMO variants, and GPQA, with dynamic thinking enabled, $S^{t-1}$1 or $S^{t-1}$2 is reported as sufficient to match baseline capability. The paper takes this as evidence that the load-bearing inter-step information can be expressed through a small number of token hypotheses per position (Engels et al., 18 Jun 2026).

The intermediate states themselves are then examined on 800 WildChat prompts, up to 1024 generated tokens or 4 canvases, with no early stopping and $S^{t-1}$3. Tokens are categorized as Final Token, Adjacent Token, Similar to Final, Similar to Adjacent, or Other. Under the probability-threshold intervention $S^{t-1}$4, the paper reports that even the earliest denoising steps have more than 85% of tokens categorized as final, adjacent, or similar. This is presented as support for variable transparency: the inter-step state is often readable as guesses about the current token or nearby tokens, rather than as an entirely opaque latent representation (Engels et al., 18 Jun 2026).

Algorithmic transparency remains more difficult, and the paper’s case studies emphasize specifically diffusion-like behaviors. One example shows early response-length prediction: for the prompt “Explain how photosynthesis works, in 2 sentences,” the model predicts padding or EOS with near-100% probability beyond around position 55 after the first denoising step, even though the actual content tokens remain unsettled. Another shows retroactive self-correction: on “How many square numbers are there between 400 and 800? State your answer first, then give your reasoning,” the leading answer token is 9 at step 4 and later changes to 8 with higher confidence. The paper also describes recurring skeleton-first code generation, token smearing over adjacent positions, sequence smearing across alternative whole continuations, and intermediate-context reasoning in a recurrence task where the model first generates the digit 3, uses it internally to continue the sequence, and only later replaces it with the token “Gold” before final output (Engels et al., 18 Jun 2026).

The paper additionally evaluates monitorability, treating it as a downstream application of transparency. On the tested benchmarks, which include intervention, process, and outcome-property tasks and are run over multiple canvases, DiffusionGemma and Gemma 4 are reported as similarly monitorable. The authors do not, however, treat this as evidence that single-canvas monitorability is fully comparable, and they explicitly note that the evaluation spans multiple canvases, where DiffusionGemma behaves more autoregressively (Engels et al., 18 Jun 2026).

6. Scope, limitations, and common misconceptions

Several limitations recur across the literature. The medical adaptation paper does not report LoRA dropout, number of epochs, total number of optimization steps, gradient clipping, weight decay, or a detailed data mixture beyond stating that matched runs use the same image-text data and that per-dataset LoRA finetuning is performed for VQA. Its report-drafting experiments are restricted to single-sentence infill in one 256-token canvas, and the paper does not claim deployment readiness or recommend unsupervised clinical use. Its framing is consistently interactive human-in-the-loop drafting, with a clinician fixing report fragments and regenerating drafts as needed (Puyvelde et al., 1 Jul 2026).

The decoding-order paper, for its part, is explicit that many order results can be manufactured by poor measurement. It identifies five distortion sources: trailing-EOS padding artifacts, within-regime confounding / Simpson’s paradox, commit non-monotonicity, block-size sensitivity, and large commit-batch ties. A plausible implication is that coarse summaries of diffusion decoding can be misleading in either direction: they can make the model look spuriously anti-sequential, or conversely more block-autoregressive than the actual commitment process warrants (Asaria et al., 12 Jun 2026).

The transparency paper is similarly cautious. Its positive result depends in part on a specific architectural fact: the self-conditioning state is formed by projecting soft token distributions back into embedding space. The paper does not establish that future latent reasoning models, or even future diffusion variants, will preserve equally token-aligned intermediates. It also does not establish that all reasoning is faithfully exposed by the token bottleneck, nor that single-canvas monitorability matches the multi-canvas results (Engels et al., 18 Jun 2026).

A common naming confusion concerns papers that analyze Gemma 4 26B A4B but not DiffusionGemma 26B. The security-agent paper “Measuring Safety Alignment Effects in Autonomous Security Agents” studies Google DeepMind’s gemma-4-26B-A4B-it and TrevorJS gemma-4-26B-A4B-it-uncensored, along with other families, under an autonomous security-agent harness; it explicitly does not mention the exact term “DiffusionGemma 26B.” The closest mapping it offers is therefore to Gemma 4 26B A4B and its uncensored derivative, not to the diffusion model itself. That distinction matters because results about authorized security-agent behavior in Gemma 4 26B A4B cannot be treated as direct evidence about DiffusionGemma 26B (David et al., 19 May 2026).

Taken together, the available evidence supports a precise but bounded characterization. DiffusionGemma 26B is a Gemma-4-derived diffusion model whose denoising decoder enables bidirectional conditioning and any-order infill, whose matched medical adaptation is at least as strong as autoregressive Gemma-4-26B on the reported VQA tasks while being 3.5–4.4× faster on a 256-token drafting workload, whose actual token commitment is governed by large multi-token accept batches with only a partial left-to-right bias, and whose intermediate denoising states appear substantially more interpretable than naive latent-space arguments would suggest. At the same time, the literature does not claim universal superiority over autoregressive models, deployment-ready clinical autonomy, or exhaustive transparency of the denoising process (Puyvelde et al., 1 Jul 2026, Asaria et al., 12 Jun 2026, Engels et al., 18 Jun 2026).