Diff Transformer: Differential Attention

• Diff Transformer is a Transformer architecture that computes attention as the difference between two softmax maps, effectively canceling noise and highlighting relevant information.
• It leverages differential attention to reduce head redundancy and stabilize training dynamics, resulting in improved long-context modeling.
• Empirical studies show enhanced performance in language modeling and in-context learning, with variants like Shared DIFF and DINT further optimizing parameter efficiency and global context integration.

Diff Transformer, also called DIFF Transformer or Differential Transformer in subsequent work, is a Transformer architecture in which attention is computed as the difference between two separate softmax attention maps before multiplying the value stream. It was introduced to address the tendency of standard Transformers to overallocate attention to irrelevant context, with the stated objective of amplifying attention to relevant context while canceling noise; later analyses linked its empirical behavior to negative attention, reduced head redundancy, and improved learning dynamics (Ye et al., 2024, Kong et al., 22 May 2025). The term should be distinguished from Diffusion Transformers (DiTs), which replace the U-Net in latent diffusion models with a transformer denoiser and belong to a different research line (Amin et al., 14 Nov 2025).

1. Formal definition and layer structure

In the original formulation, an input sequence $X\in\mathbb{R}^{N\times d_{\mathrm{model}}}$ is projected into two query groups, two key groups, and one value stream: $$[Q_1;Q_2]=XW^Q,\qquad [K_1;K_2]=XW^K,\qquad V=XW^V.$$ Two attention maps are then formed,

$$A_1=\mathrm{softmax}\!\Bigl(\tfrac{Q_1K_1^T}{\sqrt d}\Bigr),\qquad A_2=\mathrm{softmax}\!\Bigl(\tfrac{Q_2K_2^T}{\sqrt d}\Bigr),$$

and the core operator is

$\mathrm{DiffAttn}(X)=\bigl(A_1-\lambda A_2\bigr)V.$

The differential attention map is therefore

$A_{\mathrm{diff}}=A_1-\lambda A_2.$

The scalar $\lambda$ is reparameterized for stable training as

$$\lambda=\exp(\lambda_{q_1}\cdot \lambda_{k_1})-\exp(\lambda_{q_2}\cdot \lambda_{k_2})+\lambda_{\rm init},$$

with $\lambda_{q_i},\lambda_{k_i}\in\mathbb{R}^d$ and a small positive $\lambda_{\rm init}$ (Ye et al., 2024).

At the block level, the original decoder layer preserves the standard residual structure: $$Y^l=\mathrm{MultiHeadDiff}\bigl(\mathrm{RMSNorm}(X^l)\bigr)+X^l,\qquad X^{l+1}=\mathrm{SwiGLU}\bigl(\mathrm{RMSNorm}(Y^l)\bigr)+Y^l.$$ The multi-head module applies differential attention per head, then applies head-wise normalization implemented via GroupNorm per head, multiplies by $$[Q_1;Q_2]=XW^Q,\qquad [K_1;K_2]=XW^K,\qquad V=XW^V.$$0, concatenates heads, and projects with $$[Q_1;Q_2]=XW^Q,\qquad [K_1;K_2]=XW^K,\qquad V=XW^V.$$1. RMSNorm replaces LayerNorm, and SwiGLU is used in the feed-forward sublayer. The original report states that a formal gradient analysis shows that overall gradient magnitudes remain on the same order as conventional attention, so existing Transformer hyperparameters carry over with no loss of stability (Ye et al., 2024).

The initial large-scale regime reported for the architecture used a 3 Billion parameter model trained on 1 Trillion tokens, with hidden size 3072, FFN size 8192, 28 layers, 12 heads for Diff versus 24 heads for the Transformer baseline, sequence length 4096, batch 4 M tokens, AdamW$$[Q_1;Q_2]=XW^Q,\qquad [K_1;K_2]=XW^K,\qquad V=XW^V.$$2, peak learning rate $$[Q_1;Q_2]=XW^Q,\qquad [K_1;K_2]=XW^K,\qquad V=XW^V.$$3, 1 000 warmup steps, linear decay to $$[Q_1;Q_2]=XW^Q,\qquad [K_1;K_2]=XW^K,\qquad V=XW^V.$$4, weight decay 0.1, and no dropout (Ye et al., 2024).

2. Noise cancellation, signed attention, and mechanistic interpretation

The original motivation is explicitly analogical: by analogy to a differential amplifier or noise-canceling headphones, subtracting two similarly computed signals can cancel common-mode noise. In this view, if both $$[Q_1;Q_2]=XW^Q,\qquad [K_1;K_2]=XW^K,\qquad V=XW^V.$$5 and $$[Q_1;Q_2]=XW^Q,\qquad [K_1;K_2]=XW^K,\qquad V=XW^V.$$6 assign spurious mass to irrelevant tokens, $$[Q_1;Q_2]=XW^Q,\qquad [K_1;K_2]=XW^K,\qquad V=XW^V.$$7 suppresses that shared background and yields a higher-contrast attention pattern. The original paper describes this as promoting the emergence of sparse attention patterns and reports that Diff Transformer is less distracted by irrelevant context in long-context modeling, key information retrieval, hallucination mitigation, in-context learning, and activation outlier reduction (Ye et al., 2024).

Later mechanistic analysis refined that account in three directions. First, it identified enhanced expressivity via negative attention: because subtraction produces signed weights, Diff can explicitly down-weight irrelevant tokens. Empirically, Diff assigns a substantial fraction of its weights to negative values, upward of 30–40% in middle layers, and qualitative examples show negative weights canceling distracting subject or literal contexts. Second, it identified reduced redundancy among attention heads: pairwise cosine distances between per-head attention maps are larger, Centered Kernel Alignment among heads’ output features is reduced, and head-importance scores are more uniformly distributed. Third, it identified improved learning dynamics: the Hessian spectrum at the start of training contains far fewer negative eigenvalues than in a standard Transformer, while fixing or removing the learnable $$[Q_1;Q_2]=XW^Q,\qquad [K_1;K_2]=XW^K,\qquad V=XW^V.$$8 eliminates this advantage and destabilizes training (Kong et al., 22 May 2025).

A common interpretive tension concerns sparsity. The original paper attributes part of Diff Transformer’s behavior to sparse attention patterns, but the later analysis reports that classical sparsity metrics, such as the ratio of near-zero weights or entropy of the distribution, show that Diff is less sparse than softmax attention in the usual absolute-value sense, because it redistributes context with signed weights rather than merely zeroing weak contexts (Ye et al., 2024, Kong et al., 22 May 2025). This suggests that the architecture’s selectivity is better understood as signed relevance allocation than as sparsity in a strictly nonnegative-attention sense.

3. Scaling behavior and downstream evidence

On zero-shot LM Eval Harness for the 1 T-token regime, the reported average accuracy is 60.6% for Diff-3B, compared with 57.5 for OpenLLaMA-3B-v2 and 56.8 for StableLM-3B-v2. In fitted scaling laws, a 6.8 B Diff matches an 11 B Transformer, corresponding to approximately 62% of the parameters, and a 7.8 B Diff matches a 13.1 B Transformer, corresponding to 59.5% of the parameters. Under token scaling for 3 B models, Diff trained on 160 B tokens matches a Transformer trained on 251 B tokens, or approximately 64% of the data (Ye et al., 2024).

The long-context and retrieval results are similarly central to the architecture’s reception. In 4 K-context multi-needle retrieval, the reported accuracies are 85% for Transformer versus 92% for Diff at $$[Q_1;Q_2]=XW^Q,\qquad [K_1;K_2]=XW^K,\qquad V=XW^V.$$9, 62% versus 84% at $$A_1=\mathrm{softmax}\!\Bigl(\tfrac{Q_1K_1^T}{\sqrt d}\Bigr),\qquad A_2=\mathrm{softmax}\!\Bigl(\tfrac{Q_2K_2^T}{\sqrt d}\Bigr),$$0, and 55% versus 85% at $$A_1=\mathrm{softmax}\!\Bigl(\tfrac{Q_1K_1^T}{\sqrt d}\Bigr),\qquad A_2=\mathrm{softmax}\!\Bigl(\tfrac{Q_2K_2^T}{\sqrt d}\Bigr),$$1. In 64 K contexts, the original paper reports that Diff remains approximately 80–90% accurate at 8 needles while Transformer drops toward 20% as length grows. For answer-span allocation, when the correct needle is at 25% depth, Diff allocates approximately 0.30 of total attention to it versus 0.03 for Transformer, while noise outside key spans falls from approximately 0.54 to approximately 0.02 (Ye et al., 2024).

The same study reports notable advantages in many-shot in-context learning. Across TREC, TREC-fine, Banking-77, and Clinic-150, Diff outperforms Transformer by +5–22% absolute accuracy as demonstrations scale to 64 K worth of examples. Under prompt-order permutation, Transformer’s accuracy can swing by up to $$A_1=\mathrm{softmax}\!\Bigl(\tfrac{Q_1K_1^T}{\sqrt d}\Bigr),\qquad A_2=\mathrm{softmax}\!\Bigl(\tfrac{Q_2K_2^T}{\sqrt d}\Bigr),$$2, whereas Diff’s swing is reported as less than 5%. In contextual hallucination mitigation, the reported correct, non-hallucinatory scores improve from 44% to 53% on XSum, 32% to 41% on CNN/DM, 42% to 61% on MultiNews, 28% to 39% on Qasper, 36% to 46% on HotpotQA, and 29% to 36% on 2WikiMQA (Ye et al., 2024).

The architecture was also presented as reducing activation and logit outliers. Reported top-1 attention logit outliers fall from approximately 318 in Transformer to approximately 38.8 in Diff, while hidden-state top-1 outliers fall from 3608 to 1688. Under low-bit quantization on HellaSwag, the report states that the 16-bit baseline is approximately 71% for both models, but at 8-bit Transformer drops by approximately 5 points while Diff drops by approximately 2 points, at 6-bit Transformer drops by approximately 15 points while Diff drops by approximately 3 points, and at 4-bit Transformer becomes unusable while Diff remains approximately 50% (Ye et al., 2024).

4. Architectural descendants: Shared DIFF and DINT

Two direct descendants modify the differential-attention core in complementary ways. Shared DIFF Transformer reduces parameter redundancy by replacing two independent query and key branches with a shared base projection plus low-rank updates, while DINT Transformer augments differential attention with a global “integral” component and exact row normalization (Cang et al., 29 Jan 2025, Cang et al., 29 Jan 2025).

Variant	Main modification	Selected reported result
Shared DIFF Transformer	Shared $$A_1=\mathrm{softmax}\!\Bigl(\tfrac{Q_1K_1^T}{\sqrt d}\Bigr),\qquad A_2=\mathrm{softmax}\!\Bigl(\tfrac{Q_2K_2^T}{\sqrt d}\Bigr),$$3 plus low-rank updates	24–40% fewer parameters; Shared DIFF-3B average accuracy 61.5%
DINT Transformer	Adds global importance $$A_1=\mathrm{softmax}\!\Bigl(\tfrac{Q_1K_1^T}{\sqrt d}\Bigr),\qquad A_2=\mathrm{softmax}\!\Bigl(\tfrac{Q_2K_2^T}{\sqrt d}\Bigr),$$4 and row-normalized $$A_1=\mathrm{softmax}\!\Bigl(\tfrac{Q_1K_1^T}{\sqrt d}\Bigr),\qquad A_2=\mathrm{softmax}\!\Bigl(\tfrac{Q_2K_2^T}{\sqrt d}\Bigr),$$5	Validation loss 3.055; DINT-3B average accuracy 62.2%

In Shared DIFF Transformer, the query and key projections are factorized as

$$A_1=\mathrm{softmax}\!\Bigl(\tfrac{Q_1K_1^T}{\sqrt d}\Bigr),\qquad A_2=\mathrm{softmax}\!\Bigl(\tfrac{Q_2K_2^T}{\sqrt d}\Bigr),$$6

and similarly for keys. The final operator becomes

$$A_1=\mathrm{softmax}\!\Bigl(\tfrac{Q_1K_1^T}{\sqrt d}\Bigr),\qquad A_2=\mathrm{softmax}\!\Bigl(\tfrac{Q_2K_2^T}{\sqrt d}\Bigr),$$7

where $$A_1=\mathrm{softmax}\!\Bigl(\tfrac{Q_1K_1^T}{\sqrt d}\Bigr),\qquad A_2=\mathrm{softmax}\!\Bigl(\tfrac{Q_2K_2^T}{\sqrt d}\Bigr),$$8 is a learned scalar. The parameter count for the original DIFF Transformer query and key projections is $$A_1=\mathrm{softmax}\!\Bigl(\tfrac{Q_1K_1^T}{\sqrt d}\Bigr),\qquad A_2=\mathrm{softmax}\!\Bigl(\tfrac{Q_2K_2^T}{\sqrt d}\Bigr),$$9, whereas Shared DIFF requires $\mathrm{DiffAttn}(X)=\bigl(A_1-\lambda A_2\bigr)V.$0. When $\mathrm{DiffAttn}(X)=\bigl(A_1-\lambda A_2\bigr)V.$1, the dominant term is nearly halved. The reported experiments use $\mathrm{DiffAttn}(X)=\bigl(A_1-\lambda A_2\bigr)V.$2 and observe 24–40% fewer parameters to match original DIFF or standard Transformer performance curves; on zero-shot language modeling, DIFF-3B averages 60.6% and Shared DIFF-3B averages 61.5% (Cang et al., 29 Jan 2025).

DINT Transformer starts from the observation that the row sums of $\mathrm{DiffAttn}(X)=\bigl(A_1-\lambda A_2\bigr)V.$3 equal $\mathrm{DiffAttn}(X)=\bigl(A_1-\lambda A_2\bigr)V.$4, not 1, which introduces numerical instability, and that pure differential attention lacks global context modeling. It defines a global importance vector

$\mathrm{DiffAttn}(X)=\bigl(A_1-\lambda A_2\bigr)V.$5

broadcasts it to $\mathrm{DiffAttn}(X)=\bigl(A_1-\lambda A_2\bigr)V.$6, and then forms

$\mathrm{DiffAttn}(X)=\bigl(A_1-\lambda A_2\bigr)V.$7

Because $\mathrm{DiffAttn}(X)=\bigl(A_1-\lambda A_2\bigr)V.$8 and $\mathrm{DiffAttn}(X)=\bigl(A_1-\lambda A_2\bigr)V.$9, each row of $A_{\mathrm{diff}}=A_1-\lambda A_2.$0 sums to 1. The reported results give validation loss approximately 3.055 for DINT versus 3.062 for DIFF and 3.15 for a standard Transformer; the DINT-3B zero-shot average accuracy is 62.2% versus 60.6% for DIFF-3B, and the multi-needle 4 K accuracies are $A_{\mathrm{diff}}=A_1-\lambda A_2.$1 for DINT versus $A_{\mathrm{diff}}=A_1-\lambda A_2.$2 for DIFF and $A_{\mathrm{diff}}=A_1-\lambda A_2.$3 for Transformer (Cang et al., 29 Jan 2025).

5. Importing differential attention into pretrained models

A substantial difficulty for the original architecture is that it demands large-scale training from scratch. Later work therefore asked whether the empirical benefits of differential attention could be imported into pretrained Transformers with lightweight modifications rather than full retraining (Kong et al., 22 May 2025).

The most direct answer is DEX. Instead of recomputing two full attention maps in the query–key circuit, DEX reuses the pretrained softmax attention scores and applies a differential operator to the output value stream: $A_{\mathrm{diff}}=A_1-\lambda A_2.$4 where $A_{\mathrm{diff}}=A_1-\lambda A_2.$5 is a lightweight linear projection. DEX is applied only to a subset of heads, selected either by low importance or by high entropy, and training updates only $A_{\mathrm{diff}}=A_1-\lambda A_2.$6 for those heads, plus $A_{\mathrm{diff}}=A_1-\lambda A_2.$7 and $A_{\mathrm{diff}}=A_1-\lambda A_2.$8; the rest of the model remains frozen. Reported results show that entropy-based head selection on about 50% of heads yields the best gains, with +4.0 points average on Llama-3B, Needle-in-a-Haystack accuracy rising from 66.9% to 78.3%, and throughput on Llama-3B remaining within 2–3% of the original model, whereas the full Diff circuit is 30–50% slower and uses substantially more memory (Kong et al., 22 May 2025).

DiffLoRA explores a different parameter-efficient path: low-rank adapters are inserted into the positive and negative branches of differential attention. In the reported one-epoch protocol, DiffLoRA uses rank 64 on the negative branch or rank 32 on both branches, with learning rate $A_{\mathrm{diff}}=A_1-\lambda A_2.$9, batch size 64, and max length 4096 tokens. Its empirical profile is mixed. DiffLoRA-32 improves HumanEval by +11 points over LoRA, but it is 7 points below LoRA on DROP, averages approximately 0.520 on five-task RAG-QA versus approximately 0.601 for LoRA, and can devolve into very short or meaningless generations; the Group-Norm variant catastrophically fails with average approximately 0.05. The paper’s interpretation is that small adapter budgets and single-epoch tuning are insufficient to carve out a new attention manifold, and that most observed mass shifts likely arise from data distribution rather than the differential architecture itself (Misrahi et al., 31 Jul 2025).

Taken together, these results indicate that the core benefits of differential attention can be partially transferred to pretrained models, but the degree of transfer depends strongly on where the differential operator is inserted and how much adaptation capacity is available. DEX reports consistent gains with minimal adaptation data, whereas DiffLoRA reports domain-specific improvements but weaker average transfer (Kong et al., 22 May 2025, Misrahi et al., 31 Jul 2025).

6. Limitations, misconceptions, and research directions

The original paper explicitly reports a throughput cost: Diff pays a 6–12% slowdown in FlashAttention throughput, although it also notes that FlashAttention-3 or a specialized CUDA kernel could reduce that penalty. It also limits its own empirical scope to decoder-only language modeling and lists encoder-decoder, vision-language, and mixture-of-experts settings as open extensions. The same source identifies unresolved issues around $\lambda$0 and its per-layer reparameterization, and notes that differential cancellation presumes common-mode noise; adversarial or highly structured noise may require alternate formulations such as more than two sub-spaces (Ye et al., 2024).

A second limitation is structural. DINT argues that pure differential attention lacks global context modeling and violates strict row normalization, motivating its differential-integral reformulation. Shared DIFF, in turn, argues that independent signal generation in DIFF Transformer produces parameter redundancy and suboptimal utilization of information, motivating shared base projections with low-rank updates. These critiques do not reject the differential-attention idea itself; rather, they reposition the original formulation as one point in a broader design space that trades off noise suppression, global context capture, numerical stability, and parameter efficiency (Cang et al., 29 Jan 2025, Cang et al., 29 Jan 2025).

A recurring misconception is terminological. “Diff Transformer” in this literature denotes a differential-attention architecture for sequence modeling, not a diffusion-model denoiser. Diffusion Transformers, often abbreviated DiT, are a separate family of models used in image, video, audio, and other generative settings, where a transformer predicts denoising updates across diffusion timesteps. The names are similar, but the mathematical role of “Diff” is different: in Diff Transformer it refers to a difference of attention maps, whereas in DiT it refers to diffusion-based generative modeling (Ye et al., 2024, Amin et al., 14 Nov 2025).

Open directions already identified in the literature include adaptive low-rank update rank per layer or head, extension of shared differential attention to cross-attention, dynamic bases that evolve per sequence, and analytical characterization of the trade-off between base capacity and update rank. Another active direction is exploiting the resulting sparsity or signed selectivity for more efficient kernels, since the original paper suggests that future work could use block-sparse or index-sparse implementations, and later pretrained-model work highlights the continuing tension between architectural fidelity and deployment efficiency (Cang et al., 29 Jan 2025, Ye et al., 2024, Kong et al., 22 May 2025).