GDN Mixer: Efficient Transformer Decoding

• GDN Mixer is a recurrent linear-time module for transformer decoders that uses a scalar-gated delta rule to achieve constant memory and efficiency.
• It integrates with various attention mechanisms, enabling flexible performance tradeoffs through precise layer placement and surrogate modeling.
• GDN offers significant computational speedup while managing quality loss, making it ideal for high-throughput inference in long-context scenarios.

Gated DeltaNet (GDN) Mixer is a recurrent linear-time mixer module for transformer decoder layers, introduced in the Super Apriel 15B supernet as an alternative to attention-based and other efficient mixers. The GDN mixer achieves $O(n)$ computational complexity and constant-size recurrent memory by utilizing a scalar-gated delta rule, and it is optimized for high-throughput transformer inference, particularly in scenarios requiring flexible tradeoffs between quality and speed. Within Super Apriel, GDN can be assigned per-layer and intermixed with Full Attention (FA), Sliding Window Attention (SWA), and Kimi Delta Attention (KDA), allowing fine-grained performance customization at runtime (Labs et al., 21 Apr 2026).

1. Position Within the Transformer Decoder

In the Super Apriel architecture, each decoder layer contains a "mixer" that can be set to one of four options: FA, SWA, KDA, or GDN. The GDN mixer replaces the conventional attention mechanism. It operates using the following sequence:

• Fused Linear Projection: $W_{\mathrm{in}}\in\mathbb{R}^{d\times(3d_h+d_v+g)}$ generates, per token $t$, the query $\mathbf{q}_t\in\mathbb{R}^{d_h}$, key $\mathbf{k}_t\in\mathbb{R}^{d_k}$, value $\mathbf{v}_t\in\mathbb{R}^{d_v}$, a modulation vector $\mathbf{z}_t\in\mathbb{R}^{d_h}$, and raw gating parameters $$(\alpha_{\text{raw},t}, \beta_{\text{raw},t})\in\mathbb{R}^2$$.
• Causal 1D Convolution (SiLU-activated): Applied to $(\mathbf{q}, \mathbf{k}, \mathbf{v})$ for chunk-parallel training.
• Fixed-Size Recurrent State: $\mathbf{S}_{t-1}\in\mathbb{R}^{d_k\times d_v}$, updated at each step.
• Gated RMSNorm and Output Projection: After state update, an RMSNorm (with gating) and a linear output projection ($W_{\mathrm{in}}\in\mathbb{R}^{d\times(3d_h+d_v+g)}$0) yield the pre-residual output.

This strategy contrasts with FA, which maintains a growing key–value cache and performs matrix–matrix products with quadratic complexity; SWA, which restricts attention to a local window; and KDA, which implements a similar delta rule but with per-channel (vector) gating rather than a scalar.

2. Scalar-Gating Mechanism

At each token position $W_{\mathrm{in}}\in\mathbb{R}^{d\times(3d_h+d_v+g)}$1, the GDN mixer computes two scalar gating coefficients from its raw projections: $W_{\mathrm{in}}\in\mathbb{R}^{d\times(3d_h+d_v+g)}$2 where

$W_{\mathrm{in}}\in\mathbb{R}^{d\times(3d_h+d_v+g)}$3

• $W_{\mathrm{in}}\in\mathbb{R}^{d\times(3d_h+d_v+g)}$4 denotes the logistic sigmoid, so $W_{\mathrm{in}}\in\mathbb{R}^{d\times(3d_h+d_v+g)}$5, functioning as an adaptive learning rate.
• $W_{\mathrm{in}}\in\mathbb{R}^{d\times(3d_h+d_v+g)}$6 is forced negative by construction, yielding $W_{\mathrm{in}}\in\mathbb{R}^{d\times(3d_h+d_v+g)}$7 as a global decay gate.
• $W_{\mathrm{in}}\in\mathbb{R}^{d\times(3d_h+d_v+g)}$8 and $W_{\mathrm{in}}\in\mathbb{R}^{d\times(3d_h+d_v+g)}$9 are learned scalars controlling the decay dynamics.

This scalar gating mechanism provides an adjustable global forget factor ($t$0) and write-rate ($t$1), crucial for managing the hidden state size and information flow.

3. Delta Rule Recurrence

The mixer’s core computation is encapsulated by a delta rule recurrence: $t$2

• The prior state $t$3 is first decayed (shrunk by $t$4) and has its component along $t$5 erased proportional to $t$6.
• The new association $t$7 is written into the state with weight $t$8 (outer-product).
• The $t$9 recurrent state per head ensures constant memory usage regardless of sequence length.

Finally, the output is formed via gated RMSNorm and a linear projection: $\mathbf{q}_t\in\mathbb{R}^{d_h}$0 A standard residual connection and feed-forward block succeed this computation, consistent with baseline transformer architectures.

4. Computational Efficiency and Quality Tradeoffs

Empirical cost and quality comparisons in Super Apriel demonstrate the GDN mixer’s efficiency:

Mixer	State Kind	Relative Cost (FA=1.00)
FA	KV cache	1.00
SWA	Bounded KV	0.48
KDA	Recurrent	0.21
GDN	Recurrent	0.14

Aggressive use of GDN in layer placements leads to the largest speedups, with corresponding quality degradation quantified across recommended presets:

Preset	FA	SWA	KDA	GDN	Speedup	Quality (% teacher)
Reg∣Lklhd‒10	0	10	5	33	10.7×	77
Reg∣Lklhd‒13	0	16	13	19	6.9×	81
Reg∣Lklhd‒26	12	26	6	4	2.9×	96
All-FA	48	0	0	0	1.0×	100

As GDN layers increase from 4 to 33, speedup improves from $\mathbf{q}_t\in\mathbb{R}^{d_h}$1 to $\mathbf{q}_t\in\mathbb{R}^{d_h}$2, with quality retention decreasing from $\mathbf{q}_t\in\mathbb{R}^{d_h}$3 to $\mathbf{q}_t\in\mathbb{R}^{d_h}$4. GDN’s efficiency is especially pronounced for long context lengths, where its throughput advantage compounds (up to $\mathbf{q}_t\in\mathbb{R}^{d_h}$5 additional speedup from 16K to 32K context compared to more modest gains for FA-heavy presets).

5. Placement Optimization and Supernet Usage

Super Apriel enables runtime selection among $\mathbf{q}_t\in\mathbb{R}^{d_h}$6 possible layerwise assignments of its four mixers. The optimal balance of speed and quality is found through a data-driven surrogate-modelling and search procedure:

• Sample placements and estimate their proxy-benchmark quality.
• Encode placements with cluster-expansion features (unary and adjacent-pair indicators).
• Fit a Bayesian linear surrogate with a Normal-Inverse-Gamma prior; learned weights form a Markov Random Field for efficient optimization.
• Using the per-layer cost model, execute a dynamic-programming (Viterbi) search to extract Pareto-optimal placements at each speed/quality budget.
• Iteratively refine with empirical evaluation of high-confidence/uncertainty candidates until the Pareto frontier stabilizes.

This algorithm identifies which layers preferentially adopt GDN under constrained computation; under extreme cost constraints, mid-network layers tend to switch last and can become GDN-like.

6. Practical Implications and Recommendations

GDN’s fixed-size memory ensures decode cost is independent of context. High-throughput deployments, especially those serving long sequences, benefit most from GDN-heavy configurations. Best practices as observed in Super Apriel include:

• Assign GDN preferentially to late layers where long-range dependencies are lesser.
• Under moderate budgets, retain FA or SWA in early/mid layers (e.g., layers 18–28) to preserve solution quality.
• For near-match to teacher quality ($\mathbf{q}_t\in\mathbb{R}^{d_h}$796%), restrict GDN to fewer than 10 layers; for maximal throughput (8–11×), use GDN in more than 30 layers.
• Inference should use one of eight precomputed presets spanning the speed/quality Pareto frontier to avoid runtime search overhead.
• GDN is effective in speculative decoding: an all-GDN draft (constant memory/state) feeds into all-FA as verifier. Since GDN’s probability distribution remains close to FA, this maintains high acceptance and yields overall throughput beyond the fastest non-speculative preset.

Gated DeltaNet constitutes a central mechanism for attaining extreme transformer decoding efficiency in the Super Apriel supernet, balancing quality and performance through scalar-gated recurrence and precise, placement-guided deployment (Labs et al., 21 Apr 2026).