---
title: 'GDN Mixer: Efficient Transformer Decoding'
url: https://www.emergentmind.com/topics/gdn-mixer
type: topic
---

# GDN Mixer: Efficient Transformer Decoding

Gated DeltaNet (GDN) Mixer is a recurrent linear-time mixer module for transformer decoder layers, introduced in the Super Apriel 15B supernet as an alternative to attention-based and other efficient mixers. The GDN mixer achieves $O(n)$ computational complexity and constant-size recurrent memory by utilizing a scalar-gated delta rule, and it is optimized for high-throughput transformer inference, particularly in scenarios requiring flexible tradeoffs between quality and speed. Within Super Apriel, GDN can be assigned per-layer and intermixed with Full Attention (FA), Sliding Window Attention (SWA), and Kimi Delta Attention (KDA), allowing fine-grained performance customization at runtime [2604.19877].

## 1. Position Within the Transformer Decoder

In the Super Apriel architecture, each decoder layer contains a "mixer" that can be set to one of four options: FA, SWA, KDA, or GDN. The GDN mixer replaces the conventional attention mechanism. It operates using the following sequence:

- **Fused Linear Projection:** $W_{\mathrm{in}}\in\mathbb{R}^{d\times(3d_h+d_v+g)}$ generates, per token $t$, the query $\mathbf{q}_t\in\mathbb{R}^{d_h}$, key $\mathbf{k}_t\in\mathbb{R}^{d_k}$, value $\mathbf{v}_t\in\mathbb{R}^{d_v}$, a modulation vector $\mathbf{z}_t\in\mathbb{R}^{d_h}$, and raw gating parameters $(\alpha_{\text{raw},t}, \beta_{\text{raw},t})\in\mathbb{R}^2$.
- **Causal 1D Convolution (SiLU-activated):** Applied to $(\mathbf{q}, \mathbf{k}, \mathbf{v})$ for chunk-parallel training.
- **Fixed-Size Recurrent State:** $\mathbf{S}_{t-1}\in\mathbb{R}^{d_k\times d_v}$, updated at each step.
- **Gated RMSNorm and Output Projection:** After state update, an RMSNorm (with gating) and a linear output projection ($W_{\mathrm{out}}$) yield the pre-residual output.

This strategy contrasts with FA, which maintains a growing key–value cache and performs matrix–matrix products with quadratic complexity; SWA, which restricts attention to a local window; and KDA, which implements a similar delta rule but with per-channel (vector) gating rather than a scalar.

## 2. Scalar-Gating Mechanism

At each token position $t$, the GDN mixer computes two scalar gating coefficients from its raw projections:
\[
\beta_t = \sigma\bigl(\beta_{\text{raw},t}\bigr),  \qquad \alpha_t = \exp\!\bigl(g_t\bigr)
\]
where
\[
g_t = -\exp\bigl(A_{\log}\bigr)\;\mathrm{softplus}\bigl(\alpha_{\text{raw},t} + \mathrm{dt\_bias}\bigr)
\]
- $\sigma$ denotes the logistic sigmoid, so $0<\beta_t<1$, functioning as an adaptive learning rate.
- $g_t$ is forced negative by construction, yielding $0 < \alpha_t < 1$ as a global decay gate.
- $A_{\log}$ and $\mathrm{dt\_bias}$ are learned scalars controlling the decay dynamics.

This scalar gating mechanism provides an adjustable global forget factor ($\alpha_t$) and write-rate ($\beta_t$), crucial for managing the hidden state size and information flow.

## 3. Delta Rule Recurrence

The mixer’s core computation is encapsulated by a delta rule recurrence:
\[
\mathbf{S}_t = \alpha_t\Bigl(\mathbf{I} - \beta_t\mathbf{k}_t\mathbf{k}_t^{\!\top}\Bigr)\mathbf{S}_{t-1} + \beta_t\mathbf{k}_t\mathbf{v}_t^{\!\top}
\]
- The prior state $\mathbf{S}_{t-1}$ is first decayed (shrunk by $\alpha_t$) and has its component along $\mathbf{k}_t$ erased proportional to $\beta_t$.
- The new association $\mathbf{k}_t\mathbf{v}_t^{\!\top}$ is written into the state with weight $\beta_t$ (outer-product).
- The $O(1)$ recurrent state per head ensures constant memory usage regardless of sequence length.

Finally, the output is formed via gated RMSNorm and a linear projection:
\[
\tilde{\mathbf{y}_t} = \mathrm{RMSNorm}\bigl(\mathbf{S}_t\bigr) \odot \mathbf{z}_t, \quad 
\mathbf{y}_t = W_{\mathrm{out}}\,\tilde{\mathbf{y}_t}
\]
A standard residual connection and feed-forward block succeed this computation, consistent with baseline transformer architectures.

## 4. Computational Efficiency and Quality Tradeoffs

Empirical cost and quality comparisons in Super Apriel demonstrate the GDN mixer’s efficiency:

| Mixer | State Kind     | Relative Cost (FA=1.00) |
|-------|---------------|-------------------------|
| FA    | KV cache      | 1.00                    |
| SWA   | Bounded KV    | 0.48                    |
| KDA   | Recurrent     | 0.21                    |
| GDN   | Recurrent     | 0.14                    |

Aggressive use of GDN in layer placements leads to the largest speedups, with corresponding quality degradation quantified across recommended presets:

| Preset         | FA | SWA | KDA | GDN | Speedup | Quality (% teacher) |
|----------------|----|-----|-----|-----|---------|---------------------|
| Reg∣Lklhd‒10   | 0  | 10  | 5   | 33  | 10.7×   | 77                  |
| Reg∣Lklhd‒13   | 0  | 16  | 13  | 19  | 6.9×    | 81                  |
| Reg∣Lklhd‒26   | 12 | 26  | 6   | 4   | 2.9×    | 96                  |
| All-FA         | 48 | 0   | 0   | 0   | 1.0×    | 100                 |

As GDN layers increase from 4 to 33, speedup improves from $2.9\times$ to $10.7\times$, with quality retention decreasing from $96\%$ to $77\%$. GDN’s efficiency is especially pronounced for long context lengths, where its throughput advantage compounds (up to $2.6\times$ additional speedup from 16K to 32K context compared to more modest gains for FA-heavy presets).

## 5. Placement Optimization and Supernet Usage

Super Apriel enables runtime selection among $4^{48}$ possible layerwise assignments of its four mixers. The optimal balance of speed and quality is found through a data-driven surrogate-modelling and search procedure:

- Sample placements and estimate their proxy-benchmark quality.
- Encode placements with cluster-expansion features (unary and adjacent-pair indicators).
- Fit a Bayesian linear surrogate with a Normal-Inverse-Gamma prior; learned weights form a Markov Random Field for efficient optimization.
- Using the per-layer cost model, execute a dynamic-programming (Viterbi) search to extract Pareto-optimal placements at each speed/quality budget.
- Iteratively refine with empirical evaluation of high-confidence/uncertainty candidates until the Pareto frontier stabilizes.

This algorithm identifies which layers preferentially adopt GDN under constrained computation; under extreme cost constraints, mid-network layers tend to switch last and can become GDN-like.

## 6. Practical Implications and Recommendations

GDN’s fixed-size memory ensures decode cost is independent of context. High-throughput deployments, especially those serving long sequences, benefit most from GDN-heavy configurations. Best practices as observed in Super Apriel include:

- Assign GDN preferentially to late layers where long-range dependencies are lesser.
- Under moderate budgets, retain FA or SWA in early/mid layers (e.g., layers 18–28) to preserve solution quality.
- For near-match to teacher quality ($\geq$96%), restrict GDN to fewer than 10 layers; for maximal throughput (8–11×), use GDN in more than 30 layers.
- Inference should use one of eight precomputed presets spanning the speed/quality Pareto frontier to avoid runtime search overhead.
- GDN is effective in speculative decoding: an all-GDN draft (constant memory/state) feeds into all-FA as verifier. Since GDN’s probability distribution remains close to FA, this maintains high acceptance and yields overall throughput beyond the fastest non-speculative preset.

Gated DeltaNet constitutes a central mechanism for attaining extreme transformer decoding efficiency in the Super Apriel supernet, balancing quality and performance through scalar-gated recurrence and precise, placement-guided deployment [2604.19877].

Source: https://www.emergentmind.com/topics/gdn-mixer