Super Apriel: Adaptive Multi-Mixer Supernet

• Super Apriel is a 15B-parameter supernet that uses four mixer types per decoder layer to dynamically balance inference speed and quality.
• It incorporates a surrogate model with Bayesian linear regression and Viterbi-style dynamic programming to optimize layerwise mixer placements.
• The system supports flexible serving and speculative decoding, eliminating the need for separate draft models for different speed regimes.

Super Apriel is a 15B-parameter supernet designed to expose multiple inference-speed regimes from a single checkpoint by replacing the conventional single self-attention block in each decoder layer with four trained alternatives: Full Attention (FA), Sliding Window Attention (SWA), Kimi Delta Attention (KDA), and Gated DeltaNet (GDN). A layer-wise assignment of these mixers, termed a placement, determines the active computation path for a request and can be switched at serving time without reloading weights. The model is trained by stochastic distillation from a frozen Apriel 1.6 teacher and then supervised fine-tuned, yielding a single weight set that spans several points on the speed–quality Pareto frontier while also supporting speculative decoding without a separate draft model (Labs et al., 21 Apr 2026). Within the broader Apriel line, Apriel-1.5-15B-Thinker provides family context as a distinct 15B multimodal reasoning model that emphasized depth upscaling, staged continual pre-training, and high-quality supervised fine-tuning, but it does not itself instantiate the multi-mixer supernet mechanism that defines Super Apriel (Radhakrishna et al., 1 Oct 2025).

1. Definition and position within the Apriel family

Super Apriel is characterized by the slogan “one checkpoint, many speeds”: a single supernet stores all mixer variants, while each inference request activates exactly one mixer per decoder layer. This separates parameter storage from active execution: the checkpoint stores approximately 25B parameters, but any concrete placement uses only approximately 15B of them (Labs et al., 21 Apr 2026).

This design should be distinguished from ordinary checkpoint families in which different speed tiers require separate models, separate weight loading, or independent draft and target networks for speculative decoding. In Super Apriel, the serving system changes the per-layer mixer routing instead of changing the model weights. A plausible implication is that deployment concerns shift from model version management toward placement selection, cache behavior, and routing policy.

Apriel-1.5-15B-Thinker is relevant chiefly as lineage context. It described a 15B multimodal model with a 48-layer decoder obtained by depth upscaling from Pixtral-12B, a staged continual pre-training curriculum, and large-scale supervised fine-tuning, and it positioned those ingredients as a blueprint for later “Super Apriel” directions (Radhakrishna et al., 1 Oct 2025). Super Apriel, however, is technically a different object: not merely a 15B reasoning model, but a structured supernet over a combinatorial family of decoder-layer mixer assignments.

2. Supernet architecture and mixer taxonomy

Super Apriel is built on a 48-layer autoregressive decoder with hidden dimension $d = 5120$, grouped-query attention with $h_q = 32$ query heads and $h_{kv} = 8$ KV heads, per-head dimension $d_h = 128$, an FFN of intermediate width 14,336 with SiLU gating, vocabulary 131,072, and a context window of 262,400 in training and 32,768 in serving (Labs et al., 21 Apr 2026). Every decoder layer contains four parallel mixer modules, and a placement chooses one of them per layer.

The four mixer types differ primarily in asymptotic decode cost and state representation:

Mixer	Mechanism	Complexity / state
FA	Full softmax attention	$O(n^2)$, $O(n \cdot d)$ KV cache
SWA	Sliding-window attention	$O(n \cdot w)$, $O(w \cdot d)$ bounded KV cache
KDA	Kimi Delta Attention	$O(n)$, recurrent state $32 \times 128 \times 128$
GDN	Gated DeltaNet	$h_q = 32$0, recurrent state $h_q = 32$1

For FA, the step-$h_q = 32$2 computation is

$h_q = 32$3

$h_q = 32$4

SWA retains the same form but restricts attention to the most recent $h_q = 32$5 positions:

$h_q = 32$6

The recurrent mixers replace the unbounded KV cache with structured per-layer state. GDN maintains $h_q = 32$7 and updates it as

$h_q = 32$8

$h_q = 32$9

$h_{kv} = 8$0

KDA extends this with channelwise gates $h_{kv} = 8$1:

$h_{kv} = 8$2

$h_{kv} = 8$3

$h_{kv} = 8$4

$h_{kv} = 8$5

Although KDA and GDN introduce recurrent state of order $h_{kv} = 8$6, the additional parameters for each recurrent mixer remain below 1% of total parameters. This is central to the Super Apriel premise: large decode-time speed changes are obtained through mixer substitution and state-form changes rather than gross parameter-count changes.

3. Placements, presets, and the speed–quality frontier

A placement is the 48-tuple

$h_{kv} = 8$7

The configuration space therefore contains $h_{kv} = 8$8 possible placements (Labs et al., 21 Apr 2026). A preset is a curated placement chosen from the Pareto frontier for a particular speed target.

Quality is measured by a score $h_{kv} = 8$9 defined as averaged normalized log-likelihood or exact-match accuracy on the dev set $d_h = 128$0 or on all tasks. Cost is modeled additively:

$d_h = 128$1

with regression-fit per-layer costs at 16K tokens

$d_h = 128$2

Decode throughput speedup is then $d_h = 128$3 relative to the all-FA baseline, measured at 32K contexts.

Representative presets are:

Preset	Speedup @32K	Quality retention
all-FA	1.0×	100% (74.2 vs 73.9)
Reg\	Lklhd -26	2.9×
Reg\	Lklhd -18	4.8×
Reg\	Lklhd -13	6.9×
Reg\	Lklhd -10	10.7×

The all-FA preset matches the Apriel 1.6 teacher on all reported benchmarks. The recommended hybrid presets span $d_h = 128$4 to $d_h = 128$5 decode throughput at 96% to 77% quality retention. The main-text description reports a nearly linear degradation of quality as speedup increases and notes that throughput advantages compound at longer context lengths (Labs et al., 21 Apr 2026).

Two interpretive points are important. First, the frontier is not obtained by a monotone “replace attention from top to bottom” heuristic; it arises from explicit search over heterogeneous layer assignments. Second, the existence of high-quality presets with relatively few FA layers suggests that much of the model’s effective long-range behavior can be preserved even when a substantial portion of the stack uses bounded-window or recurrent mixers, though the exact layerwise allocation remains an empirical optimization problem rather than a fixed architectural rule.

4. Surrogate modeling and placement search

Because exhaustive evaluation over $d_h = 128$6 placements is infeasible, Super Apriel introduces a placement optimization toolkit centered on a surrogate model described as a “cluster expansion” (Labs et al., 21 Apr 2026). The score is approximated by truncated local interactions:

$d_h = 128$7

Here $d_h = 128$8 is a per-layer unary potential and $d_h = 128$9 is a pairwise potential. For $O(n^2)$0, mixer vocabulary size $O(n^2)$1, and interaction range $O(n^2)$2, the resulting representation has 948 features.

The statistical model is Bayesian linear regression over indicator features $O(n^2)$3:

$O(n^2)$4

fit by Ridge regression as a MAP estimator with a conjugate Normal-Inv-Gamma prior. The predictive mean $O(n^2)$5 estimates placement quality, while $O(n^2)$6 provides uncertainty. The fitted coefficients also serve as Markov-random-field-style potentials.

The additive cost model and the bounded interaction range make exact constrained search tractable:

$O(n^2)$7

Because the potentials decompose over contiguous windows of size at most $O(n^2)$8, the optimization can be solved by dynamic programming, specifically Viterbi-style DP, in

$O(n^2)$9

A single forward pass finds the best placement at every cost level.

The search loop is adaptive. It begins with approximately 1000 cost-stratified placements evaluated on the supernet, fits the surrogate, generates candidates by DP at each cost level, and scores them using either

$O(n \cdot d)$0

for exploitation or

$O(n \cdot d)$1

for exploration. Top candidates from each bucket are evaluated, added to the dataset, and the model is refit for four rounds. This procedure makes the speed–quality landscape operationally searchable rather than purely combinatorial. A plausible implication is that Super Apriel turns decoder design into a cost-constrained structured prediction problem over layerwise operator choices.

5. Distillation and supervised fine-tuning

Training proceeds in two major stages. In Stage S1, the supernet is initialized from a frozen Apriel 1.6 model: shared FFNs, norms, and embeddings remain fixed, while all four mixer modules per layer are trained (Labs et al., 21 Apr 2026). During each forward step, the placement is sampled stochastically with independent uniform layer choices,

$O(n \cdot d)$2

The composite loss is

$O(n \cdot d)$3

where $O(n \cdot d)$4 is MSE on layer-boundary activations, $O(n \cdot d)$5 is reverse KL, and $O(n \cdot d)$6 is forward KL. The weights are $O(n \cdot d)$7 in Stage 1, with $O(n \cdot d)$8 added in Stage 2. The training corpus comprises 266B tokens: 197B from a 1.5 teacher to warm up GDN and KDA, followed by 69B from the 1.6 teacher. The mixture includes reasoning traces, code, STEM, web text, and multimodal data. Optimization uses AdamW with learning rate $O(n \cdot d)$9, batch size $O(n \cdot w)$0 tokens, a ZeRO-3 to ZeRO-2 schedule, and approximately 70K H100-GPU hours.

In Stage S2, the shared parameters remain frozen and only the mixer weights are retrained on instruction-tuning data using next-token cross-entropy. The sequence length is 32,768, the batch is $O(n \cdot w)$1 tokens, and AdamW again uses learning rate $O(n \cdot w)$2. The paper distinguishes four placement-sampling policies: local sampling, global sampling, targeted sampling, and hybrid sampling. Reported campaigns include full supernet SFT with 137B tokens using global sampling over all placements, single-placement SFT with 40B tokens, and targeted 8-preset SFT with 60B tokens over the seven Pareto presets plus all-FA (Labs et al., 21 Apr 2026).

Within the broader Apriel trajectory, this training recipe departs from Apriel-1.5-15B-Thinker. The latter emphasized depth upscaling, projection-neck realignment, two-stage continual pre-training, and text-only supervised fine-tuning over explicit reasoning traces, whereas Super Apriel freezes shared backbone components and concentrates learning on interchangeable mixer modules (Radhakrishna et al., 1 Oct 2025). This suggests that Super Apriel treats architectural flexibility itself as a learnable object.

6. Serving, speculative decoding, and methodological cautions

At inference time, a single GPU-resident supernet loads all mixer weights while shared parameters are common across placements. Switching presets consists of changing which mixer block is called at each layer and then recapturing CUDA graphs, reported as a 5–15 second step (Labs et al., 21 Apr 2026). The vLLM integration adds a per-request placement_id, enforces same-placement batching, and dispatches the corresponding pre-captured graph. This is the operational mechanism behind the “one checkpoint, many speeds” claim.

The same checkpoint also supports speculative decoding without a separate draft model. If $O(n \cdot w)$3 denotes an efficient preset and $O(n \cdot w)$4 the all-FA target preset, then for speculative horizon $O(n \cdot w)$5 the acceptance rate is defined from the expected number of accepted tokens:

$O(n \cdot w)$6

A separate linear surrogate for $O(n \cdot w)$7 is used to find a draft placement maximizing the approximate speculative speedup

$O(n \cdot w)$8

Empirically, with $O(n \cdot w)$9, the cheapest draft placement, all-GDN, gives the highest speculative speedup of approximately $O(w \cdot d)$0 over all-FA on math tasks (Labs et al., 21 Apr 2026).

Several methodological cautions are explicit. The paper investigates whether the best configurations at each speed level can be identified early in training or only after convergence. Rankings stabilize quickly at 0.5B scale, but the most efficient configurations are more unstable at 15B, which the authors present as a caution against extrapolating from smaller models. This directly constrains one possible misconception: small-scale placement ranking is not guaranteed to preserve the ordering of the fastest high-efficiency presets at full scale. Another common misunderstanding is that the model is “25B at inference”; in fact, approximately 25B parameters are stored, but each placement activates only approximately 15B. A further misconception is that speculative decoding still requires a separately trained draft network; Super Apriel explicitly removes that requirement by sourcing both target and draft behaviors from the same supernet.

Taken together, Super Apriel defines a model family whose primary object is not a single fixed decoder graph but a trained placement space over multiple mixer types. The central contribution is therefore architectural and systems-oriented at once: a supernet formulation, a tractable search procedure over an otherwise intractable configuration space, and a serving stack that turns those searched placements into request-time speed presets from a single checkpoint (Labs et al., 21 Apr 2026).