Nemotron-Flash: Hybrid SLM for Low Latency

• Nemotron-Flash is a family of hybrid small language models designed to optimize real-device latency by balancing depth-width ratios and hybridizing state-space with full-attention layers.
• It employs an evolutionary search framework to identify latency-optimal architectures under strict real-world constraints while significantly advancing accuracy and throughput trade-offs.
• The integration of weight normalization enhances training stability and convergence, leading to improved accuracy metrics and faster inference compared to traditional parameter-optimized models.

Nemotron-Flash refers to a family of hybrid small LLMs (SLMs) designed with explicit optimization for real-device latency, rather than solely parameter efficiency. While prior approaches to SLM construction stressed reducing parameter count for accuracy gains under fixed budgets, Nemotron-Flash fundamentally re-examines architectural determinants of real-world latency—specifically, the depth-width ratio and core operator selection—and operationalizes these insights through an evolutionary search over latency-constrained design spaces. The result is a set of hybrid models combining linear-time state space layers with a minimized count of full-attention blocks, extended by a weight normalization scheme to stabilize and accelerate training convergence. Empirical benchmarks show Nemotron-Flash models advancing the accuracy-latency and accuracy-throughput frontiers by significant margins relative to parameter-optimized peers such as Qwen3 (Fu et al., 24 Nov 2025).

1. Architectural Design and Parameterization

Nemotron-Flash models are parameterized in terms of the number of “blocks” $L$ and hidden dimension $d$ within Transformer-based architectures. Each block contains a token-mixing operator (e.g., attention, state-space) and a feed-forward network (FFN). For a standard configuration:

• Each block comprises $3$ linear projections for Q/K/V, an output projection, and a two-layer FFN with intermediate width $4d$.
• The dominant parameter count is $N ≃ L \times (4d^2 + 8d^2) = 12L d^2$, or $O(L d^2)$.

Previous SLM design favored deep-thin models (large $L$, small $d$), yielding optimal parameter efficiency for accuracy at fixed $N$. However, such shapes are suboptimal for real-device latency, as batch-one (“small-batch”) decoding latency scales linearly in $L$ and quadratically in $d$, given GPU kernel launch and matrix multiply considerations. Empirical profiling of (A100/H100) cards demonstrates a latency-minimizing “sweet spot” in the $(L,d)$ plane that is less deep and wider than previously favored schemes.

Nemotron-Flash extends the classical parameter-data scaling law (cf. Hoffmann et al.) via:

$$L_{\text{loss}}(D, W) = L_0 + a D^{-\alpha} + b W^{-\beta} + c N_{\text{data}}^{-\gamma}$$

with loss $L_{\text{loss}}$ predicted from depth ($D$), width ($W$), and data size. For any latency budget $\tau^*$, candidate $D, W$ pairs are profiled, and the loss-minimizing combination is selected within the feasible latency regime, guaranteeing an optimal trade-off of accuracy under real-device constraints.

2. Efficient Attention Mechanisms and Operator Selection

Candidate “token-mixing” primitives considered at a 500M parameter scale include:

• Standard Multi-Head Attention (MHA)
• FlashAttention-2 (I/O-optimized variant, $O(L d^2)$)
• Sliding-Window Attention (SWA; window $w$, $O(L w d)$)
• Mamba, Mamba2 (linear SSM-based, $O(L d)$)
• DeltaNet, Gated DeltaNet (state-space delta rule, $O(L d)$)
• Gated Linear Attention (GLA, $O(L d)$)
• RWKV (RNN-like, $O(L d)$)

The following computational complexities and memory characteristics are outlined:

Operator	Compute	Memory
Full MHA, FlashA	$O(L d^2)$	$O(d^2 + L d^2)$
SWA ($w$ window)	$O(L w d)$	$O(w d)$
SSM (Mamba etc.)	$O(L d)$ (or $O(L d \log d)$)	$O(d)$ (per time step)

Empirical Pareto frontier evaluation indicates DeltaNet and Gated DeltaNet dominate the PPL–latency trade-off among pure models, while hybrids mixing DeltaNet with Mamba2 outperform those combining only full-attention or GLA. Interleaving full-attention with linear-time operators provides enhanced recall and throughput. The final Nemotron-Flash hybrid designs are the product of such interleavings, optimizing both accuracy and step-wise latency (Fu et al., 24 Nov 2025).

3. Evolutionary Search Framework for Latency-Optimal Topology

Nemotron-Flash discovery leverages an evolutionary search supported by aging-evolution with tournament selection. The search objective is:

• Minimize proxy PPL: $\operatorname{proxy\_PPL}(\text{arch})$
• Subject to $\text{latency}(\text{arch}) \leq \tau^*$

Short-training (10B tokens) PPL is used as a proxy for final model performance (yielding $88.8\%$ Spearman correlation). Model architectures are encoded as three “stages” (early/mid/late), each repeating a building-block type (from token-mixing choices and FFN layouts). Mutation occurs by randomly altering operator selection, FFN ratio, or block counts within constraints, and the hidden size is selected post facto to exactly saturate the latency budget.

Aging-evolution iterates over roughly 100 generational cycles, replacing the oldest candidate with a new “child.” The best architecture under decoding-latency constraints (837M parameters) alternates DeltaNet, Mamba2, and full-attention blocks with interleaved FFNs, yielding PPL 20.70 (WikiText), CR Acc 51.04%, and 8k-token batch-one decoding latency of 17.71s on A100. This hybrid systematically outperforms all pure model baselines at matched latency (Fu et al., 24 Nov 2025).

4. Weight Normalization Enhancement for Training Stability

To prevent degenerate growth of weight matrix norms (which cause vanishing relative updates), Nemotron-Flash applies unit-sphere normalization of weight rows or columns after each optimization step, as determined by block connectivity:

• For $W$ mapping hidden to hidden (output not added back), normalize each row:

$W_{i,:} \leftarrow \frac{W_{i,:}}{||W_{i,:}||_2}$

• For $W$ whose outputs are added back to hidden, normalize each column:

$W_{:,j} \leftarrow \frac{W_{:,j}}{||W_{:,j}||_2}$

This is derived from nGPT but omits activation normalization, introducing less than 10% overhead. Integrated with AdamW and cosine learning-rate scheduling (no weight decay), this method leads to smoother weight distribution, stabilized loss curves, and slightly increased gradient magnitudes late in training. Across Llama, DeltaNet, and Mamba2 SLMs trained on 100B tokens, average PPL improves by $0.66$ and CR Acc by $+1.20\%$ (Fu et al., 24 Nov 2025).

5. Empirical Benchmarks and Comparative Evaluation

Key Results

Nemotron-Flash sets new records along key performance axes when compared to Qwen3 baselines:

Model	Params	Latency (8k, BS=1)	Throughput (32k, max BS)	Avg Acc
Qwen3-0.6B	0.6B	27.55s	160 tok/s	44.11%
Nemotron-Flash-1B	0.96B	14.45s	7,289 tok/s	49.63%
Qwen3-1.7B	1.7B	36.20s	157 tok/s	55.47%
Nemotron-Flash-3B	2.7B	28.71s	2,939 tok/s	60.98%

Nemotron-Flash-1B achieves $+5.5$ percentage points accuracy, $1.9\times$ lower latency, and $45.6\times$ throughput over Qwen3-0.6B; Nemotron-Flash-3B yields similar gains over Qwen3-1.7B.

Framework and Latency Profiling

Comparison of decoding latency (seconds) for 8k–32k token lengths across frameworks (A100):

Framework	6	64	256	1,024	8,192	32,768
PyTorch	0.16	0.96	3.64	14.28	113.93	457.72
vLLM	0.02	0.14	0.55	2.16	17.82	169.74
TensorRT-LLM	0.17	0.30	0.64	1.81	13.65	60.41
Ours (hybrid)	0.01	0.14	0.54	2.17	17.54	71.65

Throughput also scales by up to $2$–$3\times$ over full-attention SLMs when including Mamba2/DeltaNet for large batches.

6. Synthesis and Implications

Nemotron-Flash demonstrates that explicit co-optimization of model depth/width for latency, hybridization of quadratic (attention) and linear (state-space) token-mixing operators, and a minimalistic weight normalization step are collectively necessary and sufficient to advance the accuracy–latency frontier for SLMs. In edge and cloud inference scenarios with small-batch or tail-latency constraints, these models yield substantial improvements in both speed and accuracy.

The evolutionary architecture search framework and weight normalization technique employed are generalizable, suggesting their applicability to future SLM families under intensifying deployment constraints. A plausible implication is that future SLM development will increasingly separate architecture choices for accuracy from those for latency, with hybrid operator compositions playing a central role in practical model deployments (Fu et al., 24 Nov 2025).