---
title: 'Nemotron Nano 2: Hybrid Mamba-Transformer'
url: https://www.emergentmind.com/topics/nemotron-nano-2
type: topic
---

# Nemotron Nano 2: Hybrid Mamba-Transformer

Nemotron Nano 2 denotes a series of compact, high-throughput, reasoning-oriented language and vision-language models based on hybrid Mamba-Transformer backbones, pioneered by NVIDIA and collaborators. The models are engineered for computational efficiency and state-of-the-art (SoTA) reasoning accuracy within strict memory and hardware footprints, targeting both text and multimodal domains. Key distinguishing features include extensive adoption of Mamba-2 state-space layers, aggressive quantization (FP8), advanced knowledge distillation, highly-structured model compression, and novel support for reasoning control and deployment flexibility across a unified weight space.

## 1. Hybrid Mamba–Transformer Model Architecture

The core innovation of Nemotron Nano 2 lies in its hybridization of the Transformer architecture with Mamba-2 state-space model (SSM) layers. In the canonical Nano 2 backbone, approximately 92% of standard self-attention layers are replaced by Mamba-2 blocks, resulting in a structure exemplified by 62 layers: 6 multi-head self-attention, 28 Mamba-2, and 28 feed-forward layers [2508.14444]. The Mamba-2 layer architecture partitions the hidden state (e.g., $d_m = 5120$) into grouped subspaces, each processed by parallel, causal 1D convolutions and SSM modules. For a group $g$:
\[
U_g = X W_{x,g}^{\top} \in \mathbb{R}^{B \times S \times d_h}, \quad Z_g = X W_{z,g}^{\top}, \quad C_g = \mathrm{Conv1D}(U_g; w_{dt}) + \mathrm{SSM}_g(Z_g; W_{B,g}, W_{C,g}, W_{dt,g})
\]
A gated RMSNorm (GateNorm) modulates the SSM output, maintaining long-range information propagation while greatly reducing $O(S^2)$ complexity of conventional attention.

The backbone enables scaling to unprecedented context lengths (tested up to $128\,\mathrm{k}$ tokens on \texttt{A10G} 22GB cards) with minimal key-value cache overhead. Architectural pruning and parameter reduction are guided by empirical importance metrics at both layer and neuron (for feed-forward networks) and head (for Mamba groups) levels. FP8 quantization (E4M3) is utilized throughout the majority of training, halving memory consumption compared to bfloat16, with no observed instability [2508.14444].

## 2. Training and Compression Pipeline

Nemotron Nano 2 relies on a multi-stage training regime combining aggressive pretraining, alignment, and compression. The canonical sequence comprises:
- Pretraining a 12B-parameter base model (“Nano-12B-v2-Base”) on 20T tokens in FP8 [2508.14444].
- Multi-stage alignment: supervised fine-tuning (SFT), direct preference optimization (DPO), generalized RPO (GRPO), and RLHF, including data with "think-trace" truncation for reasoning trace calibration.
- Minitron-style compression through importance-based pruning and distillation, reducing the backbone (e.g., from 12B to 9B parameters) while retaining ≳98% SoTA reasoning performance at vastly lower compute/memory.

Pruning is informed by measuring logit sensitivity to candidate layer removals ($\Delta_i = {\mathbb{E}_X} \lVert \text{Logits}_{\text{orig}}(X) - \text{Logits}_{\text{pruned} \to i}(X)\rVert^2 / |V|$). FFN neurons and Mamba heads are similarly ranked and retained to meet target memory budgets, subject to hardware constraints (e.g., 19.66 GiB usable on A10G at 128K context). Distillation replay and checkpoint merging (e.g., 50/50 interpolation for reasoning/chat balance) finalize model tuning [2508.14444].

## 3. Inference, Reasoning Modes, and Deployment

Nemotron Nano 2 is designed for efficient inference and supports dynamic reasoning control. The models expose a "reasoning-on/off" toggle implemented via prompt prefixes or (in larger Nemotron family variants) through dedicated router MLPs that select active subspaces at runtime [2505.00949].

Empirically, the 9B pruned model achieves 156.4 tokens/sec (vs. Qwen3-8B's 24.8) on 8K input/16K output, batch size 8—amounting to a 6.3× throughput advantage for long-range reasoning [2508.14444]. Reasoning accuracy is state-of-the-art: e.g., 91.4% on GSM8K-CoT, 63.6% on MATH Level 5, and 82.2% on RULER@128K. Memory usage for weights and cache is tightly scoped, staying <20 GiB for a 9B model in bf16 (excluding context-dependent activation overhead).

The Llama-Nemotron Nano ("LN-Nano," 8B) variant demonstrates further reasoning efficiency by leveraging neural architecture search (NAS, via the Puzzle system), FFN fusion, and vertical compression. Dynamic mode switching is performed by inserting "detailed thinking on/off" flags in the input; no additional gating network is required [2505.00949].

## 4. Vision-Language and Multimodal Extensions

Nemotron Nano 2 underpins vision-language models such as Nemotron Nano V2 VL (12B), which combine a c-RADIOv2 vision encoder, a two-layer MLP connector, and the hybrid Mamba-Transformer backbone [2511.03929]. These models use both patch-merging and Efficient Video Sampling (EVS) to prune redundant spatial tokens in images and videos, facilitating long-context multimodal inference at high throughput (e.g., up to 85 tokens/sec with 60% EVS on FP8).

Vision–language SFT is applied in multiple stages, involving both frozen and joint training of the connector, encoder, and Mamba-Transformer decoder. The backend maintains >98% of the text-only accuracy after heavy multimodal training. Quantization-aware distillation (QAD) to FP4 further boosts inference speed with negligible accuracy loss [2511.03929].

## 5. Nested Models and Efficient Elasticity

Extending the architecture, "Nemotron Elastic" enables nesting of multiple extractable submodels (“many-in-one”) within a single 12B parent, each optimized for a different deployment budget (e.g., 6B, 9B, 12B), with zero-shot slicing at inference [2511.16664]. This process is managed by a trained router—conditioned via differentiable Gumbel-Softmax sampling over budget vectors for each elastic axis (embedding, Mamba, attention, FFN, depth)—producing binary masks for activation. Zero-shot submodel extraction imposes no additional fine-tuning or retraining overhead.

Elastic training is performed over only 110B tokens, representing a 360× reduction in training cost versus independent models from scratch and a 7× gain over SoTA compression. All models reside in a shared 24 GB bf16 checkpoint (+<1 GB metadata), saving ≈43% deployment memory. Submodels reach or surpass baseline SoTA accuracy: e.g., Elastic-6B achieves 70.61 vs. NanoV2-9B at 75.99 and Elastic-12B at 77.41 on reasoning benchmarks. Extended-context training yields up to +19.8% absolute accuracy improvement for smaller submodels [2511.16664].

## 6. Model Access, Licensing, and Practical Usage

Nemotron Nano 2 and associated family members are released under the NVIDIA Open Model License, supporting broad academic and commercial use. Model checkpoints (e.g., bf16, FP8, FP4) and datasets are hosted on HuggingFace (see references [2508.14444], [2511.03929]). The codebase supports common deep learning frameworks including NeMo, Megatron-LM, vLLM for fast inference, and context-parallel optimization for extremely long sequences.

Python code snippets for model loading, inference (including batch multimodal pipelines and EVS toggling), and quantization settings are included in the official documentation [2511.03929]. The architecture is compatible with state-of-the-art scaling strategies such as pipeline and model parallelism through DeepSpeed or HuggingFace Accelerate.

---

### References

- "NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Model" [2508.14444]
- "Nemotron Elastic: Towards Efficient Many-in-One Reasoning LLMs" [2511.16664]
- "Llama-Nemotron: Efficient Reasoning Models" [2505.00949]
- "NVIDIA Nemotron Nano V2 VL" [2511.03929]

Source: https://www.emergentmind.com/topics/nemotron-nano-2