---
title: 'Nemotron 3 Nano: Hybrid MoE LLM'
url: https://www.emergentmind.com/topics/nemotron-3-nano
type: topic
---

# Nemotron 3 Nano: Hybrid MoE LLM

Nemotron 3 Nano is a 30-billion-parameter large language model employing a Mixture-of-Experts (MoE) hybrid Mamba-Transformer architecture, developed by NVIDIA for high-throughput, cost-efficient inference with advanced reasoning, agentic behavior, and long-context support. As the foundational model for recent agentic, code, and multimodal systems, it sets a Pareto frontier for active parameter efficiency and hardware utilization in the open-weight LLM ecosystem [2512.20848][2512.20856][2601.20088][2604.24954][2606.26493].

## 1. Model Architecture and MoE Hybridization

Nemotron 3 Nano integrates Mamba-2 state-space layers with standard Transformer self-attention layers and sparse MoE feed-forward networks in an interleaved stack. Its typical backbone consists of 52 layers: 23 Mamba-2 layers, 6 attention layers (often grouped), and 23 MoE layers for a total of approximately 30–31.6 billion parameters, of which ~3.6 billion are "active" per token due to MoE sparsity [2512.20848][2606.26493]. The key architectural principles include:

- **Mamba-2 layers:** State-space models providing constant-memory O($dL$) recurrences for long-range dependencies without resorting to rotary encodings or classic position embeddings.
- **Self-attention blocks:** Grouped-query attention (GQA) with e.g., 32 (query) × 2 (key/value) heads and head dimension ~128.
- **MoE FFN Blocks:** Gating routers select top-$k$ out of $E$ experts per token (e.g., $k=6$, $E=128$ typical for Nano 30B-A3B), using softmax gating and squared-ReLU or other custom nonlinearities.

All primitive types share a hidden dimensionality $d$ in the 2–4k range (e.g., $d=2688$–$4096$), with MoE expert FFNs having local dimensions up to ~16k. The MoE blocks use GShard-style load balancing loss with DeepSeek-inspired aux-loss-free regularization to prevent expert collapse. The model's parameter count is summarized in the table:

| Layer Type                 | Count        | Hidden Dim | Experts/Block | Active/tok |
|----------------------------|-------------|------------|--------------|------------|
| Transformer/self-attention | 6–8         | 4096       | N/A          | Full       |
| Mamba-2 SSM                | 20–30       | 4096       | N/A          | Full       |
| MoE FFN                    | ~20–30      | 4096/1856  | 128          | 6          |

This hybridization enables efficient context extension and expert capacity [2512.20848][2606.26493].

## 2. Training Regimen: Pretraining, SFT, and RL

The Nemotron 3 Nano training pipeline involves three distinct phases:

- **Massive pretraining:** Up to 25 trillion tokens, with 4T+ CommonCrawl, 2T code, over 3T new or specialized tokens (including math and synthetic agentic dialogues). The cross-entropy objective is augmented with a load-balance loss for MoE utilization [2512.20848][2512.20856][2606.26493].
- **Supervised fine-tuning (SFT):** Over 18 million examples covering math, chain-of-thought (CoT), code, multilinguality, tool use, proofs, and synthetic agentic tasks. SFT data incorporate both CoT-augmented and "truncated" reasoning traces to support reasoning toggles and budget-aware control. The SFT stage involves context expansion up to 256k tokens and batch sizes of hundreds to several thousand [2512.20848][2512.20856].
- **Multi-environment RL:** A Generative Reward-PPO (GRPO) or RLVR approach using masked importance sampling, across code execution, tool use, hard math/coding, verifiable QA, and conversation. Policy rollout and optimization maintain frozen router weights for MoE stability and append regularization or bonuses (e.g., for chain-of-thought brevity). RLHF is layered atop RLVR via generative reward models and circular pairwise comparisons [2512.20848][2512.20856]. 

For long-context capability, Nemotron 3 Nano employs a curriculum culminating in a "CPT" (context parallel training) phase with up to 1M-token simulated contexts and mixed 4k/512k sequences, leveraging 8-way context parallelism and 8-way expert/tensor parallel scaling.

## 3. Quantization Methodologies: NVFP4 and QAD Recovery

Reducing inference cost is critical; Nemotron 3 Nano employs multi-tier quantization:

- **NVFP4 Quantization:** A 4-bit floating-point format with (sign, 2-bit exponent, 1-bit mantissa), leveraging local per-block FP8 scales ($s_{\rm E4M3}$) and a global FP32 scale ($s_{\rm FP32}$), enabling $\ge2\times$ reduction from BF16 and $2{-}3\times$ higher throughput on NVIDIA Blackwell NVFP4 cores. Quantization is first applied post-training (PTQ) to all weights except a few stabilizing attention/Mamba layers (kept in BF16). Activations are quantized layerwise at runtime [2601.20088].
- **Quantization-Aware Distillation (QAD):** To recover accuracy lost in PTQ and avoid instability in quantization-aware training (QAT), QAD distills a full precision teacher into the NVFP4-quantized student using KL divergence between softmax outputs:
  \[
  \mathcal{L}_{\mathrm{QAD}} = D_{\mathrm{KL}}\bigl(P_T\,\|\,P_S\bigr) = \sum_{y}P_T(y|x) \log\frac{P_T(y|x)}{P_S(y|x)}
  \]
  Only soft logits from the teacher are required, and QAD runs as a single posttraining stage on $\approx2.5$B SFT/RL tokens, robust to data quality and source. QAD recovers $\gtrsim95\%$ of BF16 benchmark accuracy and yields superior performance to QAT—especially for RL-heavy skills [2601.20088].

| Variant          | AA-LCR | AIME25 | GPQA-D | LiveCode-v5 | SciCode |
|------------------|--------|--------|--------|-------------|---------|
| BF16             | 35.9   | 89.1   | 73.0   | 72.1        | 33.0    |
| NVFP4 PTQ        | 31.3   | 85.0   | 71.6   | 68.9        | 30.5    |
| QAD (SFT+RL mix) | 34.3   | 87.9   | 72.7   | 68.9        | 32.3    |
| QAT              | 24.8   | 83.3   | 66.0   | 62.0        | 25.8    |

QAD is thus a best-practice for quantized post-training in large hybrid models [2601.20088].

## 4. Performance Benchmarks and Efficiency Analysis

Nemotron 3 Nano achieves best-in-class open-model performance on reasoning, coding, math, and long-context benchmarks:

- **MMLU-Pro (5-shot):** 78.3%
- **AIME25:** 89.06%
- **LiveCodeBench:** 68.25%
- **MiniF2F pass@1:** 50.03%
- **RULER-100 @1M tokens:** 86.34%

MoE and hybridization yield a 3.3×–3.8× real throughput gain versus dense 30B models (e.g., GPT-OSS-20B, Qwen3-30B-A3B), with context length scaling up to 1M tokens [2512.20848][2512.20856]. Memory and compute costs are $1/4$ to $1/2$ those of FP16/FP8 models under NVFP4. Inference on Blackwell GB300 achieves up to 1.8M tokens/sec per GPU at NVFP4, with total weight storage $<12$GB for 1M-token contexts [2512.20856].

## 5. Advanced Features: Long-Context, Multimodality, and Parallel Generation

Nemotron 3 Nano supports features that distinguish it from previous large models:

- **Long-context handling:** Elimination of rotary position encodings; Mamba state-space recurrences enable native extrapolation to $>$1M context. KV cache and memory scale only with the handful of attention layers, not model depth.
- **Multimodal expansion:** Nemotron 3 Nano Omni, based on the Nano 30B-A3B backbone, integrates vision (ViT), audio (FastConformer), and video into a unified context using patch/fusion, Conv3D temporal compression, and Efficient Video Sampling (EVS) to reduce memory and attention costs. Quantized NVFP4 deployment achieves $<1/3$ original memory with $<1\%$ accuracy loss; real-world throughput is $7$–$9\times$ that of comparably sized models on key multimodal tasks [2604.24954].
- **Diffusion language modeling:** As the frozen context tower in Nemotron-TwoTower, Nano 30B-A3B supports blockwise masked denoising, delivering $2.4\times$ generation throughput for a $<1.5\%$ accuracy drop [2606.26493].

## 6. Use Cases, Model Releases, and Ecosystem

Nemotron 3 Nano is released under a commercially permissive license, enabling its use in research, industry, and collaborative environments [2512.20848][2512.20856]. It is aimed at cost-sensitive large-scale inference, edge/low-memory deployment, agentic research, and bulk long-context applications. Its weights, training code (NeMo, Megatron-LM), and RL+SFT data are available via NVIDIA and Hugging Face releases.

Notable ecosystem developments include:

- Enhanced post-SFT/RL gen reasoning and full dynamic reasoning/collapse via token budget control.
- Open multimodal checkpoints (Nano Omni) supporting text, vision, audio, and video, with public recipes for SFT and RL.
- Benchmark results and sample code for quantized inference and distributed sharded serving on Blackwell, Hopper, and Ampere GPUs.

## 7. Comparative Context and Derivative Models

Nemotron 3 Nano has influenced a range of derivative and adjacent models:

- **Llama-Nemotron family:** Efficient reasoning transformers in $\{8,49,253\}$ billion-parameter classes, integrating post-training and reasoning toggles [2505.00949].
- **Nemotron 3 Super/Ultra:** Larger counterparts employing LatentMoE, more extensive RL pipelines, and MTP layers for ultra-fast text generation [2512.20856].
- **TwoTower diffusion modeling:** Establishing the utility of frozen backbone + lightweight denoiser towers for high-throughput, high-fidelity text generation [2606.26493].

Nemotron 3 Nano thus delineates the current design space for high-efficiency, long-context, RL-robust, quantized LLMs in the open research landscape.

Source: https://www.emergentmind.com/topics/nemotron-3-nano