---
title: 'Nemotron 3 Nano 30B-A3B: Hybrid MoE Language Model'
url: https://www.emergentmind.com/topics/nemotron-3-nano-30b-a3b
type: topic
---

# Nemotron 3 Nano 30B-A3B: Hybrid MoE Language Model

Nemotron 3 Nano 30B-A3B is NVIDIA’s open-weight language model in the Nano line, described as an efficient Mixture-of-Experts hybrid Mamba-Transformer model for agentic reasoning. The technical report gives the model as a 31.6B-parameter sparse MoE system with 3.2B active parameters per forward pass, 3.6B active parameters including embeddings, pretraining on 25 trillion text tokens, and support for context lengths up to 1M tokens; it was released in both pretrained Base and post-trained forms, with later work also studying FP8 and NVFP4 deployment variants [2512.20848].

## 1. Identity, naming, and release status

The model appears in the literature under closely related names: **Nemotron 3 Nano 30B-A3B**, **Nemotron-3-Nano-30B-A3B**, and **Nemotron-3-Nano-30B-A3B-Base**. The technical report distinguishes the pretrained checkpoint from the post-trained checkpoint, while subsequent papers often use the base model as a backbone for further adaptation or evaluation [2512.20848].

The suffix **“30B-A3B”** is not explicitly defined in prose in the technical report, but the report states enough to make the intended interpretation clear: the model is at roughly the 30B total-parameter scale, while **A3B** denotes about 3B activated parameters per token or forward pass. Later papers often shorten this to **“30B-parameter”** and **“3B active parameters”**, especially when comparing MoE systems of similar scale [2512.20848][2606.22586].

Core specification reported in the technical report is summarized below [2512.20848].

| Aspect | Reported value |
|---|---|
| Total parameters | 31.6B |
| Active parameters per forward pass | 3.2B |
| Active parameters including embeddings | 3.6B |
| Context length | up to 1M tokens |
| Pretraining budget | 25 trillion text tokens |

The release record in the technical report includes **Nemotron 3 Nano 30B-A3B BF16**, **Nemotron 3 Nano 30B-A3B FP8**, and **Nemotron 3 Nano 30B-A3B Base BF16**. A later quantization report additionally names the NVFP4 checkpoint **NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4**, indicating that the model family was subsequently extended into lower-precision inference formats rather than remaining BF16-only [2512.20848][2601.20088].

## 2. Architectural design

Nemotron 3 Nano 30B-A3B is described as a **hybrid Mamba-Transformer backbone plus Mixture-of-Experts FFN replacement**. The technical report lists the following architectural hyperparameters: **52 layers**, hidden size **2688**, **32** query heads, **2** KV heads, head dimension **128**, Mamba state dimension **128**, Mamba groups **8**, Mamba heads **64**, Mamba head dimension **64**, expert dimension **1856**, **128** routable experts, **6** activated experts, and **2** shared experts. It further specifies **Squared ReLU** inside the experts, a **standard learned MLP router** with **sigmoid gating**, **RMSNorm**, **no positional embeddings**, **no dropout**, **no bias on linear layers**, and **untied embedding and projection weights** [2512.20848].

The same report emphasizes that only **6 self-attention layers** exist, so the model is heavily biased toward state-space computation. The family white paper adds that Nano retains only “a select few” attention layers, uses **GQA**, and assigns **2 KV heads** per attention layer. This combination is central to the model’s deployment profile: GQA reduces KV-cache growth, while Mamba-2 carries most sequence modeling burden outside the few attention layers [2512.20848][2512.20856].

Later papers describe the same backbone with a more explicit hybrid-block decomposition. The Nemotron-TwoTower paper treats Nemotron-3-Nano-30B-A3B as a **52-layer** model containing **23 Mamba-2 layers**, **6 self-attention layers**, and **23 mixture-of-experts layers**, and the low-resource routing study describes the base BF16 model as a hybrid Mamba-2/Transformer MoE with **23 MoE layers**, **23 Mamba-2 layers**, **6 attention layers using grouped-query attention**, and **128 routed experts plus 1 shared expert** per MoE layer, with **top-5 routed experts plus the always-on shared expert** active for each token [2606.26493][2605.17598].

These descriptions are not identical. The base-model technical report lists **128 routable experts, 6 activated experts, and 2 shared experts**, whereas the routing study describes **128 routed experts plus 1 shared expert** and **6 experts active in total**. Likewise, one report infers that the non-attention remainder of the 52-layer stack is Mamba-heavy, while later papers count Mamba and MoE layers separately. This suggests differing counting conventions for hybrid blocks or expert substructures rather than a fully reconciled standalone specification, and later adaptation papers explicitly defer several internals—such as tokenizer details, routing equations, expert counts under alternative accounting, and full context-handling internals—to the Nemotron-3 report [2512.20848][2605.17598][2606.26493].

## 3. Pretraining corpus and long-context extension

The pretrained base model was trained on **25 trillion text tokens**, including **more than 3 trillion new unique tokens over Nemotron 2**. Pretraining is organized into **Phase 1** with **23.5T tokens** of diverse data and **Phase 2** with **1.5T tokens** of higher-quality data. The corpus spans **15 categories**, with web crawl data split into five quality buckets and additional components including **math**, **Wikipedia**, **code**, **nemotron-cc-code**, **academic text**, **Crawl++**, **multilingual**, and synthetic SFT-style data grouped as **general-sft**, **stem-sft**, and **code-sft** [2512.20848].

The multilingual pretraining mix covers **19 languages**: Arabic, Chinese, Czech, Danish, Dutch, Finnish, French, German, Hebrew, Hindi, Italian, Japanese, Korean, Portuguese, Polish, Russian, Spanish, Swedish, and Thai. The report also highlights several new corpora relative to Nemotron 2, including **Nemotron-CC-Code-v1** at **427.92B tokens**, **Nemotron-CC-v2.1** with **2.1T new tokens** from rephrased medium-high-quality Common Crawl data and **over 2.5T new tokens from Common Crawl data** overall, and **Nemotron-Pretraining-Specialized-v1**, whose **RQA** component alone contains **4.3M demonstrations** and **31.7B unique tokens** [2512.20848].

Optimization follows **Warmup-Stable-Decay** with **AdamW**, weight decay **0.1**, $\beta_1 = 0.9$, $\beta_2 = 0.95$, sequence length **8192**, batch size **3072**, and roughly **25M tokens per batch**. The warmup lasts **8.4B tokens**, the peak learning rate is **\(10^{-3}\)**, the stable phase covers **20T tokens**, the decay phase covers **5T tokens**, and the minimum learning rate is **\(10^{-5}\)** [2512.20848].

Long-context capability is not treated as a purely architectural by-product. The report adds a dedicated **LC-Phase** with **121B tokens**, trained on **H100 GPUs** with **8-way context parallelism**, **8-way tensor parallelism**, **8-way expert parallelism**, and **4-way pipeline parallelism**. Its data blend consists of **20%** long-context document QA data, **1%** synthetic retrieval-focused data, and **79%** downscaled Phase 2 data. The authors report that training only on **524,288-token (512K)** batches hurt short-context benchmarks slightly, whereas mixing **512K** and **4K** sequences improved both short-context and long-context performance [2512.20848].

A related family report places Nemotron Nano 3 in a broader long-context recipe with **continued pre-training at 512k**, **supervised fine-tuning at 256k**, and an RL long-context environment with inputs up to **32k** tokens, while also stating that the models are designed to support **up to 1M token context** and that Nano’s attention layers do **not** use RoPE [2512.20856]. Taken together, these reports portray long-context support as a combined consequence of the Mamba-heavy hybrid design, low-attention-depth stack, GQA, and explicit long-context training.

## 4. Post-training, reasoning control, and precision adaptation

Post-training combines **supervised fine-tuning (SFT)**, **multi-environment reinforcement learning from verifiable rewards (RLVR)**, and **reinforcement learning from human feedback (RLHF)**. The SFT corpus spans **over 18M total samples** across competition math, competition code, conversational tool use, long context, formal proofs, multilingual data, terminal use, general chat, instruction following, safety, software engineering, science, GenSelect-style selection reasoning, and CUDA code generation. Long-context SFT examples have **mean length 128K** and maximum **256K** [2512.20848].

The chat template explicitly supports **reasoning on/off**, **reasoning budget control**, and **tool-integrated reasoning**. To teach those controls, **10%** of samples have reasoning traces stripped and **3%** of traces are randomly truncated. The SFT run itself uses **13,000 steps**, batch size **64**, sequence packing to **256K**, learning rate **\(5 \cdot 10^{-5}\)**, **800 warmup steps**, and a sequence-level MoE load-balancing regularizer with coefficient **\(10^{-4}\)** [2512.20848]. The family white paper adds a concrete interface detail: when a specified maximum budget for internal thinking is reached, one can append `</think>` and let the model continue from the partial trace [2512.20856].

The RLVR stage is unusually broad. It includes competition math (**17K** DAPO tasks and **104K** SkyWorks math tasks), competition coding (**22K** tasks), difficult STEM multiple-choice QA (**135K** tasks), structured output / JSON schema adherence (**9K** tasks), instruction following (**46K** IF-style tasks and **3K** subtle multi-turn tasks), long-context multi-document synthesis QA (**12K** tasks), and two agentic tool-use environments, including a workplace assistant with **5 databases**, **26 tools**, and **690 tasks**. Training uses **synchronous GRPO** with **masked importance sampling**, **128 prompts per step**, **16 generations per prompt**, effective batch size **2048**, maximum generation length **49K**, overlong filtering, and frozen MoE router weights while expert bias continues to update under aux-loss-free balancing [2512.20848].

RLHF is mediated by a **generative reward model** trained from **Qwen3-235B-A22B-Thinking-2507**. The policy-side RLHF stage uses **128 prompts** and **16 responses per prompt**, together with a **circular comparison strategy** to reduce pairwise reward-model comparisons from quadratic to linear complexity. A distinctive addition is **Group Relative Length Control**, which rewards shorter reasoning and answer components relative to other candidates for the same prompt; the report states that this reduced verbosity by **30% during training without sacrificing accuracy** [2512.20848].

Precision adaptation is an important secondary thread in the literature. The technical report’s deployment recipe uses **post-training quantization (PTQ)** to **FP8**, keeping all **6 self-attention layers**, the **6 Mamba layers immediately preceding those attention layers**, and Conv1D in all Mamba layers in **BF16**; it reports approximately **99% median accuracy recovery** relative to BF16 [2512.20848]. A later quantization study then examines **NVFP4** recovery for the post-trained model, arguing that the model is **post-trained with multi-stage RL**, that **PTQ** alone causes losses, that **QAT** can break RL-acquired behavior, and that **QAD** recovers performance to near-BF16 using the original BF16 model as teacher, **KL divergence**, **temperature \(T=1\)**, **learning rate \(1\mathrm{e}{-5}\)**, and about **2.5B tokens** [2601.20088].

## 5. Evaluation profile, throughput, and benchmark positioning

The released **Base** checkpoint is already strong on math, code, and long context. Against **Qwen3-30B-A3B-Base**, the technical report gives **MMLU-Pro 65.05 vs 61.71**, **AGIEval-En 68.32 vs 63.12**, **HumanEval 78.05 vs 70.73**, **MBPP-Sanitized 75.49 vs 73.15**, **GSM8K 92.34 vs 89.01**, **MATH 82.88 vs 61.14**, **MATH-500 avg@32 78.63 vs 55.08**, **RULER 64K 87.50 vs 63.55**, and **RULER 128K 82.92 vs 60.69**; MMLU itself is a counterexample, where Qwen is reported higher at **81.07 vs 78.56** [2512.20848].

The final post-trained model is positioned as a reasoning- and agent-oriented open model rather than a uniform win on every benchmark. Reported post-training numbers include **MMLU-Pro 78.30**, **AIME25 89.06** without tools and **99.17** with tools, **GPQA 73.04** without tools and **75.00** with tools, **LiveCodeBench 68.25**, **SciCode 33.28**, **MiniF2F pass@1 50.03** and **pass@32 79.92**, **SWE-Bench (OpenHands) 38.76**, **TauBench V2 average 49.04**, **BFCL v4 53.76**, **IFBench 71.51**, **Arena-Hard-V2 Average 67.65**, and **WMT24++ en→xx 86.20** [2512.20848].

Long-context evaluation is particularly central to the model’s public identity. The post-trained report gives **RULER-100 @ 256K = 92.92**, **RULER-100 @ 512K = 91.25**, and **RULER-100 @ 1M = 86.34**, all above the corresponding Qwen3 values reported in the same table. The same section also records **AA-LCR = 35.85**, where Qwen is higher at **59.00**. This pattern recurs elsewhere: the model is very strong on long retrieval/stability-style evaluation but not uniformly best on every long-context benchmark [2512.20848].

Throughput is another headline result. In an **8K input / 16K output** scenario on a **single H200 GPU**, the technical report states **2.2×** faster inference throughput than **GPT-OSS-20B** and **3.3×** faster inference throughput than **Qwen3-30B-A3B-Thinking-2507**, using the better of **vLLM** and **TRT-LLM** for each model. The Nemotron 3 white paper repeats the **3.3× higher throughput than Qwen3-30B-A3B** figure for the same **8k input / 16k output** workload and adds that the speedups increase further for longer sequences [2512.20848][2512.20856].

The model’s benchmark profile therefore resists a simple “best overall” characterization. It is stronger than many similarly sized open models on math, theorem proving, long-context retrieval, agentic tool-use interfaces, and several chat-alignment tasks, but it is weaker than Qwen on **MMLU-ProX avg** and **AA-LCR**, and it does not dominate GPT-OSS on every reasoning or agentic metric. This mix is consistent with the training recipe: the system is optimized for high-throughput reasoning, tool use, and long-context deployment rather than for a single homogeneous leaderboard target [2512.20848].

## 6. Later adaptations, external evaluations, and unresolved specification issues

Subsequent papers largely treat Nemotron 3 Nano 30B-A3B as a **backbone** rather than reintroducing it from first principles. The most direct example is **Nemotron-TwoTower**, which instantiates a two-tower diffusion language model by taking **two copies** of the pretrained Nemotron-3-Nano-30B-A3B checkpoint: a **frozen** causal context tower and a **trainable** denoiser tower. In that setting the adapted system is reported to retain **98.7%** of the autoregressive baseline’s quality while offering **2.42×** higher wall-clock generation throughput, with lower confidence thresholds pushing throughput **beyond 3×** at larger quality loss [2606.26493].

The backbone also underlies later descendants. **Nemotron-Cascade-2-30B-A3B** is explicitly described as **based on Nemotron-3-Nano-30B-A3B-Base** and shows what the architecture can support under a more elaborate cascade RL pipeline; **Nemotron 3 Nano Omni** is described as being built on the highly efficient **Nemotron 3 Nano 30B-A3B backbone**, then extended to text, images, video, and audio with modality-specific encoders and projectors [2603.19220][2604.24954]. In the Omni paper, the text-only comparison shows the multimodal system remains close to the standalone language model on many text benchmarks, reinforcing the role of the 30B-A3B model as the central reasoning substrate [2604.24954].

Independent evaluations further clarify both strengths and failure modes. In **Text2DSL**, Nemotron-3-Nano-30B-A3B is one of two MoE models evaluated on **PolkitBench**; with structured prompt context, syntax validity rises from **97.6%** to **98.6%**, structure validity from **88.9%** to **98.6%**, and CodeBLEU from **0.518** to **0.829**, yet the model still produces **1,075** invalid `action.id` instances affecting **908 of 4,147 structurally valid outputs (21.9%)**. The authors interpret this as evidence that prompt-time grounding helps substantially but does not eliminate closed-vocabulary hallucination [2606.22586].

A separate multilingual-routing study uses Nemotron-3-Nano-30B-A3B as the hybrid-architecture MoE case study for low-resource behavior. There the base model exhibits **deep-layer routing collapse** for Hebrew, while balanced bilingual adaptation reorganizes routing toward overwhelmingly shared experts: mean Hebrew-English cosine similarity rises from **0.818** in the base model to **0.957** after CPT, and average Hebrew benchmark score improves from **65.61** to **68.00** [2605.17598]. This does not redefine the base model’s architecture, but it establishes that MoE routing behavior in Nemotron 3 Nano is analyzable as an independent object and remains plastic under further adaptation.

Compression and quantization work has likewise used the model as a hard test case. A block-removal study applies constrained binary optimization to **NVIDIA-Nemotron-3-Nano-30B-A3B-FP8**, restricting removal to **MoE blocks only** and showing that removing **2** or **3** out of **23 MoE blocks** retains **94% (91%)** of original GPQA accuracy and **88% (74%)** of original AIME25 accuracy, respectively, **without retraining** [2602.00161]. The NVFP4 study, by contrast, argues that for the RL-heavy post-trained model **QAT** is actively harmful, whereas **QAD** brings metrics such as **AA-LCR**, **AIME25**, **GPQA-D**, and **SciCode** back near BF16 [2601.20088].

One persistent interpretive issue is that many later papers only partially specify the base model. The TwoTower paper states explicitly that from that paper alone one cannot recover hidden size, number of experts, active parameters per token, tokenizer details, context length, routing formula, or the full pretraining setup, because these are outside its scope and deferred to the Nemotron-3 report [2606.26493]. More broadly, the literature uses slightly different conventions for active-parameter accounting, layer decomposition, and shared-expert description. For that reason, the most defensible encyclopedic reading is that **Nemotron 3 Nano 30B-A3B** is a 30B-class open-weight sparse hybrid backbone whose stable, well-attested properties are its Mamba-heavy MoE design, approximately 3B active compute footprint, 25T-token pretraining, 1M-token context support, strong reasoning/agentic orientation, and repeated reuse as a foundation for subsequent diffusion, multimodal, RL, quantization, pruning, and multilingual-routing studies [2512.20848].

Source: https://www.emergentmind.com/topics/nemotron-3-nano-30b-a3b