Nemotron 3 Nano 30B-A3B: Hybrid MoE Language Model

• Nemotron 3 Nano 30B-A3B is a 31.6B-parameter hybrid Mixture-of-Experts language model featuring 3.2B active parameters per forward pass for efficient agentic reasoning.
• It employs a hybrid Mamba-Transformer architecture with 52 layers, integrating few self-attention layers and multiple MoE layers to optimize long-context processing.
• The model is pretrained on 25 trillion tokens and further refined with supervised fine-tuning, reinforcement learning, and quantization to enhance both reasoning performance and deployment efficiency.

Nemotron 3 Nano 30B-A3B is NVIDIA’s open-weight LLM in the Nano line, described as an efficient Mixture-of-Experts hybrid Mamba-Transformer model for agentic reasoning. The technical report gives the model as a 31.6B-parameter sparse MoE system with 3.2B active parameters per forward pass, 3.6B active parameters including embeddings, pretraining on 25 trillion text tokens, and support for context lengths up to 1M tokens; it was released in both pretrained Base and post-trained forms, with later work also studying FP8 and NVFP4 deployment variants (NVIDIA et al., 23 Dec 2025).

1. Identity, naming, and release status

The model appears in the literature under closely related names: Nemotron 3 Nano 30B-A3B, Nemotron-3-Nano-30B-A3B, and Nemotron-3-Nano-30B-A3B-Base. The technical report distinguishes the pretrained checkpoint from the post-trained checkpoint, while subsequent papers often use the base model as a backbone for further adaptation or evaluation (NVIDIA et al., 23 Dec 2025).

The suffix “30B-A3B” is not explicitly defined in prose in the technical report, but the report states enough to make the intended interpretation clear: the model is at roughly the 30B total-parameter scale, while A3B denotes about 3B activated parameters per token or forward pass. Later papers often shorten this to “30B-parameter” and “3B active parameters”, especially when comparing MoE systems of similar scale (NVIDIA et al., 23 Dec 2025, Kozachok et al., 21 Jun 2026).

Core specification reported in the technical report is summarized below (NVIDIA et al., 23 Dec 2025).

Aspect	Reported value
Total parameters	31.6B
Active parameters per forward pass	3.2B
Active parameters including embeddings	3.6B
Context length	up to 1M tokens
Pretraining budget	25 trillion text tokens

The release record in the technical report includes Nemotron 3 Nano 30B-A3B BF16, Nemotron 3 Nano 30B-A3B FP8, and Nemotron 3 Nano 30B-A3B Base BF16. A later quantization report additionally names the NVFP4 checkpoint NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4, indicating that the model family was subsequently extended into lower-precision inference formats rather than remaining BF16-only (NVIDIA et al., 23 Dec 2025, Xin et al., 27 Jan 2026).

2. Architectural design

Nemotron 3 Nano 30B-A3B is described as a hybrid Mamba-Transformer backbone plus Mixture-of-Experts FFN replacement. The technical report lists the following architectural hyperparameters: 52 layers, hidden size 2688, 32 query heads, 2 KV heads, head dimension 128, Mamba state dimension 128, Mamba groups 8, Mamba heads 64, Mamba head dimension 64, expert dimension 1856, 128 routable experts, 6 activated experts, and 2 shared experts. It further specifies Squared ReLU inside the experts, a standard learned MLP router with sigmoid gating, RMSNorm, no positional embeddings, no dropout, no bias on linear layers, and untied embedding and projection weights (NVIDIA et al., 23 Dec 2025).

The same report emphasizes that only 6 self-attention layers exist, so the model is heavily biased toward state-space computation. The family white paper adds that Nano retains only “a select few” attention layers, uses GQA, and assigns 2 KV heads per attention layer. This combination is central to the model’s deployment profile: GQA reduces KV-cache growth, while Mamba-2 carries most sequence modeling burden outside the few attention layers (NVIDIA et al., 23 Dec 2025, NVIDIA et al., 24 Dec 2025).

Later papers describe the same backbone with a more explicit hybrid-block decomposition. The Nemotron-TwoTower paper treats Nemotron-3-Nano-30B-A3B as a 52-layer model containing 23 Mamba-2 layers, 6 self-attention layers, and 23 mixture-of-experts layers, and the low-resource routing study describes the base BF16 model as a hybrid Mamba-2/Transformer MoE with 23 MoE layers, 23 Mamba-2 layers, 6 attention layers using grouped-query attention, and 128 routed experts plus 1 shared expert per MoE layer, with top-5 routed experts plus the always-on shared expert active for each token (Reda et al., 25 Jun 2026, Joseph et al., 17 May 2026).

These descriptions are not identical. The base-model technical report lists 128 routable experts, 6 activated experts, and 2 shared experts, whereas the routing study describes 128 routed experts plus 1 shared expert and 6 experts active in total. Likewise, one report infers that the non-attention remainder of the 52-layer stack is Mamba-heavy, while later papers count Mamba and MoE layers separately. This suggests differing counting conventions for hybrid blocks or expert substructures rather than a fully reconciled standalone specification, and later adaptation papers explicitly defer several internals—such as tokenizer details, routing equations, expert counts under alternative accounting, and full context-handling internals—to the Nemotron-3 report (NVIDIA et al., 23 Dec 2025, Joseph et al., 17 May 2026, Reda et al., 25 Jun 2026).

3. Pretraining corpus and long-context extension

The pretrained base model was trained on 25 trillion text tokens, including more than 3 trillion new unique tokens over Nemotron 2. Pretraining is organized into Phase 1 with 23.5T tokens of diverse data and Phase 2 with 1.5T tokens of higher-quality data. The corpus spans 15 categories, with web crawl data split into five quality buckets and additional components including math, Wikipedia, code, nemotron-cc-code, academic text, Crawl++, multilingual, and synthetic SFT-style data grouped as general-sft, stem-sft, and code-sft (NVIDIA et al., 23 Dec 2025).

The multilingual pretraining mix covers 19 languages: Arabic, Chinese, Czech, Danish, Dutch, Finnish, French, German, Hebrew, Hindi, Italian, Japanese, Korean, Portuguese, Polish, Russian, Spanish, Swedish, and Thai. The report also highlights several new corpora relative to Nemotron 2, including Nemotron-CC-Code-v1 at 427.92B tokens, Nemotron-CC-v2.1 with 2.1T new tokens from rephrased medium-high-quality Common Crawl data and over 2.5T new tokens from Common Crawl data overall, and Nemotron-Pretraining-Specialized-v1, whose RQA component alone contains 4.3M demonstrations and 31.7B unique tokens (NVIDIA et al., 23 Dec 2025).

Optimization follows Warmup-Stable-Decay with AdamW, weight decay 0.1, $\beta_1 = 0.9$, $\beta_2 = 0.95$, sequence length 8192, batch size 3072, and roughly 25M tokens per batch. The warmup lasts 8.4B tokens, the peak learning rate is $10^{-3}$, the stable phase covers 20T tokens, the decay phase covers 5T tokens, and the minimum learning rate is $10^{-5}$ (NVIDIA et al., 23 Dec 2025).

Long-context capability is not treated as a purely architectural by-product. The report adds a dedicated LC-Phase with 121B tokens, trained on H100 GPUs with 8-way context parallelism, 8-way tensor parallelism, 8-way expert parallelism, and 4-way pipeline parallelism. Its data blend consists of 20% long-context document QA data, 1% synthetic retrieval-focused data, and 79% downscaled Phase 2 data. The authors report that training only on 524,288-token (512K) batches hurt short-context benchmarks slightly, whereas mixing 512K and 4K sequences improved both short-context and long-context performance (NVIDIA et al., 23 Dec 2025).

A related family report places Nemotron Nano 3 in a broader long-context recipe with continued pre-training at 512k, supervised fine-tuning at 256k, and an RL long-context environment with inputs up to 32k tokens, while also stating that the models are designed to support up to 1M token context and that Nano’s attention layers do not use RoPE (NVIDIA et al., 24 Dec 2025). Taken together, these reports portray long-context support as a combined consequence of the Mamba-heavy hybrid design, low-attention-depth stack, GQA, and explicit long-context training.

4. Post-training, reasoning control, and precision adaptation

Post-training combines supervised fine-tuning (SFT), multi-environment reinforcement learning from verifiable rewards (RLVR), and reinforcement learning from human feedback (RLHF). The SFT corpus spans over 18M total samples across competition math, competition code, conversational tool use, long context, formal proofs, multilingual data, terminal use, general chat, instruction following, safety, software engineering, science, GenSelect-style selection reasoning, and CUDA code generation. Long-context SFT examples have mean length 128K and maximum 256K (NVIDIA et al., 23 Dec 2025).

The chat template explicitly supports reasoning on/off, reasoning budget control, and tool-integrated reasoning. To teach those controls, 10% of samples have reasoning traces stripped and 3% of traces are randomly truncated. The SFT run itself uses 13,000 steps, batch size 64, sequence packing to 256K, learning rate $5 \cdot 10^{-5}$, 800 warmup steps, and a sequence-level MoE load-balancing regularizer with coefficient $10^{-4}$ (NVIDIA et al., 23 Dec 2025). The family white paper adds a concrete interface detail: when a specified maximum budget for internal thinking is reached, one can append </think> and let the model continue from the partial trace (NVIDIA et al., 24 Dec 2025).

The RLVR stage is unusually broad. It includes competition math (17K DAPO tasks and 104K SkyWorks math tasks), competition coding (22K tasks), difficult STEM multiple-choice QA (135K tasks), structured output / JSON schema adherence (9K tasks), instruction following (46K IF-style tasks and 3K subtle multi-turn tasks), long-context multi-document synthesis QA (12K tasks), and two agentic tool-use environments, including a workplace assistant with 5 databases, 26 tools, and 690 tasks. Training uses synchronous GRPO with masked importance sampling, 128 prompts per step, 16 generations per prompt, effective batch size 2048, maximum generation length 49K, overlong filtering, and frozen MoE router weights while expert bias continues to update under aux-loss-free balancing (NVIDIA et al., 23 Dec 2025).

RLHF is mediated by a generative reward model trained from Qwen3-235B-A22B-Thinking-2507. The policy-side RLHF stage uses 128 prompts and 16 responses per prompt, together with a circular comparison strategy to reduce pairwise reward-model comparisons from quadratic to linear complexity. A distinctive addition is Group Relative Length Control, which rewards shorter reasoning and answer components relative to other candidates for the same prompt; the report states that this reduced verbosity by 30% during training without sacrificing accuracy (NVIDIA et al., 23 Dec 2025).

Precision adaptation is an important secondary thread in the literature. The technical report’s deployment recipe uses post-training quantization (PTQ) to FP8, keeping all 6 self-attention layers, the 6 Mamba layers immediately preceding those attention layers, and Conv1D in all Mamba layers in BF16; it reports approximately 99% median accuracy recovery relative to BF16 (NVIDIA et al., 23 Dec 2025). A later quantization study then examines NVFP4 recovery for the post-trained model, arguing that the model is post-trained with multi-stage RL, that PTQ alone causes losses, that QAT can break RL-acquired behavior, and that QAD recovers performance to near-BF16 using the original BF16 model as teacher, KL divergence, temperature $T=1$, learning rate $1\mathrm{e}{-5}$, and about 2.5B tokens (Xin et al., 27 Jan 2026).

5. Evaluation profile, throughput, and benchmark positioning

The released Base checkpoint is already strong on math, code, and long context. Against Qwen3-30B-A3B-Base, the technical report gives MMLU-Pro 65.05 vs 61.71, AGIEval-En 68.32 vs 63.12, HumanEval 78.05 vs 70.73, MBPP-Sanitized 75.49 vs 73.15, GSM8K 92.34 vs 89.01, MATH 82.88 vs 61.14, MATH-500 avg@32 78.63 vs 55.08, RULER 64K 87.50 vs 63.55, and RULER 128K 82.92 vs 60.69; MMLU itself is a counterexample, where Qwen is reported higher at 81.07 vs 78.56 (NVIDIA et al., 23 Dec 2025).

The final post-trained model is positioned as a reasoning- and agent-oriented open model rather than a uniform win on every benchmark. Reported post-training numbers include MMLU-Pro 78.30, AIME25 89.06 without tools and 99.17 with tools, GPQA 73.04 without tools and 75.00 with tools, LiveCodeBench 68.25, SciCode 33.28, MiniF2F pass@1 50.03 and pass@32 79.92, SWE-Bench (OpenHands) 38.76, TauBench V2 average 49.04, BFCL v4 53.76, IFBench 71.51, Arena-Hard-V2 Average 67.65, and WMT24++ en→xx 86.20 (NVIDIA et al., 23 Dec 2025).

Long-context evaluation is particularly central to the model’s public identity. The post-trained report gives RULER-100 @ 256K = 92.92, RULER-100 @ 512K = 91.25, and RULER-100 @ 1M = 86.34, all above the corresponding Qwen3 values reported in the same table. The same section also records AA-LCR = 35.85, where Qwen is higher at 59.00. This pattern recurs elsewhere: the model is very strong on long retrieval/stability-style evaluation but not uniformly best on every long-context benchmark (NVIDIA et al., 23 Dec 2025).

Throughput is another headline result. In an 8K input / 16K output scenario on a single H200 GPU, the technical report states 2.2× faster inference throughput than GPT-OSS-20B and 3.3× faster inference throughput than Qwen3-30B-A3B-Thinking-2507, using the better of vLLM and TRT-LLM for each model. The Nemotron 3 white paper repeats the 3.3× higher throughput than Qwen3-30B-A3B figure for the same 8k input / 16k output workload and adds that the speedups increase further for longer sequences (NVIDIA et al., 23 Dec 2025, NVIDIA et al., 24 Dec 2025).

The model’s benchmark profile therefore resists a simple “best overall” characterization. It is stronger than many similarly sized open models on math, theorem proving, long-context retrieval, agentic tool-use interfaces, and several chat-alignment tasks, but it is weaker than Qwen on MMLU-ProX avg and AA-LCR, and it does not dominate GPT-OSS on every reasoning or agentic metric. This mix is consistent with the training recipe: the system is optimized for high-throughput reasoning, tool use, and long-context deployment rather than for a single homogeneous leaderboard target (NVIDIA et al., 23 Dec 2025).

6. Later adaptations, external evaluations, and unresolved specification issues

Subsequent papers largely treat Nemotron 3 Nano 30B-A3B as a backbone rather than reintroducing it from first principles. The most direct example is Nemotron-TwoTower, which instantiates a two-tower diffusion LLM by taking two copies of the pretrained Nemotron-3-Nano-30B-A3B checkpoint: a frozen causal context tower and a trainable denoiser tower. In that setting the adapted system is reported to retain 98.7% of the autoregressive baseline’s quality while offering 2.42× higher wall-clock generation throughput, with lower confidence thresholds pushing throughput beyond 3× at larger quality loss (Reda et al., 25 Jun 2026).

The backbone also underlies later descendants. Nemotron-Cascade-2-30B-A3B is explicitly described as based on Nemotron-3-Nano-30B-A3B-Base and shows what the architecture can support under a more elaborate cascade RL pipeline; Nemotron 3 Nano Omni is described as being built on the highly efficient Nemotron 3 Nano 30B-A3B backbone, then extended to text, images, video, and audio with modality-specific encoders and projectors (Yang et al., 19 Mar 2026, NVIDIA et al., 27 Apr 2026). In the Omni paper, the text-only comparison shows the multimodal system remains close to the standalone LLM on many text benchmarks, reinforcing the role of the 30B-A3B model as the central reasoning substrate (NVIDIA et al., 27 Apr 2026).

Independent evaluations further clarify both strengths and failure modes. In Text2DSL, Nemotron-3-Nano-30B-A3B is one of two MoE models evaluated on PolkitBench; with structured prompt context, syntax validity rises from 97.6% to 98.6%, structure validity from 88.9% to 98.6%, and CodeBLEU from 0.518 to 0.829, yet the model still produces 1,075 invalid action.id instances affecting 908 of 4,147 structurally valid outputs (21.9%). The authors interpret this as evidence that prompt-time grounding helps substantially but does not eliminate closed-vocabulary hallucination (Kozachok et al., 21 Jun 2026).

A separate multilingual-routing study uses Nemotron-3-Nano-30B-A3B as the hybrid-architecture MoE case study for low-resource behavior. There the base model exhibits deep-layer routing collapse for Hebrew, while balanced bilingual adaptation reorganizes routing toward overwhelmingly shared experts: mean Hebrew-English cosine similarity rises from 0.818 in the base model to 0.957 after CPT, and average Hebrew benchmark score improves from 65.61 to 68.00 (Joseph et al., 17 May 2026). This does not redefine the base model’s architecture, but it establishes that MoE routing behavior in Nemotron 3 Nano is analyzable as an independent object and remains plastic under further adaptation.

Compression and quantization work has likewise used the model as a hard test case. A block-removal study applies constrained binary optimization to NVIDIA-Nemotron-3-Nano-30B-A3B-FP8, restricting removal to MoE blocks only and showing that removing 2 or 3 out of 23 MoE blocks retains 94% (91%) of original GPQA accuracy and 88% (74%) of original AIME25 accuracy, respectively, without retraining (Jansen et al., 29 Jan 2026). The NVFP4 study, by contrast, argues that for the RL-heavy post-trained model QAT is actively harmful, whereas QAD brings metrics such as AA-LCR, AIME25, GPQA-D, and SciCode back near BF16 (Xin et al., 27 Jan 2026).

One persistent interpretive issue is that many later papers only partially specify the base model. The TwoTower paper states explicitly that from that paper alone one cannot recover hidden size, number of experts, active parameters per token, tokenizer details, context length, routing formula, or the full pretraining setup, because these are outside its scope and deferred to the Nemotron-3 report (Reda et al., 25 Jun 2026). More broadly, the literature uses slightly different conventions for active-parameter accounting, layer decomposition, and shared-expert description. For that reason, the most defensible encyclopedic reading is that Nemotron 3 Nano 30B-A3B is a 30B-class open-weight sparse hybrid backbone whose stable, well-attested properties are its Mamba-heavy MoE design, approximately 3B active compute footprint, 25T-token pretraining, 1M-token context support, strong reasoning/agentic orientation, and repeated reuse as a foundation for subsequent diffusion, multimodal, RL, quantization, pruning, and multilingual-routing studies (NVIDIA et al., 23 Dec 2025).