Nemotron 3 Ultra: Hybrid MoE Model

• Nemotron 3 Ultra is NVIDIA’s largest Nemotron 3 model, combining a hybrid Mixture-of-Experts and Mamba-Attention architecture to deliver 550B total and 55B active parameters with a 1M-token context.
• The model leverages advanced methods such as LatentMoE, NVFP4 pre-training, and Multi-Token Prediction, optimizing inference throughput and boosting performance on agentic reasoning tasks.
• An open release strategy with full checkpoints, training recipes, and reproducible code on Hugging Face and NVIDIA NeMo distinguishes Nemotron 3 Ultra from related models like Llama-Nemotron Ultra.

Searching arXiv for the cited Nemotron 3 Ultra papers to ground the article and verify the latest technical report. arxiv_search query: "Nemotron 3 Ultra Open Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning" Nemotron 3 Ultra is NVIDIA’s largest Nemotron 3 model: a Mixture-of-Experts Hybrid Mamba-Attention LLM with 550 billion total parameters and 55 billion active parameters per token, a 1M-token context length, and a post-training stack built for agentic reasoning, tool use, and controllable reasoning effort. The Nemotron 3 family white paper introduced Ultra as the top tier of the Nano–Super–Ultra line and identified the core technologies associated with the family—LatentMoE, Multi-Token Prediction, NVFP4 pre-training, multi-environment reinforcement learning, and reasoning budget control—while the later dedicated Ultra report specified the released Ultra configuration, training pipeline, inference benchmarks, and open-source artifacts (NVIDIA et al., 24 Dec 2025, NVIDIA et al., 12 Jun 2026).

1. Family position and model identity

Within the Nemotron 3 family, Ultra is the largest variant, following Nano and Super. The family paper characterizes Nano as optimized for cost-efficient inference, Super as optimized for collaborative agents and high-volume workloads such as IT ticket automation, and Ultra as the model that “provides state-of-the-art accuracy and reasoning performance” (NVIDIA et al., 24 Dec 2025). The dedicated Ultra report then makes the scale explicit, describing Nemotron 3 Ultra as a 550 billion total and 55 billion active parameter Mixture-of-Experts Hybrid Mamba-Attention LLM intended for long-running autonomous agentic tasks (NVIDIA et al., 12 Jun 2026).

Two documentary layers are relevant to its public description. The 2025 family white paper presents architectural principles and relative scale-ups across Nano, Super, and Ultra, and explicitly notes that where the paper did not specify Ultra-specific hyperparameters, only the principles and relative scale-ups are given (NVIDIA et al., 24 Dec 2025). The 2026 Ultra report provides the full released model specification, including parameter counts, pre-training curriculum, long-context extension, post-training stages, throughput measurements, and checkpoint names (NVIDIA et al., 12 Jun 2026). Read together, these documents define Ultra both as the capstone model of the Nemotron 3 family and as a concrete open model release.

2. Hybrid Mamba–Attention and LatentMoE design

Nemotron 3 Ultra uses a hybrid architecture in which sparse state-space modeling, sparse expert routing, and attention are combined rather than treated as mutually exclusive design choices. The family white paper describes the common Nemotron 3 pattern as a Hybrid Mamba–Transformer MoE architecture in which most layers are Mamba-2 blocks, interleaved with sparse Mixture-of-Experts blocks, plus a handful of GQA self-attention layers with 2 KV heads; approximately 90% of layers are Mamba-2 or MoE blocks and approximately 10% are GQA self-attention blocks (NVIDIA et al., 24 Dec 2025). The later Ultra report describes the released model as a Hybrid “Mamba-Attention” stack that interleaves sparse state-space layers (“Mamba”) with low-dimensional global anchors and local multi-head self-attention, together with LatentMoE layers that route each token through only $k=22$ of $E=512$ experts (NVIDIA et al., 12 Jun 2026).

The core MoE routing mechanism is sparse top-$k$ dispatch. In the family formulation, for hidden state $x \in \mathbb{R}^d$, gate weights are

$$g_i(x)=\frac{\exp(w_i^T x)}{\sum_{j=1}^N \exp(w_j^T x)},$$

after which the model selects the top-$K$ experts and computes

$$\mathrm{MoE}(x)=\sum_{i\in\mathcal{T}(x)} g_i(x)\,E_i(x),$$

followed by a standard residual and LayerNorm (NVIDIA et al., 24 Dec 2025). The Ultra report presents the same principle in tokenwise form: the router computes $g=\mathrm{softmax}(W_1 s_t)\in\mathbb{R}^E$, selects the top-$k$ experts, and aggregates

$$y_t=\sum_{j=1}^k g_{i_j}\cdot \mathrm{Expert}_{i_j}(s_t),$$

so that expert compute is incurred only along the selected paths and the active compute remains proportional to 55B rather than 550B parameters (NVIDIA et al., 12 Jun 2026).

A distinctive architectural feature of Super and Ultra is LatentMoE. Instead of routing in the full hidden dimension $E=512$0, LatentMoE projects into a smaller latent space $E=512$1, performs gating and expert computation there, then projects back:

$E=512$2

Because the routed dimension is $E=512$3 rather than $E=512$4, routed weight loads and all-to-all traffic shrink by $E=512$5, and Nemotron 3 re-invests those savings to scale both the number of experts and the number of selected experts:

$E=512$6

In the 8B-active-parameter ablation reported in the family paper, $E=512$7 and $E=512$8, so $E=512$9 becomes $k$0, with downstream gains from 48.30 to 52.87 on MMLU-Pro, from 70.10 to 72.11 on MMLU, from 51.95 to 55.14 on Code, from 78.32 to 80.19 on Math, and from 81.73 to 82.10 on Commonsense (NVIDIA et al., 24 Dec 2025). Ultra is stated to leverage exactly this LatentMoE design at larger scale.

The attention sub-layer in the Ultra report remains conventional where attention is used:

$k$1

This placement of attention inside a predominantly Mamba-and-MoE stack is central to the model’s stated balance of throughput, context length, and routing fidelity (NVIDIA et al., 12 Jun 2026).

3. Pre-training, NVFP4, and the 1M-token context extension

The released Ultra report states that Nemotron 3 Ultra Base was trained on $k$2 text tokens using an NVFP4 quantized recipe, then extended to a context length of 1,048,576 tokens by a continuous pre-training phase (NVIDIA et al., 12 Jun 2026). The family white paper, by contrast, describes Super and Ultra at the family level as using NVFP4 and states that Ultra is “pretrained to 25 T tokens” (NVIDIA et al., 24 Dec 2025). The documents therefore preserve both a family-level description and a later Ultra-specific training account.

The NVFP4 recipe is one of the defining systems contributions attached to Super and Ultra. The family paper specifies block quantization over every 16-element micro-block with an E4M3 scale factor and an E2M1 element format, plus one global FP32 scale layer. It also specifies Random Hadamard Transforms on wgrad inputs, stochastic rounding on gradients, and retention of the final 15% of the network in BF16 for stability: specifically QKV and attention projections in BF16, Mamba output projections in MXFP8, and latent projections and MTP in BF16. Across scales, the reported train/valid loss gaps are below 1% for Nano and below 0.6% at 8B, while downstream metrics track BF16 baselines (NVIDIA et al., 24 Dec 2025). The Ultra report restates the recipe at the released-model level as “most weights and activations in the 32 A→4 W (E2M1) format with blockwise quantization, stochastic rounding on gradients, and higher precision on attention and MTP heads” (NVIDIA et al., 12 Jun 2026).

The released pre-training curriculum is two-phase. Phase 1 covers 15T tokens of diverse web crawl plus synthetic data; Phase 2 covers 5T tokens of high-quality filtered datasets, including PDFs, code, legal, and QA. The learning-rate schedule warms up for 200B tokens to $k$3, stays stable at peak until 15T tokens, then decays proportionally to $k$4 to $k$5 by 20T tokens (NVIDIA et al., 12 Jun 2026).

Long-context extension is described differently in the two sources. The family white paper states that Ultra inherits the same long-context recipe as Nano: No RoPE; continued pretraining on 512K sequences; SFT at 256K; an RL environment up to 32K; and synthetic data focusing on multi-document retrieval and multi-hop reasoning. It also explains the scaling argument for throughput, namely that Mamba-2 layers carry only a constant state, so per-token inference cost remains $k$6 and does not grow with context length (NVIDIA et al., 24 Dec 2025). The released Ultra report specifies a continuous pre-training phase to 1,048,576 tokens using rotary embeddings with frequency scaling applied to $k$7 and $k$8, a data mixture of 46% long-document QA and 54% Phase 2 SFT-style data, and a schedule in which 92% of steps use 1M tokens and 8% use 4K tokens, with no mixing of sequence lengths within an iteration (NVIDIA et al., 12 Jun 2026).

This long-context stage is attached to a concrete positional mechanism in the released report. For each dimension pair $k$9,

$x \in \mathbb{R}^d$0

The family paper’s empirical long-context evidence is reported for Nano rather than Ultra: on RULER, Nemotron 3 Nano-30B (hybrid) scores 74.48 at 128K, 71.67 at 256K, 66.02 at 512K, and 54.19 at 1M, compared with a dense hybrid baseline that drops from 85.13 to 23.43 over the same range (NVIDIA et al., 24 Dec 2025). For Ultra, the significance is architectural rather than benchmark-specific: the hybrid Mamba-Attention design and sparse routing are presented as the mechanism enabling 1M-token operation with high throughput (NVIDIA et al., 12 Jun 2026).

4. Multi-Token Prediction and post-training for agentic reasoning

Super and Ultra include Multi-Token Prediction (MTP) layers near the output. In the family paper, standard language-model prediction at position $x \in \mathbb{R}^d$1 is

$x \in \mathbb{R}^d$2

and MTP adds an auxiliary projection

$x \in \mathbb{R}^d$3

to predict the next $x \in \mathbb{R}^d$4 tokens in one forward pass, with loss

$x \in \mathbb{R}^d$5

On an 8B-active MoE base, the reported gains include 70.06 to 71.26 on MMLU, 45.05 to 47.84 on MMLU-Pro, 65.58 to 66.89 on MBPP-Sanitized, 86.43 to 88.05 on ARC-Challenge, 74.59 to 75.45 on WinoGrande, 84.02 to 85.36 on RACE, and 82.49 to 84.46 on GSM8K, summarized in the paper as an approximately 2.4% absolute average gain across the task set (NVIDIA et al., 24 Dec 2025). The Ultra report adds that the released system uses two identical MTP heads, retains an auxiliary weight $x \in \mathbb{R}^d$6, and exposes a native speculative-decoding path via a 7-token draft head (NVIDIA et al., 12 Jun 2026).

The released Ultra post-training pipeline has three stages: Supervised Fine-Tuning, Reinforcement Learning, and Multi-teacher On-Policy Distillation. In SFT, Stage 1 uses packed sequences of 294,912 tokens, batch size 64, 204,800 samples, and a cosine learning-rate schedule from a peak of $x \in \mathbb{R}^d$7 to $x \in \mathbb{R}^d$8. Stage 2 uses packed sequences of 515,000 tokens, including 512K long-context SFT data, batch size 64, 19,200 samples, and a peak learning rate of $x \in \mathbb{R}^d$9 decaying to $$g_i(x)=\frac{\exp(w_i^T x)}{\sum_{j=1}^N \exp(w_j^T x)},$$0. The data blend includes tool use, search trajectories, terminal use, code, math/proofs, safety, multilingual examples, and reasoning-truncated samples used to teach budget control, while the MTP objective with $$g_i(x)=\frac{\exp(w_i^T x)}{\sum_{j=1}^N \exp(w_j^T x)},$$1 is retained throughout (NVIDIA et al., 12 Jun 2026).

For RL, the family paper describes a single simultaneous multi-environment run covering code, math, tools, long context, chat, and more, using GRPO with masked importance sampling and an asynchronous actor-learner architecture that decouples rollout generation from gradient updates; it also notes that MTP speeds up multi-step rollouts by predicting multiple tokens per forward pass (NVIDIA et al., 24 Dec 2025). The dedicated Ultra report describes this RL stage more specifically as a unified on-policy RLVR phase over math, code, search, terminal use, office workflows, safety, chat, instruction-following, structured outputs, and long-context QA, trained with an asynchronous PPO-style algorithm (GRPO) with trust-region clipping, global batch size 8192, 16 rollouts per prompt, rollout lengths up to 64K tokens, and multi-environment scheduling (NVIDIA et al., 12 Jun 2026).

The third stage, Multi-teacher On-Policy Distillation (MOPD), is specific to the released Ultra report. Over a diverse set of more than 10 specialized teacher policies spanning software engineering, office/productivity, search, terminal-use, conversational tool use, STEM reasoning, safety, and factuality/instruction, the student generates on-policy rollouts and receives token-level reward signals from each teacher. The domain-weighted reverse-KL objective is

$$g_i(x)=\frac{\exp(w_i^T x)}{\sum_{j=1}^N \exp(w_j^T x)},$$2

The implementation uses asynchronous rollouts, behavior/proximal policy decoupling, and PPO-style clipping on importance ratios, and two iterative MOPD iterations are reported to achieve 67–172% recovery toward teacher performance across agentic and reasoning benchmarks (NVIDIA et al., 12 Jun 2026).

Reasoning-budget control is built into inference rather than added as a separate controller. The family paper states that the user specifies a maximum “thinking” token budget $$g_i(x)=\frac{\exp(w_i^T x)}{\sum_{j=1}^N \exp(w_j^T x)},$$3; when the model reaches $$g_i(x)=\frac{\exp(w_i^T x)}{\sum_{j=1}^N \exp(w_j^T x)},$$4 or sees </think>, it stops chain-of-thought and emits its final answer (NVIDIA et al., 24 Dec 2025). The released report states that three inference modes—“off,” “regular,” and “medium-effort”—were taught during SFT and RL by mixing short versus truncated reasoning samples and adjusting RL rewards; on the Artificial Analysis Intelligence Index V4 tasks, medium-effort mode uses approximately 2.5 times fewer tokens with approximately 7% loss in accuracy, enabling a latency–accuracy trade-off through a single API switch (NVIDIA et al., 12 Jun 2026).

5. Throughput, efficiency, and benchmark positioning

Nemotron 3 Ultra is positioned as a high-throughput sparse model whose active compute resembles a much smaller dense model while retaining a far larger total parameter count. The released report states that per-token active FLOP and memory cost are equivalent to a 55B dense model, because only the top-$$g_i(x)=\frac{\exp(w_i^T x)}{\sum_{j=1}^N \exp(w_j^T x)},$$5 expert paths are active at inference time (NVIDIA et al., 12 Jun 2026). The family paper attributes the throughput profile to the combination of constant-state Mamba layers, sparse routing, chunked all-to-all launches, and bucketing of routed tokens to balance expert loads, and states that Ultra “sustains high throughput even at 1 M tokens” (NVIDIA et al., 24 Dec 2025).

The released inference-stack description is explicit: LatentMoE supplies sparse expert compute, Hybrid Mamba-Attention provides $$g_i(x)=\frac{\exp(w_i^T x)}{\sum_{j=1}^N \exp(w_j^T x)},$$6 KV cache and linear-time SSM decode, and native MTP supplies speculative decoding through the 7-token draft head (NVIDIA et al., 12 Jun 2026). The report further states that prefill is FLOP-bound and decode is I/O-bound, giving the workload-level latency model

$$g_i(x)=\frac{\exp(w_i^T x)}{\sum_{j=1}^N \exp(w_j^T x)},$$7

with tensor parallelism, expert parallelism, and disaggregated prefill/decode replicas tuned per workload (NVIDIA et al., 12 Jun 2026).

The published throughput benchmarks are on GB200 NVL72 at 8K input and 64K output in NVFP4. Under that configuration, the report lists throughput advantages of +5.9× versus GLM-5.1-754B-A40B, +4.8× versus Kimi-K2.6-1T-A32B, and +1.6× versus Qwen-3.5-397B-17B (NVIDIA et al., 12 Jun 2026). The abstract summarizes this as “up to ~6x higher inference throughput as compared to state-of-the-art publicly available LLMs while attaining on-par accuracy” (NVIDIA et al., 12 Jun 2026).

Benchmark positioning is partly family-level and partly Ultra-specific. The family paper states that Ultra delivers state-of-the-art accuracy and chain-of-thought performance across MMLU-Pro, MATH, HumanEval, tool-use benchmarks, and 1M-token tasks, outperforming Super and Nano at comparable inference cost, while also noting that exact Ultra numbers were withheld pending full release (NVIDIA et al., 24 Dec 2025). The released report then restates the model’s profile as state-of-the-art accuracy plus high inference throughput and a 1M-token context length, emphasizing suitability for autonomous agentic tasks rather than a single benchmark specialization (NVIDIA et al., 12 Jun 2026).

6. Open release, reproducibility, and distinction from Llama-Nemotron

The release strategy for Nemotron 3 Ultra combines model checkpoints, data releases, and training code. The family paper states that NVIDIA would openly release the model weights, pre- and post-training software, recipes, and all data for which it holds redistribution rights, under a permissive NVIDIA license and with Apache 2.0 for software; it also gives a roadmap in which Nemotron 3 Ultra was expected in November 2025 after Nano and Super (NVIDIA et al., 24 Dec 2025). The released Ultra report states that the base, post-trained, and quantized checkpoints, along with the training data and recipe, are open-sourced on Hugging Face, and that training recipes and RL environments are in NVIDIA’s NeMo repository (NVIDIA et al., 12 Jun 2026).

Artifact	Identifier or description
Post-trained NVFP4 checkpoint	nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4
Post-trained BF16 checkpoint	nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16
Base BF16 checkpoint	nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-Base-BF16
Generative Reward Model	nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-GenRM
Pretraining data	nvidia/Nemotron-Pretraining-Code-v3, nvidia/Nemotron-Pretraining-Legal-v1, nvidia/Nemotron-Pretraining-Specialized-v1.2, nvidia/Nemotron-Pretraining-Multiple-Choice, nvidia/Nemotron-Pretraining-Generative, nvidia/Nemotron-Pretraining-Fact-Seeking, nvidia/Nemotron-Pretraining-Moral-Scenarios
Post-training data	nvidia/Nemotron-Posttraining-v3, nvidia/Nemotron-RL-Super-Training-Blends

A recurrent point of confusion is the distinction between Nemotron 3 Ultra and Llama-Nemotron Ultra. Llama-Nemotron Ultra, reported in “Llama-Nemotron: Efficient Reasoning Models,” is a separate model line: a 253B-parameter student derived from Meta’s Llama-3.1-405B-Instruct via the Puzzle NAS framework, with 96 Transformer layers, a context window of up to 128K tokens, and a dynamic reasoning toggle implemented by system-token prefixes such as “<system> detailed thinking on” and “<system> detailed thinking off” (Bercovich et al., 2 May 2025). Nemotron 3 Ultra, by contrast, is the 550B total / 55B active Mixture-of-Experts Hybrid Mamba-Attention model with 1M-token context length, LatentMoE, NVFP4 pre-training, MOPD, and the Nemotron 3 release artifacts on Hugging Face (NVIDIA et al., 12 Jun 2026).

The distinction matters conceptually as well as taxonomically. Llama-Nemotron Ultra is a NAS-optimized heterogeneous Transformer aimed at efficient reasoning under the NVIDIA Open Model License, whereas Nemotron 3 Ultra is an open Hybrid Mamba-Attention MoE system whose published emphasis is on the joint optimization of sparse activation, very long context, inference throughput, and agentic post-training (Bercovich et al., 2 May 2025, NVIDIA et al., 12 Jun 2026).