--- title: Nemotron-3 Super & Ultra Models url: https://www.emergentmind.com/topics/nemotron-3-super-and-ultra-models type: topic --- # Nemotron-3 Super & Ultra Models The Nemotron-3 Super and Ultra models constitute the two largest instantiations of NVIDIA’s Nemotron-3 family, designed to deliver scalable, high-throughput agentic and reasoning capabilities suitable for large context windows and demanding production workloads. Employing hybrid architectures, low-precision quantization, and advanced reinforcement learning protocols, these models facilitate efficient serving and advanced planning over long input sequences. ## 1. Hybrid Mamba–Transformer and LatentMoE Architecture Nemotron-3 Super and Ultra are architected as hybrid Mixture-of-Experts (MoE) models centered on Mamba-2 sequence-modeling layers. The layer order interleaves: - Mamba-2 layers, which maintain a constant, small fixed internal state and supplant standard feed-forward and self-attention modules with efficient state-space recurrences. - Sparse expert MoE blocks, providing high nonlinear capacity per computational step. - Intermittent full self-attention layers (GQA multi-query heads), allowing global context mixing within the sequence. In the LatentMoE variant employed by both Super and Ultra, input representations $x\in\mathbb{R}^d$ are projected via a learnable $U\in\mathbb{R}^{\ell\times d}$ to a lower-dimensional latent space $z=Ux$, where $\ell\ll d$. All subsequent gating and expert feed-forward processing occur in this reduced latent space, with post-expert outputs $y_\text{lat}$ returned to the full dimension by $V\in\mathbb{R}^{d\times\ell}$: $y=x+V y_\text{lat}$. The reduction in communication and computation, $d/\ell$, enables both a multiplication of expert diversity and increased token-level parallelism. For Super/Ultra, $\ell\approx d/4$ (e.g., $d=4096$, $\ell=1024$), thus quadrupling both total experts and active expert selection per token compared to standard MoEs. Multi-Token Prediction (MTP) blocks are incorporated near the end of the network for speculative decoding: the model outputs $M$ subsequent token distributions per position via a linear head, trained to maximize acceptance rate on batch-1, long-output scenarios. This configuration yields nearly constant memory utilization per token, with throughput scaling linearly as context length increases to 1M tokens. ## 2. Training Procedures and Quantization Both Nemotron-3 Super and Ultra are initialized via large-scale supervised pretraining. The corpus comprises over 10 trillion tokens, integrating filtered web text, curated code, mathematical sources, and extensive long-context documents. Deduplication and curation is performed using NeMo Data Designer, which supports document-level deduplication and long-horizon sequence structuring up to 512K tokens. All major parameters are trained natively in NVFP4 quantization using the NVIDIA Transformer Engine. NVFP4 consists of fused 4-bit floating point GEMMs for forward, backward, and weight update passes. Approximately 85% of model layers (apart from projections and certain attention modules) reside entirely in NVFP4, with key “sensitive” components retained in BF16 or MXFP8. On Blackwell Ultra hardware, NVFP4 achieves $\sim3\times$ peak GEMM throughput vs FP8, with empirical loss difference to BF16 remaining under 1%. After supervised training and SFT, a consolidated multi-environment RLHF-style stage is conducted. Here, actor-learners gather trajectory rollouts concurrently from diverse environments, including mathematical chain-of-thought (GSM8K, MATH), embodied code tasks (APPS, HumanEval++), tool invocation (When2Call), long-context retrieval, and chat agents. Generalized Reward-Penalized Optimization (GRPO) with KL-regularization and masked importance sampling is applied, maximizing: $$ \mathcal{L}_\text{RL}(\theta) = - \mathbb{E}_{\tau\sim\pi_\text{old}} \left[\sum_t A_t \log \pi_\theta(a_t|s_t) \right] + \lambda \mathrm{KL}[\pi_\theta\,||\,\pi_\text{ref}] $$ Here, the advantage $A_t$ incorporates GAE with a reference policy anchor. Empirically, joint RL yields monotonic improvements across reasoned tasks, planning, and agentic skills. ## 3. Performance, Token Throughput, and Scalability Nemotron-3 Super sustains approximately 1.6M tokens/sec on an 8-GPU NVIDIA Blackwell Ultra cluster (batch-8, sequence length 8K in/8K out), compared to 0.45M tokens/sec for a baseline Transformer-MoE of similar scale. Ultra achieves 2.2M tokens/sec under identical conditions, with active parameter count $\sim$70B and total $\sim$400B. For high-throughput workloads (8K+16K), Super demonstrates $4\times$ greater throughput than Qwen3-70B and $1.8\times$ that of Gemma 3-70B. Ultra runs more than $2\times$ Gemma 3 under matched conditions. Context-length support is native up to 1M tokens, with Mamba layers encoding position internally, and attention layers using RoPE only for inputs $\leq32$K tokens. Sliding-window inference and paging enable scaling without rotary or positional clipping, with accuracy on 1M-context benchmarks (RULER) sustaining above 66—compared to collapse below 25 for similarly sized dense hybrids. Super and Ultra permit inference cost per 1M-token query of $<$0.25 GPU-hours and $\approx$0.35 GPU-hours, respectively, on Blackwell Ultra hardware. Batch-1 latencies average 40ms/token (Super) and 50ms/token (Ultra), both under 100ms end-to-end for initial tokens, markedly below dense baselines. ## 4. Quantitative Reasoning and Agentic Capability Benchmarks After RL post-training, core benchmarks include: - MMLU-Pro (5-shot CoT EM): Super $\approx$49%, Ultra $\approx$52% - GSM8K (8-shot): Super $\approx$85%, Ultra $\approx$87% - HumanEval++ (3-shot coding): Super $\approx$65%, Ultra $\approx$69% - Code function calling (APIGen-MT): Super $\approx$84%$\,\mathrm{pass@1}$, Ultra $\approx$88%$\,\mathrm{pass@1}$ - Chat agentic tasks (WildChat, 1M logs): Super wins 65% ($vs.$ GPT-3.5), Ultra wins 74% ($vs.$ GPT-4) - Tool invocation (When2Call): Super selects correct pipeline 91% of the time, Ultra 94% Performance curves position Ultra at or above GPT-4 in chain-of-thought tasks and long-context agentic benchmarks, with Super outperforming comparable open models (Qwen2.5, Gemma 3). ## 5. Practical Deployment and Resource Considerations Nemotron-3 Super and Ultra are architected for agentic, high-volume deployments such as IT ticketing, code review, and long-document reasoning. - Inference cost and scalability are optimized for linear efficiency across $\geq$64 GPU pods, with minimal bandwidth overhead due to LatentMoE’s reduced communication footprint. - Recommended hardware comprises NVIDIA Blackwell Ultra or GB300 platforms, utilizing Transformer Engine, GPUDirect RDMA, and NVSwitch for scale-out. - Software stack integrates NVIDIA Triton with custom NVFP4 kernels, a fused Mamba-2 operator in the backend, and an MTP speculative-decoding scheduler; distributed RL and fine-tuning utilize open-source NeMo-RL and NeMo-Gym. All weight checkpoints, training recipes (data designer, NVFP4 schedules, RL environments), and evaluation harnesses are slated for open release [2512.20856]. ## 6. Comparative and Evolutionary Context Nemotron-3 Super and Ultra extend the architecture class introduced in Llama-Nemotron (Super: 49B, Ultra: 253B) but contrast via their hybrid Mamba–Transformer backbone, LatentMoE block routing, native NVFP4 quantization, and full 1M-token context support [2505.00949]. Whereas early Nemotron variants leveraged heterogeneous attention and FFN block mixtures, Super and Ultra pursue sparse expert parallelism and constant-memory scaling for long context and generative planning tasks. Both model families employ RL optimization protocols (GRPO, RPO, multi-environment RL) and open training pipelines (NeMo, Megatron-LM, vLLM), supporting enterprise deployment with commercially permissive licenses and reproducible recipes. A plausible implication is that the shift to Mamba-State Space MoE and LatentMoE designs is likely to inform future long-context agentic models targeting efficient bandwidth, scalability, and speculatively decoded generation with minimal hardware overhead. ## 7. Codebases, Datasets, and Open Resources Key resources announced for Nemotron-3 Super and Ultra include: - Model weights and checkpoints for both Super and Ultra. - Data-designer toolchain for corpus curation and formatting, including deduplication and sequence structuring. - NVFP4 training schedules and quantization recipes. - RL environment definitions and trajectory sampling infrastructure (NeMo-RL, NeMo-Gym). - MTP scheduling and fast-generation methods in Triton/vLLM backends. - Empirical evaluation scripts and harnesses for benchmark replication. These resources are being released under open-access terms, with datasets for pre- and post-training provided where redistribution rights allow [2512.20856]. --- Nemotron-3 Super and Ultra represent hybrid MoE–Mamba architecture designs with advanced RL post-training, NVFP4 throughput optimization, and native million-token context capacity, establishing a benchmark for agentic, scalable, and commercially deployable open language models. Source: https://www.emergentmind.com/topics/nemotron-3-super-and-ultra-models