Llama 3.3 Nemotron Super 49B Overview

• Llama 3.3 Nemotron Super 49B is an open large-scale heterogeneous reasoning model integrating Llama and Nemotron innovations for enterprise-grade performance.
• It employs NAS-driven architectural optimizations, dynamic reasoning toggles, and specialized fine-tuning with RL to enhance reasoning accuracy and inference efficiency.
• LN-Super provides open access assets under a permissive license, ensuring reproducible research and broad industrial deployment.

Llama 3.3 Nemotron Super 49B is an open large-scale heterogeneous reasoning LLM that integrates architectural innovations from both the Llama and Nemotron series, targeting enterprise-grade reasoning, inference efficiency, and broad deployment flexibility. The model, officially designated as "Llama-3.3-Nemotron-Super-49B" or "LN-Super," is a 49B parameter model distinct for its Neural Architecture Search–derived structure and extensive reasoning-focused post-training regime. LN-Super is positioned in the literature as a leading open alternative to proprietary or larger-scale models, offering unique features such as a dynamic reasoning toggle and superior throughput per hardware compared to both earlier Llama and SOTA reasoning models, all released under a commercially-permissive license (Bercovich et al., 2 May 2025).

1. Architectural Foundations and Model Design

Llama 3.3 Nemotron Super 49B is based on the Llama 3.3 70B-Instruct backbone but extensively refactored using a heterogeneous Transformer design. This architecture allows individual layers to diverge from canonical transformer blocks, leveraging Neural Architecture Search (NAS) via the Puzzle framework:

• Block-wise Local Distillation: Each layer may be replaced with a variant block. Variants differ in the inclusion/exclusion of self-attention, with some blocks omitting attention altogether to reduce computation and memory footprint. Feedforward dimensions are variable, with reductions down to 10% of original intermediate size for certain sub-blocks.
• Mixed-Integer Programming Solver: Optimal sequence of block variants is selected (minimizing latency under throughput and memory constraints) for a targeted deployment profile, such as single H100 GPU inference with large token cache support (e.g., up to 300k tokens with FP8).
• FFN Fusion: Parts of the model—primarily in larger "Ultra" variants but also in Super—use feedforward merging (concatenation of FFNs across layers where attention is removed) to further improve latency and parallelism.

The resulting model exhibits a blend of full/self-attention and compressed/efficient layers, distinguishing LN-Super from prior dense or MoE transformer baselines.

2. Pretraining Corpus and Knowledge Transfer

LN-Super exploits both original Llama 3.3 data curation and Nemotron-CC corpus practices. Training utilizes tens of trillions of tokens, with particular attention to:

• Source corpus scale: The Nemotron-CC curation pipeline emphasizes unique, high-quality language data via classifier ensembling and model-based rather than heuristic filtering, delivering >4× unique real tokens over comparable datasets (Su et al., 3 Dec 2024).
• Distillation Mix: LN-Super is trained on 40B tokens from Distillation Mix, supervised on Llama 3.3-70B outputs to regain capacity potentially lost during NAS-based compression and blockwise substitution.
• Synthetic Data and Reasoning Annotations: Post-pretraining datasets contain millions of reasoning traces (e.g., mathematics, scientific, code, multi-step instructions), partially distilled from strong teacher models such as DeepSeek-R1 and encompassing extensive multi-domain supervised data.

This ensures LN-Super reaches and often exceeds the reasoning and instruction-following abilities of both its Llama-based ancestor and contemporary SOTA alternatives.

3. Reasoning-Focused Post-Training and Dynamic Inference Regime

LN-Super introduces a multi-stage reasoning-centric post-training regime:

• Supervised Fine-Tuning (SFT): One epoch over a reasoning-annotated dataset, with explicit tagging in the system prompt: "detailed thinking on" (triggering full chain-of-thought, verbose justification) or "off" (concise replies). SFT uses both reasoning and non-reasoning outputs for the same prompt, allowing the model to support both modes.
• Reinforcement Learning (RL) with RPO: Reward-Preference Optimization (RPO) and Reinforcement Learning from Human Feedback (RLHF) optimizes for both helpfulness and correct reasoning using synthetic instruction datasets and reward models. RL phases are targeted to enforce control tag adherence and context-appropriate response style selection.
• Reasoning Toggle: At inference, LN-Super responds to system prompt toggles, allowing user-level control over output style. This unifies assistant/chat-style and deep reasoner inference within a single model deployment.

This pipeline differentiates LN-Super as the first open model supporting robust user-governed reasoning depth via simple prompt modification.

4. Empirical Performance, Inference Efficiency, and Benchmarks

LN-Super's empirical profile is characterized by high reasoning accuracy, inference efficiency, and broad benchmark coverage. Results (systematically reported) include:

Task/Benchmark	LN-Super (Reasoning on)	DeepSeek-R1	Llama-3.1-405B
GPQA-Diamond	66.7	65.2	58.8
AIME24/25	67.5/60.0	70.0/55.0	79.5/65.8
MATH500	96.6	94.5	96.2
BFCL V2	73.7	65.5	71.6
LiveCodeBench	45.5	57.5	63.4
Arena Hard (helpful)	88.3	65.4	90.5

Key performance notes:

• Reasoning Accuracy: LN-Super matches or exceeds DeepSeek-R1 on scientific and mathematical reasoning (notably GPQA-Diamond).
• Inference Throughput: On single H100 GPU (TP1), achieves 5× speedup over Llama 3.3-70B-Instruct at equivalent batch size. Model design supports up to 300k cached tokens for large-context conversational agents.
• Reasoning Toggle Sensitivity: Benchmarks demonstrate marked performance deltas between reasoning-enabled responses ("on") and standard outputs ("off"), e.g., AIME24: 67.5% vs 16.7%. This confirms the toggle’s operational effectiveness.
• Generalized Judging: LN-Super strongly generalizes in judge-as-model roles (JudgeBench), outperforming both open and proprietary baselines in alignment and helpfulness domains.

5. Technical Innovations and Optimization Methodology

Model optimization in LN-Super is underpinned by formal objectives and algorithmic advances:

• NAS Objective: The architectural configuration is selected to minimize total system latency under required throughput $T^*$ and memory $M^*$:

$$\min_{\{B_i \in V\}} \text{Latency}(\{B_i\}) \quad \text{s.t.} \quad \text{Throughput}(\{B_i\}) \geq T^*, \;\; \text{Memory}(\{B_i\}) \leq M^*$$

• Distillation Loss:

$$\mathcal{L}_{\text{distill}} = -\sum_{t} \sum_{w} p^{(t)}_t(w) \log p^{(s)}_t(w)$$

• RL Preference Optimization:

$$\max_\theta \mathbb{E}_{x \sim \mu, y \sim \pi_\theta}\left[r(y) - \beta\, \text{KL}(\pi_\theta(y|x) \Vert \pi_{\text{ref}}(y|x))\right]$$

where $r(y)$ is a reward assigned by external models, and $\pi_{\text{ref}}$ is a reference policy.

LN-Super leverages FP8 quantization and flexible token cache allocation for efficient high-throughput inference on commodity GPU hardware.

6. Release Assets, Licensing, and Ecosystem

LN-Super is released with a full complement of assets:

• Model Weights: Available on public HuggingFace repositories, with configuration and metadata.
• Post-training Datasets: The Llama-Nemotron-Post-Training-Dataset comprises all SFT and RL data for reasoning and general assistant tasks.
• Training Codebases: NeMo, NeMo-Aligner, and Megatron-LM code supporting both pretraining and post-training workflows.
• License: NVIDIA Open Model License Agreement grants broad research and enterprise permission, enabling unrestricted deployment.
• Compatibility: Efficient design for single-node H100 deployment, as well as large-scale distributed inference clusters.

This level of transparency and ecosystem openness supports both academic benchmarking and industrial integration.

7. Comparative Assessment and Significance

Llama 3.3 Nemotron Super 49B represents a significant advance in heterogeneous, open-source LLM research:

• Delivers SOTA or near-SOTA reasoning, chat, and instruction-following performance compared to models with equal or greater scale.
• Embeds architectural choices (attentive block selection, FFN fusion, NAS, distillation) enabling high inference throughput and cost-effective deployment.
• Introduces a functioning, user-governed reasoning toggle during inference—providing operational flexibility not available in prior open models.
• Is released under an enterprise-friendly license, with full code and dataset availability, facilitating reproducibility and downstream model development.

LN-Super thus sets a benchmark for efficiency, reasoning controllability, and open research infrastructure among large-scale foundation models (Bercovich et al., 2 May 2025).