Nemotron-T 8B: Scalable LLM with Pruning & Distillation

• Nemotron-T 8B is a compact, high-performing language model created by compressing a 15B parameter teacher using structured pruning and knowledge distillation.
• Structured pruning and distillation enable efficient training, reducing compute cost and token usage while achieving competitive MMLU and ARC-Challenge scores.
• The model leverages the Nemotron-CC and Nemotron-CC-Math datasets to enhance general and mathematical reasoning, driving significant performance gains.

Nemotron-T 8B refers to a compact, high-performing LLM derived from the Nemotron-4 family via a systematic procedure centered on structured pruning and knowledge distillation (Muralidharan et al., 19 Jul 2024). This model exemplifies a new paradigm for LLM scaling: rather than training every model size from scratch, Nemotron-T 8B is constructed by compressing a larger, fully trained model (15B parameters) to a target size using empirical, data-efficient algorithms. The process achieves strong downstream task performance, including competitive Massive Multitask Language Understanding (MMLU) scores, while dramatically reducing compute and data requirements. Its training exploits the Nemotron-CC dataset (Su et al., 3 Dec 2024) for general web-scale pretraining and Nemotron-CC-Math (Mahabadi et al., 20 Aug 2025) for mathematical and scientific content, accounting for significant improvements in reasoning and general-domain capabilities.

1. Structured Pruning Pipeline

Nemotron-T 8B is generated by sequentially applying depth-wise and width-wise structured pruning, guided by activation-based importance scores (Muralidharan et al., 19 Jul 2024). Each variant of the compression pipeline includes:

• Width-wise pruning: Attention heads, MLP neurons, and embedding channels are ranked by activation-based metrics. For attention heads, importance is computed as $$F^{(i)}_{head} = \sum_{B,S} \|Attn(\mathbf{X}W^{Q,i}, \mathbf{X}W^{K,i}, \mathbf{X}W^{V,i})\|_2$$, aggregating over batch and sequence dimensions.
• Depth-wise pruning: Entire layers are ranked by either validation perplexity increase or Block Importance ($$BI_{i} = 1-\mathbb{E}_{X,t}\left[\frac{\mathbf{X}_{i,t}^T \mathbf{X}_{i+1,t}}{\|\mathbf{X}_{i,t}\|_2 \|\mathbf{X}_{i+1,t}\|_2}\right]$$), measuring the network’s sensitivity to layer removal.
• Residual information aggregation: When pruning attention heads, residuals from pruned heads are projected onto those retained, partially preserving lost knowledge.

These techniques eliminate the need for computationally expensive gradient-based importance estimation. Aggregation schemes for scores (L2 norm over batches and mean over sequence dimension) are empirically validated for optimal ranking.

2. Knowledge Distillation and Retraining

Following compression, Nemotron-T 8B undergoes knowledge distillation (KD)–based retraining, where the pruned “student” mimics the output distributions of the “teacher” (the original, uncompressed model) (Muralidharan et al., 19 Jul 2024). The distillation process utilizes:

• Logit distillation loss: $$L_{\text{logits}} = \frac{1}{\ell} \sum_{k=1}^\ell \text{Loss}(p_t^k(x,\tau), p_s^k(x,\tau))$$, with $$p(x_i,\tau) = \frac{\exp(x_i/\tau)}{\sum_j \exp(x_j/\tau)}$$ as the temperature-scaled probability for token $i$.
• Optional intermediate state matching: Matching hidden state activations at select layers between teacher and student, $$L_{\text{is}} = \frac{1}{\ell} \sum_{k \in H}\sum_{i=1}^{\ell} \text{Loss}(h^k_{t,i}, h^k_{s,i})$$, where $H$ is the set of layers.
• Total loss formulation: $$L = L_{\text{CLM}} + L_{\text{logits}} + \alpha \times L_{\text{is}}$$, where $L_{\text{CLM}}$ is the standard LLMing cross-entropy and $\alpha$ is a dynamically computed weighting coefficient.

In most architectures, the best results are achieved by using logit distillation alone, particularly when depth is preserved to a sufficient degree. Retraining typically uses a tiny fraction (about 1.8B tokens per candidate; <3% of the full training corpus) to stabilize performance.

3. Neural Architecture Search for Compression and Scaling

An empirical, lightweight neural architecture search (NAS) explores the design space—layer count, number of attention heads, MLP expansion ratios, embedding sizes—for Nemotron-T 8B (Muralidharan et al., 19 Jul 2024). This search involves:

• Candidate selection: Architectures with 29–32 layers, and either 32 or 48 heads, are pruned and minimally retrained to reach target parameter counts.
• One-shot vs. iterative pruning: Width-wise pruning benefits from single-pass elimination, while more aggressive depth pruning may require iterative steps.
• Ablation studies: Empirically, the choice of aggregation metric and Kullback–Leibler-based distillation loss (outperforming alternatives like cosine similarity or reverse KL) are critical for final accuracy.

NAS enables efficient exploration of hyperparameter and topology choices to maximize performance for the fixed parameter budget.

4. Pretraining Data: Nemotron-CC and Nemotron-CC-Math

Nemotron-T 8B leverages Nemotron-CC (Su et al., 3 Dec 2024), a web-scale corpus emphasizing high token uniqueness and data quality:

• Scale and diversity: 6.3T total tokens, with 4.4T deduplicated real tokens, plus 1.9T synthetically generated tokens. This is four times more unique tokens than DCLM or FineWeb-Edu.
• Quality filtering: Classifier ensembling (using FineWeb-Edu and DCLM classifiers) and selective filtering maximize retention of valuable material.
• Synthetic augmentation: High-quality documents are enriched using instruct models to produce diverse QA, summaries, and knowledge lists; low-quality documents are rephrased to improve clarity.

For domain-specific enhancement, Nemotron-CC-Math (Mahabadi et al., 20 Aug 2025) supplies high-quality mathematical and code data:

• Extraction pipeline: Utilizes Lynx-based layout-preserving HTML rendering, followed by LLM cleaning (Phi-4). Mathematical content across formats is unified to LaTeX and code blocks are precisely preserved.
• Corpus scale: Nemotron-CC-Math-3+ (133B tokens) and -4+ (52B tokens) subsets, both substantially larger than previous math pretraining sets.
• Performance impact: Pretraining with Nemotron-CC-Math produces +4.8 to +12.6 gains on the MATH benchmark, +4.6 to +14.3 on MBPP+, and measurable improvements on MMLU and MMLU-Stem.

This pretraining data composition yields demonstrably higher MMLU and ARC-Challenge scores compared to comparably sized models trained on less unique or lower-fidelity corpora.

5. Benchmarks, Performance, and Efficiency

Nemotron-T 8B demonstrates competitive or improved accuracy on major benchmarks:

• MMLU: The model achieves 63.8% accuracy—comparable to Mistral 7B (64.1%) and LLaMa-3 8B, and up to 16% better than training from scratch for some variants (Muralidharan et al., 19 Jul 2024).
• ARC-Challenge and others: Nemotron-CC pretraining yields +5.6 on MMLU and +3.1 on ARC-Challenge relative to DCLM, with an average improvement across diverse benchmarks (Su et al., 3 Dec 2024).
• Mathematical and code reasoning: Incorporating Nemotron-CC-Math leads to direct, state-of-the-art gains over prior open math datasets (Mahabadi et al., 20 Aug 2025).

Efficiency improvements are pronounced:

• Token usage: Up to 40× fewer training tokens required per model for additional size variants in the family, relative to from-scratch training (Muralidharan et al., 19 Jul 2024).
• Compute cost: Total compute for training (measured in FLOPs) is reduced by 1.8× for the family (15B, 8B, 4B).
• Practical implications: This allows for rapid model scaling, deployment flexibility, and cost reduction without sacrificing accuracy.

6. Comparison to Related Models and Methods

Nemotron-T 8B’s compression and retraining methodology contrasts with the practice of full retraining for every model size. Compared to other compression methods and community models:

Model	Parameters	MMLU Accuracy (%)	Pretraining Tokens	Compression Type
Nemotron-T 8B	~8B	63.8	1.8B (empirical retrain)	Structured pruning + KD
Mistral 7B	~7B	64.1	Full training	Conventional training
LLaMa-3 8B	~8B	≈63.8	Full training	Conventional training

This approach achieves competitive accuracy with dramatically reduced data and computational costs. Additionally, Nemotron-T 8B outperforms state-of-the-art compression techniques from the literature and maintains key properties for real-world deployment.

7. Broader Implications and Future Directions

The Nemotron-T 8B model, supported by Nemotron-CC and Nemotron-CC-Math datasets, illustrates the practical feasibility of scaling LLMs via model compression pipelines. The demonstrated gains establish benchmarks for both general and mathematical reasoning tasks. This suggests that careful data curation and structured compression, coupled with knowledge distillation-based retraining, can produce models as performant as those trained from scratch, while enabling multi-size scaling with substantial resource savings. A plausible implication is that future LLM development will increasingly favor compression-first workflows—leveraging large, diverse, and domain-specific corpora—to better utilize compute and facilitate deployment across a spectrum of resource environments.

These outcomes also highlight the importance of maintaining token-level diversity and structural fidelity in pretraining corpora, particularly for tasks demanding higher-order reasoning. The open availability of model weights and datasets further catalyzes research into model scaling, data pipeline innovations, and the development of specialized models for technical and logical domains.