Papers
Topics
Authors
Recent
Search
2000 character limit reached

Strong Teacher Not Needed? On Distillation in LLM Pretraining

Published 22 May 2026 in cs.LG and cs.CL | (2605.23857v1)

Abstract: Knowledge distillation generally assumes a strong-to-weak relationship where stronger teachers yield better students. In this work, we examine this assumption about distillation in LLM pretraining. By varying architecture sizes and training token budgets, we create strong-to-weak, same-level, and weak-to-strong teacher-student relationships, and study distillation's effectiveness under each. We find that the teacher need not be strong: with proper mixing of the language modeling and knowledge distillation losses, even small and undertrained teachers improve larger students. At the same time, a stronger teacher is not always better: pushing the teacher further, through more parameters or more training tokens, can saturate or even reverse the distillation gains. We further observe that distillation improves generalization (out-of-distribution and downstream performance) more readily than in-domain fitting. Together, these results challenge the common belief that distillation pretraining always requires a strong teacher.

Authors (2)

Summary

  • The paper demonstrates that effective distillation does not rely solely on a strong teacher, but on optimizing teacher-student compatibility with appropriate loss mixing.
  • Empirical results reveal that weaker or similarly sized teachers, when paired with tuned α, can boost generalization and downstream accuracy even with high in-domain perplexity.
  • The study provides practical insights for LLM pretraining by emphasizing careful teacher selection and hyperparameter tuning to prevent performance degradation due to overpowered teachers.

Distillation in LLM Pretraining: Challenging the Strong-Teacher Paradigm

Background and Motivation

The prevailing paradigm for knowledge distillation in LLM pretraining suggests that a stronger teacher—usually a larger or better-trained model—will always yield a superior student. This assumption has permeated both teacher selection and distillation loss design, and has positioned distillation primarily as a compression mechanism. Despite its broad adoption, particularly given scaling trends and computational constraints, this paper demonstrates that the strong-to-weak teacher-student relationship is neither necessary nor invariably optimal.

Systematic Experimental Framework

To interrogate classic assumptions, the authors vary teacher architectures (0.7B, 1.7B, 3.8B, 8.0B parameters) and training token budgets (10B–300B) to build a cross-product of teacher-student relationships: strong-to-weak, same-level, and weak-to-strong. Distillation is implemented by a convex combination of language modeling loss and KL-divergence with the teacher’s distribution, regulated by the mixing coefficient α∈[0,1]\alpha \in [0,1].

Models are consistently evaluated on three dimensions: in-domain perplexity (FineWeb-Edu), out-of-domain perplexity (across 11 corpora), and downstream accuracy (15 benchmarks from QA, reasoning, and comprehension).

Key Findings on Teacher-Student Compatibility

Effective Weak-to-Strong and Same-Level Distillation

Contrary to conventional wisdom, teacher models smaller or less trained than the student—even those with higher in-domain perplexity—can improve distillation outcomes when α\alpha is tuned appropriately. Same-architecture and same-token teachers also yield measurable gains, supporting the insight that architectural parity does not preclude effective knowledge transfer. Figure 1

Figure 1: Teacher-student compatibility, not just teacher strength, governs supervision effectiveness; weak-to-strong and same-level distillation can improve students, while strongest teachers can saturate or degrade transfer.

Saturation and Reversal with Stronger Teachers

Increasing teacher size or training tokens yields diminishing returns and, above certain thresholds, reversal: stronger teachers (especially at high token budgets or larger architectures) can lead to suboptimal or degraded transfer relative to smaller, similarly trained teachers. This trend is pronounced in downstream and out-of-distribution evaluations, contradicting monotonic scaling assumptions. Figure 2

Figure 2: Distillation improvement versus loss-mixing coefficient α\alpha; optimal α\alpha depends strongly on teacher-student compatibility.

Figure 3

Figure 3: Optimal α\alpha varies with teacher strength; stronger teachers favor higher α\alpha, but improvement saturates or declines past certain points.

Domain-Specific Performance Patterns

Out-of-distribution generalization and downstream accuracy are improved more easily and more markedly than in-domain perplexity. Weak teachers frequently fail to improve in-domain fitting but can enhance generalization metrics, highlighting the specificity of transfer effects and the importance of evaluating models on broader distributions. Figure 4

Figure 4: Improvement trends versus teacher in-domain perplexity; performance gains depend not just on loss but on architecture and compatibility.

Empirical Analysis of Loss Mixing

Each teacher configuration demonstrates an optimal α\alpha for distillation, with weaker teachers consistently favoring lower α\alpha to dilute unreliable supervision, and strong teachers favoring higher α\alpha. However, even strong teachers may require reduced α\alpha at extreme token budgets, reinforcing the importance of explicit hyperparameter tuning over heuristics.

Mechanistic Analysis: Token-Level and Distributional Effects

Improvement Concentrated on Difficult Tokens

Improvements from distillation are concentrated on tokens with high entropy under the baseline student ("difficult tokens"), not uniformly distributed as regularization would predict. This refutes hypotheses equating distillation benefits to label smoothing or regularization. Figure 5

Figure 5: Gains are concentrated on hard tokens; monotonic trend is universal across teacher strengths.

Figure 6

Figure 6: Heatmap across all α\alpha0 values highlights robust improvement for difficult tokens under strong teachers, but degradation as weak teachers dominate supervision.

Information Gain and Causality

Distillation improvement is causally tied to tokens where the teacher provides correct predictions absent from the baseline student’s top-10—these "teacher only" tokens drive transfer. In contrast, tokens where the baseline is correct but the teacher is not ("baseline only") yield substantial performance degradation, explaining the necessity of α\alpha1 to prevent harmful supervision. Figure 7

Figure 7: The cost-benefit profile of distillation; improvement is causally explained by teacher-provided novel correct information.

Distributional Effects and Probability Redistribution

Strong teachers enable the student distribution to converge efficiently toward the teacher’s prediction, measured via top-10 token overlap. Weak teachers distort the student distribution when supervised too strongly (α\alpha2 high), causing performance collapse. Probability redistribution, not mere sharpening, underlies empirical improvements: entropy typically increases as perplexity decreases (student becomes less confident but more accurate). Figure 8

Figure 8: Weak teachers distort student distribution; strong teachers enable knowledge transfer without distortion.

Figure 9

Figure 9: Probability redistribution (not sharpening) is the main effect of distillation; beneficial entropy changes accompany improved perplexity.

Practical Implications and Future Directions

These results fundamentally challenge the necessity of strong-to-weak teacher selection and monotonic improvement assumptions in distillation. In practical LLM development:

  • Teacher selection must incorporate compatibility and architectural parity, not absolute strength.
  • Hyperparameter tuning, especially α\alpha3, is critical for maximizing effective transfer.
  • In scenarios where the best available teacher is smaller or less trained, distillation remains viable and can enhance the student’s capabilities.
  • Overtraining or excessive scale in the teacher may reduce transfer utility, particularly for generalization.

This work raises broader questions about the mechanisms that govern cross-model information transfer, the limits of dark knowledge utility at scale, and the prospects for leveraging weaker models or prior-generation models as teachers in computationally constrained settings.

Conclusion

By systematically varying teacher-student relationships and characterizing both global and token-local behavior, the paper demonstrates that distillation in LLM pretraining is governed by compatibility, not merely teacher strength. Weak-to-strong and same-level distillation can improve students, overpowered teachers sometimes impede transfer, and loss mixing must be explicitly optimized. These results imply flexibility in teacher selection, mandate careful hyperparameter tuning, and refute regularization-based explanations for distillation efficacy. The practical and theoretical implications broaden the applicability of distillation and emphasize the importance of nuanced design choices for building efficient, capable models (2605.23857).

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Explain it Like I'm 14

What this paper is about (big picture)

This paper looks at a common idea in AI called “knowledge distillation.” That’s when a “teacher” model helps a “student” model learn. Most people assume the teacher must be much stronger and smarter than the student to help. The authors test that assumption for LLMs and find something surprising: a teacher doesn’t have to be very strong to help, and making the teacher bigger and bigger doesn’t always make the student better. What matters more is how well the teacher and student “fit” each other and how you mix the teacher’s advice with real data during training.

The main questions (in simple terms)

  • Can a smaller or less-trained teacher still help a bigger or better student model learn?
  • Is a stronger teacher always better, or can an “overly strong” teacher actually hurt?

How they tested it (methods, with plain-language analogies)

Think of training an AI like preparing for a test:

  • The “student” is the model you want to improve.
  • The “teacher” is another model that gives hints about what answers are likely.
  • The “data” are the practice questions the student studies (called “tokens”; more tokens = more reading practice).
  • “Perplexity” is how surprised the model is by the correct answer. Lower perplexity means the model is doing a better job predicting the next word.

What they varied:

  • Model sizes: They tried four sizes (about 0.7B, 1.7B, 3.8B, and 8.0B parameters). Bigger usually means more capable.
  • Training data for the teacher: Teachers were trained on different amounts of text (from 10 billion up to 300 billion tokens). More tokens = more practice.
  • The mixing “dial” (alpha, written as α\alpha): This controls how much the student listens to the teacher versus the real answers in the data.
    • Low α\alpha: follow the real answers more, the teacher less.
    • High α\alpha: follow the teacher more.

What they measured:

  • In-domain perplexity: How well the student predicts text that’s similar to its training data (like taking another test from the same textbook).
  • Out-of-distribution perplexity: How well it predicts text from different sources (new topics and styles).
  • Downstream accuracy: How well it does on various tasks (like multiple-choice tests in science, math, logic, and reading comprehension).

Important setup details:

  • Students were all trained on 50B tokens (same study time).
  • Teachers varied in both size and how much they had studied (10B–300B tokens).
  • They tried different α\alpha values to find the best mix of teacher advice and real answers.
  • They used standard, well-known data and model designs to keep things fair.

What they found (key results)

Here are the main takeaways, explained simply:

  • Weak teachers can help stronger students if you mix their advice correctly.
    • Even a smaller or less-trained teacher can give helpful hints, as long as you don’t rely on it too much. In practice, this means using a lower α\alpha so the student still listens more to the real data than to a weak teacher.
  • Stronger is not always better.
    • Sometimes, making the teacher larger or training it for longer doesn’t help—and can even reduce the student’s improvement. Why? If the teacher becomes too specialized or too different from the student, its advice may be less useful or even confusing for the student’s learning style.
  • Distillation improves generalization more than memorization.
    • The biggest gains showed up on new or different kinds of data and on real tasks (out-of-distribution and downstream), not as much on the training-like data (in-domain). In other words, distillation tends to teach the student “how to think” rather than just “what the training data looks like.”
  • The mixing dial (α\alpha) really matters—and it depends on the teacher.
    • Each teacher has a “sweet spot” for α\alpha.
    • Weaker teachers work best with smaller α\alpha (listen less to the teacher).
    • Stronger teachers can use larger α\alpha (listen more)—but if the teacher becomes “too strong” or overtrained for the student, the best α\alpha drops again.
  • “Teacher score” isn’t a perfect guide.
    • A teacher’s own training score (like low perplexity) predicts how well it helps the student on training-like data, but not necessarily on new domains or real tasks. A smaller teacher trained on more varied data sometimes helps more with generalization than a bigger teacher.
  • Pure imitation is worse than mixing.
    • Training the student to only copy the teacher (very high α\alpha) was usually worse than combining teacher advice with real answers.

Why this matters (impact and implications)

  • You don’t need a super-strong teacher to benefit from distillation.
    • This is great news when training cutting-edge models: if the “best” teacher you have is from an older generation or smaller size, it can still help.
  • Bigger isn’t always better—compatibility is key.
    • Choose teachers that “fit” the student in size and training style, and tune how much the student should listen (α\alpha). This can save time, money, and computing power.
  • Expect better transfer to new tasks and topics.
    • Distillation often helps models do better on things they weren’t directly trained on. That’s valuable for building more useful and flexible AI systems.
  • Practical advice:
    • Try teachers of different sizes and training amounts, not just the largest one available.
    • Tune the mixing dial α\alpha for each teacher-student pair.
    • Don’t judge a teacher only by its own training score—check how it helps the student on new data and tasks.

In short, this paper challenges a long-held belief: a strong teacher is not always needed for knowledge distillation to work. With the right balance and a compatible teacher, even smaller or less-trained models can boost a larger student—especially for real-world, out-of-training tasks.

View Paper Prompt  View All Prompts 

Knowledge Gaps

Knowledge gaps, limitations, and open questions

The paper advances understanding of distillation in LLM pretraining, but several aspects remain uncertain or unexplored. Future work could address the following gaps:

  • Limited scale: all results are within 0.7B–8B architectures; it is unclear whether the findings (e.g., weak-to-strong benefits, over-strong teacher degradation) hold at frontier scales (tens to hundreds of billions of parameters).
  • Single pretraining corpus: models are trained on FineWeb-Edu only; sensitivity to corpus composition, quality, multilinguality, and domain mixtures is untested.
  • Student scope: core analyses fix the student to 1.7B and 50B tokens; whether trends generalize across different student sizes and student token budgets (e.g., very small or much larger students, longer student training) remains open.
  • Compute-efficiency: the paper reports percent improvements but not compute-normalized efficiency; what is the best teacher choice per unit of total compute (teacher pretraining + student distillation) is unresolved.
  • Teacher–student data overlap: teacher and student draw from the same pool with different seeds; the causal effect of overlap vs disjoint teacher-student datasets on transfer is not isolated.
  • Teacher “compatibility”: compatibility is informally characterized; a concrete, predictive metric (beyond in-domain ppl) for matching teachers to students is missing.
  • Over-strong teachers: the mechanism by which very strong teachers “overwhelm” students is not explained; are issues due to calibration, entropy mismatch, overconfident logits, distributional mismatch, or optimization dynamics?
  • Temperature and calibration: only τ=1 is used; the role of softening (τ>1), confidence calibration, and teacher entropy in mitigating over-strong-teacher degradation is unexplored.
  • Alpha scheduling: α is chosen by a small grid and fixed per run; dynamic or curriculum schedules (e.g., increasing α over time, per-example/per-token α) and principled selection rules are not investigated.
  • Alternative KD objectives: only forward-KL on logits is examined; effects of reverse-KL, JS divergence, margin/logit-distance losses, label smoothing, or top-k/thresholded distillation are untested.
  • Intermediate representation transfer: no layer-wise, attention/feature, or hidden-state matching is explored; whether feature-level KD alters the weak-to-strong/over-strong patterns is unknown.
  • Multi-teacher and ensembling: the potential of ensembles or heterogeneous teachers (e.g., mixing sizes/tokens or domains) to improve compatibility and avoid saturation is not assessed.
  • Architecture mismatch beyond size: teachers and students share the Llama3 family; how choices like depth/width trade-offs, normalization variants, rotary settings, MoE vs dense, or tokenizer mismatches affect transfer is open.
  • Tokenizer and vocabulary: both models share tokenization; cross-tokenizer distillation (e.g., different vocabularies or segmentation) and its impact on KD efficacy are unexamined.
  • Context length: all training uses 8K context; whether teacher–student context-length mismatch or long-context regimes change distillation dynamics is unknown.
  • Downstream breadth and metrics: downstream evaluation emphasizes multiple-choice accuracy and perplexity; effects on generative tasks (summarization, reasoning CoT quality, code pass@k, factuality, hallucination) are not measured.
  • Safety, bias, and calibration: impacts of distillation (especially weak-to-strong) on bias, fairness, toxicity, privacy/memorization, and probability calibration (e.g., ECE) are not evaluated.
  • Robustness and variance: limited discussion of variance over random seeds; statistical significance and sensitivity to optimization hyperparameters (lr schedules, weight decay, clipping) are not quantified.
  • Online vs offline teacher costs: the practical trade-offs of online teacher querying vs stored logits (bandwidth, memory), and whether logit compression (e.g., top-k, quantization) preserves benefits, are not addressed.
  • Data curriculum and selection: whether targeting examples with high teacher–student disagreement, high loss, or specific domains improves KD efficiency is not studied.
  • Iterative/born-again distillation: only one-shot distillation is considered; whether repeated self-distillation or teacher refresh cycles compound gains is unknown.
  • Comparing KD to more student data: the paper fixes student tokens; a direct compute-controlled comparison of KD gains vs simply training the student longer or on higher-quality data is absent.
  • Theory: there is no formal framework explaining when weak-to-strong works and why over-strong teachers saturate; a predictive theory tying teacher entropy, mismatch, and α to generalization gains is missing.
  • Practical selection policy: despite empirical patterns, a prescriptive algorithm for choosing teacher size/tokens and α under a fixed compute budget (and for a target evaluation type) is not provided.
View Paper Prompt  View All Prompts 

Practical Applications

Immediate Applications

Below are concrete, deployable applications that leverage the paper’s findings on loss mixing, teacher–student compatibility, and the observation that generalization gains (OOD and downstream) are easier to realize than in-domain fit. These can be applied today with existing tooling for LLM pretraining and distillation.

  • Budget-aware LLM pretraining with weak or same-level teachers
    • Sector: software/ML platforms, cloud AI providers, open-source model builders
    • What to do: Use a smaller or equally sized teacher (e.g., previous-generation or partially trained model) for distillation during pretraining of a mid-sized student (on the order of ~0.7B–8B parameters), and tune the loss mixing coefficient α rather than assuming “bigger teacher is better.”
    • Why it works: The paper shows weak-to-strong and same-level distillation can yield measurable gains, especially on OOD and downstream tasks, if α is chosen appropriately.
    • Practical workflow:
    • Freeze the available teacher and run mixed LM+KD loss training for the student.
    • Conduct a small α sweep (e.g., α ∈ {0.2, 0.4, 0.6, 0.8}) and pick the α that optimizes OOD and downstream metrics.
    • Avoid pure KD (α=1.0) unless validated; mixed loss generally performs better.
    • Assumptions/dependencies: Student and teacher are reasonably architecture-compatible (e.g., Llama-like); teacher logits accessible; evaluation suite includes OOD and downstream tasks; results are strongest in the 0.7B–8B regime and 10B–300B-token training ranges studied.
  • Compute- and energy-efficient teacher selection
    • Sector: energy, cloud operations, sustainability, policy within AI labs
    • What to do: Stop overtraining the teacher (e.g., hundreds of billions of tokens) when the student is smaller and/or not trained as long. Prefer teacher configurations that are compatible, not just stronger.
    • Why it works: The paper finds diminishing or even negative returns from “over-strong” teachers and that compatibility plus tuned α often beats raw scale.
    • Practical workflow:
    • Track OOD perplexity and downstream accuracy while increasing teacher tokens; early-stop the teacher when student gains plateau.
    • Document α alongside teacher size/tokens to justify compute budgets for audits.
    • Assumptions/dependencies: Comparable data quality; availability of intermediate teacher checkpoints; governance/reporting processes to capture α and evaluation.
  • OOD-first evaluation for distillation success
    • Sector: MLOps, research, applied AI teams in healthcare/finance/education
    • What to do: Make OOD perplexity and downstream accuracy the primary “go/no-go” gates for distillation choices (teacher, α), rather than relying on in-domain perplexity alone.
    • Why it works: Generalization gains are easier and larger; teacher train loss alone is a misleading signal for OOD/downstream transfer.
    • Practical workflow:
    • Maintain an OOD suite (diverse corpora and downstream tasks).
    • Select teacher/α by maximizing OOD/downstream composite metrics.
    • Assumptions/dependencies: Access to high-quality, diverse OOD corpora and downstream benchmarks relevant to target use cases.
  • α-tuning heuristic integrated into training pipelines
    • Sector: ML tooling/software
    • What to build: A simple “α tuner” that:
    • Estimates teacher strength (e.g., teacher in-domain ppl or proxy).
    • Starts with lower α for weaker teachers and higher α for stronger ones, then refines via a small sweep.
    • Why it works: Optimal α correlates with teacher strength; weaker teachers favor lower α; overly strong teachers may require dialing α back.
    • Assumptions/dependencies: Teacher perplexity estimates; automation hooks in training stack; compute budget for a small sweep.
  • Cost-effective iterative model upgrades
    • Sector: product engineering, open-source models, mid-market AI teams
    • What to do: When shipping a new mid-size model version, use the current production model (same-size or slightly smaller) as a teacher during pretraining to improve generalization without waiting for a stronger teacher.
    • Why it works: Born-again/same-level distillation benefits hold in LLM pretraining; the paper demonstrates measurable gains even with same-size teachers under mixed loss.
    • Assumptions/dependencies: Access to teacher inference; alignment with current deployment constraints; careful α selection.
  • Edge and mobile model preparation
    • Sector: edge/robotics/mobile AI
    • What to do: Distill small models (sub-2B) using modest teachers to improve OOD robustness for on-device tasks (e.g., offline assistance, basic robotics control dialogue).
    • Why it works: Gains show up more strongly in OOD/generalization; small teachers plus tuned α are practical for edge-focused pretraining.
    • Assumptions/dependencies: On-device or near-edge training workflows, or server-side pretraining then deployment; lightweight evaluation harnesses.
  • Domain adaptation with budget-limited teachers
    • Sector: healthcare, finance, legal, education
    • What to do: For a domain student model, distill from an earlier or smaller domain teacher trained on fewer tokens rather than waiting for a very large, thoroughly trained domain teacher.
    • Why it works: Weak-to-strong distillation with the right α provides generalization gains that matter in domains with distribution shifts and limited data.
    • Assumptions/dependencies: Domain-appropriate OOD/downstream evaluations (e.g., MedMCQA for healthcare); teacher access consistent with data governance and licensing.
  • Governance and reporting checklists for distillation
    • Sector: policy/compliance within AI organizations
    • What to do: Require reporting of teacher size/tokens, teacher in-domain ppl, α, and OOD/downstream results when documenting a distilled model release.
    • Why it works: Prevents over-reliance on teacher train loss as a quality proxy; encourages compute-efficient teacher selection aligned with generalization.
    • Assumptions/dependencies: Organizational commitment to measurement and transparency; standardization of evaluation suites.

Long-Term Applications

These opportunities require further research, scaling experiments beyond 8B, new tooling, or validation across modalities and tasks.

  • Automated teacher–student compatibility search and scheduling
    • Sector: ML tooling/platforms, AutoML
    • What to build: An “AutoDistill” system that jointly searches teacher size/tokens, α schedule over training, and stopping criteria, optimizing OOD/downstream metrics under a compute budget.
    • Rationale: The paper shows compatibility and α interplay matter more than raw teacher strength; automated search can formalize this.
    • Dependencies: Efficient multi-run orchestration, robust OOD/downstream metrics, budget-aware objective functions.
  • Multi-teacher, curriculum-style distillation
    • Sector: research, advanced ML engineering
    • Idea: Sequence teachers by compatibility (e.g., weak → same-level → modestly strong), or mix heterogeneous teachers (different sizes/token budgets) with adaptive α weights through training.
    • Rationale: Observed saturation/overtraining effects suggest staggered or blended teachers may avoid being “overwhelmed” and better transfer broad knowledge.
    • Dependencies: Mixing policies, stability studies, scheduling strategies, and evaluation at scale.
  • New scaling laws for distillation pretraining
    • Sector: academia, frontier labs
    • Idea: Develop compute-optimality frameworks that incorporate teacher choice and α (not just model size and data), with targets oriented toward OOD/downstream metrics.
    • Rationale: The “stronger is not always better” result implies current scaling heuristics should be extended to include compatibility and mixing effects.
    • Dependencies: Broader sweeps across larger models (>8B), diverse datasets, and domains.
  • Cross-modality extensions (vision, speech, robotics, code)
    • Sector: robotics, multimodal systems, developer tooling
    • Idea: Test whether weak-to-strong and same-level distillation benefits transfer to multimodal pretraining (e.g., VLMs) and specialized code models; emphasize OOD/generalization gains.
    • Rationale: The generalization-first improvement pattern could be especially valuable in robotics and code (domain shifts are common).
    • Dependencies: Modality-specific KD formulations, teacher logit access, and reliable OOD/generalization benchmarks.
  • α scheduling policies and theory-informed regularization
    • Sector: research/tooling
    • Idea: Learn α as a function of training progress and teacher signals (e.g., confidence/entropy), or design new regularizers that prevent “overwhelming” from overly strong teachers.
    • Rationale: Optimal α varies with teacher strength and training tokens; dynamic policies could automate balance between ground truth and teacher guidance.
    • Dependencies: Online estimators of teacher reliability, stable training under dynamic objectives.
  • Federated and distributed KD with heterogeneous participants
    • Sector: federated learning, privacy-preserving AI
    • Idea: Aggregate signals from many weaker local teachers (clients) to distill a stronger central student, with adaptive α per teacher/client.
    • Rationale: Findings legitimize using weaker teachers; in federated contexts, it’s realistic that many teachers are small and undertrained.
    • Dependencies: Privacy-preserving logit sharing, weighting schemes, robustness against noisy/biased teachers.
  • Robustness, fairness, and bias-aware teacher selection
    • Sector: policy, responsible AI
    • Idea: Select teachers and α by optimizing OOD fairness/robustness metrics (not only accuracy), given the paper’s emphasis on generalization benefits.
    • Rationale: If KD primarily boosts generalization, use that leverage to target robustness and bias reduction in shifted populations.
    • Dependencies: Standardized OOD fairness/robustness benchmarks; evaluation protocols; careful monitoring for unintended bias transfer.
  • Sector-specific “distillability” diagnostics
    • Sector: healthcare, finance, legal, education
    • Idea: Develop diagnostic tools to predict whether a candidate teacher will improve a target student on domain-critical OOD tasks before training.
    • Rationale: Teacher train loss is not predictive; practical diagnostics (e.g., small-sample OOD probes, entropy/consistency measures) could prevent costly failed runs.
    • Dependencies: Domain-relevant OOD probes; correlation studies linking diagnostics to downstream gains.

Cross-cutting assumptions and dependencies to monitor

  • External validity: The study spans Llama-style architectures (~0.7B–8.0B) and specific token budgets (10B–300B) on FineWeb-Edu; extrapolation to much larger models, other architectures, or vastly different corpora requires validation.
  • Loss design: Mixed LM+KD with τ=1 and tuned α underpins most gains; pure KD often underperforms. Tooling must support loss mixing and α sweeps/schedules.
  • Evaluation breadth: Gains are clearest on OOD/downstream metrics. Organizations should maintain strong OOD suites aligned to their real-world distributions.
  • Data/licensing: Teacher logits access and data governance (especially in regulated sectors) constrain feasibility; distillation may propagate biases from the teacher—evaluate fairness/robustness explicitly.
  • Compute budgets: Although teachers need not be strongest-available, α tuning and OOD evaluation add overhead; plan small, targeted sweeps and early-stopping criteria.
View Paper Prompt  View All Prompts 

Glossary

  • AdamW: An optimizer variant of Adam that decouples weight decay from the gradient-based update, often improving generalization. Example: "We train with AdamW using β1=0.9, β2=0.95, and weight decay 0.1."
  • architecture mismatch: A disparity between teacher and student model architectures that can hinder knowledge transfer. Example: "Architecture mismatch and teacher overtraining can degrade transfer even when the teacher attains a lower training loss."
  • autoregressive language modeling: A training paradigm where the model predicts the next token given all previous tokens in a sequence. Example: "We review the standard autoregressive language modeling and knowledge distillation (KD) objectives."
  • batch size: The number of sequences processed together in a single training step. Example: "We use an 8192-token context length and a 256-sequence batch size."
  • bf16 precision: A 16-bit floating-point format (bfloat16) used to speed up training and reduce memory while keeping sufficient numeric range. Example: "The peak learning rate is 3×10−43 \times 10^{-4} with a cosine decay to 10\% of the peak, linear warm-up for the first 5\% of steps, gradient clipping of 1.0, and bf16 precision."
  • Born-Again Networks: A distillation approach where a model is trained from the outputs of another model of the same architecture, often improving performance without capacity changes. Example: "This aligns with Born-Again Networks~\citep{BornAgain} in vision, where same-level distillation helps despite no capacity advantage."
  • context length: The maximum number of tokens the model attends to when predicting the next token. Example: "We use an 8192-token context length and a 256-sequence batch size."
  • cosine decay: A learning rate schedule that decreases the learning rate following a cosine curve. Example: "The peak learning rate is 3×10−43 \times 10^{-4} with a cosine decay to 10\% of the peak, linear warm-up for the first 5\% of steps, gradient clipping of 1.0, and bf16 precision."
  • cross-entropy: A loss function measuring the difference between two probability distributions, commonly used for classification and language modeling. Example: "We refer to this as the language modeling loss (LM loss), the cross-entropy between the student distribution and a one-hot target (the ground-truth next token)."
  • distillation pretraining: Training a student model to match a teacher model’s output distributions during pretraining. Example: "In distillation pretraining~\citep{hintonKD}, the target is no longer a one-hot vector but the teacher's relative preferences among all tokens."
  • downstream accuracy: Performance on tasks beyond next-token prediction that assess how well learned representations transfer. Example: "and average downstream accuracy (acc) across 15 benchmarks (higher is better)."
  • forward KL divergence: The Kullback–Leibler divergence measured as KL(teacher || student), encouraging the student to cover teacher probability mass. Example: "Student distribution ptp_t will try to match the teacher distribution qtq_t by minimizing the forward KL divergence, as in \eqrefc{eq:kd_loss}."
  • frozen teacher: A teacher model whose parameters are not updated during distillation and only provide guidance signals. Example: "During distillation, the teacher model is kept frozen and provides logits only."
  • generalization: The model’s ability to perform well on data or tasks different from those seen during training. Example: "We further observe that distillation improves generalization (out-of-distribution and downstream performance) more readily than in-domain fitting."
  • gradient clipping: Limiting the norm of gradients to stabilize training and prevent exploding gradients. Example: "The peak learning rate is 3×10−43 \times 10^{-4} with a cosine decay to 10\% of the peak, linear warm-up for the first 5\% of steps, gradient clipping of 1.0, and bf16 precision."
  • in-domain perplexity: Perplexity measured on held-out data from the same distribution as the training set. Example: "We measure in-domain perplexity (id ppl) on a held-out test split from training dataset FineWeb-Edu, which was not seen during training."
  • knowledge distillation (KD): Transferring knowledge from a teacher model to a student model by matching output distributions. Example: "Knowledge distillation generally assumes a strong-to-weak relationship where stronger teachers yield better students."
  • language modeling loss (LM loss): The cross-entropy loss between the predicted distribution and the ground-truth next token. Example: "We refer to this as the language modeling loss (LM loss), the cross-entropy between the student distribution and a one-hot target (the ground-truth next token)."
  • logits: The unnormalized scores (pre-softmax) output by a model for each token in the vocabulary. Example: "During distillation, the teacher model is kept frozen and provides logits only."
  • loss mixing coefficient α: A hyperparameter that balances the LM loss and the KD loss in the mixed objective. Example: "The loss mixing coefficient α\alpha controls the trade-off:"
  • mixed loss: A training objective that combines standard LM loss with KD loss using a mixing coefficient. Example: "We refer to this as the mixed loss and use it for all distillation pretraining in this work."
  • one-hot target: A target vector with probability 1 on the ground-truth class/token and 0 elsewhere. Example: "the cross-entropy between the student distribution and a one-hot target (the ground-truth next token)."
  • out-of-distribution perplexity: Perplexity measured on corpora that differ from the training distribution to assess generalization. Example: "We also measure out-of-distribution perplexity (ood ppl) on 11 high-quality corpora:"
  • overtraining: Training a model so extensively that its signals become less helpful for transferring to a student. Example: "Architecture mismatch and teacher overtraining can degrade transfer even when the teacher attains a lower training loss."
  • pretraining corpus: The dataset from which training sequences are drawn for LLM pretraining. Example: "Given a token sequence x=(x1,…,xT)x = (x_1,\ldots,x_T) drawn from a pretraining corpus D\mathcal{D}, a student model with parameters θS\theta_S defines a next-token distribution"
  • standard pretraining baseline: A reference model trained without distillation (pure LM loss) for fair comparison. Example: "The standard pretraining baseline for each student corresponds to training with α=0\alpha = 0 (language modeling loss only) under the same data ordering and random seed as distillation pretraining."
  • teacher–student compatibility: How well the teacher’s knowledge aligns with the student’s capacity and training, affecting distillation effectiveness. Example: "Effective distillation in LLM pretraining depends on teacher--student compatibility, not strong teacher."
  • teacher–student relationships: The relative strengths of teacher and student defined by architecture size, token budget, and training loss. Example: "Together, size and tokens create a spectrum of teacher-student relationships:"
  • temperature (τ): A scaling applied to logits to soften or sharpen probability distributions during distillation. Example: "We use a temperature of τ=1\tau=1 for the knowledge distillation loss to apply no scaling to the logits."
  • training token budgets: The total number of tokens used to pretrain the teacher (or student), controlling training compute. Example: "Training token budgets."
  • train loss: The model’s loss on its training data; used here to categorize teacher strength relative to the student. Example: "Train loss: lower-to-higher or higher-to-lower, defined by teacher train loss compared to student baseline."
  • warm-up: A schedule that gradually increases the learning rate at the start of training to stabilize optimization. Example: "The peak learning rate is 3×10−43 \times 10^{-4} with a cosine decay to 10\% of the peak, linear warm-up for the first 5\% of steps, gradient clipping of 1.0, and bf16 precision."
  • weight decay: An L2 regularization term applied to weights to prevent overfitting. Example: "We train with AdamW using β1=0.9, β2=0.95, and weight decay 0.1."
View Paper Prompt  View All Prompts 

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 2 tweets with 157 likes about this paper.

Sign Up for Free