Scaling Laws for Task-Specific LLM Distillation

Abstract: LLMs achieve strong performance across a growing range of domains, yet their scale poses deployment challenges in applications where latency and cost constraints are critical. This paper derives empirical scaling laws for domain-specific LLM compression, quantifying how in-domain and general knowledge performance scale with dataset size, compression ratio, supervision format, and iterative pruning schedule. Using quantitative finance as our application domain, we compare logit-based and LoRA-based distillation under iterative structural pruning, introducing a blended chain-of-thought supervision loss that stabilizes KL-divergence distillation over reasoning traces. In-domain task quality degrades predictably under compression while general-knowledge benchmarks collapse well before the same point; supervision format is the key driver of this tradeoff, with chain-of-thought supervision actively recovering general knowledge that pruning erases. We release the headline dataset FinHeadlineMix, scaling law results, and practical recommendations to provide a reusable framework for domain-specific compression decisions.

• The paper demonstrates that blended chain-of-thought distillation enhances general knowledge retention and model calibration in compressed LLMs.
• It systematically quantifies how dataset size, pruning schedules, and supervision formats affect in-domain performance on a financial news task.
• The study reveals that decayed iterative pruning outperforms uniform reduction, enabling extreme compression with minimal performance loss.

Scaling Laws for Task-Specific LLM Distillation: A Technical Essay

Motivation and Scope

The paper "Scaling Laws for Task-Specific LLM Distillation" (2606.24747) delineates a systematic empirical study of domain-specific LLM compression using knowledge distillation. Addressing critical deployment challenges in production scenarios with acute latency and cost constraints, the work quantifies how in-domain and general-knowledge performance scale as functions of dataset size, compression ratio, supervision format (label-only vs. blended chain-of-thought, CoT), and iterative pruning schedules. The authors ground their experiments in a financial news classification task with a synthetic, large-scale dataset and apply structured pruning and distillation (logit-based and LoRA-based) to a dense LLM teacher (Qwen3-32B). The study introduces a blended CoT supervision loss that stabilizes sequence-level KL divergence, anchoring the analysis in practical and theoretical scaling laws guiding model compression in real-world workloads.

Data Foundation and Task Taxonomy

The financial domain dataset (FinHeadlineMix) comprises 500,000 synthetic headlines annotated with 35 event classes representing a taxonomy typically encountered in quantitative finance applications. The labels are assigned via LLM reasoning mode, and the class distribution is heavily imbalanced, reflecting industry data realities (Figure 1).

Figure 1: Train set label distribution over the 35 event classes in the financial news dataset (training split, classes sorted by decreasing frequency).

The labeling protocol enforces deterministic evaluation splits and rigorous reproducibility, employing greedy decoding in the teacher's non-thinking mode to ensure stability of gold labels, eliminating artifacts from GPU reduction non-determinism.

Structured Pruning and Knowledge Distillation Methods

Pruning

Joint depth and width pruning is executed, guided by lightweight NAS proxy metrics, optimizing for hardware-optimal architectures. Depth pruning leverages block influence scores based on hidden-state cosine similarity, while width pruning judiciously removes attention heads and feedforward neurons, maintaining kernel compatibility.

Distillation Configurations

Four distilled configurations are systematically compared:

• LoRA distillation: Adapter-focused, label-only SFT, scaling adapter rank with dataset size to match effective information content.
• Logit-based distillation: Full model weight updates via sequence-level KL minimization on teacher logits, supporting both label-only and blended CoT supervision.
• Blended CoT supervision: Convex-weighting of loss between label and reasoning trace tokens, enabling stable gradient propagation across sequence regions and richer knowledge recovery.

Blended CoT was found unstable with LoRA; only logit-based distillation reliably leverages CoT traces.

Scaling Law Results

Data Efficiency and Performance Tradeoffs

Fixing compression at 50\% of teacher size, the effect of training dataset size and distillation method is isolated. Blended CoT KD and LoRA perform comparably on Macro F1 at low data, but CoT KD yields consistently superior calibration (NLL, Brier). LoRA overtakes CoT KD on all in-domain metrics at higher data volumes. Direct-label KD is highly fragile, failing to converge and exhibiting non-monotonic degradation.

General-knowledge benchmarks (MMLU, MMLU-Pro) show that LoRA delivers marginal improvements over the pruned baseline, recovering little general knowledge. In contrast, blended CoT KD actively recovers erased knowledge, outperforming the pruned baseline. Direct-label KD further erodes general-knowledge performance as data increase, highlighting supervision format as the critical driver of preservation vs. erosion of broad capabilities.

Figure 2: In-domain performance as a function of training dataset size. (a) Macro F1; (b) gold-label NLL; (c) Brier score.

Figure 3: General-knowledge benchmarks: MMLU and MMLU-Pro scores as a function of training dataset size for distillation methods at 50\% student size.

Iterative Pruning: Uniform and Decayed Schedules

Empirical scaling laws are established by varying compression ratios across iterative schedules. Uniform steps from teacher to 16\% model size are compared to decayed schedules (exponential, polynomial, cosine, linear) with progressively smaller later reductions. Multi-step iterative pruning surpasses direct-to-target compression in recoverable accuracy; the compression path is as crucial as the final size. Schedules that front-load reductions (exponential, polynomial) exhibit more stable late-stage recovery and permit further compression beyond uniform limits.

Figure 4: In-domain performance under uniform iterative pruning and distillation. (a) Macro F1; (b) gold-label NLL; (c) Brier score.

Figure 5: General-knowledge benchmarks for uniform iterative compression: MMLU and MMLU-Pro scores as a function of model size (compression ratio).

Decayed schedules demonstrate that late-stage refining steps allow models to be pushed below 16\% with competitive performance, with exponential decay offering maximal headroom and stability.

Figure 6: In-domain performance under decayed and uniform iterative schedules. (a) Macro F1; (b) gold-label NLL; (c) Brier score.

Figure 7: General-knowledge benchmarks for decayed and uniform iterative schedules: MMLU and MMLU-Pro scores at intermediate and final steps.

Scaling Laws Tables

Scaling law tables catalog model size, pruning steps, and achieved scores, quantifying how each compression schedule mediates tradeoffs between recoverable task performance and general-knowledge retention.

Empirical Observations and Contradictory Claims

The paper asserts that training distribution shift, not compression, is the dominant driver of student-teacher performance gaps; self-distillation on the unpruned teacher accounts for nearly twice the gap induced by compression to 16\%. LoRA leaves general knowledge at the pruned baseline, and blended CoT supervision is the only method to actively recover it. Direct-label KD degrades general knowledge further, contradicting any claim that label-only supervision suffices for general capability retention.

Macro F1, calibration, and accuracy metrics demonstrate quantitative degradation curves and scaling laws, with explicit reference baselines for both teacher and self-distilled students.

Practical, Theoretical, and Future Implications

Practical Recommendations

The results yield actionable guidance for model deployment:

• Use blended CoT KD for data efficiency and general-knowledge retention, especially when reasoning and broad capabilities are relevant.
• Employ LoRA for best in-domain classification when general knowledge is dispensable and ample data are available.
• Avoid direct-label KD in any scenario where general-knowledge preservation is required.
• Prefer decayed pruning schedules (exponential, polynomial, cosine) for extreme compression while maintaining high recovery probability; linear decay leads on in-domain scores at moderate targets.

Theoretical Implications

The empirically observed scaling laws reinforce the bias-variance tradeoff in distillation at the multi-billion parameter scale. Knowledge transfer is governed not just by model size or distillation method but crucially by supervision richness. Priors learned via CoT traces anchor broader reasoning, mitigating catastrophic forgetting endemic to label-only fine-tuning. Compression is feasible down to 16\% of teacher parameters given sufficient iterative refinement; the compression trajectory is non-trivially path-dependent.

Future Directions

The blended CoT formulation remains unstable in LoRA; addressing this could unlock data efficiency gains in adapter-centric training. Adaptive, schedule-free optimizers may further improve distillation stability, especially given sparse and heterogeneous gradients in blended CoT regimes. Validation across more domains, generative tasks, other teacher architectures (e.g., mixture-of-experts), and integration with quantization are open avenues.

Conclusion

The study presents a rigorous empirical investigation of scaling laws governing task-specific LLM distillation, with explicit characterization of tradeoffs between deployability, in-domain accuracy, and preservation of general knowledge. Blended CoT supervision is shown to be pivotal for data-efficient distillation and recovery of erased knowledge, and iterative pruning schedules (particularly decayed) are necessary for stable extreme compression. Detailed scaling laws, practical recommendations, and nuanced limitations facilitate principled domain-specific compression decisions for production LLM deployments.