Scaling Laws for CPT

• Scaling Laws for CPT are predictive models that define quantitative relationships between training tokens, perplexity measures, and domain adaptation performance.
• They incorporate diverse formulations—from perplexity-aware to token-per-parameter laws—to optimize data curation and learning rate schedules in continual pre-training.
• These laws enable rapid, cost-effective tuning by analytically determining optimal data mixtures and hyperparameters, significantly reducing experimental overhead.

Scaling laws for CPT (Continual Pre-Training) establish predictive, quantitative relationships between model performance, data characteristics, corpus mixtures, and training scale in the context of adapting large pre-trained models to new domains via further pre-training. Influential recent work has generalized classical scaling concepts to the unique setting and constraints of CPT, introducing new structures that account for domain shifts, mixing of general and domain-specific data, learning rate schedules, and sample informativeness. This article synthesizes the major theoretical advances, empirical validations, and current limitations underlying contemporary CPT scaling law research.

1. Fundamental Scaling Law Structures in CPT

Classical scaling laws for LLMs encode the monotonic, sublinear improvement in test loss when increasing training corpus size, typically formalized as:

$L(N) = A N^{-\alpha} + C$

where $L$ is validation/test loss, $N$ is the number of training tokens, $\alpha$ the scaling exponent, and $C$ the irreducible loss floor. In continual pre-training, this relationship is disrupted by non-homogeneous data and domain/adaption effects, demanding multi-axis and context-aware generalizations.

Several principal formulations have emerged for modeling CPT loss:

• Perplexity-Aware Scaling Law: Introduces a mapping from the perplexity landscape—mean and standard deviation ($\mu, \sigma$) of base-model-assigned domain perplexities—directly to held-out loss after CPT, replacing naive token-counting with an importance-weighted measure (Liu et al., 25 Dec 2025):

  $$\hat{L}(\mu, \sigma, D) = E + \frac{D_c}{\mu^{\alpha_\mu(\sigma)} \sigma^{\alpha_\sigma(\mu)} D^{\alpha_D}}$$

• Mixture & Domain Scaling Laws: Predicts held-out loss as a joint function of model size $N$, token budget $D$, and mixture ratio $r$ between general and domain corpora. The D-CPT law (Que et al., 2024) and the CMR law (Gu et al., 2024) exemplify this approach, introducing scaling and trade-off terms for mixture optimization:

  $L$0

• Token-Per-Parameter-Scaled Laws: Recent PTPP-aware laws (Goffinet et al., 27 Oct 2025) make the pretraining tokens-per-parameter, $L$1, an explicit variable to improve extrapolation at various adaptation scales, fitting

  $L$2

  with $L$3 “gated” as a function of $L$4.

• Learning Dynamics–Based Laws: Models the full CPT loss curve by decoupling the impact of learning rate annealing from distribution shift, producing accurate predictions for arbitrary CPT schedules and replay ratios (Wang et al., 12 May 2025).

2. Data Informativeness and Perplexity Landscapes

CPT performance is not solely a function of data volume. Heterogeneity in the domain-specific corpus leads to sharply variable informativeness, relevance, and redundancy. The perplexity-aware scaling law addresses this via the statistical moments ($L$5) of base-model perplexity on candidate samples:

• Low-PPL samples are too predictable (redundant); high-PPL samples are often noisy or off-topic; intermediate-PPL samples most closely close the knowledge gap characteristic of continual pre-training (Liu et al., 25 Dec 2025).
• The loss surface plotted over ($L$6) exhibits a “bowl-shaped” minimum. The law admits a functional minimizer $L$7 which guides adaptive selection of the highest-utility training subsets.
• Empirical validation shows that sampling to match this perplexity optimum leads to faster convergence, higher domain-task accuracy, and avoids both redundancy and overfitting.

3. Predictive Laws for Mixture Optimization and Resource Allocation

A central question in CPT is the optimal general–domain mixture: how much domain-specific data maximizes domain specialization without catastrophic forgetting of base/general competencies. Scaling law–driven approaches provide analytic strategies:

• Critical Mixture Ratio (CMR) Law: Defines the unique mixture ratio $L$8 that maximizes domain performance while constraining general loss to an acceptable tolerance. Both domain and general losses follow power-laws in $L$9:

  $N$0

  Subject to a Lagrangian constraint balancing general and domain loss drift over token budget $N$1, $N$2 itself empirically follows a two-parameter power-law in $N$3:

  $N$4

  This law enables closed-form mixture planning for any token budget, reducing or eliminating grid-search (Gu et al., 2024).

• D-CPT Law and Cross-Domain Generalization: D-CPT law enables prediction of domain and general losses for arbitrary mixture, size, and budget via low-cost pilot runs, with cross-domain extension incorporating a single “domain learnability coefficient” $N$5 determined from a minimal pilot run. Performance is maintained with $N$6 (Que et al., 2024).
• Replay Ratio and Budget Planning: Laws parameterized for replay ratio ($N$7) and token-per-parameter constraints enable analytic minimization of adaptation token count under multi-domain retention constraints. The optimal replay for a loss-bounded CPT is solved directly from the scaling law system, obviating expensive trial-and-error approaches (Goffinet et al., 27 Oct 2025, Wang et al., 12 May 2025).

4. Scaling Law Fitting Methodologies

Robust scaling law development for CPT adopts systematic data collection and fitting pipelines:

• Subset Sampling Over Multi-Axis Grids: Calibration is performed via small-scale experiments spanning model sizes, token budgets, mixture ratios, and candidate data statistics. The standard approach evaluates hundreds of mixture/loss points, then fits generative laws using Huber or squared error in log-loss space.
• Validation and Extrapolation: Laws are tested on holdout scales, mixture ratios, and, when feasible, entirely new domains (e.g., academic vs finance). Cross-domain generalization is assessed with coefficients learned from limited pilot adaptions.
• Optimization Procedures: Constrained L-BFGS fitting is commonly employed, and analytic solutions for optimal ratios or replay parameters are derived via closed-form critical points (often involving Lagrange multipliers).
• Perplexity Surface Matching: For perplexity-aware laws, the optimum is found via minimization over the empirical perplexity landscape; data curation is then performed via distance-to-optimum matching in ($N$8)-space.

5. Empirical Characterization and Data Regimes

Recent studies provide detailed empirical characterizations of scaling exponents, learning curves, and trade-offs:

Law / Domain	Scaling Exponent(s)	Key Regimes / Effects	Reference
Perplexity-aware	$N$9, $\alpha$0 etc.	Loss falls as tokens, mean-PPL, and std-PPL are simultaneously optimized	(Liu et al., 25 Dec 2025)
CMR Law (Finance)	$\alpha$1	$\alpha$2 rises with model size and token budget	(Gu et al., 2024)
D-CPT Law	$\alpha$3	Accurate from small pilot runs; cross-domain generalization via $\alpha$4	(Que et al., 2024)
Recommendation CPT	$\alpha$5 (modality)	UIH: 0.55, CF: 0.35, item–text: 0.16; curves strictly power-law only for high-quality synthetic data	(Zhang et al., 7 Feb 2026)
PTPP-aware Law	$\alpha$6	Loss reduction saturates with higher PTPP; analytic replay/compute planning	(Goffinet et al., 27 Oct 2025)

Significance includes:

• High-quality, low-redundancy data is far more valuable than sheer volume; the correct data schedule (modality mix, perplexity, diversity) dramatically amplifies learning exponents.
• For recommendation and other specialized domains, robust scaling is attainable only via bias-free synthetic data; repeated raw logs yield subscaling or plateau regimes.
• Data repetition beyond a critical threshold (UIH example: 16× repeat) breaks scaling, producing flat or even worsening validation loss (Zhang et al., 7 Feb 2026).
• In all validated domains, mixture optimization and early law-based tuning can eliminate up to 99% of unnecessary CPT computation compared to heuristic grid-search.

6. Limitations and Open Directions

Despite strong goodness-of-fit and boundary-crossing generalizations, CPT scaling laws remain circumscribed by domain, mixture, and architecture:

• Most empirical studies are limited to pre-determined domains (e.g., medical, legal, finance, code) and a single or small family of base LLMs; transfer to highly low-resource, domain-shifted, or multilingual scenarios is not yet fully substantiated (Que et al., 2024, Gu et al., 2024).
• Near-zero or near-unity mixture ratios, and extremely small or large batch sizes, can produce inflection or breakdown points absent from the scaling formulas.
• Many laws treat model size, data scale, and time as separable axes; real effects (e.g., interplay between scaling exponents and PTPP at large scales) are likely more complex (Goffinet et al., 27 Oct 2025).
• Most current selection algorithms for optimal data subsets (e.g., distance-to-optimum in perplexity) are greedy and local; more global or multi-objective optimization may yield further efficiency.
• Extensions to three-axis scaling (model size, data size, perplexity/informativeness) and integration of compute scaling (FLOPs) are active areas for future development.

7. Impact and Practical Application

The predictive accuracy and tractability of modern CPT scaling laws have directly enabled highly efficient domain adaptation, principled resource allocation, and reproducible assessment of downstream specialization capabilities:

• Avoidance of catastrophic forgetting is now achievable with bounded computational cost via closed-form critical mixture prediction.
• Fast pilot-run–based law fitting allows rapid evaluation of new domains or corpora, substantially reducing experimental time and energy requirements.
• Data-centric approaches—specifically the curation of high-utility, unique, and relevance-optimal samples—have supplanted model-centric scaling in settings where compositional and domain shift render naive scaling suboptimal (Zhang et al., 7 Feb 2026).
• Analytical laws now support pipeline integration, dynamically guiding hyperparameter choices, replay/adaptation planning, and mixture policies to maximize delivered performance within compute/throughput constraints.

Recent research converges on a unified perspective: scaling laws for CPT, when rigorously calibrated, provide the foundation for systematic, cost-effective, and high-performing domain adaptation at scale, superseding heuristic and brute-force paradigms. The continued refinement and domain extension of such laws are poised to determine the future of efficient LLM adaptation across scientific, technical, and commercial domains.