Task-Stratified Scaling Laws in ML

• Task-stratified scaling laws are conditional relations that model downstream performance based on specific tasks, metrics, and evaluation conditions.
• They reveal that universal scaling trends can mask important variations, making tailored laws essential for accurate predictability in different domains.
• These insights guide compute-efficient strategies across NLP, robotics, and scientific ML by highlighting the impact of task-specific variables on scaling behavior.

to=arxiv_search 天天中彩票谁=10 大发pkingិនိုင်ငံjson սխեմա {"query":"Task-Stratified Scaling Laws downstream scaling laws unreliable (Lourie et al., 1 Jul 2025) machine translation (Isik et al., 2024) compute-efficient model ladders (Bhagia et al., 2024) context-aware scaling laws (Montgomery et al., 16 Oct 2025)", "max_results": 10, "sort_by": "relevance"} RTLR to=arxiv_search code 天天中彩票派奖json {"query":"(Lourie et al., 1 Jul 2025) OR (Isik et al., 2024) OR (Bhagia et al., 2024) OR (Montgomery et al., 16 Oct 2025)", "max_results": 10, "sort_by": "relevance"} Task-stratified scaling laws are scaling-law formulations in which downstream performance is modeled conditionally rather than by a single universal relation: at minimum by task, and often also by metric, validation distribution, benchmark implementation, objective family, context regime, or capability stratum. In current work on LLMs, retrieval, robotics, scientific ML, and compression, this perspective arises from a common empirical observation: smooth aggregate laws for pretraining loss, model size, data size, or compute do not reliably induce a single downstream law for all evaluations. Instead, scaling behavior is repeatedly found to be local to a “model-task-objective-metric-pipeline slice,” or to a closely related stratum, and extrapolation succeeds chiefly when that stratum is held fixed (Lourie et al., 1 Jul 2025, Bhagia et al., 2024, Montgomery et al., 16 Oct 2025).

1. Definition and scope

In the downstream-scaling literature, a scaling law is often defined operationally as a stable functional relation from pretraining loss or perplexity to downstream task performance that supports extrapolation. A favorable case is one in which task performance is “roughly linear in some monotonic transformation of validation loss,” for example

$y = a \exp\{c \cdot x\} + b,$

where $y$ is the downstream metric and $x$ is validation perplexity (Lourie et al., 1 Jul 2025). Task-stratified scaling laws reject the assumption that one such relation should hold globally across tasks and conditions.

The same conceptual shift appears in transfer learning. For machine translation, the object of interest is not upstream language-model loss but downstream cross-entropy and translation quality after finetuning, and the law is explicitly conditioned on pretraining data size $D_p$, finetuning data size $D_f$, and alignment between pretraining and downstream distributions (Isik et al., 2024). For pretrained LLMs evaluated on multiple-choice tasks, one line of work goes further and fits a separate law for each task: first predicting a task-specific loss from $(N,D)$, then predicting task accuracy from that task loss (Bhagia et al., 2024).

This produces a broader definition. Task-stratified scaling laws are conditional laws in which the fitted function, its exponents, its asymptote, and even its qualitative regime are indexed by task-specific or stratum-specific variables. In some papers those strata are explicit tasks; in others they are capability types, evaluation metrics, reranking paradigms, seen versus unseen regimes, or pretraining-composition conditions (Zhou et al., 26 Aug 2025, Seetharaman et al., 5 Mar 2026, Sartor et al., 2024, Uslu et al., 18 Jun 2026).

2. Why universal downstream laws are contested

The strongest direct challenge to universal downstream laws comes from the re-analysis in “Scaling Laws Are Unreliable for Downstream Tasks: A Reality Check” (Lourie et al., 1 Jul 2025). In its 46-task audit of Gadre et al., only 18 tasks—39%—are classified as “predictable,” meaning well-described by a linear scaling law after transforming the cross-entropy loss. The remaining 28 tasks fall into irregular categories: inverse, nonmonotonic, noisy, trendless, or breakthrough. On this evidence, clean linear downstream scaling is a minority case, not the default (Lourie et al., 1 Jul 2025).

The same paper shows that apparently benign design choices can change the inferred law. On HellaSwag, C4- and RedPajama-trained models align when validation perplexity is measured on C4, but diverge when it is measured on Paloma’s 100 Programming Languages; on CoQA, the relative superiority of C4 versus RedPajama flips depending on the validation corpus. Cross-study comparisons reinforce the point: MMLU trends upward in both compared setups, but CommonsenseQA is nonmonotonic in one and cleanly scalable in another, while BoolQ appears trendless in one setup and trendful in another (Lourie et al., 1 Jul 2025).

Task-level heterogeneity also appears in finetuning-time studies. “Scaling Laws Under the Microscope” reports that SQuAD 1.1, MNLI, and QNLI show relatively clean power laws, whereas BoolQ is not even monotonic in parameter count; MRPC and RACE improve materially if the smallest models are excluded, suggesting a critical minimum scale for those tasks (Ivgi et al., 2022). In the long-context setting, “Predicting Task Performance with Context-aware Scaling Laws” shows that compute alone predicts only an average performance level unless prompt length is modeled jointly with compute, and that arithmetic reasoning, commonsense reasoning, and machine translation exhibit different context exponents and different saturation points (Montgomery et al., 16 Oct 2025).

A recurrent misconception is therefore that a smooth upstream metric or a smooth scale variable should automatically yield a smooth downstream law. The literature instead shows that downstream regularity is contingent: it may hold in a given task-stratum, but it does not survive arbitrary pooling across tasks, metrics, or setups (Lourie et al., 1 Jul 2025).

3. Principal stratification axes

The literature does not yet provide a single formal taxonomy of strata, but several recurrent axes appear across domains.

Stratification axis	Representative instantiation	Evidence
Task or benchmark identity	MMLU, BoolQ, CommonsenseQA, HellaSwag, CoQA	Same nominal setup yields qualitatively different scaling curves by task (Lourie et al., 1 Jul 2025)
Validation distribution or loss operationalization	C4 vs 100 Programming Languages	Changing only the validation corpus can exaggerate differences or reverse conclusions (Lourie et al., 1 Jul 2025)
Objective family or pipeline stage	Pointwise, pairwise, listwise reranking	Different reranking paradigms exhibit different scaling behaviors (Seetharaman et al., 5 Mar 2026)
Metric	BLEU vs downstream CE; NDCG vs CE/MRR; neural vs behavioral alignment	The same model family can scale predictably for one metric and not for another (Isik et al., 2024, Seetharaman et al., 5 Mar 2026, Gokce et al., 2024)
Capability stratum	Knowledge memorization vs knowledge utilization	PTQ sensitivity differs sharply across these strata (Zhou et al., 26 Aug 2025)
Generalization regime	Seen/familiar vs unseen/novel tasks	Data-scaling exponents differ between seen and unseen robotics tasks (Sartor et al., 2024)
Context regime or task structure	Prompt length, context limit, hierarchy depth $h$	ICL and long-context downstream laws depend on context and task structure (Montgomery et al., 16 Oct 2025, Mehta et al., 9 Nov 2025)

Two points follow. First, “task” in this literature is often narrower than a benchmark label and broader than a single metric. A retrieval system, for example, may need separate laws for reranking rather than first-stage retrieval, and separate laws again for pointwise, pairwise, or listwise objectives (Seetharaman et al., 5 Mar 2026). Second, some papers support broader grouping only weakly. The reality-check paper strongly supports stratification by individual task and by exact experimental condition, but does not establish a benchmark-family taxonomy such as reasoning versus language understanding (Lourie et al., 1 Jul 2025).

4. Mathematical forms

There is no single canonical task-stratified equation. Instead, recent work uses a family of conditional forms tailored to the relevant stratum.

For pretrained LLMs in the overtrained regime, “Establishing Task Scaling Laws via Compute-Efficient Model Ladders” fits, for each task separately, a Chinchilla-style task-loss law

$$L(N,D) = \frac{A}{N^\alpha} + \frac{B}{D^\beta} + E,$$

followed by a sigmoidal task-loss-to-accuracy map

$Acc(L) = \frac{a}{1 + e^{-k(L-L_0)}} + b.$

The chained predictor

$\hat{Acc}(N,D)=Acc(\hat{L}(N,D))$

predicts four ranked-classification tasks within 2 absolute points on both target models, while higher-variance tasks such as ARC-Challenge, ARC-Easy, and OpenBookQA are materially less predictable (Bhagia et al., 2024).

In machine translation transfer learning, different downstream observables require different forms. “Scaling Laws for Downstream Task Performance in Machine Translation” models BLEU by

$y$0

and downstream cross-entropy by

$y$1

In aligned regimes, both behave monotonically with pretraining data size; under moderate misalignment, downstream CE can keep improving while BLEU fluctuates or worsens (Isik et al., 2024).

For reranking, the fitted family is a saturating power law on the metric itself:

$y$2

with a joint form

$y$3

This works well for NDCG@10 and MAP in many settings, but not uniformly for CE or MRR (Seetharaman et al., 5 Mar 2026).

For context-sensitive downstream performance, the law is multiplicative rather than additive. “Predicting Task Performance with Context-aware Scaling Laws” models performance as the product of a saturating transformed power law in training compute, a saturating transformed power law in prompt length, and a sigmoid penalty when prompt length exceeds the model’s context limit (Montgomery et al., 16 Oct 2025). This is explicitly a task-conditioned law family: arithmetic reasoning, commonsense reasoning, and machine translation share the template but not the fitted exponents or characteristic scales (Montgomery et al., 16 Oct 2025).

Under post-training quantization, the law incorporates PTQ-specific variables. “Scaling Laws for Task-Stratified Knowledge in Post-Training Quantized LLMs” defines effective bit-width as

$y$4

and fits a multiplicative law over model size $y$5, calibration set size $y$6, group size $y$7, and $y$8. The fitted exponents differ sharply between knowledge memorization and knowledge utilization, with memorization much more sensitive to model size, calibration, and effective precision (Zhou et al., 26 Aug 2025).

The mathematical moral is that task stratification does not merely shift intercepts. It often changes the functional family, the relevant independent variables, or the nonlinear link between an intermediate proxy and the task metric.

5. Empirical evidence across domains

In NLP transfer and multilingual MT, the main recurring pattern is conditional monotonicity. In encoder-decoder MT, well-aligned pretraining mixtures yield smooth BLEU and CE scaling, whereas moderately misaligned pretraining can make BLEU non-monotonic even while CE continues to improve (Isik et al., 2024). In decoder-only multilingual MT, separate scaling curves differ across language directions and domains; EN→DE and EN→FR do not scale identically, and domain-specific metrics can diverge, as CometKiwi does on finance (Caillaut et al., 2024). These results support the view that “translation” is not a single homogeneous scaling target.

Retrieval gives a closely related but more metric-centric picture. For dense retrieval, contrastive perplexity follows precise power-law scaling with model size and annotated data size, but the fitted exponents differ between MSMARCO and T2Ranking, and annotation quality changes the slope of the data-scaling curve (Fang et al., 2024). In reranking, NDCG@10 and MAP show reliable scaling behavior and can be forecast accurately, while CE and MRR are less stable; TREC DL ’19 MRR, in particular, does not show predictable model-scaling trends (Seetharaman et al., 5 Mar 2026). These papers indicate that retrieval scaling is stratified not only by task stage but by metric and supervision regime.

Embodied and scientific domains exhibit the same theme under different observables. In robotics, aggregate meta-analysis supports power-law scaling in failure rate with data, model size, and compute, but the clearest task-related stratification is between seen/familiar and unseen/novel tasks: filtered data-scaling exponents are markedly more negative for seen tasks than for unseen ones (Sartor et al., 2024). In power systems, scenario generalization follows an approximate power law with demonstrations and scenarios, while moving from single-task to 3-task and then 6-task multitask training does not materially degrade per-task performance; parameter scaling from 1B to 8B is comparatively weak and non-monotonic across tasks (Liu et al., 25 Mar 2025).

Task-stratified behavior is also prominent when the evaluation target is not ordinary task accuracy. For task-optimized visual models of the primate ventral stream, behavioral alignment continues to scale with compute according to

$y$9

whereas aggregate neural alignment saturates according to

$x$0

The scaling benefit is ordered by stratum as

$x$1

and the fitted compute-optimal recipe allocates more compute to data than to model size, approximately $x$2 and $x$3 (Gokce et al., 2024).

Finally, some of the sharpest examples of task-conditioned exponents come from scientific ML. In jet-tagging transfer, changing only the pretraining composition shifts the compute-optimal frontier from roughly balanced scaling

$x$4

under scratch training to a strongly data-favoring regime

$x$5

when pretraining uses the full QCD + res2p + res34p mixture (Uslu et al., 18 Jun 2026). In neural force fields, equivariance changes the exponents themselves: the compute exponent rises from about $x$6 for an unconstrained MPNN to about $x$7 for eSEN, with higher-order equivariant representations scaling better than lower-order ones (Ngo et al., 10 Oct 2025). These cases show that stratification can be induced not only by downstream benchmark identity but by alignment structure and symmetry class.

6. Theory, interpretation, and open problems

Theoretical work helps explain why task-stratified laws should exist. “Explaining Neural Scaling Laws” distinguishes variance-limited and resolution-limited regimes, with universal exponent $x$8 in the variance-limited case and task-dependent exponents in the resolution-limited case. In the random-feature and kernel setting, if

$x$9

then

$D_p$0

so $D_p$1 (Bahri et al., 2021). “Neural Scaling Laws Rooted in the Data Distribution” offers a complementary picture with two regimes: a subcritical/critical regime in which the data decompose into power-law-distributed discrete subtasks, and a supercritical regime dominated by a single manifold-like cluster (Brill, 2024). “Scaling Laws and In-Context Learning” then makes task structure explicit, stating that ICL error scales with exponent

$D_p$2

where $D_p$3 is hierarchy depth, and that the critical emergence scale obeys

$D_p$4

again tying exponents and thresholds to task structure rather than to model size alone (Mehta et al., 9 Nov 2025).

Several limitations recur across empirical studies. The reality-check paper classifies 46 downstream curves into six qualitative categories but does not specify a numerical fitting pipeline, threshold, or exact $D_p$5 cutoff for those assignments (Lourie et al., 1 Jul 2025). The PTQ paper reports adjusted $D_p$6 for fitted laws but no confidence intervals, bootstrap uncertainty, held-out prediction error, or formal cross-validation (Zhou et al., 26 Aug 2025). The robotics meta-analysis explicitly states that embodied AI lacks standardized general-purpose benchmarks, preventing clean comparability across tasks, environments, and embodiments (Sartor et al., 2024). These limitations matter because task stratification is easiest to motivate empirically and hardest to formalize statistically.

A second open problem is granularity. The evidence strongly supports fitting separate laws per task in many settings, but broader groupings remain unsettled. Some papers support coarse strata such as memorization versus utilization, seen versus unseen, neural versus behavioral alignment, or pointwise versus pairwise versus listwise objectives (Zhou et al., 26 Aug 2025, Sartor et al., 2024, Gokce et al., 2024, Seetharaman et al., 5 Mar 2026). Broader semantic families—reasoning, language understanding, coding, safety, multilinguality—are often plausible but not yet established by the cited studies (Lourie et al., 1 Jul 2025).

A third open problem is transferability. One of the most robust conclusions in the literature is methodological rather than theoretical: a law validated in one experimental context should not be assumed to hold in another without diagnostics. This applies to validation corpus choice, metric, prompt format, answer set, architecture, candidate-generation stage, context window, and pretraining mixture (Lourie et al., 1 Jul 2025, Seetharaman et al., 5 Mar 2026, Montgomery et al., 16 Oct 2025, Uslu et al., 18 Jun 2026).

Task-stratified scaling laws therefore describe not a single settled model class but an emerging research program. Its common thesis is that downstream scaling relations are conditional laws whose stability depends on the stratum being held fixed. The strongest empirical support is at the individual-task level; the broader theoretical ambition is to explain when those task-level laws can be pooled, when they must remain separate, and which latent structural variables determine their exponents, thresholds, and breakdowns (Lourie et al., 1 Jul 2025, Bahri et al., 2021, Brill, 2024).