---
title: Task-Stratified Scaling Laws in ML
url: https://www.emergentmind.com/topics/task-stratified-scaling-laws
type: topic
---

# Task-Stratified Scaling Laws in ML

to=arxiv_search  天天中彩票谁=10  大发pkingិនိုင်ငံjson սխեմա
{"query":"Task-Stratified Scaling Laws downstream scaling laws unreliable 2507.00885 machine translation 2402.04177 compute-efficient model ladders 2412.04403 context-aware scaling laws 2510.14919", "max_results": 10, "sort_by": "relevance"}	RTLR to=arxiv_search code  天天中彩票派奖json
{"query":"2507.00885 OR 2402.04177 OR 2412.04403 OR 2510.14919", "max_results": 10, "sort_by": "relevance"}
Task-stratified scaling laws are scaling-law formulations in which downstream performance is modeled conditionally rather than by a single universal relation: at minimum by task, and often also by metric, validation distribution, benchmark implementation, objective family, context regime, or capability stratum. In current work on language models, retrieval, robotics, scientific ML, and compression, this perspective arises from a common empirical observation: smooth aggregate laws for pretraining loss, model size, data size, or compute do not reliably induce a single downstream law for all evaluations. Instead, scaling behavior is repeatedly found to be local to a “model-task-objective-metric-pipeline slice,” or to a closely related stratum, and extrapolation succeeds chiefly when that stratum is held fixed [2507.00885] [2412.04403] [2510.14919].

## 1. Definition and scope

In the downstream-scaling literature, a scaling law is often defined operationally as a stable functional relation from pretraining loss or perplexity to downstream task performance that supports extrapolation. A favorable case is one in which task performance is “roughly linear in some monotonic transformation of validation loss,” for example
$$
y = a \exp\{c \cdot x\} + b,
$$
where \(y\) is the downstream metric and \(x\) is validation perplexity [2507.00885]. Task-stratified scaling laws reject the assumption that one such relation should hold globally across tasks and conditions.

The same conceptual shift appears in transfer learning. For machine translation, the object of interest is not upstream language-model loss but downstream cross-entropy and translation quality after finetuning, and the law is explicitly conditioned on pretraining data size \(D_p\), finetuning data size \(D_f\), and alignment between pretraining and downstream distributions [2402.04177]. For pretrained language models evaluated on multiple-choice tasks, one line of work goes further and fits a separate law for each task: first predicting a task-specific loss from \((N,D)\), then predicting task accuracy from that task loss [2412.04403].

This produces a broader definition. Task-stratified scaling laws are conditional laws in which the fitted function, its exponents, its asymptote, and even its qualitative regime are indexed by task-specific or stratum-specific variables. In some papers those strata are explicit tasks; in others they are capability types, evaluation metrics, reranking paradigms, seen versus unseen regimes, or pretraining-composition conditions [2508.18609] [2603.04816] [2405.14005] [2606.19781].

## 2. Why universal downstream laws are contested

The strongest direct challenge to universal downstream laws comes from the re-analysis in “Scaling Laws Are Unreliable for Downstream Tasks: A Reality Check” [2507.00885]. In its 46-task audit of Gadre et al., only 18 tasks—39%—are classified as “predictable,” meaning well-described by a linear scaling law after transforming the cross-entropy loss. The remaining 28 tasks fall into irregular categories: inverse, nonmonotonic, noisy, trendless, or breakthrough. On this evidence, clean linear downstream scaling is a minority case, not the default [2507.00885].

The same paper shows that apparently benign design choices can change the inferred law. On HellaSwag, C4- and RedPajama-trained models align when validation perplexity is measured on C4, but diverge when it is measured on Paloma’s 100 Programming Languages; on CoQA, the relative superiority of C4 versus RedPajama flips depending on the validation corpus. Cross-study comparisons reinforce the point: MMLU trends upward in both compared setups, but CommonsenseQA is nonmonotonic in one and cleanly scalable in another, while BoolQ appears trendless in one setup and trendful in another [2507.00885].

Task-level heterogeneity also appears in finetuning-time studies. “Scaling Laws Under the Microscope” reports that SQuAD 1.1, MNLI, and QNLI show relatively clean power laws, whereas BoolQ is not even monotonic in parameter count; MRPC and RACE improve materially if the smallest models are excluded, suggesting a critical minimum scale for those tasks [2202.06387]. In the long-context setting, “Predicting Task Performance with Context-aware Scaling Laws” shows that compute alone predicts only an average performance level unless prompt length is modeled jointly with compute, and that arithmetic reasoning, commonsense reasoning, and machine translation exhibit different context exponents and different saturation points [2510.14919].

A recurrent misconception is therefore that a smooth upstream metric or a smooth scale variable should automatically yield a smooth downstream law. The literature instead shows that downstream regularity is contingent: it may hold in a given task-stratum, but it does not survive arbitrary pooling across tasks, metrics, or setups [2507.00885].

## 3. Principal stratification axes

The literature does not yet provide a single formal taxonomy of strata, but several recurrent axes appear across domains.

| Stratification axis | Representative instantiation | Evidence |
|---|---|---|
| Task or benchmark identity | MMLU, BoolQ, CommonsenseQA, HellaSwag, CoQA | Same nominal setup yields qualitatively different scaling curves by task [2507.00885] |
| Validation distribution or loss operationalization | C4 vs 100 Programming Languages | Changing only the validation corpus can exaggerate differences or reverse conclusions [2507.00885] |
| Objective family or pipeline stage | Pointwise, pairwise, listwise reranking | Different reranking paradigms exhibit different scaling behaviors [2603.04816] |
| Metric | BLEU vs downstream CE; NDCG vs CE/MRR; neural vs behavioral alignment | The same model family can scale predictably for one metric and not for another [2402.04177] [2603.04816] [2411.05712] |
| Capability stratum | Knowledge memorization vs knowledge utilization | PTQ sensitivity differs sharply across these strata [2508.18609] |
| Generalization regime | Seen/familiar vs unseen/novel tasks | Data-scaling exponents differ between seen and unseen robotics tasks [2405.14005] |
| Context regime or task structure | Prompt length, context limit, hierarchy depth \(h\) | ICL and long-context downstream laws depend on context and task structure [2510.14919] [2511.06232] |

Two points follow. First, “task” in this literature is often narrower than a benchmark label and broader than a single metric. A retrieval system, for example, may need separate laws for reranking rather than first-stage retrieval, and separate laws again for pointwise, pairwise, or listwise objectives [2603.04816]. Second, some papers support broader grouping only weakly. The reality-check paper strongly supports stratification by individual task and by exact experimental condition, but does not establish a benchmark-family taxonomy such as reasoning versus language understanding [2507.00885].

## 4. Mathematical forms

There is no single canonical task-stratified equation. Instead, recent work uses a family of conditional forms tailored to the relevant stratum.

For pretrained language models in the overtrained regime, “Establishing Task Scaling Laws via Compute-Efficient Model Ladders” fits, for each task separately, a Chinchilla-style task-loss law
$$
L(N,D) = \frac{A}{N^\alpha} + \frac{B}{D^\beta} + E,
$$
followed by a sigmoidal task-loss-to-accuracy map
$$
Acc(L) = \frac{a}{1 + e^{-k(L-L_0)}} + b.
$$
The chained predictor
$$
\hat{Acc}(N,D)=Acc(\hat{L}(N,D))
$$
predicts four ranked-classification tasks within 2 absolute points on both target models, while higher-variance tasks such as ARC-Challenge, ARC-Easy, and OpenBookQA are materially less predictable [2412.04403].

In machine translation transfer learning, different downstream observables require different forms. “Scaling Laws for Downstream Task Performance in Machine Translation” models BLEU by
$$
f(D_p) = (\log (A \cdot D_p^{\alpha}))^{\beta},
$$
and downstream cross-entropy by
$$
L(D_p) = E + \frac{A}{D_p^{\alpha}}.
$$
In aligned regimes, both behave monotonically with pretraining data size; under moderate misalignment, downstream CE can keep improving while BLEU fluctuates or worsens [2402.04177].

For reranking, the fitted family is a saturating power law on the metric itself:
$$
\mathcal{M}(M)=a-bM^{-c}, \qquad
\mathcal{M}(S)=a-bS^{-c},
$$
with a joint form
$$
\mathcal{M}(M,S)=a-bM^{-\alpha}-cS^{-\beta}.
$$
This works well for NDCG@10 and MAP in many settings, but not uniformly for CE or MRR [2603.04816].

For context-sensitive downstream performance, the law is multiplicative rather than additive. “Predicting Task Performance with Context-aware Scaling Laws” models performance as the product of a saturating transformed power law in training compute, a saturating transformed power law in prompt length, and a sigmoid penalty when prompt length exceeds the model’s context limit [2510.14919]. This is explicitly a task-conditioned law family: arithmetic reasoning, commonsense reasoning, and machine translation share the template but not the fitted exponents or characteristic scales [2510.14919].

Under post-training quantization, the law incorporates PTQ-specific variables. “Scaling Laws for Task-Stratified Knowledge in Post-Training Quantized Large Language Models” defines effective bit-width as
$$
B_{eff} = W_{base} + \frac{b_s + b_z}{G},
$$
and fits a multiplicative law over model size \(N\), calibration set size \(C_b\), group size \(G\), and \(B_{eff}\). The fitted exponents differ sharply between knowledge memorization and knowledge utilization, with memorization much more sensitive to model size, calibration, and effective precision [2508.18609].

The mathematical moral is that task stratification does not merely shift intercepts. It often changes the functional family, the relevant independent variables, or the nonlinear link between an intermediate proxy and the task metric.

## 5. Empirical evidence across domains

In NLP transfer and multilingual MT, the main recurring pattern is conditional monotonicity. In encoder-decoder MT, well-aligned pretraining mixtures yield smooth BLEU and CE scaling, whereas moderately misaligned pretraining can make BLEU non-monotonic even while CE continues to improve [2402.04177]. In decoder-only multilingual MT, separate scaling curves differ across language directions and domains; EN→DE and EN→FR do not scale identically, and domain-specific metrics can diverge, as CometKiwi does on finance [2409.15051]. These results support the view that “translation” is not a single homogeneous scaling target.

Retrieval gives a closely related but more metric-centric picture. For dense retrieval, contrastive perplexity follows precise power-law scaling with model size and annotated data size, but the fitted exponents differ between MSMARCO and T2Ranking, and annotation quality changes the slope of the data-scaling curve [2403.18684]. In reranking, NDCG@10 and MAP show reliable scaling behavior and can be forecast accurately, while CE and MRR are less stable; TREC DL ’19 MRR, in particular, does not show predictable model-scaling trends [2603.04816]. These papers indicate that retrieval scaling is stratified not only by task stage but by metric and supervision regime.

Embodied and scientific domains exhibit the same theme under different observables. In robotics, aggregate meta-analysis supports power-law scaling in failure rate with data, model size, and compute, but the clearest task-related stratification is between seen/familiar and unseen/novel tasks: filtered data-scaling exponents are markedly more negative for seen tasks than for unseen ones [2405.14005]. In power systems, scenario generalization follows an approximate power law with demonstrations and scenarios, while moving from single-task to 3-task and then 6-task multitask training does not materially degrade per-task performance; parameter scaling from 1B to 8B is comparatively weak and non-monotonic across tasks [2503.20040].

Task-stratified behavior is also prominent when the evaluation target is not ordinary task accuracy. For task-optimized visual models of the primate ventral stream, behavioral alignment continues to scale with compute according to
$$
S=1-1.4\tilde{C}^{-0.06},
$$
whereas aggregate neural alignment saturates according to
$$
S=0.48-0.55\tilde{C}^{-0.16}.
$$
The scaling benefit is ordered by stratum as
$$
\text{Behavior} > \text{IT} > \text{V4} > \text{V2} > \text{V1},
$$
and the fitted compute-optimal recipe allocates more compute to data than to model size, approximately \(D \approx C^{0.7}\) and \(N \approx C^{0.3}\) [2411.05712].

Finally, some of the sharpest examples of task-conditioned exponents come from scientific ML. In jet-tagging transfer, changing only the pretraining composition shifts the compute-optimal frontier from roughly balanced scaling
$$
N^* \propto C^{0.517}, \qquad D^* \propto C^{0.483}
$$
under scratch training to a strongly data-favoring regime
$$
N^* \propto C^{0.224}, \qquad D^* \propto C^{0.776}
$$
when pretraining uses the full QCD + res2p + res34p mixture [2606.19781]. In neural force fields, equivariance changes the exponents themselves: the compute exponent rises from about \(0.142\) for an unconstrained MPNN to about \(0.403\) for eSEN, with higher-order equivariant representations scaling better than lower-order ones [2510.09768]. These cases show that stratification can be induced not only by downstream benchmark identity but by alignment structure and symmetry class.

## 6. Theory, interpretation, and open problems

Theoretical work helps explain why task-stratified laws should exist. “Explaining Neural Scaling Laws” distinguishes variance-limited and resolution-limited regimes, with universal exponent \(1\) in the variance-limited case and task-dependent exponents in the resolution-limited case. In the random-feature and kernel setting, if
$$
\lambda_i \sim i^{-(1+\alpha_K)},
$$
then
$$
L(D)\propto D^{-\alpha_K}, \qquad L(P)\propto P^{-\alpha_K},
$$
so \(\alpha_D=\alpha_P=\alpha_K\) [2102.06701]. “Neural Scaling Laws Rooted in the Data Distribution” offers a complementary picture with two regimes: a subcritical/critical regime in which the data decompose into power-law-distributed discrete subtasks, and a supercritical regime dominated by a single manifold-like cluster [2412.07942]. “Scaling Laws and In-Context Learning” then makes task structure explicit, stating that ICL error scales with exponent
$$
\alpha=\frac{1}{2(h+1)},
$$
where \(h\) is hierarchy depth, and that the critical emergence scale obeys
$$
N_c=\Theta((k\cdot h)^{2(h+1)}),
$$
again tying exponents and thresholds to task structure rather than to model size alone [2511.06232].

Several limitations recur across empirical studies. The reality-check paper classifies 46 downstream curves into six qualitative categories but does not specify a numerical fitting pipeline, threshold, or exact \(R^2\) cutoff for those assignments [2507.00885]. The PTQ paper reports adjusted \(R^2\) for fitted laws but no confidence intervals, bootstrap uncertainty, held-out prediction error, or formal cross-validation [2508.18609]. The robotics meta-analysis explicitly states that embodied AI lacks standardized general-purpose benchmarks, preventing clean comparability across tasks, environments, and embodiments [2405.14005]. These limitations matter because task stratification is easiest to motivate empirically and hardest to formalize statistically.

A second open problem is granularity. The evidence strongly supports fitting separate laws per task in many settings, but broader groupings remain unsettled. Some papers support coarse strata such as memorization versus utilization, seen versus unseen, neural versus behavioral alignment, or pointwise versus pairwise versus listwise objectives [2508.18609] [2405.14005] [2411.05712] [2603.04816]. Broader semantic families—reasoning, language understanding, coding, safety, multilinguality—are often plausible but not yet established by the cited studies [2507.00885].

A third open problem is transferability. One of the most robust conclusions in the literature is methodological rather than theoretical: a law validated in one experimental context should not be assumed to hold in another without diagnostics. This applies to validation corpus choice, metric, prompt format, answer set, architecture, candidate-generation stage, context window, and pretraining mixture [2507.00885] [2603.04816] [2510.14919] [2606.19781].

Task-stratified scaling laws therefore describe not a single settled model class but an emerging research program. Its common thesis is that downstream scaling relations are conditional laws whose stability depends on the stratum being held fixed. The strongest empirical support is at the individual-task level; the broader theoretical ambition is to explain when those task-level laws can be pooled, when they must remain separate, and which latent structural variables determine their exponents, thresholds, and breakdowns [2507.00885] [2102.06701] [2412.07942].

Source: https://www.emergentmind.com/topics/task-stratified-scaling-laws