Measuring Task-Agnostic Training Data Influence Across Language Model Pretraining

Abstract: Measuring training data influence consistently across LLM pretraining is challenging. It is difficult to select downstream tasks or validation sets representative of a model's general capabilities, and reliance on task performance at intermediate checkpoints complicates comparisons across training. We propose a measure of training data influence that does not require selecting a downstream task or validation set as the attribution target. Specifically, we define an example's influence by how much its gradient update reduces the squared distance to the final parameters of a given pretraining run, and estimate this quantity from intermediate checkpoints without retraining. Applying the method to 18 configurations from the Pythia and PolyPythia suites, we find systematic temporal changes in influential data. Early in training, literature-related data are more strongly aligned with the trajectory toward the final parameters, whereas STEM data become more strongly aligned in later stages. This qualitative crossover is broadly consistent across model configurations. Our results provide a tractable trajectory-level view of how influential data change throughout pretraining, complementing influence analyses defined with respect to specific downstream tasks or validation sets.

• The paper introduces a task-agnostic influence measure that scores each training example by how much its update reduces squared parameter distance to the run’s final model, with negative scores identifying opponent examples.
• The checkpoint-based estimator correlates with exact step-level contributions at roughly 0.6 early in training and above 0.94 late, enabling post hoc analysis without retraining across 18 Pythia and PolyPythia configurations.
• The analysis finds that difficult data contribute most during mid-training, opponent examples rise to 8–10% near training’s end, and contribution patterns shift from Literature toward STEM domains, while rankings remain strongly endpoint-dependent.

Overview

"Measuring Task-Agnostic Training Data Influence Across LLM Pretraining" (2608.13515) addresses a structural limitation of training data attribution: nearly all existing influence measures—Influence Functions, Data Shapley, TracIn, TRAK—condition influence on a specific downstream target such as a test loss or task metric. For LLM pretraining, where the objective is broad general capability rather than a single task, this conditioning introduces two problems: no validation set is clearly representative of general capabilities, and task-conditioned influence estimates are hard to compare consistently across checkpoints, since intermediate models may not yet exhibit the target behavior. The paper proposes a task-agnostic alternative: an example's influence is defined as the reduction in squared L2 distance between the current parameters and the final parameters $\theta^*$ of a given pretraining run. This reframing, inspired by trajectory matching in dataset distillation but applied retrospectively to real training data, yields a common reference point throughout training and admits a checkpoint-based approximation that requires no retraining.

Method

At the mini-batch level, the contribution of update $t$ is defined exactly as $\mathrm{Cont}(B_t) = S_t - S_{t+1}$ where $S_t = \|\theta^* - \theta_t\|_2^2$. Expanding gives a closed form with two interpretable components: an alignment term $2\Delta_t^\top(\theta^* - \theta_t)$ and a squared-norm penalty $-\|\Delta_t\|_2^2$. Contributions can be negative ("opponent" examples whose updates move the model away from $\theta^*$).

The example-level decomposition assumes standard SGD (a stated simplification, since Pythia is trained with Adam; the authors follow TracIn's convention and note that an exact adaptive-optimizer decomposition would require accounting for optimizer state history). Under this decomposition, the penalty splits into a self-norm term and cross-example interaction terms within the batch. Under a pairwise-orthogonality simplification, the self-norm term recovers the TracIn self-influence term, and the framework inherits TracIn's observation that noisy or mislabeled examples receive large penalties.

Because per-step computation is infeasible, the method approximates $\theta_t \approx \theta_c$ and $\Delta_t \approx \theta_{c'} - \theta_c$ over checkpoint intervals $[c, c')$, enabling post hoc estimation from publicly released checkpoints. This approximation is the practical core of the paper and is validated empirically rather than theoretically.

Empirical findings

The analysis covers 18 configurations—six deduplicated Pythia scales (70M to 12B) plus PolyPythia variants at 160M and 410M—each with 154 checkpoints over 300B tokens of The Pile. For each interval, 1,000 examples are sampled from the actual training stream.

Contribution dynamics. Mean contribution peaks around 40k steps and declines thereafter. Most strikingly, the share of opponent examples rises from roughly 0–1% before 70k steps to 8–10% near the end of training, even while the learning rate decays smoothly. This indicates that late-stage updates become progressively misaligned with the direction toward the final parameters, not merely smaller—a pattern the closed-form definition predicts but a norm-based measure would miss.

Text difficulty. Binning examples by perplexity under the final Pythia-12B checkpoint, higher-PPL examples gain share during the middle stage: the highest-PPL bin's contribution share rises from about 20% to 25% while the lowest-PPL bin falls from 20% to 15%. Notably, this does not support a simple easy-to-difficult curriculum narrative—difficult data matter most mid-training, not monotonically.

Domain crossover. Using the NeMo Curator domain classifier over a domain-balanced sample (100 examples per domain per interval), STEM domains (Science; Computers and Electronics) dominate the bottom 5% of contributions in the first half of training, while Books and Literature becomes prominent among low contributors later; STEM share among the top 5% rises correspondingly. This Literature-to-STEM crossover is the paper's central qualitative claim and is broadly consistent across configurations.

Validation and robustness

Checkpoint approximation. On Pythia-70M-Deduped, retraining three intervals with full step-level state yields Pearson/Spearman correlations between exact and approximate contributions of about 0.6 (early), above 0.8 (middle), and above 0.94 (late). Agreement improves as inter-checkpoint parameter drift shrinks, but the early-training approximation is only moderately faithful—an acknowledged limitation, since the approximation was validated at 70M scale only.

Endpoint sensitivity. The measure is explicitly endpoint-dependent: using a 140k-step reference instead of the 143k final checkpoint leaves rankings nearly unchanged (mean Spearman 0.997, top-5% overlap 0.972), but earlier references diverge sharply—a 30k reference yields a negative mean Spearman correlation of −0.554 with the final-reference scores. Contribution rankings are therefore locally stable but globally reference-dependent, which the authors state plainly.

Cross-configuration consistency. The crossover pattern is preserved across scales and PolyPythia seeds, with two caveats: transitions occur later for smaller models (suggesting phase timing depends on scale), dynamics weaken in magnitude at larger scales, and data ordering introduces somewhat more variation than weight initialization. The anomalous PolyPythia-410M seed-3 run—a known training failure—shows qualitatively different dynamics, indicating the measure tracks training health.

Divergence from task-conditioned influence. A TracIn-style score computed against domain-specific held-out validation losses does not consistently recover the Literature-to-STEM crossover; its attribution patterns depend heavily on the chosen validation domain. This is a substantive result: task-agnostic and task-conditioned attribution yield different, not merely redundant, pictures of pretraining data roles.

Limitations and open questions

The authors are explicit about scope. The measure is relative to one run's endpoint and sensitive to parameterization; squared parameter distance is a geometric, not functional, notion of progress, and information-geometric alternatives would be substantially more expensive. The SGD assumption for example-level decomposition diverges from the Adam-based training actually used. Validation of the approximation is confined to the 70M model and three intervals. All experiments are within the Pythia/PolyPythia family and The Pile, so generality to other architectures, datasets, and recipes remains unestablished. The analysis is observational: it characterizes contribution variation but does not establish causal effects of changing data mixtures, and disentangling domain effects from correlated difficulty remains open.

Conclusion

This paper contributes a tractable, task-agnostic influence measure defined by progress toward a run's final parameters, with a checkpoint-based estimator validated to correlate strongly with exact computation in mid-to-late training. Its principal empirical findings—middle-stage dominance of high-perplexity data, a late-stage rise in opponent examples, and a robust Literature-to-STEM contribution crossover—provide trajectory-level evidence for stage-aware data composition and lend support to the common practice of increasing STEM data late in pretraining. The measure complements rather than replaces task-conditioned attribution, and its endpoint-dependence and scale-limited validation define the boundaries within which its conclusions should be interpreted.