Influence Score (IS) in Code LLM Pretraining

Updated 5 July 2026

Influence Score (IS) is a metric that measures the drop in downstream validation loss after one training step on a sample, directly indicating its benefit or harm.
It ranks code pretraining examples by comparing loss before and after updating the model, effectively identifying samples that improve or degrade performance.
Empirical studies using CodeLLMs show that IS-based filtering enhances model performance on code tasks, though its computation is intensive and highly task-dependent.

Searching arXiv for the cited paper and closely related influence-score work in language/data selection. Influence Score (IS), written as $DIScore$ in "An Empirical Study on Influence-Based Pretraining Data Selection for Code LLMs" (Xing et al., 9 Apr 2026), is a sample-level measure of how much a single pretraining example helps or hurts a code LLM on a downstream programming task. In that study, the score is defined for code pretraining data by the change in downstream-task validation loss after one training step on an individual sample, making it an oracle, validation-loss-based criterion for ranking pretraining data by downstream usefulness. The concept is introduced to address a gap between prior pretraining-data filtering work on general datasets and the specific demands of programming datasets, where downstream evaluation is typically generative and often execution-based rather than classification-based (Xing et al., 9 Apr 2026).

1. Formal definition

For a training example $x_i$ and a downstream validation set $D_r$ , the paper defines influence as

$I_M(x_i; D_r) = L(D_r \mid M) - L(D_r \mid A(M, x_i)),$

where $M$ is the current model, $A(M, x_i)$ is the model after one training step on $x_i$ , and $L(\cdot)$ is the loss on the validation set $D_r$ (Xing et al., 9 Apr 2026). The score is therefore the drop in validation loss after a single update on that sample.

The sign has an immediate interpretation. If the loss on $D_r$ decreases after training on $x_i$ 0, the score is positive and the sample is treated as beneficial. If the loss increases, the score is negative and the sample is treated as harmful. In this formulation, IS is not an abstract importance heuristic; it is a direct estimate of whether a pretraining sample improves or degrades downstream-task performance as measured through validation loss (Xing et al., 9 Apr 2026).

A defining feature of the method is that it is an oracle score computed from validation-set loss, not a score predicted from sample features or from a small proxy model. This distinguishes it from several strands of earlier language influence research, including TracIn-style gradient scoring for language classification (Yeh et al., 2022), influence-based pruning studies in NLU (Anand et al., 2023), and multi-stage influence formulations that trace finetuned predictions back to pretraining examples (Chen et al., 2020). This suggests that the paper’s IS should be understood as a task-grounded data-selection signal rather than merely another checkpoint-gradient proxy.

2. Adaptation to generative programming tasks

The central methodological problem is that code benchmarks are usually generative and are often judged by execution-based metrics such as pass@k, accuracy, or exact match, whereas the influence definition above requires a loss-based signal (Xing et al., 9 Apr 2026). The paper resolves this by constructing validation sets from downstream coding tasks and using validation loss on those sets as a proxy for downstream performance.

The downstream benchmarks used to derive validation sets are HumanEval, MBPP, DS-1000, CrossCodeEval, and Bird-SQL (Xing et al., 9 Apr 2026). For tasks with ground-truth solutions, benchmark problems and reference answers are used directly. HumanEval is treated specially: because it lacks ground-truth answers in the required benchmark format, the authors generate reference solutions using CodeLlama-34B, and they exclude especially hard problems that would be unreliable for a 1B model to evaluate (Xing et al., 9 Apr 2026). To keep computation manageable and to make validation sets comparable, each benchmark-derived validation set is capped at 200 problems when possible; if a dataset has fewer than 200 problems, all of it is used (Xing et al., 9 Apr 2026).

Benchmark	Setting or note
HumanEval	Python, Java, C++
MBPP	Python
DS-1000	Python/data science
CrossCodeEval	Exact match benchmark
Bird-SQL	SQL

This construction is the paper’s main bridge from downstream code evaluation to influence scoring. Rather than trying to optimize pass@1 or exact match directly at the per-sample level, the method evaluates whether a sample reduces validation loss on a benchmark-derived set that represents a downstream programming task (Xing et al., 9 Apr 2026). A plausible implication is that IS in this setting is less a universal property of code data than a conditional property relative to a chosen validation set.

3. Filtering pipeline and experimental setting

The empirical study is conducted with a 1B-parameter CodeLLM called CodeShell-1B, trained from scratch on 100 billion code tokens sampled from StarCoderData (Xing et al., 9 Apr 2026). The authors save checkpoints every 5B tokens and analyze 20 checkpoints in total. The downstream evaluation suite spans HumanEval, MBPP, DS-1000, CrossCodeEval, and Bird-SQL, with standard task-level metrics including Pass@1, Accuracy, and Exact Match; HumanEval, MBPP, DS-1000, and Bird-SQL are execution-based, whereas CrossCodeEval uses exact match (Xing et al., 9 Apr 2026). They also use Spearman rank correlation to study how well validation loss tracks downstream performance and how influence scores correlate across checkpoints and tasks (Xing et al., 9 Apr 2026).

The filtering procedure is explicit. For each candidate pretraining sample $x_i$ 1, the model is updated for one training step on $x_i$ 2, and the change in validation loss on a downstream-task validation set is measured. Samples are then sorted by $x_i$ 3. The highest-scoring examples are designated top $x_i$ 4 samples and treated as beneficial, whereas the lowest-scoring examples are designated bottom $x_i$ 5 samples and treated as harmful (Xing et al., 9 Apr 2026). The authors then continue pretraining on either the top or bottom subset to test whether the ranking truly separates useful from harmful code data.

Prediction-based scoring appears only as a later comparison. The paper trains a RoBERTa-Base model to predict oracle influence scores from a small labeled subset, in a setup described as similar to MATES, but reports that this approximation performs poorly for code (Xing et al., 9 Apr 2026). Consequently, the main pipeline is not prediction-based filtering but validation-set-loss filtering.

4. Empirical effectiveness

The main empirical result is that IS-based filtering using validation-set loss improves programming performance in the reported setting (Xing et al., 9 Apr 2026). The paper shows, first, that validation loss strongly correlates with downstream performance across tasks and checkpoints, and more strongly than training loss or held-out loss. Second, training on top-influence samples outperforms training on bottom-influence samples, indicating that the influence score identifies data that is genuinely useful for code LLM pretraining (Xing et al., 9 Apr 2026).

These results matter because they validate the score operationally rather than only analytically. A sample ranked highly by $x_i$ 6 is not merely associated with a desirable representation statistic; continuing pretraining on such samples yields better downstream programming behavior than continuing on low-scoring samples (Xing et al., 9 Apr 2026). The paper therefore concludes that validation-loss-based influence filtering is effective for code data, at least in this experimental setting.

The study also establishes that validation loss is a stronger proxy for downstream coding quality than generic losses measured on training or held-out data (Xing et al., 9 Apr 2026). In effect, the paper treats downstream-aligned validation loss as the relevant objective for data selection. This suggests that the usefulness of IS depends critically on the quality and representativeness of the constructed validation sets.

5. Dynamic, task-dependent, and language-dependent behavior

A major finding is that beneficial data is not fixed across training stages (Xing et al., 9 Apr 2026). The correlation of influence scores between early checkpoints is only moderate, while correlations between adjacent late-stage checkpoints become much higher. The paper interprets this as a stabilization of the notion of useful data later in training. IS is therefore a dynamic criterion rather than a static tag attached permanently to a sample.

Task dependence is equally pronounced. Tasks in the same family, especially the HumanEval variants, have relatively high influence-score correlation, but many other task pairs have weak or even negative correlation (Xing et al., 9 Apr 2026). Importantly, sharing a programming language does not guarantee similar influence patterns; different Python tasks can still behave very differently. For this reason, the authors argue that validation sets should be general and practical rather than narrowly tailored to a single benchmark (Xing et al., 9 Apr 2026).

The language analysis adds a further asymmetry. Same-language training data does not necessarily have a higher mean influence than other-language data, but it tends to have a much larger variance in influence scores (Xing et al., 9 Apr 2026). The consequence is that language-matched code contains both especially helpful and especially harmful examples. The paper therefore argues for targeted filtering rather than simply increasing the volume of same-language data.

These observations correct several common simplifications. One simplification is that “good” code pretraining data can be identified once and then reused unchanged throughout training; the checkpoint analysis rejects that view. Another is that language matching alone is a sufficient criterion for useful code data; the variance results reject that view as well (Xing et al., 9 Apr 2026).

6. Alternative filters, failed approximations, and limitations

The paper compares influence-based selection against perplexity filtering and LLM-based educational scoring using GPT-4o-style scores (Xing et al., 9 Apr 2026). Its conclusion is that neither method reliably separates high- from low-influence code samples. Although trends vary across tasks, the absolute differences in influence across perplexity or LLM-score buckets are small, so these methods do not align well with oracle $x_i$ 7 (Xing et al., 9 Apr 2026).

The study is similarly skeptical about scalable prediction-based influence estimation. When a RoBERTa-Base model is trained to predict oracle influence scores from a labeled subset, the resulting Spearman correlation with oracle influence is very low, with the best reported value only around 0.16 (Xing et al., 9 Apr 2026). Increasing the number of labeled influence examples does not solve the problem, and filtering based on predicted scores does not significantly outperform random selection. The bottom line is therefore asymmetric: oracle influence-score filtering based on validation-set loss works for code pretraining data selection, but influence prediction with a small proxy model does not (Xing et al., 9 Apr 2026).

The main practical limitation is computational cost. Computing the score requires a one-step update and a validation-loss evaluation for each sample, making the procedure expensive and difficult to scale to trillion-token corpora without better approximations (Xing et al., 9 Apr 2026). In that respect, the paper stands in contrast to work that emphasizes scalability through cheaper influence proxies, including one-shot data-sampling scores in NLU (Anand et al., 2023) and layer-restricted influence approximations in LLMs (Vitel et al., 6 Nov 2025). This suggests that the current formulation of IS is strongest as an oracle analysis tool and as proof that downstream-aligned filtering is possible, rather than as a turnkey large-scale preprocessing pipeline.

In summary, the paper defines Influence Score for code LLM pretraining as a validation-loss delta after one training step on an individual sample, uses it to rank code data by downstream usefulness, and shows that the resulting oracle filter improves programming performance in a 1B-model, 100B-token setting (Xing et al., 9 Apr 2026). Its broader significance lies in three claims established by the study: validation loss can serve as an effective bridge from generative programming benchmarks to influence estimation, beneficial pretraining data is strongly dependent on both training stage and downstream task, and practical scalability remains unresolved.