---
title: Influence Score (IS) in Code LLM Pretraining
url: https://www.emergentmind.com/topics/influence-score-is
type: topic
---

# Influence Score (IS) in Code LLM Pretraining

Searching arXiv for the cited paper and closely related influence-score work in language/data selection.
Influence Score (IS), written as \(DIScore\) in "An Empirical Study on Influence-Based Pretraining Data Selection for Code Large Language Models" [2604.07769], is a sample-level measure of how much a single pretraining example helps or hurts a code large language model on a downstream programming task. In that study, the score is defined for code pretraining data by the change in downstream-task validation loss after one training step on an individual sample, making it an oracle, validation-loss-based criterion for ranking pretraining data by downstream usefulness. The concept is introduced to address a gap between prior pretraining-data filtering work on general datasets and the specific demands of programming datasets, where downstream evaluation is typically generative and often execution-based rather than classification-based [2604.07769].

## 1. Formal definition

For a training example \(x_i\) and a downstream validation set \(D_r\), the paper defines influence as

\[
I_M(x_i; D_r) = L(D_r \mid M) - L(D_r \mid A(M, x_i)),
\]

where \(M\) is the current model, \(A(M, x_i)\) is the model after one training step on \(x_i\), and \(L(\cdot)\) is the loss on the validation set \(D_r\) [2604.07769]. The score is therefore the drop in validation loss after a single update on that sample.

The sign has an immediate interpretation. If the loss on \(D_r\) decreases after training on \(x_i\), the score is positive and the sample is treated as beneficial. If the loss increases, the score is negative and the sample is treated as harmful. In this formulation, IS is not an abstract importance heuristic; it is a direct estimate of whether a pretraining sample improves or degrades downstream-task performance as measured through validation loss [2604.07769].

A defining feature of the method is that it is an oracle score computed from validation-set loss, not a score predicted from sample features or from a small proxy model. This distinguishes it from several strands of earlier language influence research, including TracIn-style gradient scoring for language classification [2202.11844], influence-based pruning studies in NLU [2311.16298], and multi-stage influence formulations that trace finetuned predictions back to pretraining examples [2007.09081]. This suggests that the paper’s IS should be understood as a task-grounded data-selection signal rather than merely another checkpoint-gradient proxy.

## 2. Adaptation to generative programming tasks

The central methodological problem is that code benchmarks are usually generative and are often judged by execution-based metrics such as pass@k, accuracy, or exact match, whereas the influence definition above requires a loss-based signal [2604.07769]. The paper resolves this by constructing validation sets from downstream coding tasks and using validation loss on those sets as a proxy for downstream performance.

The downstream benchmarks used to derive validation sets are HumanEval, MBPP, DS-1000, CrossCodeEval, and Bird-SQL [2604.07769]. For tasks with ground-truth solutions, benchmark problems and reference answers are used directly. HumanEval is treated specially: because it lacks ground-truth answers in the required benchmark format, the authors generate reference solutions using CodeLlama-34B, and they exclude especially hard problems that would be unreliable for a 1B model to evaluate [2604.07769]. To keep computation manageable and to make validation sets comparable, each benchmark-derived validation set is capped at 200 problems when possible; if a dataset has fewer than 200 problems, all of it is used [2604.07769].

| Benchmark | Setting or note |
|---|---|
| HumanEval | Python, Java, C++ |
| MBPP | Python |
| DS-1000 | Python/data science |
| CrossCodeEval | Exact match benchmark |
| Bird-SQL | SQL |

This construction is the paper’s main bridge from downstream code evaluation to influence scoring. Rather than trying to optimize pass@1 or exact match directly at the per-sample level, the method evaluates whether a sample reduces validation loss on a benchmark-derived set that represents a downstream programming task [2604.07769]. A plausible implication is that IS in this setting is less a universal property of code data than a conditional property relative to a chosen validation set.

## 3. Filtering pipeline and experimental setting

The empirical study is conducted with a 1B-parameter CodeLLM called CodeShell-1B, trained from scratch on 100 billion code tokens sampled from StarCoderData [2604.07769]. The authors save checkpoints every 5B tokens and analyze 20 checkpoints in total. The downstream evaluation suite spans HumanEval, MBPP, DS-1000, CrossCodeEval, and Bird-SQL, with standard task-level metrics including Pass@1, Accuracy, and Exact Match; HumanEval, MBPP, DS-1000, and Bird-SQL are execution-based, whereas CrossCodeEval uses exact match [2604.07769]. They also use Spearman rank correlation to study how well validation loss tracks downstream performance and how influence scores correlate across checkpoints and tasks [2604.07769].

The filtering procedure is explicit. For each candidate pretraining sample \(x_i\), the model is updated for one training step on \(x_i\), and the change in validation loss on a downstream-task validation set is measured. Samples are then sorted by \(DIScore\). The highest-scoring examples are designated top \(DIScore\) samples and treated as beneficial, whereas the lowest-scoring examples are designated bottom \(DIScore\) samples and treated as harmful [2604.07769]. The authors then continue pretraining on either the top or bottom subset to test whether the ranking truly separates useful from harmful code data.

Prediction-based scoring appears only as a later comparison. The paper trains a RoBERTa-Base model to predict oracle influence scores from a small labeled subset, in a setup described as similar to MATES, but reports that this approximation performs poorly for code [2604.07769]. Consequently, the main pipeline is not prediction-based filtering but validation-set-loss filtering.

## 4. Empirical effectiveness

The main empirical result is that IS-based filtering using validation-set loss improves programming performance in the reported setting [2604.07769]. The paper shows, first, that validation loss strongly correlates with downstream performance across tasks and checkpoints, and more strongly than training loss or held-out loss. Second, training on top-influence samples outperforms training on bottom-influence samples, indicating that the influence score identifies data that is genuinely useful for code LLM pretraining [2604.07769].

These results matter because they validate the score operationally rather than only analytically. A sample ranked highly by \(DIScore\) is not merely associated with a desirable representation statistic; continuing pretraining on such samples yields better downstream programming behavior than continuing on low-scoring samples [2604.07769]. The paper therefore concludes that validation-loss-based influence filtering is effective for code data, at least in this experimental setting.

The study also establishes that validation loss is a stronger proxy for downstream coding quality than generic losses measured on training or held-out data [2604.07769]. In effect, the paper treats downstream-aligned validation loss as the relevant objective for data selection. This suggests that the usefulness of IS depends critically on the quality and representativeness of the constructed validation sets.

## 5. Dynamic, task-dependent, and language-dependent behavior

A major finding is that beneficial data is not fixed across training stages [2604.07769]. The correlation of influence scores between early checkpoints is only moderate, while correlations between adjacent late-stage checkpoints become much higher. The paper interprets this as a stabilization of the notion of useful data later in training. IS is therefore a dynamic criterion rather than a static tag attached permanently to a sample.

Task dependence is equally pronounced. Tasks in the same family, especially the HumanEval variants, have relatively high influence-score correlation, but many other task pairs have weak or even negative correlation [2604.07769]. Importantly, sharing a programming language does not guarantee similar influence patterns; different Python tasks can still behave very differently. For this reason, the authors argue that validation sets should be general and practical rather than narrowly tailored to a single benchmark [2604.07769].

The language analysis adds a further asymmetry. Same-language training data does not necessarily have a higher mean influence than other-language data, but it tends to have a much larger variance in influence scores [2604.07769]. The consequence is that language-matched code contains both especially helpful and especially harmful examples. The paper therefore argues for targeted filtering rather than simply increasing the volume of same-language data.

These observations correct several common simplifications. One simplification is that “good” code pretraining data can be identified once and then reused unchanged throughout training; the checkpoint analysis rejects that view. Another is that language matching alone is a sufficient criterion for useful code data; the variance results reject that view as well [2604.07769].

## 6. Alternative filters, failed approximations, and limitations

The paper compares influence-based selection against perplexity filtering and LLM-based educational scoring using GPT-4o-style scores [2604.07769]. Its conclusion is that neither method reliably separates high- from low-influence code samples. Although trends vary across tasks, the absolute differences in influence across perplexity or LLM-score buckets are small, so these methods do not align well with oracle \(DIScore\) [2604.07769].

The study is similarly skeptical about scalable prediction-based influence estimation. When a RoBERTa-Base model is trained to predict oracle influence scores from a labeled subset, the resulting Spearman correlation with oracle influence is very low, with the best reported value only around 0.16 [2604.07769]. Increasing the number of labeled influence examples does not solve the problem, and filtering based on predicted scores does not significantly outperform random selection. The bottom line is therefore asymmetric: oracle influence-score filtering based on validation-set loss works for code pretraining data selection, but influence prediction with a small proxy model does not [2604.07769].

The main practical limitation is computational cost. Computing the score requires a one-step update and a validation-loss evaluation for each sample, making the procedure expensive and difficult to scale to trillion-token corpora without better approximations [2604.07769]. In that respect, the paper stands in contrast to work that emphasizes scalability through cheaper influence proxies, including one-shot data-sampling scores in NLU [2311.16298] and layer-restricted influence approximations in language models [2511.04715]. This suggests that the current formulation of IS is strongest as an oracle analysis tool and as proof that downstream-aligned filtering is possible, rather than as a turnkey large-scale preprocessing pipeline.

In summary, the paper defines Influence Score for code LLM pretraining as a validation-loss delta after one training step on an individual sample, uses it to rank code data by downstream usefulness, and shows that the resulting oracle filter improves programming performance in a 1B-model, 100B-token setting [2604.07769]. Its broader significance lies in three claims established by the study: validation loss can serve as an effective bridge from generative programming benchmarks to influence estimation, beneficial pretraining data is strongly dependent on both training stage and downstream task, and practical scalability remains unresolved.

Source: https://www.emergentmind.com/topics/influence-score-is