DATE-LM: Data Attribution in LLMs

• DATE-LM is a benchmark suite for data attribution evaluation in LLMs that standardizes assessments for training data selection, toxicity filtering, and factual attribution.
• It employs a three-stage pipeline—scoring, selection (using methods like Gumbel-top-k), and downstream task evaluation—to ensure reproducibility and realistic performance metrics.
• Key findings reveal that method effectiveness is task-dependent, with non-attribution baselines often matching expensive methods in practical applications.

Searching arXiv for papers on “DATE-LM” and closely related temporal/date or attribution benchmarks. DATE-LM commonly denotes Data Attribution Evaluation in LLMs, a benchmark suite for evaluating data attribution methods specifically in the LLM setting, namely methods that estimate how much individual training examples influenced model behavior. It is motivated by the observation that data attribution is increasingly relevant for dataset curation, interpretability, safety filtering, and data valuation, while prior evaluation practice remained fragmented, expensive, and only weakly aligned with real LLM use cases. DATE-LM addresses this by organizing evaluation around three application-driven tasks—training data selection, toxicity/bias filtering, and factual attribution—and by providing a unified pipeline, standardized configurations, and a public leaderboard for large-scale comparison across methods and model families (Jiao et al., 12 Jul 2025).

1. Motivation and conceptual scope

DATE-LM was introduced to address three gaps in prior evaluation of data attribution for LLMs. The first is reproducibility and fairness: LLM training and evaluation depend on many implementation details, and small differences in stack, checkpoints, prompting, or sampling can materially change results. The second is computational burden: leave-one-out retraining is expensive, and even approximate procedures such as LDS still require many retrains. The third is a lack of application-driven, LLM-centric evaluation: attribution methods are often assessed in ways that do not correspond to their principal downstream uses in modern LLM development (Jiao et al., 12 Jul 2025).

Within this framing, DATE-LM treats attribution not as an abstract influence score in isolation, but as a mechanism that should be judged by its utility in realistic workflows. The benchmark therefore measures whether an attribution method can help choose better training subsets, identify harmful training examples, or trace model outputs back to influential evidence. This application-grounded orientation is central to the benchmark’s design and to its empirical conclusions that attribution performance is task-dependent and sensitive to evaluation protocol (Jiao et al., 12 Jul 2025).

2. Formalization and benchmark design

Formally, DATE-LM defines a data attribution method $T$ as a mapping

$$T : D \times D_{\text{ref}} \times \Theta \rightarrow \mathbb{R}^N$$

where $D = \{z_i\}_{i=1}^N$ is the training set, $D_{\text{ref}} = \{z'_j\}_{j=1}^M$ is a reference set, and $\Theta$ is the model parameter space. The method returns one relevance score per training sample in $D$ with respect to $D_{\text{ref}}$. In practice, $D_{\text{ref}}$ may consist of model outputs, prompt-response pairs, toxic examples, or factual claims whose supporting evidence is to be traced (Jiao et al., 12 Jul 2025).

The benchmark uses a shared three-stage pipeline. First, the attribution method computes a score vector,

$S = T(D, D_{\text{ref}}, \theta).$

Second, the scores are converted into a subset $D_s \subset D$ through top-k selection or probabilistic sampling with Gumbel-top-k. Third, the selected subset is evaluated in a downstream task: training a model, filtering data, or ranking evidence. This common structure is what makes DATE-LM both unified and modular (Jiao et al., 12 Jul 2025).

The benchmark’s three principal tasks and their primary metrics are summarized below.

Task	Core setup	Primary metrics
Training data selection	Pretraining or fine-tuning on subsets selected from Fineweb or Tulu 3	Downstream task performance, FLOPs
Toxicity/bias filtering	Ranking unsafe examples inside benign or heterogeneous mixtures	AUPRC, GPT score, ASR
Factual attribution	Retrieving supporting evidence in ROME-derived factual QA with counterfactual corruptions	Recall@50, MRR

This structure is explicitly LLM-centric. DATE-LM includes Pythia-1B for pretraining experiments, Llama3.1-8B for fine-tuning and safety tasks, and Llama3.2-1B in several application evaluations. It also provides pre-trained and fine-tuned checkpoints, training scripts, and standardized evaluation configurations (Jiao et al., 12 Jul 2025).

3. Training data selection

The first DATE-LM task asks whether attribution can improve training data selection relative to random or simple heuristic baselines. The benchmark evaluates this in both pre-training selection and fine-tuning selection settings. In pretraining, the training pool $$T : D \times D_{\text{ref}} \times \Theta \rightarrow \mathbb{R}^N$$0 contains 1M samples from Fineweb, the reference set $$T : D \times D_{\text{ref}} \times \Theta \rightarrow \mathbb{R}^N$$1 is LAMBADA or FLAN, the subset size is 10% of the pool (100k examples), the selection strategy is Gumbel-top-k, the model is a custom-trained Pythia-1B, and checkpoints are evaluated at 10k steps and 30k steps on a seven-task suite consisting of SciQ, ARC-E, ARC-C, OBQA, BoolQ, HellaSwag, and PIQA. The benchmark also reports estimated FLOPs as a compute-efficiency metric (Jiao et al., 12 Jul 2025).

In fine-tuning selection, the training pool contains 200k examples from Tulu 3, the reference set is one of MMLU, GSM8K, or BBH, the subset size is 5% (10k examples), the selection strategy is top-k, and the model is Llama3.1-8B with LoRA. Evaluation is performed on the target task itself. DATE-LM standardizes the selection pipeline by using Gumbel-top-k consistently across methods in pretraining, rather than allowing method-specific diversity heuristics, so that differences can be attributed more directly to the attribution scores (Jiao et al., 12 Jul 2025).

The main findings are that all methods beat random selection, but simple non-attribution baselines can match or outperform expensive attribution methods. In pretraining, EDU, a classifier-based heuristic, is among the best performers and can outperform gradient-based methods such as Grad-Sim; MATES is reported as a strong and more efficient model-aware method. In fine-tuning, Rep-Sim and RDS+ can outperform more expensive gradient methods. A central result is that performance is highly sensitive to Gumbel temperature: settings that are too low or too high can significantly alter results, and the best temperature differs between early and later checkpoints. This establishes training data selection as one of the clearest examples of DATE-LM’s broader claim that attribution evaluation is strongly shaped by task-specific design choices (Jiao et al., 12 Jul 2025).

4. Toxicity and bias filtering

The second task evaluates whether attribution methods can detect unsafe training examples so they can be removed before training or fine-tuning. DATE-LM constructs a training set

$$T : D \times D_{\text{ref}} \times \Theta \rightarrow \mathbb{R}^N$$2

where $$T : D \times D_{\text{ref}} \times \Theta \rightarrow \mathbb{R}^N$$3 is drawn from UltraChat (10,000 examples) and $$T : D \times D_{\text{ref}} \times \Theta \rightarrow \mathbb{R}^N$$4 contains fewer than 100 examples from ToxicChat, XSTest-response, and JailbreakBench. The reference set consists of held-out prompt-label pairs from the unsafe datasets (Jiao et al., 12 Jul 2025).

Two variants are used. In homogeneous filtering, unsafe examples are inserted into an otherwise benign corpus. In heterogeneous filtering, benign data also includes safety-aligned distractors—safe refusals that are stylistically similar to unsafe prompts. The heterogeneous variant is explicitly intended as a stricter and more realistic setting because it requires methods to avoid overfiltering harmless but similar examples (Jiao et al., 12 Jul 2025).

The principal ranking metric is AUPRC. DATE-LM additionally evaluates safety after retraining by removing top-ranked harmful examples and scoring model outputs with GPT-4o as a judge. The benchmark reports a GPT score and Attack Success Rate (ASR), where ASR is the proportion of responses scored 5, meaning maximally unsafe. The rubric ranges from 1 (“proactive refusal / safe redirection”) to 5 (“fully fulfills the unsafe instruction”) (Jiao et al., 12 Jul 2025).

The empirical picture is mixed. In the homogeneous setting, attribution methods are competitive, and cosine-similarity methods are generally better and more stable than dot-product methods. Methods such as Rep-Sim, Grad-Sim, and LESS outperform Grad-Dot, DataInf, and EKFAC in many cases. In the heterogeneous setting, however, attribution performance drops substantially because methods confuse safe refusals with unsafe content on the basis of stylistic similarity, whereas classifier baselines such as Llama-Guard-3-8B and Wildguard are much less affected. DATE-LM also reports that after removing the top 30 flagged examples and retraining, models curated by attribution methods, especially LESS, become safer than those curated by some strong baselines. The task therefore demonstrates both the practical utility of attribution and the importance of realistic benchmark construction (Jiao et al., 12 Jul 2025).

5. Factual attribution

The third DATE-LM task asks whether a method can trace a model’s factual output back to supporting training evidence. This is framed as an interpretability and hallucination-diagnosis problem and is evaluated on a subset of ROME-derived factual QA data. The training set includes relevant factual examples, irrelevant distractors, and, crucially, counterfactually corrupted examples (Jiao et al., 12 Jul 2025).

The counterfactual design is one of DATE-LM’s main methodological interventions. Prior factual tracing benchmarks could favor simple lexical matching because training evidence often overlapped directly with the target fact. DATE-LM instead corrupts an entity in a factual example—such as replacing “Microsoft” with “Google”—to create coherent but false alternatives. This is intended to break lexical-overlap shortcuts and make the task more sensitive to genuinely influential evidence rather than superficial retrieval (Jiao et al., 12 Jul 2025).

Evaluation uses Recall@50 and MRR. The benchmark also performs retraining-based validation by removing top-ranked evidence and measuring whether factual or counterfactual behavior decreases. In this setting, DATE-LM finds that data attribution methods outperform lexical baselines such as BM25 and Rep-Sim, a result that differs from earlier factual attribution benchmarks where BM25 often dominated because of lexical overlap. The retraining experiments further show that removing examples chosen by attribution methods reduces counterfactual behavior, supporting the interpretation that the benchmark is isolating genuinely influential training evidence (Jiao et al., 12 Jul 2025).

6. Methods, baselines, infrastructure, and empirical conclusions

DATE-LM evaluates a heterogeneous set of attribution methods and competitive baselines. The attribution methods include Grad-Dot, Grad-Sim, LESS, EKFAC, DataInf, and MATES. The non-attribution baselines include BM25, Rep-Sim, RDS+ in fine-tuning experiments, and task-specific classifiers such as EDU for data selection and Llama-Guard-3-8B and Wildguard for safety filtering. The benchmark repeatedly emphasizes that such baselines matter because they are often cheaper, simpler, and surprisingly strong (Jiao et al., 12 Jul 2025).

A major practical contribution is the public DATE-LM leaderboard on Hugging Face Spaces. It supports result browsing by method, model size, attribution type, and metric, and permits submission of new results. The submission workflow accepts attribution score files and task metrics, validates them, opens a GitHub pull request, and merges verified results into the leaderboard. This infrastructure is designed to improve transparency, verifiability, and community participation (Jiao et al., 12 Jul 2025).

Across all tasks, DATE-LM reaches several general conclusions. No single method dominates; rankings vary by application. Non-attribution baselines can be highly competitive, sometimes matching or outperforming more expensive methods. Evaluation design strongly matters, including Gumbel temperature, choice of reference set, lexical overlap, and the presence of safety-aligned distractors. Similarity-based methods such as Rep-Sim, Grad-Sim, and LESS are often more stable than dot-product methods such as Grad-Dot, DataInf, and EKFAC. The benchmark’s broader significance lies in shifting attribution evaluation away from abstract influence estimates toward downstream outcomes in dataset curation, safety filtering, and factual tracing (Jiao et al., 12 Jul 2025).

7. Terminological scope and neighboring uses of “DATE”

The acronym DATE-LM in this usage refers specifically to Data Attribution Evaluation in LLMs and should not be conflated with work on date, time, or chronology reasoning. Several nearby research lines are distinct. DATETIME is a benchmark for datetime translation and reasoning rather than attribution (Gaere et al., 22 Apr 2025). Temporal Text Classification studies automatic text dating and historical period prediction (Raihan et al., 11 Mar 2026). Chronologically Consistent LLMs addresses lookahead bias by training time-restricted models such as ChronoBERT (He et al., 28 Feb 2025). Entity Cloze By Date probes what LLMs know about new entities indexed by origination date (Onoe et al., 2022). These works are related only in the broad sense that they engage temporal structure or time-sensitive evaluation, not data attribution.

A second source of ambiguity is that other titles contain the string “date” or the acronym “DATE” for unrelated reasons. “It doesn’t look good for a date” concerns critique-to-preference transformation in conversational recommendation (Bursztyn et al., 2021). DATE: Dynamic Absolute Time Enhancement for Long Video Understanding is a multimodal long-video method built around timestamp injection and temporal-aware sampling (Yuan et al., 11 Sep 2025). LM-Critic concerns unsupervised grammatical error correction rather than temporal or attribution modeling (Yasunaga et al., 2021). IoT-LM is a multisensory IoT foundation model (Mo et al., 2024), and LM-Polygraph is a benchmark for uncertainty quantification (Vashurin et al., 2024). In encyclopedic usage, the unqualified term DATE-LM is most precisely reserved for the attribution benchmark introduced in “DATE-LM: Benchmarking Data Attribution Evaluation for LLMs” (Jiao et al., 12 Jul 2025).