---
title: 'DATE-LM: Data Attribution in LLMs'
url: https://www.emergentmind.com/topics/date-lm
type: topic
---

# DATE-LM: Data Attribution in LLMs

Searching arXiv for recent papers on “DATE-LM” and closely related temporal/date or attribution benchmarks.
DATE-LM commonly denotes **Data Attribution Evaluation in Language Models**, a benchmark suite for evaluating **data attribution methods specifically in the LLM setting**, namely methods that estimate how much individual training examples influenced model behavior. It is motivated by the observation that data attribution is increasingly relevant for **dataset curation, interpretability, safety filtering, and data valuation**, while prior evaluation practice remained fragmented, expensive, and only weakly aligned with real LLM use cases. DATE-LM addresses this by organizing evaluation around three application-driven tasks—**training data selection**, **toxicity/bias filtering**, and **factual attribution**—and by providing a unified pipeline, standardized configurations, and a public leaderboard for large-scale comparison across methods and model families [2507.09424].

## 1. Motivation and conceptual scope

DATE-LM was introduced to address three gaps in prior evaluation of data attribution for large language models. The first is **reproducibility and fairness**: LLM training and evaluation depend on many implementation details, and small differences in stack, checkpoints, prompting, or sampling can materially change results. The second is **computational burden**: leave-one-out retraining is expensive, and even approximate procedures such as LDS still require many retrains. The third is a **lack of application-driven, LLM-centric evaluation**: attribution methods are often assessed in ways that do not correspond to their principal downstream uses in modern LLM development [2507.09424].

Within this framing, DATE-LM treats attribution not as an abstract influence score in isolation, but as a mechanism that should be judged by its utility in realistic workflows. The benchmark therefore measures whether an attribution method can help choose better training subsets, identify harmful training examples, or trace model outputs back to influential evidence. This application-grounded orientation is central to the benchmark’s design and to its empirical conclusions that attribution performance is task-dependent and sensitive to evaluation protocol [2507.09424].

## 2. Formalization and benchmark design

Formally, DATE-LM defines a data attribution method \(T\) as a mapping
\[
T : D \times D_{\text{ref}} \times \Theta \rightarrow \mathbb{R}^N
\]
where \(D = \{z_i\}_{i=1}^N\) is the training set, \(D_{\text{ref}} = \{z'_j\}_{j=1}^M\) is a reference set, and \(\Theta\) is the model parameter space. The method returns one relevance score per training sample in \(D\) with respect to \(D_{\text{ref}}\). In practice, \(D_{\text{ref}}\) may consist of model outputs, prompt-response pairs, toxic examples, or factual claims whose supporting evidence is to be traced [2507.09424].

The benchmark uses a shared three-stage pipeline. First, the attribution method computes a score vector,
\[
S = T(D, D_{\text{ref}}, \theta).
\]
Second, the scores are converted into a subset \(D_s \subset D\) through **top-k** selection or **probabilistic sampling** with **Gumbel-top-k**. Third, the selected subset is evaluated in a downstream task: training a model, filtering data, or ranking evidence. This common structure is what makes DATE-LM both unified and modular [2507.09424].

The benchmark’s three principal tasks and their primary metrics are summarized below.

| Task | Core setup | Primary metrics |
|---|---|---|
| Training data selection | Pretraining or fine-tuning on subsets selected from Fineweb or Tulu 3 | Downstream task performance, FLOPs |
| Toxicity/bias filtering | Ranking unsafe examples inside benign or heterogeneous mixtures | AUPRC, GPT score, ASR |
| Factual attribution | Retrieving supporting evidence in ROME-derived factual QA with counterfactual corruptions | Recall@50, MRR |

This structure is explicitly LLM-centric. DATE-LM includes **Pythia-1B** for pretraining experiments, **Llama3.1-8B** for fine-tuning and safety tasks, and **Llama3.2-1B** in several application evaluations. It also provides pre-trained and fine-tuned checkpoints, training scripts, and standardized evaluation configurations [2507.09424].

## 3. Training data selection

The first DATE-LM task asks whether attribution can improve **training data selection** relative to random or simple heuristic baselines. The benchmark evaluates this in both **pre-training selection** and **fine-tuning selection** settings. In pretraining, the training pool \(D\) contains **1M samples from Fineweb**, the reference set \(D_{\text{ref}}\) is **LAMBADA** or **FLAN**, the subset size is **10% of the pool (100k examples)**, the selection strategy is **Gumbel-top-k**, the model is a custom-trained **Pythia-1B**, and checkpoints are evaluated at **10k steps and 30k steps** on a seven-task suite consisting of **SciQ, ARC-E, ARC-C, OBQA, BoolQ, HellaSwag, and PIQA**. The benchmark also reports estimated **FLOPs** as a compute-efficiency metric [2507.09424].

In fine-tuning selection, the training pool contains **200k examples from Tulu 3**, the reference set is one of **MMLU, GSM8K,** or **BBH**, the subset size is **5% (10k examples)**, the selection strategy is **top-k**, and the model is **Llama3.1-8B with LoRA**. Evaluation is performed on the target task itself. DATE-LM standardizes the selection pipeline by using **Gumbel-top-k consistently** across methods in pretraining, rather than allowing method-specific diversity heuristics, so that differences can be attributed more directly to the attribution scores [2507.09424].

The main findings are that **all methods beat random selection**, but **simple non-attribution baselines can match or outperform expensive attribution methods**. In pretraining, **EDU**, a classifier-based heuristic, is among the best performers and can outperform gradient-based methods such as **Grad-Sim**; **MATES** is reported as a strong and more efficient model-aware method. In fine-tuning, **Rep-Sim** and **RDS+** can outperform more expensive gradient methods. A central result is that performance is highly **sensitive to Gumbel temperature**: settings that are too low or too high can significantly alter results, and the best temperature differs between early and later checkpoints. This establishes training data selection as one of the clearest examples of DATE-LM’s broader claim that attribution evaluation is strongly shaped by task-specific design choices [2507.09424].

## 4. Toxicity and bias filtering

The second task evaluates whether attribution methods can detect **unsafe training examples** so they can be removed before training or fine-tuning. DATE-LM constructs a training set
\[
D = D_{\text{benign}} \cup D_{\text{unsafe}}
\]
where \(D_{\text{benign}}\) is drawn from **UltraChat (10,000 examples)** and \(D_{\text{unsafe}}\) contains **fewer than 100 examples** from **ToxicChat**, **XSTest-response**, and **JailbreakBench**. The reference set consists of held-out prompt-label pairs from the unsafe datasets [2507.09424].

Two variants are used. In **homogeneous filtering**, unsafe examples are inserted into an otherwise benign corpus. In **heterogeneous filtering**, benign data also includes **safety-aligned distractors**—safe refusals that are stylistically similar to unsafe prompts. The heterogeneous variant is explicitly intended as a stricter and more realistic setting because it requires methods to avoid overfiltering harmless but similar examples [2507.09424].

The principal ranking metric is **AUPRC**. DATE-LM additionally evaluates safety after retraining by removing top-ranked harmful examples and scoring model outputs with **GPT-4o as a judge**. The benchmark reports a **GPT score** and **Attack Success Rate (ASR)**, where **ASR** is the proportion of responses scored **5**, meaning maximally unsafe. The rubric ranges from **1** (“proactive refusal / safe redirection”) to **5** (“fully fulfills the unsafe instruction”) [2507.09424].

The empirical picture is mixed. In the homogeneous setting, attribution methods are competitive, and **cosine-similarity methods are generally better and more stable than dot-product methods**. Methods such as **Rep-Sim**, **Grad-Sim**, and **LESS** outperform **Grad-Dot**, **DataInf**, and **EKFAC** in many cases. In the heterogeneous setting, however, attribution performance drops substantially because methods confuse **safe refusals** with unsafe content on the basis of stylistic similarity, whereas classifier baselines such as **Llama-Guard-3-8B** and **Wildguard** are much less affected. DATE-LM also reports that after removing the **top 30 flagged examples** and retraining, models curated by attribution methods, especially **LESS**, become safer than those curated by some strong baselines. The task therefore demonstrates both the practical utility of attribution and the importance of realistic benchmark construction [2507.09424].

## 5. Factual attribution

The third DATE-LM task asks whether a method can trace a model’s factual output back to **supporting training evidence**. This is framed as an interpretability and hallucination-diagnosis problem and is evaluated on a subset of **ROME-derived factual QA** data. The training set includes relevant factual examples, irrelevant distractors, and, crucially, **counterfactually corrupted examples** [2507.09424].

The counterfactual design is one of DATE-LM’s main methodological interventions. Prior factual tracing benchmarks could favor simple lexical matching because training evidence often overlapped directly with the target fact. DATE-LM instead corrupts an entity in a factual example—such as replacing “Microsoft” with “Google”—to create coherent but false alternatives. This is intended to break lexical-overlap shortcuts and make the task more sensitive to genuinely influential evidence rather than superficial retrieval [2507.09424].

Evaluation uses **Recall@50** and **MRR**. The benchmark also performs retraining-based validation by removing top-ranked evidence and measuring whether factual or counterfactual behavior decreases. In this setting, DATE-LM finds that **data attribution methods outperform lexical baselines** such as **BM25** and **Rep-Sim**, a result that differs from earlier factual attribution benchmarks where BM25 often dominated because of lexical overlap. The retraining experiments further show that removing examples chosen by attribution methods reduces counterfactual behavior, supporting the interpretation that the benchmark is isolating genuinely influential training evidence [2507.09424].

## 6. Methods, baselines, infrastructure, and empirical conclusions

DATE-LM evaluates a heterogeneous set of attribution methods and competitive baselines. The attribution methods include **Grad-Dot, Grad-Sim, LESS, EKFAC, DataInf,** and **MATES**. The non-attribution baselines include **BM25**, **Rep-Sim**, **RDS+** in fine-tuning experiments, and task-specific classifiers such as **EDU** for data selection and **Llama-Guard-3-8B** and **Wildguard** for safety filtering. The benchmark repeatedly emphasizes that such baselines matter because they are often cheaper, simpler, and surprisingly strong [2507.09424].

A major practical contribution is the **public DATE-LM leaderboard** on **Hugging Face Spaces**. It supports result browsing by method, model size, attribution type, and metric, and permits submission of new results. The submission workflow accepts attribution score files and task metrics, validates them, opens a **GitHub pull request**, and merges verified results into the leaderboard. This infrastructure is designed to improve transparency, verifiability, and community participation [2507.09424].

Across all tasks, DATE-LM reaches several general conclusions. **No single method dominates**; rankings vary by application. **Non-attribution baselines can be highly competitive**, sometimes matching or outperforming more expensive methods. **Evaluation design strongly matters**, including Gumbel temperature, choice of reference set, lexical overlap, and the presence of safety-aligned distractors. **Similarity-based methods** such as **Rep-Sim**, **Grad-Sim**, and **LESS** are often more stable than dot-product methods such as **Grad-Dot**, **DataInf**, and **EKFAC**. The benchmark’s broader significance lies in shifting attribution evaluation away from abstract influence estimates toward downstream outcomes in dataset curation, safety filtering, and factual tracing [2507.09424].

## 7. Terminological scope and neighboring uses of “DATE”

The acronym **DATE-LM** in this usage refers specifically to **Data Attribution Evaluation in Language Models** and should not be conflated with work on date, time, or chronology reasoning. Several nearby research lines are distinct. **DATETIME** is a benchmark for datetime translation and reasoning rather than attribution [2504.16155]. **Temporal Text Classification** studies automatic text dating and historical period prediction [2603.11295]. **Chronologically Consistent Large Language Models** addresses lookahead bias by training time-restricted models such as **ChronoBERT** [2502.21206]. **Entity Cloze By Date** probes what language models know about new entities indexed by origination date [2205.02832]. These works are related only in the broad sense that they engage temporal structure or time-sensitive evaluation, not data attribution.

A second source of ambiguity is that other titles contain the string “date” or the acronym “DATE” for unrelated reasons. **“It doesn’t look good for a date”** concerns critique-to-preference transformation in conversational recommendation [2109.07576]. **DATE: Dynamic Absolute Time Enhancement for Long Video Understanding** is a multimodal long-video method built around timestamp injection and temporal-aware sampling [2509.09263]. **LM-Critic** concerns unsupervised grammatical error correction rather than temporal or attribution modeling [2109.06822]. **IoT-LM** is a multisensory IoT foundation model [2407.09801], and **LM-Polygraph** is a benchmark for uncertainty quantification [2406.15627]. In encyclopedic usage, the unqualified term **DATE-LM** is most precisely reserved for the attribution benchmark introduced in “DATE-LM: Benchmarking Data Attribution Evaluation for Large Language Models” [2507.09424].

Source: https://www.emergentmind.com/topics/date-lm