---
title: Counterfactual Memorization in Neural Models
url: https://www.emergentmind.com/topics/counterfactual-memorization
type: topic
---

# Counterfactual Memorization in Neural Models

to=arxiv_search.search code 񹚎json
{"query":"all:counterfactual memorization language models", "max_results": 10, "sort_by": "relevance"}
to=arxiv_search.search code 】【。】【”】【json
{"query":"ti:\"Counterfactual Memorization in Neural Language Models\" OR ti:\"Recite, Reconstruct, Recollect\" OR ti:\"Causal Estimation of Memorisation Profiles\"", "max_results": 10, "sort_by": "relevance"}
to=arxiv_search.search code ՞նչjson
{"query":"ti:\"Counterfactual Influence as a Distributional Quantity\" OR ti:\"Finding Memo: Extractive Memorization in Constrained Sequence Generation Tasks\"", "max_results": 10, "sort_by": "relevance"}
Counterfactual memorization is a family of research notions that define memorization through a comparison between observed model behavior and a counterfactual world in which a training example, or a training neighborhood around it, was absent. In the language-model literature, the canonical formulation asks whether a model predicts a document substantially better because that document was included during training, rather than merely because the document is common, templatic, or otherwise easy to predict [2112.12938]. Subsequent work broadened this picture in two directions: first, by treating memorization as heterogeneous across duplicated, reconstructible, and residual examples [2406.17746]; second, by arguing that the relevant counterfactual object is often not self-influence alone but a sample’s entire influence profile over the training set [2506.20481].

## 1. Conceptual scope and historical framing

The classical counterfactual formulation in neural language models defines memorization as dependence on an example’s own inclusion in training. Under this view, a document is memorized when the model predicts it accurately if and only if the model was trained on it, and memorization is therefore distinguished from mere familiarity with public facts, duplicated boilerplate, or repeated templates [2112.12938]. This framing was motivated by the observation that overlap-based or extraction-based criteria often conflate rare-example dependence with common repeated text.

A related distinction became important in constrained generation, especially neural machine translation. There, prior work had focused on counterfactual memorization as inclusion–exclusion dependence, whereas later work introduced extractive memorization as exact generation from insufficient context under greedy decoding. The two notions were treated as related but non-identical: counterfactual memorization is about training dependence, while extractive memorization is about inference-time elicitation under truncated conditioning [2210.12929].

Later work further expanded the concept. One line treated counterfactual influence as a distributional quantity over all training samples, arguing that self-influence alone can severely underestimate memorization risk when near-duplicates share responsibility for a target’s predictability [2506.20481]. Another line distinguished counterfactual memorization from recollection-based measures and from a stricter contextual memorization criterion, arguing that different definitions disagree about which strings are memorized and when [2507.14777]. This suggests that “counterfactual memorization” is not a single settled metric, but a research program centered on causal or quasi-causal dependence on training exposure.

## 2. Formal definitions and operational criteria

The literature contains several operationalizations of counterfactual memorization, each anchored in a different observable. In the original language-model formulation, the memorization score of an example \(x\) is the difference between expected performance when trained on subsets containing \(x\) and expected performance when trained on subsets excluding \(x\) [2112.12938]:
\[
\operatorname{mem}(x) \triangleq \mathbb{E}_{S\subset D,\, x\in S}[M(A(S), x)] - \mathbb{E}_{S\subset D,\, x\notin S}[M(A(S), x)].
\]
The corresponding counterfactual influence of \(x\) on another example \(x'\) generalizes this self-influence notion by replacing the target \(x\) with \(x'\) [2112.12938].

A later influence-based formulation writes pairwise counterfactual influence as a loss difference over models trained with and without a given sample \(x_i\) [2506.20481]:
\[
\mathcal{I}(x_i \Rightarrow x_t) =
\underset{A_j: x_i \notin D_j}{\mathbb{E}\left[\mathcal{L}_{A_j}(x_t)\right]}
-
\underset{A_j: x_i \in D_j}{\mathbb{E}\left[\mathcal{L}_{A_j}(x_t)\right]}.
\]
In that notation, traditional counterfactual memorization is the diagonal term \(\mathcal{I}(x_t \Rightarrow x_t)\), while the full influence distribution \(\{\mathcal{I}(x_i \Rightarrow x_t)\}_i\) captures distributed support from duplicates and near-duplicates [2506.20481].

A different causal formalization defines memorization for a specific trained model instance over time using potential outcomes. For an instance \(x\) trained at step \(g\), evaluated at checkpoint \(c\), instance-level memorization is [2406.04327]:
\[
M_{c,g}(x) = Y_c(x; g) - Y_c(x; \infty),
\]
where \(Y_c(x; g)\) is performance if \(x\) was trained on at step \(g\), and \(Y_c(x; \infty)\) is the counterfactual outcome if \(x\) had never been trained on. The corresponding expected estimand averages this effect over instances sharing the same treatment step [2406.04327].

A more recent formal-language treatment defines counterfactual memorization through loss reduction caused by including a string \(s\) in training. If \(D' = D \setminus \{s\}\), then \(s\) begins to be counterfactually memorized at epoch \(e=e_s^{cf}\) when \(loss(M_e(D), s) < loss(M_e(D'), s)\), and for \(e \ge e_s^{cf}\) the score is [2507.14777]:
\[
\mathtt{mem}^{cf}(s, e, D) \triangleq
\frac{loss(M_{e}(D'), s) - loss(M_{e}(D), s)}{loss(M_{e}(D'), s)}.
\]

A concise comparison of the main variants is useful.

| Variant | Operational object | Representative paper |
|---|---|---|
| Self-influence counterfactual memorization | IN vs OUT performance on the same example | [2112.12938] |
| Distributional influence | Full \(\{\mathcal{I}(x_i \Rightarrow x_t)\}_i\) over training samples | [2506.20481] |
| Dynamic causal memorization | Potential-outcome effect \(M_{c,g}\) across checkpoints | [2406.04327] |

These counterfactual notions coexist with extractive criteria. In language models, one widely used operational test is \(k\)-extractability with \(k=32\): a sample is memorized if, given the first 32 tokens as prompt, the model generates the next 32 tokens exactly [2406.17746]. In constrained generation, an NMT training pair is extractively memorized if a source prefix \(x_{1:l}\) with \(\frac{l}{|x|} \le 0.75\) still yields the exact training translation under greedy decoding [2210.12929]. The important distinction is that extraction measures observable regeneration, whereas counterfactual memorization measures dependence on training inclusion.

## 3. Heterogeneity of mechanisms

A major advance in the literature is the rejection of “memorization” as a homogeneous phenomenon. The three-part taxonomy of recitation, reconstruction, and recollection was introduced precisely to explain why exact extraction events can arise for qualitatively different reasons [2406.17746]. Under that taxonomy, **recitation** refers to highly duplicated sequences, operationalized as having at least 6 duplicates; **reconstruction** refers to repeating or incrementing template-like sequences; and **recollection** is the residual category of memorized sequences that are neither heavily duplicated nor easily explained by template heuristics [2406.17746].

This decomposition matters directly for counterfactual reasoning. If a duplicated legal disclaimer is extracted, increasing or decreasing the exact duplicate count is the natural intervention. If a numeric pattern is extracted because its continuation is logically forced, then changing the template structure matters more than removing one exact training occurrence. The taxonomy therefore reframes “what would happen if property \(X\) were changed?” as a category-dependent question rather than a global one [2406.17746].

A second mechanistic refinement comes from the influence-distribution view. Near-duplicates can make self-influence small even when a sample is near-extractable, because the sample’s training neighborhood shares responsibility for the prediction. In that regime, memorization is diffused away from the diagonal of the influence matrix rather than absent [2506.20481]. The paper introduces the Top-1 Influence Margin,
\[
\mathrm{IM}(x_t)=\frac{\max_i \mathcal{I}(x_i \Rightarrow x_t)}{\max_{i \ne i^\star}\mathcal{I}(x_i \Rightarrow x_t)},
\]
as a summary of whether one training point dominates or whether several points contribute similarly [2506.20481].

The literature also increasingly recognizes a distinction between verbatim and non-verbatim forms. In code models, functional memorization denotes recovery of functional logic even when textual overlap is low. A sample is counterfactually functionally memorized when the exposed model’s generation has low textual overlap but high functional similarity to the ground truth, while the unexposed reference model’s generation does not [2606.12764]. This extends counterfactual memorization beyond surface-form regeneration to cases where business rules, thresholds, and control flow are recovered in textually dissimilar form.

A broader theoretical critique argues that recollection-based memorization overstates the phenomenon because high recollection may arise from contextual learning rather than rote storage. On this view, counterfactual memorization improves on recollection by using a string-specific adaptive threshold derived from training without the string, while contextual memorization imposes an even stricter threshold based on the best context-only performance over all epochs [2507.14777]. This suggests that definitional disagreements are themselves part of the field’s core scientific content.

## 4. Estimation strategies and controlled experimental designs

Exact counterfactual estimation is computationally expensive because it requires comparing worlds with and without specific training examples. Early language-model work addressed this by training many models on random subsets and reusing the same bank of models to estimate IN/OUT differences for all examples [2112.12938]. A closely related strategy estimates the full pairwise influence matrix by randomized inclusion vectors over many models, making it possible to approximate \(\mathcal{I}(x_i \Rightarrow x_t)\) for all pairs in a small target dataset [2506.20481].

A more recent approach avoids retraining counterfactual models by adapting difference-in-differences from econometrics. For a treatment cohort \(G(x)=g\) and control cohort \(G(x)=\infty\), expected memorization is identified under parallel-trends and no-anticipation assumptions as a treated-group improvement minus untreated-group improvement [2406.04327]:
\[
\hat{M}_{c,g}
=
\big(\bar{Y}_c(g)-\bar{Y}_{g-1}(g)\big)
-
\big(\bar{Y}_c(\infty)-\bar{Y}_{g-1}(\infty)\big).
\]
This produces a memorization profile over checkpoint step \(c\) and treatment step \(g\), allowing analysis of instantaneous, persistent, and residual memorization within a single realized training trajectory [2406.04327].

In some settings, the field uses direct model comparisons rather than subset Monte Carlo. In RLHF for code completion, counterfactual memorization is defined by comparing a model trained on dataset \(X\) with a control model trained on \(X \setminus \{x\}\). An example \(x\) is counterfactually memorized if the trained model produces the target completion under greedy decoding while the control model does not; a \(k\)-approximate variant uses normalized edit distance with \(k=0.1\) [2406.11715]. This design supports stage-by-stage comparisons across supervised fine-tuning, reward-model training, and RL fine-tuning.

Other studies build controlled data worlds rather than exact leave-one-out pipelines. A synthetic fictional Q&A corpus creates facts that are deliberately absent from the real world, so later success on question answering can be attributed to the injected data rather than latent world knowledge [2506.05639]. This is not counterfactual memorization in the strict inclusion–exclusion sense, but it functions as a clean-room approximation of counterfactual worlds by making factual support experimentally manipulable [2506.05639].

A complementary training-time intervention is Memory Dial, which holds architecture, data, optimizer, and schedule fixed while sweeping a scalar parameter \(\alpha\) in a mixed objective,
\[
\mathcal{L}_{\mathrm{MD}}(\theta;\alpha,\tau)
=
(1-\alpha)\mathcal{L}_{\mathrm{std}}(\theta)
+
\alpha \mathcal{L}_{\mathrm{mem}}(\theta;\tau).
\]
This produces matched model families that differ primarily in additional memorization pressure, providing a quasi-counterfactual axis for asking how predictions change if the same training setup is made more or less memorizing [2604.05074].

## 5. Empirical regularities across tasks and modalities

Several empirical patterns recur across the literature. First, duplication is central but not uniform in effect. In the taxonomy work, recitation is associated with heavy exact duplication, and the probability of memorization saturates once a sample is already in the highly duplicated regime; beyond the 5-duplicate threshold, greater exposure hardly leads to memorization for recitation candidates [2406.17746]. In the influence-distribution view, near-duplicates have the opposite effect on self-influence: they can make self-influence much lower while making extraction risk higher, because memorization is shared across a cluster [2506.20481].

Second, model scale and training time matter. In Pythia models from 70M to 12B parameters, all taxonomy categories increase in absolute memorized count, but recollection grows fastest proportionally, from 4.49% of memorized examples in the 70M model to 11.34% in the 12B model [2406.17746]. A dynamic causal analysis similarly finds that memorization is stronger and more persistent in larger models, that Pythia 70M shows no persistent memorization, and that memorization depends on treatment order and learning-rate schedule [2406.04327].

Third, prompting and context can govern whether memorization is observable. In constrained generation, extractive memorization in NMT survives suffix perturbations far more often than prefix perturbations, supporting the hypothesis that early prefixes act as memorization triggers [2210.12929]. In large language models, longer prompt context substantially increases extractability, a phenomenon described as discoverability rather than a change in underlying storage [2202.07646].

Fourth, the alignment pipeline matters. In code completion, RLHF greatly reduces memorization of reward-model and RL-prompt data relative to directly fine-tuning on the same data, yet examples already memorized during supervised fine-tuning often remain memorized after RLHF [2406.11715]. This yields a stage-specific picture: memorization inherited from supervised fine-tuning is persistent, whereas propagation from reward-model training data into the final policy is much weaker [2406.11715].

Fifth, modality-specific extensions show that counterfactual memorization is not restricted to natural-language continuation. In code models, the exposed OLMo-3-32B checkpoint shows systematic positive similarity deltas over an otherwise matched unexposed checkpoint across textual, structural, judge-based, and execution-based metrics, including execution-verified counterfactual functional memorization cases where textual overlap is low but executable logic is recovered [2606.12764]. This suggests that code auditing based only on string overlap understates memorization of proprietary logic.

Finally, some studies show that apparent memorization can coexist with or even support apparent quality. An empirical evaluation of nine LLMs reported that memorized outputs often correlated with fewer factual errors and better discourse quality, while more unique outputs were often more error-prone [2304.08637]. This does not negate privacy or copyright concerns, but it complicates a simple “more memorization is always worse” narrative.

## 6. Limitations, controversies, and open problems

The main limitation across the literature is that most evidence remains either observational or quasi-experimental rather than truly sample-interventional. Random-subset retraining and full influence matrices are expensive, difference-in-differences depends on parallel-trends and no-anticipation assumptions, and checkpoint-based exposed-versus-unexposed comparisons isolate corpus exposure more cleanly than they isolate the effect of one exact sample [2406.04327]. Even when the language of counterfactuals is used, the underlying estimand often differs from strict leave-one-out retraining.

A second controversy concerns definition. Recollection-based measures, extractive measures, self-influence, full influence distributions, contextual memorization, and functional memorization do not induce the same ranking over examples. In formal-language experiments, recollection, counterfactual, and contextual memorization disagree on memorization order, and improved learning decreases contextual and counterfactual memorization while increasing recollection-based memorization [2507.14777]. This means that empirical claims about “how much memorization” exists are inseparable from the operational definition.

A third limitation is computational scale. Full influence distributions were demonstrated for a small GPT-Neo 1.3B finetuning setup with \(M=1000\) models, but the authors explicitly present this as a small-scale direct study rather than a scalable method for frontier pretraining [2506.20481]. Similar scale constraints appear in retraining-based CM approximations for NMT and code, where exact sample-level omission is replaced by subset ensembles or checkpoint comparisons.

A fourth open problem is mechanism. The taxonomy of recitation, reconstruction, and recollection explains heterogeneity at the level of examples, but not yet at the level of internal circuits [2406.17746]. Related work outside the language-model setting has begun to localize memorization to layers and units in SSL vision encoders, and MemSinks proposes training-time routing of repeated-sequence memorization into designated subnetworks to make selective forgetting more plausible [2507.09937]. This suggests that a mature theory of counterfactual memorization may need both sample-level causal estimands and mechanistic localization.

The overarching implication is that counterfactual memorization should be understood neither as a single privacy score nor as a synonym for extraction. It is a structured family of causal and quasi-causal questions about how training exposure changes model behavior, with answers that depend on the sample’s duplication, predictability, neighborhood support, modality, training stage, and the metric used to define “memorization.”

Source: https://www.emergentmind.com/topics/counterfactual-memorization