Memorization-Guided Data Reuse

• Memorization-guided Data Reuse is a paradigm where measurable signals determine when to replay, retain, or modify training data for optimal learning.
• It uses loss-retention gaps, sample-level attribution, and dynamic replay scheduling to mitigate overfitting and manage privacy risks.
• Empirical methods like DynamicCut and TabCutMix demonstrate that selective reuse improves model performance while reducing memorization-induced leakage.

Memorization-guided Data Reuse is a data-centric training and governance paradigm in which memorization signals are used to decide when data should be replayed, how often it should be replayed, which records should be retained or pruned, and which learned contents should be transformed or suppressed. In its most explicit formulation for LLM pretraining, the paradigm uses loss-retention dynamics and downstream evaluation scores to determine the number of useful repetitions and the spacing between reuses rather than fixing an epoch heuristic in advance (Zuo et al., 6 Jul 2026). Closely related work applies the same logic at sample level in tabular generative modeling, where memorization is concentrated in a small subset of records that can be identified, filtered, and retrained against (Fang et al., 28 May 2025). Across these formulations, the common premise is that memorization is measurable, unevenly distributed, and operationally useful: it can indicate privacy risk, but it can also support selective retention, selective replay, and selective intervention.

1. Conceptual scope and problem setting

The paradigm emerged from a shift in how repeated data exposure is interpreted. In older one-pass or fixed-epoch views, reuse was often treated either as a necessary engineering convenience or as a generic source of overfitting. The newer literature instead asks whether reuse should be conditioned on the model’s memorization state. In LLM pretraining, this appears as the claim that repeated data should not be governed by a fixed rule such as a “four-epoch limit,” but by when the model has forgotten enough for replay to be useful again and by when further replay starts to hurt downstream performance (Zuo et al., 6 Jul 2026). In tabular diffusion, the corresponding shift is from asking whether a model memorizes at dataset level to asking which individual records are driving leakage and whether those records can be selectively filtered (Fang et al., 28 May 2025).

This problem framing is not limited to privacy. In continual learning, memorization is treated as both a vulnerability and a resource: highly memorized examples are forgotten faster after task shifts, yet memorization is also necessary to achieve the highest performance, especially for long-tail or atypical cases (Kozal et al., 23 May 2025). In spurious-correlation settings, memorization-derived training dynamics are used to select a small but informative subset of data for later reuse in a standard ERM stage, with the explicit goal of improving worst-group accuracy while using as little as $10\%$ of the original training data (Fazla et al., 1 Jun 2026). This suggests that “data reuse” in the literature is broader than repeated epochs over a static corpus. It includes replay scheduling, sample pruning, subset selection, recombinative augmentation, and post-hoc transformation of already learned content.

A common misconception is that memorization-guided reuse means maximizing memorization. The literature does not support that interpretation. The explicit proposals instead use memorization to find a safe replay interval, to identify high-risk records for exclusion, to distinguish buffer policies under different memory regimes, or to select subsets that are informative without being dominated by shortcut features (Zuo et al., 6 Jul 2026).

2. Operational definitions and memorization signals

The formalization of memorization-guided reuse depends on the domain. In tabular generative modeling, a standard starting point is the relative distance ratio. For a generated sample $x$ and training set $\mathcal{D}$,

$$r(x) = \frac{d(x, \mathrm{NN}_1(x, \mathcal{D}))}{d(x, \mathrm{NN}_2(x, \mathcal{D}))}.$$

A generated sample is flagged as memorized if

$r(x) < \tau,\qquad \tau=\frac{1}{3}.$

The memorization ratio over a generated set $\mathcal{G}$ is then the fraction of generated samples with $r(x)<\frac13$, and the threshold-free summary is

$$\mathrm{Mem\text{-}AUC}=\int_0^1 \mathrm{Mem.\ Ratio}(\tau)\,d\tau.$$

The crucial extension for reuse is sample-level attribution: a real training sample $x_r$ is implicated when a generated sample $x_g$ both satisfies $x$0 and has $x$1. This supports per-record ranking rather than only dataset-level diagnosis (Fang et al., 28 May 2025).

In LLM training, memorization-guided reuse is framed through retention and generalization windows. For a sample $x$2 first seen at token position $x$3, the loss retention gap is

$x$4

with per-sample loss

$x$5

The retention window is

$x$6

the generalization window is

$x$7

and the effective memorization window is

$x$8

The resulting training rule is

$x$9

where $\mathcal{D}$0 is the epoch size or reuse interval and $\mathcal{D}$1 is the maximum safe number of epochs defined by downstream performance (Zuo et al., 6 Jul 2026).

A different but related causal notion appears in continual learning through Feldman’s definition: $\mathcal{D}$2 Because this is expensive, the practical reuse signal in that setting is a proxy based on the earliest training iteration after which a sample remains correctly classified: $\mathcal{D}$3 This proxy is used online for replay-buffer construction rather than for post-hoc analysis alone (Kozal et al., 23 May 2025).

These definitions already indicate an important point: memorization-guided reuse is not tied to a single metric. It is a design pattern in which a measurable signal of retention, replication, or sample dependence is converted into a decision about reuse.

3. Record-level reuse in tabular generative modeling

The clearest sample-level realization of the paradigm is the data-centric analysis of tabular diffusion models. That work shows a heavy-tailed distribution of memorization counts: most training samples are rarely memorized, while a small subset is reproduced much more frequently. The paper operationalizes “top-memorized” records by sorting training examples by memorization count and tagging the top $\mathcal{D}$4, $\mathcal{D}$5, or $\mathcal{D}$6, then validates that ranking causally through removal experiments (Fang et al., 28 May 2025).

On the Adult dataset with TabDDPM, baseline memorization ratio is $\mathcal{D}$7. Removing random $\mathcal{D}$8 of samples changes the metric to $\mathcal{D}$9, whereas removing the top-labeled $$r(x) = \frac{d(x, \mathrm{NN}_1(x, \mathcal{D}))}{d(x, \mathrm{NN}_2(x, \mathcal{D}))}.$$0 reduces it to $$r(x) = \frac{d(x, \mathrm{NN}_1(x, \mathcal{D}))}{d(x, \mathrm{NN}_2(x, \mathcal{D}))}.$$1. At $$r(x) = \frac{d(x, \mathrm{NN}_1(x, \mathcal{D}))}{d(x, \mathrm{NN}_2(x, \mathcal{D}))}.$$2 removal, random removal gives $$r(x) = \frac{d(x, \mathrm{NN}_1(x, \mathcal{D}))}{d(x, \mathrm{NN}_2(x, \mathcal{D}))}.$$3 while labeled top-$$r(x) = \frac{d(x, \mathrm{NN}_1(x, \mathcal{D}))}{d(x, \mathrm{NN}_2(x, \mathcal{D}))}.$$4 removal gives $$r(x) = \frac{d(x, \mathrm{NN}_1(x, \mathcal{D}))}{d(x, \mathrm{NN}_2(x, \mathcal{D}))}.$$5. These results directly support a sparse intervention principle: a modest high-risk fraction can dominate leakage. The same work also shows that top-memorized samples are memorized slightly earlier, have stronger early Mem-AUC spikes, and can be predicted from a warm-up phase with AUC $$r(x) = \frac{d(x, \mathrm{NN}_1(x, \mathcal{D}))}{d(x, \mathrm{NN}_2(x, \mathcal{D}))}.$$6 on Default, $$r(x) = \frac{d(x, \mathrm{NN}_1(x, \mathcal{D}))}{d(x, \mathrm{NN}_2(x, \mathcal{D}))}.$$7 on Shoppers, and $$r(x) = \frac{d(x, \mathrm{NN}_1(x, \mathcal{D}))}{d(x, \mathrm{NN}_2(x, \mathcal{D}))}.$$8 on Adult. This motivates selective reuse after a probe run rather than only after full convergence (Fang et al., 28 May 2025).

The proposed method, DynamicCut, is explicitly a two-stage reuse policy. First, a generative model is trained for a warm-up period of $$r(x) = \frac{d(x, \mathrm{NN}_1(x, \mathcal{D}))}{d(x, \mathrm{NN}_2(x, \mathcal{D}))}.$$9 epochs while collecting instance-level Mem-AUC trajectories

$r(x) < \tau,\qquad \tau=\frac{1}{3}.$0

Second, each training sample receives a score $r(x) < \tau,\qquad \tau=\frac{1}{3}.$1 by top-$r(x) < \tau,\qquad \tau=\frac{1}{3}.$2 pooling over the warm-up epochs with $r(x) < \tau,\qquad \tau=\frac{1}{3}.$3. Third, all samples are ranked by $r(x) < \tau,\qquad \tau=\frac{1}{3}.$4, the top $r(x) < \tau,\qquad \tau=\frac{1}{3}.$5 fraction is pruned with default $r(x) < \tau,\qquad \tau=\frac{1}{3}.$6, and the model is retrained on the filtered dataset. Across Default, Shoppers, and Adult, and across TabDDPM, CTGAN, and TVAE, this reduces memorization with limited utility loss; for example, on Adult the memorization ratio of CTGAN drops from $r(x) < \tau,\qquad \tau=\frac{1}{3}.$7 to $r(x) < \tau,\qquad \tau=\frac{1}{3}.$8, and TVAE drops from $r(x) < \tau,\qquad \tau=\frac{1}{3}.$9 to $\mathcal{G}$0 (Fang et al., 28 May 2025).

This record-level view is complemented by augmentation-based reuse. “TabCutMix” exchanges randomly selected feature segments between random same-class training sample pairs, while “TabCutMixPlus” clusters features by correlation and swaps correlated groups together. Both methods reduce memorization while maintaining high-quality data generation, but they do so by structured recombination rather than by sample removal (Fang et al., 2024). Taken together, these results define two distinct reuse logics for tabular data: selective exclusion of high-risk records and controlled recombination of the retained set.

4. Replay scheduling and reuse budgets in LLMs

The language-model formulation is centered on the “Memorization Window.” In the reported 100M-parameter experiments, rollback-gap analysis on FineWeb shows substantial decay between $\mathcal{G}$1GT and $\mathcal{G}$2GT, suggesting

$\mathcal{G}$3

In downstream evaluation with a fixed $\mathcal{G}$4B high-quality subset from OpenMathInstruct2 mixed with FineWeb, all schedules show a rise-then-fall pattern on MATH500, and the overfitting peak shifts later as epoch size $\mathcal{G}$5 increases. Approximate peak locations are $\mathcal{G}$6 epochs for $\mathcal{G}$7B, $\mathcal{G}$8 epochs for $\mathcal{G}$9B, and $r(x)<\frac13$0 epochs for $r(x)<\frac13$1B, while $r(x)<\frac13$2B and $r(x)<\frac13$3B behave similarly, indicating a saturation regime (Zuo et al., 6 Jul 2026).

This directly turns memorization into a reuse budget. If replay occurs before the forgetting boundary, re-exposure is likely wasteful and can accelerate overfitting. If replay occurs near $r(x)<\frac13$4, the model refreshes decaying knowledge when reuse is most informative. If replay occurs well after $r(x)<\frac13$5, the model may relearn, but the gains may saturate. The same work also shows that adding only $r(x)<\frac13$6 FineWeb between high-quality repeats dramatically improves over pure high-quality replay, which suggests that spacing alone is not sufficient: interleaved diversity matters (Zuo et al., 6 Jul 2026).

Two adjacent results reinforce this scheduling perspective. “Memory Dial” makes memorization pressure an explicit training variable by interpolating between standard cross-entropy and a temperature-sharpened objective with a scalar parameter $r(x)<\frac13$7. Across six architectures and five benchmarks, seen-example accuracy increases monotonically with $r(x)<\frac13$8 while unseen accuracy remains stable; larger models are more responsive, and frequent sequences are easier to memorize than rare ones (Zhang et al., 6 Apr 2026). The “Entropy-Memorization Law” then suggests that data entropy is linearly correlated with memorization score, so low token-entropy spans are easier to memorize than high-entropy spans in the studied OLMo setting (Huang et al., 8 Jul 2025). This suggests that replay scheduling and reuse budgeting can be conditioned not only on training dynamics but also on data-specific memorization difficulty.

5. Other operational forms: replay buffers, subset selection, recombination, and editing

In continual learning, memorization-guided reuse appears as a buffer policy. The central finding is regime-dependent: with a 500-sample buffer, selecting samples by bottom-k or middle-k proxy memorization outperforms random-style replay, while top-k memorization performs worst. On Split-CIFAR100, buffer size 500, reservoir gives $r(x)<\frac13$9 accuracy, Rainbow Memory gives $$\mathrm{Mem\text{-}AUC}=\int_0^1 \mathrm{Mem.\ Ratio}(\tau)\,d\tau.$$0, and bottom-k memscores give $$\mathrm{Mem\text{-}AUC}=\int_0^1 \mathrm{Mem.\ Ratio}(\tau)\,d\tau.$$1; top-k memscores drop to $$\mathrm{Mem\text{-}AUC}=\int_0^1 \mathrm{Mem.\ Ratio}(\tau)\,d\tau.$$2. As buffer size grows, the value of highly memorized samples rises, and mixing some top-k samples becomes mildly beneficial at $$\mathrm{Mem\text{-}AUC}=\int_0^1 \mathrm{Mem.\ Ratio}(\tau)\,d\tau.$$3 to $$\mathrm{Mem\text{-}AUC}=\int_0^1 \mathrm{Mem.\ Ratio}(\tau)\,d\tau.$$4 buffer sizes (Kozal et al., 23 May 2025). The implication is that memorization-guided reuse is not a universal “replay the most memorized points” rule; it depends on memory regime.

In dataset de-biasing, memorization-derived cumulative loss is used to select a small but robust subset. The two-stage score $$\mathrm{Mem\text{-}AUC}=\int_0^1 \mathrm{Mem.\ Ratio}(\tau)\,d\tau.$$5 separates spurious-feature difficulty from core-feature difficulty and drives coreset selection. An ERM model trained on the selected subset achieves strong worst-group accuracy while using $$\mathrm{Mem\text{-}AUC}=\int_0^1 \mathrm{Mem.\ Ratio}(\tau)\,d\tau.$$6 of the original data, including $$\mathrm{Mem\text{-}AUC}=\int_0^1 \mathrm{Mem.\ Ratio}(\tau)\,d\tau.$$7 WGA on Waterbirds and $$\mathrm{Mem\text{-}AUC}=\int_0^1 \mathrm{Mem.\ Ratio}(\tau)\,d\tau.$$8 WGA on cMNIST (Fazla et al., 1 Jun 2026). This is a reuse policy through subset construction rather than through epochs or buffers.

Post-hoc editing is another form of memorization-guided intervention. “Private Memorization Editing” first detects memorized PII through successful extraction, then allocates edits across feed-forward layers using layerwise contribution coefficients

$$\mathrm{Mem\text{-}AUC}=\int_0^1 \mathrm{Mem.\ Ratio}(\tau)\,d\tau.$$9

and applies a MEMIT-style update to replace the leaked continuation with a dummy privacy-preserving target. The reported result is a $x_r$0 average reduction in memorization-attack accuracy, in some cases to zero, while largely preserving standard benchmark performance (Ruzzetti et al., 9 Jun 2025). This is not reuse in the sense of replaying data, but it uses memorization as a signal for selective retention and selective suppression of already internalized content.

A common misconception is that memorization-guided reuse is limited to record deletion. The literature shows four distinct mechanisms: pruning high-risk records, replay scheduling by forgetting state, recombinative augmentation of retained data, and post-hoc editing of memorized contents.

6. Extensions, limits, and open research questions

The paradigm extends across modalities and threat models. In 3D latent diffusion models for medical image synthesis, nearest-neighbor analysis in a contrastive embedding space shows substantial memorization, including $x_r$1 of the PCCTA training data in one setting, motivating sample-level screening and auditing before synthetic release (Dar et al., 2023). In code models, exact clone detection over generated outputs shows over $x_r$2 memorized code snippets from $x_r$3 outputs of CodeParrot, with strong positive correlation between training-data occurrence count and output memorization frequency, which directly supports deduplication and frequency-aware reuse policies (Yang et al., 2023). In federated learning, pairwise memorization ratios

$x_r$4

separate intra-client from inter-client leakage, showing that FL models do memorize client data and that intra-client memorization generally exceeds inter-client memorization, though both are nonzero (Udsa et al., 9 Oct 2025).

The literature also marks several limits. Many operational definitions are heuristic and depend on the chosen similarity function, distance metric, threshold, or decoding strategy. The relative distance ratio with $x_r$5 is widely used in tabular work, but it remains a heuristic rather than a guarantee of privacy leakage (Fang et al., 28 May 2025). The Memorization Window is estimated in a preliminary 100M-scale setting and does not yet yield a fully implemented online scheduler (Zuo et al., 6 Jul 2026). Entropy-based difficulty estimation is strongly aggregate rather than instance-level in its best-performing form, which limits immediate deployment as a hard filtering rule (Huang et al., 8 Jul 2025). Cross-client memorization auditing in FL currently requires centralized evaluation over client suffixes, so it is better understood as an offline diagnostic than an online controller (Udsa et al., 9 Oct 2025).

Open problems are correspondingly consistent across papers. One recurrent question is how to move from descriptive memorization signals to adaptive controllers that operate online and at per-sample granularity. Another is how to separate beneficial memorization from harmful memorization: continual learning explicitly shows that some memorization is necessary to approach stationary-training performance, while privacy-oriented editing treats memorization as content to be neutralized (Kozal et al., 23 May 2025). A third is transfer across domains and models. Cross-model transfer of risky-sample rankings from diffusion models to GANs and VAEs is reported for tabular data (Fang et al., 28 May 2025), but a general theory of transferability across modalities is not yet established.

Taken together, the literature defines Memorization-guided Data Reuse as a selective rather than uniform doctrine. It is not the generic claim that “more reuse is better” or that “memorization should always be minimized.” It is the more specific claim that reuse decisions should be conditioned on measurable memorization signals, because those signals expose when repeated exposure is still informative, which records disproportionately drive leakage, which subsets are worth retaining, and which learned contents should be transformed or removed.