Palimpsestic Memorization in LLMs

• The paper demonstrates that larger LLMs memorize data faster through a layered retention mechanism that preserves core information despite continual training.
• It uses rigorous metrics like time-to-memorization and POS-specific analyses to quantify how specific data, such as nouns and numerals, are preferentially retained.
• The study highlights significant privacy risks and suggests that understanding memorization dynamics can inform data curation and model improvement strategies.

Palimpsestic memorization in LLMs refers to the phenomenon whereby information from the training data is not simply imprinted as a static trace in model parameters, but is instead layered, overwritten, and sometimes partially “forgotten” as training progresses without ever being completely erased. The analogy to a palimpsest—a manuscript whose original writing is partly erased and overwritten by later text—captures the nuanced, layered accumulation of knowledge in deep neural networks. Empirical studies of LLMs have revealed that memorization emerges as a persistent, yet dynamically evolving, process during training across both causal and masked language modeling tasks. This process depends intricately on model scale, data properties, hyperparameters, and the linguistic nature of the memorized content.

1. Definitions and Quantification of Memorization

Exact memorization is operationalized in terms of the model’s ability to predict target tokens from the training set given their original context. For a set of examples $C = \{(s, y)\}$ where $s$ is a context and $y$ the correct (next) token, a model $f$ is said to memorize an example if the predicted token (via $\arg\max$ over the output distribution) matches $y$. The memorization score is thus:

$$M(f) = \frac{1}{|C|} \sum_{(s, y) \in C} \mathbf{1}\{f(s) = y\}$$

This is functionally equivalent to reporting token-level accuracy on the training set. Time-to-memorization, $T(N, \tau)$, denotes the minimal number of data passes or updates after which a model with $N$ parameters memorizes at least a fraction $\tau$ of the data.

A key finding is that larger LLMs require fewer passes to reach a given memorization rate and achieve higher sample efficiency. Empirical log–log plots of $T(N, \tau)$ versus $N$ typically show exponential or power-law trends, demonstrating that as models scale, memorization accelerates and becomes more robust (Tirumala et al., 2022).

2. Dynamics of Training and Forgetting

Palimpsestic memorization manifests in the training process as a persistent core of memorized content that resists overwriting, even as the model continues to adapt to new data. Detailed experimental protocols involve introducing a special held-out batch, training on it at a checkpoint, and tracking the retention of its content as regular training resumes. Memorization on this batch decays rapidly at first (as initial representations are overwritten) but rapidly stabilizes at a baseline—termed the “forgetting baseline.” Larger models both memorize more quickly and preserve a higher long-term baseline of memorized data.

The forgetting baseline phenomenon stands in sharp contrast to cross-entropy (perplexity) dynamics: while perplexity may worsen (i.e., the model's confidence on the held-out batch decreases), the exact-match memorization rate does not decline beyond a certain level. This indicates a layering effect: models retain a deep substrate of training details even as their predictive confidence adapts to new distributions. The persistence of this memory, despite continued weight updates, typifies palimpsestic behavior.

Training experiments are performed on models ranging from the 12-layer, 125M-parameter configuration up to 40-layer, 13B-parameter LLMs, all with standard cross-entropy loss and Adam optimization, confirming that the phenomenon is broadly robust to architecture and optimizer choices.

3. Linguistic Structure and Differential Memorization

Memorization is not uniform across linguistic categories. Precise part-of-speech (POS) tagging and ratio calculations ($R(p)$ for POS accuracy and $R_{\text{mem}}(p)$ for POS-specific memorization rates) reveal that nouns (including proper nouns) and numerals are memorized more rapidly and robustly than, for example, verbs or adjectives. This is interpreted as an effect of “identifier tokens”—nouns and numbers often form unique, context-specific signatures for training examples, making it easier for the model to encode specific associations early in training.

Empirically, the average length of memorized token spans (the “memory unit length” $L$), while increasing with training, remains below overall sequence length, indicating that even with substantial token-level memorization, fragmentary gaps often exist in reconstructed content. This further supports a layered, non-monolithic model of memory traces.

4. Scaling, Generalization, and the Role of Memorization

Contrary to the simplistic association of memorization with overfitting, the paper demonstrates that a large fraction of training data may be memorized before any increase in validation perplexity—i.e., before any sign of generalization breakdown. The process of memorization is therefore not antithetical to generalization; rather, the two are intertwined in high-capacity models. As models grow larger, not only do they memorize more, but they are increasingly resistant to “forgetting,” and this happens even prior to the onset of overfitting as classically defined.

The sample efficiency of memorization and the observed stability of memory traces suggest that tracking memorization dynamics, rather than cross-entropy alone, provides a more nuanced diagnostic of model capacity and learning dynamics. The effect is particularly pronounced in settings involving large or highly duplicated training datasets.

5. Implications for Privacy, Data Leakage, and Model Improvement

Palimpsestic memorization has direct implications for privacy: the substrate of memorized knowledge includes details that might be reconstructed—even after further training or adaptation—posing risks of inadvertent data leakage. The persistence and resistance-to-forgetting of memorized examples are accentuated in larger models. Moreover, since certain tokens (e.g., rare proper nouns, numerals) act as unique keys for memorization, their repeated presence in training data can elevate privacy risk even if overall memorization rates are controlled.

From a model-improvement perspective, understanding which data fragments are most persistent and which are quickly overwritten can inform both training-data curation (e.g., deduplication, diversity maximization) and architecture or algorithmic choices (e.g., introducing regularization targeted at memory formation, or modulating optimization schedules to tune the layer-depth of memorization).

The observed structural bias towards retaining nouns/numbers also suggests strategies for improving controllability or factual recall in downstream tasks: augmenting training data with such “identifier” elements or structuring data to bias model memory toward desired knowledge fragments may enhance controllability.

6. Mathematical Formulations and Data Representations

Key mathematical representations supplement the empirical analysis:

• Memorization score: $$M(f) = (1/|C|) \sum_{(s,y)\in C} \mathbf{1}\{f(s)=y\}$$
• Time-to-memorization: $T(N, \tau)$, empirically modeled in log–log scaling plots
• POS-specific accuracy and memorization: $R(p)$ and $R_\text{mem}(p)$
• Forgetting (change per epoch): monitors difference in $M(f)$ across training checkpoints

Figure-based data representations visualize the exponential decline in $T(N, \tau)$ with increasing $N$ and depict the stabilization of the forgetting baseline over training epochs.

7. Significance for Theoretical Understanding and Future Research

The results demonstrate that in LLMs, memorization is not a trivial byproduct of overfitting but an emergent, measurable, and dynamically persistent feature of learning. The palimpsestic aspect—whereby a core layer of memorized knowledge persists and is resistant to both forgetting and over-writing—serves as a warning about privacy, a diagnostic for model capacity, and a design target for improved learning and generalization strategies.

Future research may focus on targeted regularization for controlling the “depth” and structure of memory, interpretability methods for tracing memorized fragments to their origin, and more nuanced metrics that capture the complex interplay between memorization, generalization, and privacy risk in modern NLP systems.