---
title: 'Palimpsa: Metaplastic Attention Memory'
url: https://www.emergentmind.com/topics/palimpsa
type: topic
---

# Palimpsa: Metaplastic Attention Memory

Searching arXiv for the cited "Palimpsa" paper and closely related palimpsest-related work to ground the article in current arXiv records.
Palimpsa is a self-attention model that casts in-context learning (ICL) as a continual learning problem under a fixed-size attention memory, and addresses the associated stability–plasticity dilemma through Bayesian metaplasticity [2602.09075]. In Palimpsa, the memory used by linear or gated linear attention is not represented solely by an attention state; it is paired with an importance, or precision, state that modulates plasticity online at the level of individual memory parameters. This design is intended to expand effective memory capacity, reduce interference in long sequences, and avoid both catastrophic forgetting and catastrophic remembering [2602.09075]. The name also invites comparison with the broader notion of palimpsest-like layering found in manuscript restoration and memory theory. In manuscript studies, palimpsests are overwritten documents whose earlier traces remain partially recoverable [2311.01130], while in neuromorphic hardware palimpsest memory denotes the coexistence of short-term overwriting with longer-term consolidated traces [2109.13198]. This suggests that Palimpsa belongs to a broader family of “layered retention” concepts, although the formalism in the model is specific to attention-based sequence processing [2602.09075].

## 1. Conceptual basis in fixed-size attention memory

Palimpsa starts from the view that ICL in transformers functions as an online associative memory: as a sequence $\{x_t\}$ arrives, keys $k_t$, queries $q_t$, and values $v_t$ are formed, and outputs $y_t$ depend on stored key–value associations [2602.09075]. In full attention this memory is realized as an explicit KV cache, whereas in linear attention, gated linear attention (GLA), and state space models (SSMs), it is replaced by a fixed-size state $S_t$ updated online [2602.09075]. The core problem is that fixed-size memories accumulate interference as sequences grow: new information must be integrated, but old information that remains relevant should not be overwritten prematurely.

The canonical linear-attention memory update is
$$
S_t = S_{t-1} + v_t \otimes k_t, \qquad y_t = S_t q_t,
$$
with interference arising whenever incoming keys are not orthogonal to previously stored keys [2602.09075]. Gated variants add input and forgetting gates, but retain a fixed, non-metaplastic learning rate, so capacity remains fixed and long sequences intensify interference [2602.09075]. Palimpsa is proposed specifically to resolve this limitation by tying plasticity to an online estimate of synaptic importance.

A useful contextual analogy comes from manuscript restoration. Digital restoration of ancient papyri aims to suppress structured background signals while preserving faint letter traces, especially in palimpsest-like conditions where text is partially erased or overlaid by periodic substrate patterns [0903.5045]. In a distinct but conceptually resonant domain, Palimpsa likewise attempts to preserve important earlier traces while allowing selective incorporation of new ones [2602.09075]. This analogy is interpretive rather than formal, but it captures the model’s emphasis on layered retention under interference.

## 2. Dual-state architecture and Bayesian metaplasticity

Palimpsa maintains two coupled states per attention head [2602.09075]. The first is an attention state
$$
S_t \in \mathbb{R}^{d_v \times d_k},
$$
or, in the variational formulation, a posterior mean $\mu_t$. The second is an importance or precision state
$$
I_t \in \mathbb{R}^{d_v \times d_k},
$$
with
$$
I_t = 1/\sigma_t^2,
$$
representing uncertainty when low and consolidated knowledge when high [2602.09075].

The probabilistic model is row-wise. For each row $i = 1,\dots,d_v$, Palimpsa assumes a Gaussian prior
$$
p(S_i) = \mathcal{N}(\mu_{\text{prior},i}, \Sigma_{\text{prior},i}),
$$
and in practice uses a factorized diagonal covariance
$$
\Sigma_{\text{prior},i} = \operatorname{diag}(\sigma_{\text{prior},i}^2),
$$
with constant prior precision
$$
I_{\text{prior},i} = 1/\sigma_{\text{prior},i}^2
$$
[2602.09075]. The likelihood of values conditioned on key and row-wise importance weights $\beta_t \in \mathbb{R}^{d_v}$ is
$$
p(v_t \mid k_t, \beta_t, S) \propto \exp\!\left(-\tfrac12 \lVert S k_t - v_t \rVert^2_{\operatorname{diag}(\beta_t)}\right),
$$
equivalently,
$$
-\log p(v_t \mid k_t, \beta_t, S) = \tfrac12 \sum_i \beta_{t,i}(S_i^T k_t - v_{t,i})^2
$$
[2602.09075]. Here $\beta_t$ acts as an input-gating term.

To enable forgetting without replay, the posterior is modified into a weighted posterior
$$
p_w(S \mid d_{1:t}) \propto p(d_t \mid S)\, p_w(S \mid d_{1:t-1}) \cdot \left(\frac{p_w(S \mid d_{1:t-1})}{p(S)}\right)^{-1/N_t},
$$
which corresponds to an online approximation to a fixed-size memory window of length $N_t$ [2602.09075]. The forgetting gate is
$$
\alpha_t = 1 - \frac{1}{N_t} \in (0,1], \qquad \alpha_t = \exp(-A d_t),
$$
with $A > 0$ learnable and $d_t \ge 0$ input-dependent [2602.09075]. Larger $N_t$ or $\alpha_t$ near $1$ slows forgetting; smaller $N_t$ accelerates it.

This structure implements Bayesian metaplasticity: synapses that have accumulated more evidence acquire larger precision and therefore smaller effective learning rates, while uncertain synapses remain plastic [2602.09075]. In the language of the paper, this dual-state design increases effective capacity, reduces interference, and prevents both catastrophic forgetting and catastrophic remembering.

## 3. Update equations, free-energy view, and diagnostics

Under the diagonal covariance approximation, Palimpsa updates precision and memory mean using local elementwise operations [2602.09075]. Writing rows implicitly and using $\odot$ for elementwise multiplication, the updates are
$$
I_t = \alpha_t I_{t-1} + (1-\alpha_t)I_{\text{prior}} + \beta_t \otimes k_t^{\circ 2}, \tag{1}
$$
$$
\mu_t = \alpha_t (I_{t-1}/I_t)\odot \mu_{t-1} + (1/I_t)\odot [(\beta_t \odot v_t)\otimes k_t], \tag{2}
$$
with readout
$$
y_t = \mu_t q_t.
$$
These equations formalize the claim that plasticity is inversely proportional to importance: the effective per-parameter learning rate is $1/I_t$ [2602.09075].

Equation (1) shows that importance increases through new evidence $\beta_t \otimes k_t^{\circ 2}$, is carried forward through $\alpha_t I_{t-1}$, and relaxes toward the prior through $(1-\alpha_t)I_{\text{prior}}$ [2602.09075]. Equation (2) combines an uncertainty-weighted carryover term with a data-driven increment scaled by $\beta_t$, $v_t$, and $k_t$. The result is a fixed-size memory in which different parameters can occupy different consolidation regimes at the same time.

The variational free energy for each row $i$ is given by
$$
F_{t,i}(\mu_i,\Sigma_i) =
\tfrac12 \beta_{t,i}(\mu_i^T k_t - v_{t,i})^2
+ (1-\alpha_t)\tfrac12 (\mu_i - \mu_{\text{prior},i})^T \Sigma_{\text{prior},i}^{-1} (\mu_i - \mu_{\text{prior},i})
+ \alpha_t \tfrac12 (\mu_i - \mu_{t-1,i})^T \Sigma_{t-1,i}^{-1} (\mu_i - \mu_{t-1,i})
+ C(\Sigma_i),
$$
where the three explicit quadratic terms are identified as plasticity, forgetting, and stability terms respectively [2602.09075]. Minimizing this objective yields the closed-form updates above.

The paper also introduces a head-level diagnostic, the metaplasticity ratio
$$
r_t = (I_{\max} - I_{\min})/I_{\min},
$$
computed per row at time $t$ [2602.09075]. Large values indicate strong differentiation between consolidated and plastic synapses. This metric is used in experiments to show that metaplastic structure emerges during training.

## 4. Relation to gated linear attention, Deltanet, Longhorn, MesaNet, and Mamba2

A central contribution of Palimpsa is to interpret several existing gated linear architectures as special cases of the same Bayesian framework under different posterior approximations [2602.09075]. This is not merely a naming exercise; it provides a transformation path from non-metaplastic to metaplastic models.

The relation can be summarized concisely:

| Model family | Posterior/architecture characterization | Relation to Palimpsa |
|---|---|---|
| Deltanet and Gated Deltanet | Fixed stability matrices, $\Sigma_{t-1,i}^{-1} \propto I$ | No metaplasticity; effectively constant $I_t$ |
| Longhorn | Diagonal stability with per-row input gating $\beta_t \in \mathbb{R}^{d_v}$ | No per-parameter metaplasticity |
| MesaNet | Approximate full covariance $\Sigma_t$ with row-wise scalar $\beta_t$ | Limited per-parameter metaplasticity |
| Mamba2 | Strong-forgetting regime with $I_t \approx I_{\text{prior}}$ | Special case where metaplasticity is effectively off |

In one canonical gated delta-rule form, the update is
$$
\mu_t = \mu_{t-1}(I - \beta_t k_t k_t^T) + \beta_t v_t k_t^T,
$$
and with forgetting,
$$
\mu_t = \mu_{t-1}\alpha_t(I - \beta_t k_t k_t^T) + \beta_t v_t k_t^T
$$
[2602.09075]. These models use fixed stability terms and therefore fixed capacity.

Palimpsa reduces to a Mamba2-like update when forgetting dominates or $\beta_t$ is extremely small, so that
$$
I_t \approx I_{\text{prior}}.
$$
Then Equation (2) becomes
$$
\mu_t \approx \alpha_t \mu_{t-1} + (1/I_{\text{prior}})\odot [(\beta_t \odot v_t)\otimes k_t], \tag{3}
$$
which the paper identifies as the asymptotic regime where Palimpsa behaves like Mamba2 [2602.09075]. On this reading, Mamba2 is a special case of Palimpsa in which forgetting dominates and importance does not evolve.

The paper further proposes a practical transformation recipe. By reparameterizing the value stream as
$$
v_t^* = \beta_t \odot v_t = \operatorname{SiLU}(\theta_v x_t),
$$
and initializing $\beta_t$ with a tiny head-wise scale so that $I_t \approx I_{\text{prior}}$, one can match the original non-metaplastic dynamics and then gradually learn $\beta_t$ and $A$ to enable metaplasticity [2602.09075]. This “upgrade path” is operationally important because the recommended training pipeline is to pretrain a fast non-metaplastic model and then fine-tune with Palimpsa.

## 5. Empirical results and training dynamics

Palimpsa is evaluated on Multi-Query Associative Recall (MQAR) and on language-model-based Commonsense Reasoning tasks [2602.09075]. In MQAR, sequences contain $L/4$ key–value pairs under maximal packing, and curriculum training uses $L \in \{128, 256, 512, 1024\}$ with two-layer GLA stacks, hidden size $128$, value-dimension expansion $\times 2$, and $8$ heads [2602.09075]. Learning rates are swept from $10^{-3}$ to $10^{-2}$, and each model is run with $8$ seeds [2602.09075].

The reported models are Palimpsa-D on a Deltanet backbone, Palimpsa-M on a Mamba2 backbone, ablations without metaplasticity, and a Gated Deltanet baseline [2602.09075]. The principal result is that metaplasticity consistently improves average MQAR accuracy across lengths for both backbones, with the gap widening at longer sequences and strongest for Palimpsa-D [2602.09075]. The interpretation given in the paper is that importance-tracked plasticity increases effective capacity and reduces interference as sequence length grows.

Training dynamics are also reported. The learned memory window $N_t$ grows rapidly, nearly exponentially at first, until it is sufficient for the task, and only then does the loss decrease [2602.09075]. The metaplasticity ratio
$$
(I_{\max}-I_{\min})/I_{\min}
$$
reaches approximately $40$, indicating that consolidated synapses become about $40\times$ more stable than plastic ones [2602.09075]. Early in training, $N_t$ is small and the dynamics are close to Mamba2, with $I_t \approx I_{\text{prior}}$; metaplasticity emerges only when longer memory is needed [2602.09075].

For language modeling and commonsense reasoning, pretraining is performed on Fineweb-Edu at two scales: $170$M and $760$M parameters, trained for $15$B and $30$B tokens, respectively [2602.09075]. Evaluation includes Wikitext perplexity, LAMBADA perplexity and accuracy, and zero-shot PIQA, HellaSwag, WinoGrande, ARC-e/c, and SIQA [2602.09075]. Fine-tuning to metaplastic Palimpsa variants preserves the total token budget; for example, at $760$M, metaplastic fine-tuning uses $2$B tokens from the $28$B checkpoint [2602.09075].

At $170$M, fine-tuned Palimpsa-D slightly outperforms baselines in average accuracy, and metaplastic variants show notably better LAMBADA perplexity, consistent with improved long-range retention [2602.09075]. At $760$M, fine-tuned Palimpsa-D and Palimpsa-M occupy the top two positions; each improves its non-metaplastic counterpart by approximately $0.6$ points in average accuracy, and Palimpsa-D exceeds Gated Deltanet by approximately $0.8$ points, with $52.27\%$ versus $51.50\%$ average [2602.09075]. The gains are again strongest on context-heavy tasks such as LAMBADA.

## 6. Implementation, computational profile, and limitations

Per token, the dominant cost of Palimpsa is $O(d_v d_k)$ for updating $\mu$ and $I$ and $O(d_v d_k)$ for readout, which is the same asymptotic order as standard GLA [2602.09075]. The principal state overhead is that both $\mu$ and $I$ must be stored, so state memory is approximately doubled relative to storing $\mu$ alone [2602.09075]. Additional parameters are limited to head-wise scalars such as $A$ and small networks or projections for producing $\beta_t$ and $d_t$, so parameter overhead is described as minor relative to the backbone [2602.09075].

The recommended implementation uses chunked associative scans across subsequences for GPU efficiency and includes numerical safeguards such as clamping $I$ within $[I_{\min}, I_{\max}]$ and optionally bounding $\beta_t$ [2602.09075]. Practical initialization is
$$
\mu_0 = 0, \qquad I_0 = I_{\text{prior}},
$$
often with constant positive prior precision per dimension [2602.09075]. Head-wise $A$ is initialized so that $\alpha_t$ corresponds to an initial memory window $N_t \approx 16$–$64$, and then learned [2602.09075]. Training guidance includes pretraining with non-metaplastic kernels for speed and switching to Palimpsa for an additional $5$–$10\%$ of tokens, while monitoring average $\log N_t$ and the metaplasticity ratio [2602.09075].

Throughput is a notable caveat. With custom kernels and associative scans, Palimpsa is reported to run at speeds comparable to Mamba1, but in large settings the non-metaplastic Mamba2 kernels can be more than $3\times$ faster [2602.09075]. This is the practical reason for the pretrain-then-fine-tune strategy.

Several limitations are stated explicitly [2602.09075]. The extra state $I$ roughly doubles per-head state memory. Unbounded $\beta_t$ or ill-conditioned $I$ can create numerical instability, motivating clipping and regularization. Under strong domain shift, learned priors and gates may require retuning, and the paper suggests possible head-wise priors or adaptive $A$ per layer. Proposed future directions include using head-level uncertainty for routing or confidence, combining metaplastic and non-metaplastic heads, and dynamic selection of metaplasticity based on context difficulty [2602.09075].

## 7. Broader “palimpsest” context across restoration, segmentation, and hardware memory

The broader research context clarifies why the name Palimpsa is apt. In manuscript studies, a palimpsest is a historical manuscript in which an older text, the *scriptio inferior*, was partially erased and overwritten with a newer text, the *scriptio superior*, producing geometric and spectral overlap between layers [2311.01130]. Digital restoration work on papyri emphasizes background suppression and letter enhancement through Fourier filtering and edge detection, especially when faint text is compromised by substrate texture, stains, or periodic fiber lines [0903.5045]. In the 7Q5 case study, custom Fourier filtering removed vertical papyrus lines and edge detection improved readability, demonstrating how structured interference can be suppressed while preserving weak textual traces [0903.5045].

That manuscript setting provides a concrete model of layered information recovery. In deep-learning work on palimpsests, semantic segmentation is proposed as a means of assigning per-pixel letter identities even when strokes overlap, using a U-Net with multi-label outputs [2311.01130]. The reported proof of concept uses synthetic overlapping letters with pixel accuracy of approximately $97\%$, precision of approximately $93\%$, recall of approximately $70\%$, and a visual success rate of about $80\%$ on the test set [2311.01130]. These studies do not define Palimpsa as a model, but they exemplify the same general challenge: preserving and disentangling superimposed traces under interference.

In neuromorphic hardware, the phrase “palimpsest memories” refers to the ability of a single synapse to carry multiple memory traces layered in time, with recent inputs dominating in the short term and older traces re-emerging after volatile dynamics decay [2109.13198]. In metal-oxide volatile memristors, this behavior arises from the interaction between fast volatile conductance changes and slower non-volatile residues. The paper reports that memristive synapses exhibit an expanded doubled capacity, can protect a consolidated long-term memory while up to hundreds of uncorrelated short-term memories temporarily overwrite it, and can implement familiarity detection of previously forgotten memories [2109.13198]. This is not the same mechanism as Palimpsa’s Bayesian precision tracking, but the family resemblance is strong: both involve selective consolidation, controlled overwrite, and differential timescales of retention.

A plausible implication is that Palimpsa’s name encodes a cross-domain metaphor rather than a purely ornamental label. In manuscript restoration, old writing remains recoverable beneath newer inscription [0903.5045; 2311.01130]. In memristive synapses, short-term traces temporarily cover but do not erase consolidated long-term ones [2109.13198]. In attention-based sequence models, Palimpsa formalizes an analogous objective: permit online rewriting while preserving important accumulated structure through per-parameter metaplasticity [2602.09075]. Within that frame, Palimpsa can be understood as a technical instantiation of palimpsest-like memory in the setting of fixed-size attention.

Source: https://www.emergentmind.com/topics/palimpsa