Palimpsa: Metaplastic Attention Memory

• Palimpsa is a self-attention model that treats in-context learning as an online continual learning problem by using a fixed-size dual-state memory architecture.
• It pairs an attention state with an importance (precision) state to implement Bayesian metaplasticity, modulating plasticity and mitigating interference and catastrophic forgetting.
• Empirical evaluations show that Palimpsa enhances effective memory capacity and accuracy on complex sequence tasks compared to traditional gated linear attention models.

Searching arXiv for the cited "Palimpsa" paper and closely related palimpsest-related work to ground the article in current arXiv records. Palimpsa is a self-attention model that casts in-context learning (ICL) as a continual learning problem under a fixed-size attention memory, and addresses the associated stability–plasticity dilemma through Bayesian metaplasticity (Bonnet et al., 9 Feb 2026). In Palimpsa, the memory used by linear or gated linear attention is not represented solely by an attention state; it is paired with an importance, or precision, state that modulates plasticity online at the level of individual memory parameters. This design is intended to expand effective memory capacity, reduce interference in long sequences, and avoid both catastrophic forgetting and catastrophic remembering (Bonnet et al., 9 Feb 2026). The name also invites comparison with the broader notion of palimpsest-like layering found in manuscript restoration and memory theory. In manuscript studies, palimpsests are overwritten documents whose earlier traces remain partially recoverable (Perino et al., 2023), while in neuromorphic hardware palimpsest memory denotes the coexistence of short-term overwriting with longer-term consolidated traces (Giotis et al., 2021). This suggests that Palimpsa belongs to a broader family of “layered retention” concepts, although the formalism in the model is specific to attention-based sequence processing (Bonnet et al., 9 Feb 2026).

1. Conceptual basis in fixed-size attention memory

Palimpsa starts from the view that ICL in transformers functions as an online associative memory: as a sequence $\{x_t\}$ arrives, keys $k_t$, queries $q_t$, and values $v_t$ are formed, and outputs $y_t$ depend on stored key–value associations (Bonnet et al., 9 Feb 2026). In full attention this memory is realized as an explicit KV cache, whereas in linear attention, gated linear attention (GLA), and state space models (SSMs), it is replaced by a fixed-size state $S_t$ updated online (Bonnet et al., 9 Feb 2026). The core problem is that fixed-size memories accumulate interference as sequences grow: new information must be integrated, but old information that remains relevant should not be overwritten prematurely.

The canonical linear-attention memory update is

$$S_t = S_{t-1} + v_t \otimes k_t, \qquad y_t = S_t q_t,$$

with interference arising whenever incoming keys are not orthogonal to previously stored keys (Bonnet et al., 9 Feb 2026). Gated variants add input and forgetting gates, but retain a fixed, non-metaplastic learning rate, so capacity remains fixed and long sequences intensify interference (Bonnet et al., 9 Feb 2026). Palimpsa is proposed specifically to resolve this limitation by tying plasticity to an online estimate of synaptic importance.

A useful contextual analogy comes from manuscript restoration. Digital restoration of ancient papyri aims to suppress structured background signals while preserving faint letter traces, especially in palimpsest-like conditions where text is partially erased or overlaid by periodic substrate patterns (0903.5045). In a distinct but conceptually resonant domain, Palimpsa likewise attempts to preserve important earlier traces while allowing selective incorporation of new ones (Bonnet et al., 9 Feb 2026). This analogy is interpretive rather than formal, but it captures the model’s emphasis on layered retention under interference.

2. Dual-state architecture and Bayesian metaplasticity

Palimpsa maintains two coupled states per attention head (Bonnet et al., 9 Feb 2026). The first is an attention state

$S_t \in \mathbb{R}^{d_v \times d_k},$

or, in the variational formulation, a posterior mean $\mu_t$. The second is an importance or precision state

$I_t \in \mathbb{R}^{d_v \times d_k},$

with

$k_t$0

representing uncertainty when low and consolidated knowledge when high (Bonnet et al., 9 Feb 2026).

The probabilistic model is row-wise. For each row $k_t$1, Palimpsa assumes a Gaussian prior

$k_t$2

and in practice uses a factorized diagonal covariance

$k_t$3

with constant prior precision

$k_t$4

(Bonnet et al., 9 Feb 2026). The likelihood of values conditioned on key and row-wise importance weights $k_t$5 is

$k_t$6

equivalently,

$k_t$7

(Bonnet et al., 9 Feb 2026). Here $k_t$8 acts as an input-gating term.

To enable forgetting without replay, the posterior is modified into a weighted posterior

$k_t$9

which corresponds to an online approximation to a fixed-size memory window of length $q_t$0 (Bonnet et al., 9 Feb 2026). The forgetting gate is

$q_t$1

with $q_t$2 learnable and $q_t$3 input-dependent (Bonnet et al., 9 Feb 2026). Larger $q_t$4 or $q_t$5 near $q_t$6 slows forgetting; smaller $q_t$7 accelerates it.

This structure implements Bayesian metaplasticity: synapses that have accumulated more evidence acquire larger precision and therefore smaller effective learning rates, while uncertain synapses remain plastic (Bonnet et al., 9 Feb 2026). In the language of the paper, this dual-state design increases effective capacity, reduces interference, and prevents both catastrophic forgetting and catastrophic remembering.

3. Update equations, free-energy view, and diagnostics

Under the diagonal covariance approximation, Palimpsa updates precision and memory mean using local elementwise operations (Bonnet et al., 9 Feb 2026). Writing rows implicitly and using $q_t$8 for elementwise multiplication, the updates are

$q_t$9

$v_t$0

with readout

$v_t$1

These equations formalize the claim that plasticity is inversely proportional to importance: the effective per-parameter learning rate is $v_t$2 (Bonnet et al., 9 Feb 2026).

Equation (1) shows that importance increases through new evidence $v_t$3, is carried forward through $v_t$4, and relaxes toward the prior through $v_t$5 (Bonnet et al., 9 Feb 2026). Equation (2) combines an uncertainty-weighted carryover term with a data-driven increment scaled by $v_t$6, $v_t$7, and $v_t$8. The result is a fixed-size memory in which different parameters can occupy different consolidation regimes at the same time.

The variational free energy for each row $v_t$9 is given by

$y_t$0

where the three explicit quadratic terms are identified as plasticity, forgetting, and stability terms respectively (Bonnet et al., 9 Feb 2026). Minimizing this objective yields the closed-form updates above.

The paper also introduces a head-level diagnostic, the metaplasticity ratio

$y_t$1

computed per row at time $y_t$2 (Bonnet et al., 9 Feb 2026). Large values indicate strong differentiation between consolidated and plastic synapses. This metric is used in experiments to show that metaplastic structure emerges during training.

4. Relation to gated linear attention, Deltanet, Longhorn, MesaNet, and Mamba2

A central contribution of Palimpsa is to interpret several existing gated linear architectures as special cases of the same Bayesian framework under different posterior approximations (Bonnet et al., 9 Feb 2026). This is not merely a naming exercise; it provides a transformation path from non-metaplastic to metaplastic models.

The relation can be summarized concisely:

Model family	Posterior/architecture characterization	Relation to Palimpsa
Deltanet and Gated Deltanet	Fixed stability matrices, $y_t$3	No metaplasticity; effectively constant $y_t$4
Longhorn	Diagonal stability with per-row input gating $y_t$5	No per-parameter metaplasticity
MesaNet	Approximate full covariance $y_t$6 with row-wise scalar $y_t$7	Limited per-parameter metaplasticity
Mamba2	Strong-forgetting regime with $y_t$8	Special case where metaplasticity is effectively off

In one canonical gated delta-rule form, the update is

$y_t$9

and with forgetting,

$S_t$0

(Bonnet et al., 9 Feb 2026). These models use fixed stability terms and therefore fixed capacity.

Palimpsa reduces to a Mamba2-like update when forgetting dominates or $S_t$1 is extremely small, so that

$S_t$2

Then Equation (2) becomes

$S_t$3

which the paper identifies as the asymptotic regime where Palimpsa behaves like Mamba2 (Bonnet et al., 9 Feb 2026). On this reading, Mamba2 is a special case of Palimpsa in which forgetting dominates and importance does not evolve.

The paper further proposes a practical transformation recipe. By reparameterizing the value stream as

$S_t$4

and initializing $S_t$5 with a tiny head-wise scale so that $S_t$6, one can match the original non-metaplastic dynamics and then gradually learn $S_t$7 and $S_t$8 to enable metaplasticity (Bonnet et al., 9 Feb 2026). This “upgrade path” is operationally important because the recommended training pipeline is to pretrain a fast non-metaplastic model and then fine-tune with Palimpsa.

5. Empirical results and training dynamics

Palimpsa is evaluated on Multi-Query Associative Recall (MQAR) and on language-model-based Commonsense Reasoning tasks (Bonnet et al., 9 Feb 2026). In MQAR, sequences contain $S_t$9 key–value pairs under maximal packing, and curriculum training uses $$S_t = S_{t-1} + v_t \otimes k_t, \qquad y_t = S_t q_t,$$0 with two-layer GLA stacks, hidden size $$S_t = S_{t-1} + v_t \otimes k_t, \qquad y_t = S_t q_t,$$1, value-dimension expansion $$S_t = S_{t-1} + v_t \otimes k_t, \qquad y_t = S_t q_t,$$2, and $$S_t = S_{t-1} + v_t \otimes k_t, \qquad y_t = S_t q_t,$$3 heads (Bonnet et al., 9 Feb 2026). Learning rates are swept from $$S_t = S_{t-1} + v_t \otimes k_t, \qquad y_t = S_t q_t,$$4 to $$S_t = S_{t-1} + v_t \otimes k_t, \qquad y_t = S_t q_t,$$5, and each model is run with $$S_t = S_{t-1} + v_t \otimes k_t, \qquad y_t = S_t q_t,$$6 seeds (Bonnet et al., 9 Feb 2026).

The reported models are Palimpsa-D on a Deltanet backbone, Palimpsa-M on a Mamba2 backbone, ablations without metaplasticity, and a Gated Deltanet baseline (Bonnet et al., 9 Feb 2026). The principal result is that metaplasticity consistently improves average MQAR accuracy across lengths for both backbones, with the gap widening at longer sequences and strongest for Palimpsa-D (Bonnet et al., 9 Feb 2026). The interpretation given in the paper is that importance-tracked plasticity increases effective capacity and reduces interference as sequence length grows.

Training dynamics are also reported. The learned memory window $$S_t = S_{t-1} + v_t \otimes k_t, \qquad y_t = S_t q_t,$$7 grows rapidly, nearly exponentially at first, until it is sufficient for the task, and only then does the loss decrease (Bonnet et al., 9 Feb 2026). The metaplasticity ratio

$$S_t = S_{t-1} + v_t \otimes k_t, \qquad y_t = S_t q_t,$$8

reaches approximately $$S_t = S_{t-1} + v_t \otimes k_t, \qquad y_t = S_t q_t,$$9, indicating that consolidated synapses become about $S_t \in \mathbb{R}^{d_v \times d_k},$0 more stable than plastic ones (Bonnet et al., 9 Feb 2026). Early in training, $S_t \in \mathbb{R}^{d_v \times d_k},$1 is small and the dynamics are close to Mamba2, with $S_t \in \mathbb{R}^{d_v \times d_k},$2; metaplasticity emerges only when longer memory is needed (Bonnet et al., 9 Feb 2026).

For language modeling and commonsense reasoning, pretraining is performed on Fineweb-Edu at two scales: $S_t \in \mathbb{R}^{d_v \times d_k},$3M and $S_t \in \mathbb{R}^{d_v \times d_k},$4M parameters, trained for $S_t \in \mathbb{R}^{d_v \times d_k},$5B and $S_t \in \mathbb{R}^{d_v \times d_k},$6B tokens, respectively (Bonnet et al., 9 Feb 2026). Evaluation includes Wikitext perplexity, LAMBADA perplexity and accuracy, and zero-shot PIQA, HellaSwag, WinoGrande, ARC-e/c, and SIQA (Bonnet et al., 9 Feb 2026). Fine-tuning to metaplastic Palimpsa variants preserves the total token budget; for example, at $S_t \in \mathbb{R}^{d_v \times d_k},$7M, metaplastic fine-tuning uses $S_t \in \mathbb{R}^{d_v \times d_k},$8B tokens from the $S_t \in \mathbb{R}^{d_v \times d_k},$9B checkpoint (Bonnet et al., 9 Feb 2026).

At $\mu_t$0M, fine-tuned Palimpsa-D slightly outperforms baselines in average accuracy, and metaplastic variants show notably better LAMBADA perplexity, consistent with improved long-range retention (Bonnet et al., 9 Feb 2026). At $\mu_t$1M, fine-tuned Palimpsa-D and Palimpsa-M occupy the top two positions; each improves its non-metaplastic counterpart by approximately $\mu_t$2 points in average accuracy, and Palimpsa-D exceeds Gated Deltanet by approximately $\mu_t$3 points, with $\mu_t$4 versus $\mu_t$5 average (Bonnet et al., 9 Feb 2026). The gains are again strongest on context-heavy tasks such as LAMBADA.

6. Implementation, computational profile, and limitations

Per token, the dominant cost of Palimpsa is $\mu_t$6 for updating $\mu_t$7 and $\mu_t$8 and $\mu_t$9 for readout, which is the same asymptotic order as standard GLA (Bonnet et al., 9 Feb 2026). The principal state overhead is that both $I_t \in \mathbb{R}^{d_v \times d_k},$0 and $I_t \in \mathbb{R}^{d_v \times d_k},$1 must be stored, so state memory is approximately doubled relative to storing $I_t \in \mathbb{R}^{d_v \times d_k},$2 alone (Bonnet et al., 9 Feb 2026). Additional parameters are limited to head-wise scalars such as $I_t \in \mathbb{R}^{d_v \times d_k},$3 and small networks or projections for producing $I_t \in \mathbb{R}^{d_v \times d_k},$4 and $I_t \in \mathbb{R}^{d_v \times d_k},$5, so parameter overhead is described as minor relative to the backbone (Bonnet et al., 9 Feb 2026).

The recommended implementation uses chunked associative scans across subsequences for GPU efficiency and includes numerical safeguards such as clamping $I_t \in \mathbb{R}^{d_v \times d_k},$6 within $I_t \in \mathbb{R}^{d_v \times d_k},$7 and optionally bounding $I_t \in \mathbb{R}^{d_v \times d_k},$8 (Bonnet et al., 9 Feb 2026). Practical initialization is

$I_t \in \mathbb{R}^{d_v \times d_k},$9

often with constant positive prior precision per dimension (Bonnet et al., 9 Feb 2026). Head-wise $k_t$00 is initialized so that $k_t$01 corresponds to an initial memory window $k_t$02–$k_t$03, and then learned (Bonnet et al., 9 Feb 2026). Training guidance includes pretraining with non-metaplastic kernels for speed and switching to Palimpsa for an additional $k_t$04–$k_t$05 of tokens, while monitoring average $k_t$06 and the metaplasticity ratio (Bonnet et al., 9 Feb 2026).

Throughput is a notable caveat. With custom kernels and associative scans, Palimpsa is reported to run at speeds comparable to Mamba1, but in large settings the non-metaplastic Mamba2 kernels can be more than $k_t$07 faster (Bonnet et al., 9 Feb 2026). This is the practical reason for the pretrain-then-fine-tune strategy.

Several limitations are stated explicitly (Bonnet et al., 9 Feb 2026). The extra state $k_t$08 roughly doubles per-head state memory. Unbounded $k_t$09 or ill-conditioned $k_t$10 can create numerical instability, motivating clipping and regularization. Under strong domain shift, learned priors and gates may require retuning, and the paper suggests possible head-wise priors or adaptive $k_t$11 per layer. Proposed future directions include using head-level uncertainty for routing or confidence, combining metaplastic and non-metaplastic heads, and dynamic selection of metaplasticity based on context difficulty (Bonnet et al., 9 Feb 2026).

7. Broader “palimpsest” context across restoration, segmentation, and hardware memory

The broader research context clarifies why the name Palimpsa is apt. In manuscript studies, a palimpsest is a historical manuscript in which an older text, the scriptio inferior, was partially erased and overwritten with a newer text, the scriptio superior, producing geometric and spectral overlap between layers (Perino et al., 2023). Digital restoration work on papyri emphasizes background suppression and letter enhancement through Fourier filtering and edge detection, especially when faint text is compromised by substrate texture, stains, or periodic fiber lines (0903.5045). In the 7Q5 case study, custom Fourier filtering removed vertical papyrus lines and edge detection improved readability, demonstrating how structured interference can be suppressed while preserving weak textual traces (0903.5045).

That manuscript setting provides a concrete model of layered information recovery. In deep-learning work on palimpsests, semantic segmentation is proposed as a means of assigning per-pixel letter identities even when strokes overlap, using a U-Net with multi-label outputs (Perino et al., 2023). The reported proof of concept uses synthetic overlapping letters with pixel accuracy of approximately $k_t$12, precision of approximately $k_t$13, recall of approximately $k_t$14, and a visual success rate of about $k_t$15 on the test set (Perino et al., 2023). These studies do not define Palimpsa as a model, but they exemplify the same general challenge: preserving and disentangling superimposed traces under interference.

In neuromorphic hardware, the phrase “palimpsest memories” refers to the ability of a single synapse to carry multiple memory traces layered in time, with recent inputs dominating in the short term and older traces re-emerging after volatile dynamics decay (Giotis et al., 2021). In metal-oxide volatile memristors, this behavior arises from the interaction between fast volatile conductance changes and slower non-volatile residues. The paper reports that memristive synapses exhibit an expanded doubled capacity, can protect a consolidated long-term memory while up to hundreds of uncorrelated short-term memories temporarily overwrite it, and can implement familiarity detection of previously forgotten memories (Giotis et al., 2021). This is not the same mechanism as Palimpsa’s Bayesian precision tracking, but the family resemblance is strong: both involve selective consolidation, controlled overwrite, and differential timescales of retention.

A plausible implication is that Palimpsa’s name encodes a cross-domain metaphor rather than a purely ornamental label. In manuscript restoration, old writing remains recoverable beneath newer inscription [(0903.5045); (Perino et al., 2023)]. In memristive synapses, short-term traces temporarily cover but do not erase consolidated long-term ones (Giotis et al., 2021). In attention-based sequence models, Palimpsa formalizes an analogous objective: permit online rewriting while preserving important accumulated structure through per-parameter metaplasticity (Bonnet et al., 9 Feb 2026). Within that frame, Palimpsa can be understood as a technical instantiation of palimpsest-like memory in the setting of fixed-size attention.