The Art of Not Forgetting

Abstract: We introduce CMP (Cognitive Memory Primitive), an architecture that represents inputs as sparse relational codes, stores them in a two-tier competitive memory, and learns entirely through local, gradient-free updates, with no backpropagation anywhere in the network. We use this architecture to test a specific hypothesis: that catastrophic forgetting, usually treated as a training-time defect to be patched with replay or regularization, is instead a structural consequence of how backpropagation assigns credit and that a learning rule that is local and sparse by construction should resist it without a patch. On a controlled domain-incremental protocol across 15 text domains, three-seed replicated, CMP's backward transfer is 15-19x better than a matched-size Transformer trained with online EWC, and the result survives a domain-order control (reported as a range, +0.24 to +0.44, rather than a single figure). We report this alongside a real, substantial accuracy gap versus the Transformer baseline, a null result on a recognized vision benchmark, and a diagnosed, unresolved failure attempting to combine this architecture with a separate mechanism that improves raw accuracy, disclosed because an honest negative result is more useful than an omitted one. The central claim is narrow and falsifiable: local, sparse, non-backpropagation learning measurably resists catastrophic forgetting better than backpropagation with its standard fix, under conditions we state precisely.

• The paper introduces CMP, a roughly 6.5-million-parameter byte-level language model that uses sparse local updates, competitive memory, predictive coding, and a movement-based weight-protect mechanism without backpropagation.
• The paper reports backward transfer of +0.1482 BPB for CMP, compared with +2.2457 for online EWC and +2.7897 for sequential fine-tuning, indicating 15–19× less forgetting under matched capacity and data.
• The paper finds that forgetting resistance is text-specific and order-sensitive, while CMP still trails Transformers in raw accuracy, shows no advantage on Split-MNIST, and requires further validation on standardized benchmarks and larger-scale experiments.

Overview and central claim

"The Art of Not Forgetting" (Atmuri, Kumar, and Bhogarajula, Arkadhi Labs) introduces CMP (Cognitive Memory Primitive), a byte-level LLM that learns entirely through local, gradient-free updates with no backpropagation anywhere in the network. The paper's purpose is not to advance a new architecture for its own sake but to test a specific, falsifiable hypothesis: that catastrophic forgetting is a structural consequence of how backpropagation assigns credit — globally, densely, to every weight via a single scalar loss — and that a learning rule that is local and sparse by construction should resist forgetting without an added correction term. The authors are careful to scope the claim narrowly: CMP is not more accurate than a Transformer, and the paper does not claim general superiority over attention-based architectures.

The central result is a domain-incremental continual-learning comparison across 15 text domains: a matched-size (~6.5M parameter) CMP with the proposed weight-protect mechanism achieves a backward transfer (BWT, in bits-per-byte, where positive means forgetting) of $+0.1482$, versus $+2.2457$ for online EWC and $+2.7897$ for naive sequential fine-tuning on a parameter-matched Transformer — a 15–19× reduction in forgetting under matched capacity and data budget. The paper is unusually explicit about what does not work: a substantial raw accuracy gap versus the Transformer, a null result on Split-MNIST, and a diagnosed, unresolved failure to merge the architecture with a separate higher-accuracy mechanism.

Architecture

CMP is assembled from established components — vector-symbolic binding (Plate's Holographic Reduced Representations, Pollack's Recursive Distributed Representations), Kanerva-style sparse distributed memory, predictive coding in the Rao–Ballard/Whittington–Bogacz tradition, and a delta-rule readout. The pipeline proceeds in six stages:

• Sparse relational binding. Two frozen random embedding matrices map consecutive bytes $x_{t-1}$, $x_t$ into left- and right-role vectors, which are combined by elementwise product, sparsified via k-winners-take-all, and renormalized into a relational code $z_t$. This is a fixed-size, $O(1)$ operation replacing self-attention's $O(n^2)$ all-pairs lookup.
• Two-tier competitive memory. A fast buffer and a slow register store relational codes via content-addressed writes gated by a calibrated match threshold; only codes surviving repeated matching are promoted from buffer to register. Retrieval is a softmax-weighted read over slots. This pairing of a fast sparse store with a slower distributed readout is explicitly motivated by McClelland et al.'s complementary learning systems account.
• Hierarchy. Consecutive relational codes are bound through a second pair of frozen mixing matrices to form a coarser, word-scale code $h_t$, analogous to multi-scale attention heads.
• Predictive coding. The coarse code predicts the fine code through a locally trained matrix $F$; the sparse residual $+2.2457$0 becomes an additional context term. $+2.2457$1 receives no gradient from the downstream loss — the top-down prediction is structurally self-supervised.
• Readout. All terms ($+2.2457$2, memory retrieval, $+2.2457$3, $+2.2457$4, plus leaky-integrator context terms and a bigram table) are summed linearly into next-byte logits. The absence of a nonlinear MLP between features and loss is what makes local credit assignment tractable.
• Local learning and weight-protect. Every readout weight is updated by a one-step delta rule using the softmax error, one hop from the loss and no further.

The paper identifies weight-protect as its one concrete mechanism-level contribution. Because EWC's Fisher-information importance signal presumes a backpropagated gradient to square, CMP instead accumulates the mean absolute movement of each readout tensor across domains as an importance score and scales future updates inversely to it. The authors are careful to state this as a gradient-free heuristic substitution, not new mathematics, and note that to their knowledge raw weight movement has not previously been used as an EWC-style importance signal.

Experimental protocol and the central result

The evaluation is a domain-incremental protocol: a single model is trained sequentially on 15 domains (Wikipedia, CPython source, Shakespeare, KJV Bible, Reuters, Moby Dick, the Federalist Papers, Darwin, Whitman, Lodash, Marcus Aurelius, Flatland, Alice in Wonderland, Python documentation, and a Hindi corpus) with no replay and no task identifier, building a $+2.2457$5 matrix of bits-per-byte scores. BWT and FWT are adapted from Lopez-Paz and Ranzato's formulation to BPB, with the sign convention stated explicitly (positive BWT means net forgetting).

The experimental design is deliberately layered, with each step closing a hole in the previous one:

1. Accuracy audit. Before any forgetting claim, the authors verify the model has something real to forget: CMP beats the bigram frequency floor on every domain and every seed, with margins of $+2.2457$6 to $+2.2457$7 BPB. Notably, an initial Transformer baseline at the original 150,000-byte budget failed this audit outright due to severe overfitting under 728× data repetition; the budget was raised to 3.8M bytes, and all Transformer results reported pass the audit.
2. Three-seed replication. CMP's BWT is tight across seeds 42–44 (mean $+2.2457$8), while FWT is noisy ($+2.2457$9), driven by one outlier seed. The paper flags the forward-transfer number as not to be over-interpreted.
3. Matched baseline comparison. At matched ~6.5M parameters and the 3.8M-byte budget, CMP's BWT of $+2.7897$0 compares against $+2.7897$1 (online EWC) and $+2.7897$2 (naive fine-tuning). The authors reconcile this figure with the replication mean explicitly: it is a re-run at the budget that makes the comparison genuinely matched, not the more favorable number.
4. Domain-order control. On a 5-domain subset, BWT ranges from $+2.7897$3 (canonical order) to $+2.7897$4 (reverse) — the canonical ordering is the best case, not the average, so the paper reports a range rather than a single figure. Even the worst tested ordering beats EWC by roughly 5×. Three orderings out of 120 possible are a sample, not an exhaustive sweep, and the true worst case may lie outside the reported range.
5. Vision anchor. On Hilbert-serialized Split-MNIST (5-task), sparse (weight-protect) and dense conditions are statistically indistinguishable in the informal sense — BWT means within one standard deviation across three seeds. This is reported as an honest null result: the forgetting-resistance mechanism shows no measurable advantage outside text, and the paper does not claim vision generality from it.

A preliminary scale probe in the appendix finds that a 5× parameter increase yields negligible improvement at the data-starved budget but a real, modest improvement (BWT $+2.7897$5) at the corrected budget — explicitly a probe, not a scaling-law derivation.

Limitations and open questions

The paper's self-accounting is thorough. The main limitations, conceded at the point where they bear on results, are:

• Accuracy gap. CMP's raw single-domain BPB (3.1–3.27) trails a properly trained matched-size Transformer by a wide margin. The claim is confined to the forgetting axis; the authors note that no non-attention architecture (Mamba, RWKV, RetNet, xLSTM) has cleanly matched Transformer short-context accuracy at matched parameters, so a real gap alongside a different real contribution has precedent.
• Order sensitivity. BWT varies by nearly a factor of two across the three tested orderings, and only three of 120 possible orderings were evaluated.
• Online EWC baseline. The comparison uses a running-Fisher variant (Schwarz et al.) rather than vanilla per-task EWC, chosen for tractability — a fair but not identical baseline.
• Self-assembled corpus. The 15-domain sequence is not a standardized benchmark; only the BWT/FWT metric formulation is established.
• Null vision result. No generalization of the effect outside text was demonstrated.
• Frozen embeddings. The binding matrices $+2.7897$6, $+2.7897$7 are fixed at initialization, a design choice acknowledged as a limitation.
• Failed merge. An attempt to combine the flagship architecture with a separate depth-line mechanism (which reaches 2.49–2.51 BPB on single-domain prediction) underperformed the flagship (3.48 vs. 3.27 BPB). The diagnosis — dense (non-kWTA) codes degrading memory slot matching, plus signal dilution across seven or more readout terms — is plausible but the fix is unresolved.
• Code availability. Training code is not publicly released, available only on request, which limits independent verification.

The two most direct open questions the paper itself identifies are closing the accuracy gap and deriving a proper scaling law for the weight-protect effect; both are explicitly left open rather than claimed.

Conclusion

This paper makes a narrow, well-scoped, and carefully hedged empirical claim: under a controlled domain-incremental protocol with matched capacity and data budget, a local, sparse, gradient-free learning system with a movement-based plasticity throttle exhibits 15–19× less backward transfer than a parameter-matched Transformer trained with online EWC, with the effect surviving seed replication and a domain-order control. It reports this alongside a substantial accuracy deficit, a null result on a recognized vision benchmark, and a diagnosed negative result on architecture combination. The strength of the paper lies less in any single component — all of which are established techniques — than in the disciplined experimental structure and the falsifiability of its central claim; its principal weaknesses are the self-assembled corpus, the partial order-sensitivity sweep, and the absence of any demonstrated generality beyond text.