User as Engram: Internalizing Per-User Memory as Local Parametric Edits

Abstract: Personal memory in a LLM is two problems: content and reasoning skill. The brain keeps the two apart (a sparse, local engram in the hippocampus for each episode, a slow neocortex for the shared skills that interpret it), so a new fact need not overwrite everything else. Most personalization today keeps a user's facts outside the weights, in a natural-language memory file or a retrieval index. When facts are written into the model instead, the standard recipe is the per-user LoRA adapter, which does the opposite of the brain, folding content and skill into one global weight delta. Writing a user's facts as a LoRA contaminates text unrelated to them; writing the same facts as local Engram rows leaves it mathematically untouched, resulting in a roughly 33,000x smaller memory footprint. We therefore propose User as Engram: store a user's content as surgical edits to the hash-keyed memory table of an Engram model, and carry the reasoning skill in one shared adapter. This layered design matches per-user LoRA's direct recall while delivering 5.6x higher indirect-reasoning accuracy on average, and never makes a single user worse at reasoning than the untouched base. The edit is a glass box: writing a fact switches on its lookup at exactly the trigger, adds the value the answer needs, leaves every other position unchanged to the last bit, and fails if written into the wrong layer. Because different users' facts land in disjoint hash slots, their edits compose: many users live in one shared table at once, stacking additively and losslessly, where a per-user LoRA, a single global weight delta, admits only one. Upon retrieval, a per-user Engram table does not grow with the population the retriever must search, so past ~100 facts it overtakes a retrieval pipeline on a 2.5x larger model.

• The paper introduces a dual memory system that decouples user-specific factual content from shared reasoning skills using local, addressable edits.
• It employs content-addressable memory tables with sparse, gated lookups and deterministic hashing to achieve surgical, zero context cost modifications.
• Empirical results demonstrate significant improvements in storage efficiency and reduced cross-user interference compared to global adaptation methods.

User as Engram: Internalizing Per-User Memory as Local Parametric Edits

Problem Formulation and Motivation

The paper addresses a fundamental challenge in LLM personalization: separating the user-specific factual memory (content) from general reasoning skill. Conventional approaches, especially per-user LoRA adapters, conflate the two by embedding facts and reasoning into shared model weights, resulting in global contamination, degraded reasoning, storage inefficiency, and lack of isolation between users. In contrast, the brain's complementary learning systems divide episodic memory (hippocampal engram) from semantic reasoning (neocortex), preventing catastrophic interference. The paper operationalizes this architectural separation in LLMs by proposing a dual-layer memory structure: sparse, addressable content edits in a hash-keyed memory table, and shared reasoning skill in a global adapter.

Figure 1: User as Engram divides memory into per-user factual content (row overrides in the Engram memory table) and shared reasoning skill (single LoRA adapter), avoiding cross-user interference.

Methodology: Local Surgical Edits for User Memory

The Engram architecture leverages content-addressable memory tables integrated into the transformer backbone, with sparse, gated lookups triggered by suffix N-gram hashing. Insertion of per-user facts is realized by computing deterministic row addresses via token-level hashes and writing optimized value vectors into those rows, using three strategies: closed-form pseudo-inverse (UNEMBED_P), gradient-based per-fact optimization (OPT), and joint optimization over all user rows (Joint OPT). This edit is strictly local: only the targeted rows and positions are touched, guaranteeing bit-level invariance elsewhere in the network.

Figure 2: The Engram lookup and row insertion mechanism reside in designated transformer layers, allowing surgical content edits at precisely computed addresses.

Figure 3: Fact insertion decomposes into address computation (suffix N-gram hashing) and row value optimization, supporting scalable per-user memory with zero leakage.

Mechanistically, the inserted rows act as glass-box edits: each write both triggers its own lookup and injects the necessary value path into the residual stream at the trigger position, and leaves all other positions strictly unchanged. Contamination from unrelated text, a core limitation of global weight adaptation methods, is eliminated; empirical results demonstrate $33{,}000\times$ reduction in loss on unrelated text compared to per-user LoRA.

Figure 4: Contrast between addressed (Engram) and global (LoRA) edits: Engram edits are localized, while LoRA changes propagate everywhere in the network.

Figure 5: The Engram mechanism activates only at the trigger position, aligns with the intended value injection, and causes zero residual change elsewhere.

Comparison with Existing Personalization Paradigms

The paper clearly situates User as Engram among competing memory methods. Non-parametric approaches (ICL, retrieval/RAG) keep facts external and pay context costs on query. Parametric approaches (adapters, knowledge editing) perform global edits to the weights without isolation. User as Engram occupies a previously unpopulated corner: local edit at zero context cost, combining private factual memory and shared reasoning.

Figure 6: Taxonomy of personal-memory methods: context cost vs. locality of edit. Engram is unique in enabling local edits at zero context cost.

Empirical evaluation shows that per-user LoRA adapters triple the loss on unrelated held-out text, demonstrating architectural contamination. Engram edits, conversely, maintain virtually perfect invariance, confirming theoretical locality. Moreover, while LoRA achieves high direct recall, it fails to support indirect reasoning, especially on weak bases, reflecting its inability to disentangle factual content from skill. The strength of the base LM determines whether LoRA damage is masked or exposed.

Figure 7: Cross-base LoRA scaling: indirect recall is severely disrupted on weak bases, less so on instruction-tuned LMs.

Scaling and Density: Storage, Recall, and Composition

Fact-insertion recall remains high as the number of facts per user increases, with Joint OPT closing much of the gap to LoRA at orders-of-magnitude smaller storage ($88\,\mathrm{KB}$ vs. $14.2\,\mathrm{MB}$ at $100$ facts/user). The key density bottleneck is caused by gradient interference among co-active rows, not by backbone or table capacity, and the deployment cost scales linearly with the number of users and facts.

Figure 8: Recall scales favorably with Joint OPT, maintaining high performance as facts per user increase.

Figure 9: Cost-quality Pareto: Engram achieves comparable quality at a fraction of LoRA and retrieval baselines’ storage footprint.

Composition is additive: override maps with disjoint triggers (per-user, per-domain) stack without interference, providing robust support for multi-tenant and corporate-user layering.

Figure 10: Heatmap shows that disjoint-trigger domains compose additively, but domains sharing triggers experience degradation.

Retrieval vs. Parametric Edits: Performance and Context Cost

Direct recall for exact trigger queries is near-perfect with retrieval but pays substantial context costs, whereas Engram achieves 68% top-1 recall with zero context tokens. Under paraphrased queries, retrieval suffers, while multi-trigger Engram insertion closes the gap, outperforming by 22 points.

Figure 11: Retrieval excels on exact-trigger queries but incurs significant context cost; Engram performs comparably at zero context.

Figure 12: Paraphrase generalization: multi-trigger Engram insertion achieves top-1 recall for all paraphrases at increased OPT cost.

LOCOMO benchmark results indicate that Engram is superior for fact-based, chainable questions (single-hop, multi-hop, reasoning) but loses in open-domain and free-form queries where long span reproduction is required.

Figure 13: LOCOMO category breakdown: Engram wins on fact-based and reasoning queries, retrieval wins on open-domain.

Layered Architecture: Complementary Learning for Reasoning

To address the limit of compositional reasoning, the paper proposes a layered design: user facts in local Engram rows are complemented by a single shared LoRA, trained on a cross-user reasoning corpus. This design preserves direct recall (100%) and delivers substantially higher indirect reasoning accuracy ($5.6\times$ LoRA across seeds), with zero additional contamination above the shared skill.

Figure 1: Layered architecture mirrors complementary learning systems, with local addressable content and global shared skill.

Figure 14: Six-condition comparison validates that only the layered design achieves both indirect reasoning and low contamination.

The approach remains invariant to KB size; retrieval degrades quickly as candidate pool grows, but Engram’s per-user table does not scale with population, giving it decisive advantage at $N \gtrsim 100$.

Figure 15: As KB size grows, retrieval-based methods degrade while Engram’s indirect reasoning accuracy remains flat.

Multi-Tenant Serving and Scalability

Deployment evaluation of EngramServer demonstrates flat scaling with the number of tenants (~232 req/s @ 4.4 ms p50 latency, 100 users × 100 facts, sub-millisecond row application overhead), confirming that practical serving of per-user Engram edits is feasible at production scale, with strict isolation and minimal overhead.

Figure 16: EngramServer throughput and latency remain invariant as tenants scale, enabling efficient multi-tenant deployment.

Limitations and Future Directions

Engram’s chaining capability is bounded: composition across triggers (multi-hop) is not supported intrinsically. Proposed future work includes mixing recite-then-reason traces into pretraining, per-user hash salt to further isolate user edits, and adaptive slot loading to alleviate density interference. Practical deployment requires Engram-pretrained backbones, which have been trained up to $1.22\mathrm{B}$ parameters; scaling to production models is a next step.

Theoretical Implications and Practical Outlook

The architectural split between content and skill—addressable, private factual memory and shared reasoning—realizes pattern separation, preventing catastrophic interference and supporting efficient multi-user personalization. It theoretically aligns with complementary learning systems theory and achieves engineering separation of pattern storage and interpretation. The practical outcome is robust, scalable, and leak-free per-user memory, with resource efficiency and compositional elegance.

Conclusion

User as Engram advances the state of personalized memory for LLMs by architecting strict locality in memory edits, isolating user content via addressable row overrides, and supporting global reasoning skill through a shared adapter. The paradigm circumvents catastrophic interference, achieves high recall and reasoning performance at orders of magnitude lower storage cost, and enables linear-scaling multi-tenant serving. Remaining limits are mechanistic, not architectural, and the method opens fertile ground for scalable, theory-inspired, and practically deployable personal memory systems in LLMs.