$δ$-mem: Efficient Online Memory for Large Language Models

Abstract: LLMs increasingly need to accumulate and reuse historical information in long-term assistants and agent systems. Simply expanding the context window is costly and often fails to ensure effective context utilization. We propose $δ$-mem, a lightweight memory mechanism that augments a frozen full-attention backbone with a compact online state of associative memory. $δ$-mem compresses past information into a fixed-size state matrix updated by delta-rule learning, and uses its readout to generate low-rank corrections to the backbone's attention computation during generation. With only an $8\times8$ online memory state, $δ$-mem improves the average score to $1.10\times$ that of the frozen backbone and $1.15\times$ that of the strongest non-$δ$-mem memory baseline. It achieves larger gains on memory-heavy benchmarks, reaching $1.31\times$ on MemoryAgentBench and $1.20\times$ on LoCoMo, while largely preserving general capabilities. These results show that effective memory can be realized through a compact online state directly coupled with attention computation, without full fine-tuning, backbone replacement, or explicit context extension.

• The paper introduces δ-mem, which integrates a compact, online associative memory into Transformer attention to boost long-context reasoning.
• It employs a delta-rule inspired update on an 8×8 memory state, enabling low-rank corrections with minimal computational overhead.
• Empirical results show δ-mem outperforms baselines by up to 1.31× on benchmark tasks while keeping parameter increases negligible.

$\delta$-mem: Efficient Online Memory for LLMs

Motivation and Problem Setting

Transformer-based LLMs are increasingly applied in scenarios requiring persistent, dynamic memory such as long-term conversational assistants and agent-based environments. Existing approaches to long-context memory—expanding the context window, textual retrieval, or pretrained parametric adaptation—are insufficient due to high computational cost, information loss from compression, retrieval noise, limited context utilization, and poor alignment with the model’s forward computation. Despite advances in context extension and retrieval-augmented paradigms, the underlying issue of efficiently retaining and exploiting relevant historical information across extended interactions remains open.

$\delta$-mem Mechanism

$\delta$-mem introduces a compact, online associative memory attached to a frozen Transformer attention backbone. Rather than extending the input sequence or performing external retrieval, it keeps a fixed-size, dynamically updated state that stores historical associations using a delta-rule-inspired update. This state is compressed into an $8 \times 8$ matrix and continuously maintained during inference. The online state directly participates in each forward pass: at every sequence position, the current hidden state is projected into a memory-specific key, value, and query; the query reads out associative signals from the prior memory state, which are then used to generate low-rank corrections to the attention computation; and then the state is written using the delta-rule residuum between predicted and target memory value.

Figure 1: Overview of $\delta$-mem: a frozen Transformer is augmented by projecting hidden states into a low-rank associative memory space; associative signals from the previous state modulate attention, and memory is updated via a delta rule.

The inference cost is independent of the full context length, as the memory read/write operations involve only the compact state. The memory readout can be injected at various points in the attention block (query, key, value, output)—enabling low-rank steering tightly coupled with the backbone’s computations.

Memory State and Update Dynamics

The memory state $\mathbf S \in \mathbb{R}^{r \times r}$ (with $r$ typically 8) is updated at every token or at coarser granularities (segment/message). For each time step, a key-value association is compressed into low-dimensional representations; the state update is governed by a gated delta rule with learnable retention/forget gates and write coefficients. This allows memory to correct for prediction errors while gradually forgetting obsolete information, maintaining stability over long histories.

Modes and Variants

Three update strategies are proposed:

• Token-State Write (TSW): Updates per-token; highest granularity, sensitive to local changes.
• Sequence-State Write (SSW): Segment/message-level updates; robust to token-level noise.
• Multi-State Write (MSW): Multiple parallel memory matrices; reduces interference and allows multi-type memory separation.

Empirical Evaluation

$\delta$-mem is evaluated on a diverse suite of general and memory-intensive benchmarks: HotpotQA (multi-hop QA), GPQA-Diamond, IFEval (instruction following), LoCoMo (long-term conversational memory), and MemoryAgentBench (multi-turn agent memory). All experiments employ a frozen Qwen3-4B-Instruct or equivalent backbone; no full model finetuning or architecture replacement is required.

Main Results

• Average score improvement: With only an $8 \times 8$ memory state, $\delta$-mem increases the average benchmark performance to $\delta$0 that of the backbone and $\delta$1 that of the best non-$\delta$2-mem baseline.
• On MemoryAgentBench: Achieves $\delta$3 improvement.
• On LoCoMo: Achieves $\delta$4 improvement.
• TTL subtask of MemoryAgentBench: Near doubling of score from $\delta$5 to $\delta$6.
• General benchmarks: Maintains or improves performance, indicating no tradeoff between memory augmentation and base capability.

The model consistently outperforms textual, outside-channel, and parametric memory baselines on all tested backbones (Qwen3-4B/Instruct, Qwen3-8B, SmolLM3-3B), confirming that memory as a compact online state yields more robust and versatile improvements than context extension or static parameterization.

Compactness and Efficiency

$\delta$7-mem introduces negligible parameter overhead (4.87M additional params for standard variants, $\delta$8 of backbone; MSW at 19.47M). Inference memory usage is comparable to standard Prefix/LoRA-based adaptation and scales favorably with prompt length. Decoding throughput is only marginally reduced relative to vanilla inference, remaining substantially more efficient than large retrieval modules or parameter-heavy memory banks.

Figure 2: Decoding throughput of memory-augmented methods, showing that $\delta$9-mem minimally affects inference efficiency.

Figure 3: Trainable parameter overhead comparison; $\delta$0-mem is orders of magnitude more compact than parametric/MLP-memory-based alternatives.

Context Recovery and Ablations

In ablation studies, $\delta$1-mem demonstrates substantial capability to recover context-relevant information even when the explicit history is removed ("no-context" setting), outperforming baselines by wide margins on both HotpotQA and LoCoMo. Correction injection at both query and output branches of the attention yields optimal performance. Memory-injection at all layers provides the strongest results, while middle-layer insertion offers a performance-efficiency compromise.

Theoretical and Practical Implications

$\delta$2-mem demonstrates that effective, scalable, dynamic test-time memory can be achieved without explicit context extension, retrieval, or parameter-heavy side modules. By treating memory as an online, compact associative state directly steering attention, it bridges the gap between efficiency, dynamic adaptation, and model alignment. The design is fully compatible with frozen model deployment, minimizing both computational and software integration costs. The approach draws connections to classical neural associative memory, online regression, and low-rank adaptation, providing a unified perspective for future dynamic memory modules in LLMs.

Prospects for Future Research

Open research directions include scaling up the memory state size per application, integrating structured or hierarchical multi-state organization, differentiating memory policy based on input semantics or task, and composing $\delta$3-mem with retrieval or editing-based paradigms. More broadly, the framework suggests a blueprint for efficient, lifelong autonomy in agentic LLMs—supporting dynamic personalization, context recovery under hard memory limits, and stable long-horizon reasoning under real-world deployment constraints.

Conclusion

$\delta$4-mem represents a significant advance in lightweight, efficient, and dynamic online memory for LLMs. Leveraging a compact associative state and direct low-rank attention corrections, it consistently boosts memory-heavy and general benchmark performance over both frozen and baseline-augmented models, with minimal inference or parameter overhead. The architecture facilitates scalable, versatile, and efficient test-time memory augmentation—positioning it as a practical solution for the next generation of persistent, continually learning LLM-based AI systems.