NextMem: Latent Memory Framework

• NextMem is a latent factual memory framework that compactly encodes detailed observations using an autoregressive autoencoder on a causal language model.
• It employs a two-stage training process—autoregressive reconstruction alignment followed by progressive latent substitution—to ensure high reconstruction fidelity and extensibility.
• The framework integrates NF4 quantization to significantly reduce storage overhead while preserving robust retrieval and accurate decompression of factual data.

NextMem is a framework for latent factual memory in LLM-based agents that is designed to occupy a middle ground between storing memory as raw text and storing it in model parameters. Its central objective is to preserve detailed factual observations compactly, reconstruct them accurately when needed, and reuse the same latent representation for retrieval. The framework is built as an autoregressive autoencoder on top of a causal LLM, uses a two-stage training process consisting of autoregressive reconstruction alignment and progressive latent substitution, and adds NF4 quantization to reduce storage overhead. In the reported experiments, NextMem emphasizes reconstructability, retrieval quality, robustness, and extensibility rather than pure direct latent instruction following (Zhang et al., 26 Feb 2026).

1. Conceptual position and problem formulation

NextMem treats factual memory as foundational for agent memory because it is intended to preserve observed facts in a near-lossless manner rather than extracting only task-specific summaries. In the framework’s formulation, if $s^t$ is the current state or observation and $M^t$ is memory at time $t$, then memory writing is

$M^{t} = \mathrm{Encode}(s^t, M^{t-1}),$

and memory reading enters policy inference as

$$f(s^t) = \mathrm{LLM} \circ \mathrm{Decode}(s^t, M^{t}).$$

This formulation makes memory an explicit encoding–decoding process rather than a passive prompt extension or a permanent parameter update (Zhang et al., 26 Feb 2026).

The framework is motivated by the limitations of two dominant paradigms. Textual memory stores facts as text and retrieves them into context at inference time, which creates heavy context and indexing burdens as memory grows. Parametric memory encodes new information into model parameters, which avoids long textual contexts but introduces catastrophic forgetting, expensive updates, and difficulty in preserving many detailed observations exactly. NextMem is explicitly proposed as a latent alternative that aims to keep the compactness and computational convenience of latent representations while retaining the reconstructability required by factual memory (Zhang et al., 26 Feb 2026).

This positioning places NextMem within a broader shift from passive storage toward specialized memory substrates. A plausible implication is that it addresses a different systems tradeoff from architectures that emphasize anticipatory conversational triggers, autonomous memory policies, or state-conditioned compiled memory, such as T-Mem (Guo et al., 13 Jun 2026), Mem-T (Yue et al., 30 Jan 2026), and MemCompiler (Ding et al., 8 May 2026).

2. Latent representation and autoregressive autoencoding

Architecturally, NextMem is an autoregressive autoencoder built on a causal LLM. The main backbone used in experiments is Qwen3-8B, though the design is presented as generic. The model is decomposed into an embedding layer, Transformer decoder blocks, and a language modeling head. The encoder and decoder share the same architecture but use different parameter sets, $\theta_{\text{encode}}$ and $\theta_{\text{decode}}$; in implementation, they are realized with a shared backbone and separate LoRA adapters, so switching between encoder and decoder amounts to swapping adapters rather than loading two full models (Zhang et al., 26 Feb 2026).

The encoder takes an input sequence $s = [s_1, s_2, \dots, s_n]$, appends a special token $[SoD]$, and then generates a short sequence of latent vectors autoregressively. The resulting latent memory is

$$\mathbf{H}^{(l)} = [\mathbf{h}^1; \mathbf{h}^2; \dots; \mathbf{h}^l] \in \mathbb{R}^{l\times d}.$$

These latent vectors are described as being directly compatible with the LLM’s input embedding space. In the reported experiments, the final models use 15 latent tokens (Zhang et al., 26 Feb 2026).

Decoding is also autoregressive, but it begins from latent vectors rather than ordinary text tokens. A suffix prompt such as $M^t$0 is embedded, concatenated with the latent memory, and passed through the decoder to generate the original text. Because the latent vectors are inserted directly into the decoder input embedding stream, they function as learned memory slots compatible with the base LLM. The paper’s core claim is that this enables a memory item to exist neither as visible text nor as a permanent weight edit, but as a reversible latent state (Zhang et al., 26 Feb 2026).

The emphasis on reversibility is central. NextMem is not framed as an extraction or summarization system; it is framed as a memory system in which encoding and decoding should preserve original text faithfully enough that the memory can later be reconstructed. This distinguishes it from profile-style memory parameterization systems such as "AI-native Memory 2.0: Second Me" (Wei et al., 11 Mar 2025), where personalization is partly encoded into model weights rather than stored as reconstructable latent items.

3. Two-stage training and quantized storage

A key contribution of NextMem is its two-stage training pipeline, which is introduced because direct text-to-latent-to-text reconstruction proved unstable in earlier failed versions. The first stage, autoregressive reconstruction alignment, trains the decoder to reconstruct the original sequence in pure text space before the encoder is asked to summarize text into latent vectors. Concretely, for each sequence $M^t$1, the training pair is

$M^t$2

$M^t$3

The first $M^t$4 positions are ignored for loss, and the model learns to generate the second copy autoregressively after $M^t$5 (Zhang et al., 26 Feb 2026).

The second stage, progressive latent substitution, freezes the aligned decoder and trains the encoder to replace progressively larger source-text prefixes with latent vectors while the decoder reconstructs the missing content. At step $M^t$6, the encoder generates a $M^t$7-length latent prefix $M^t$8, the source is chunked into blocks of size $M^t$9, and the first $t$0 blocks are considered substituted by latent memory. The progression is essential: removing progressive latent substitution causes a severe collapse in reconstruction performance, and removing the progressive schedule also lowers performance substantially (Zhang et al., 26 Feb 2026).

The appendix specifies the concrete schedule: maximum input and output lengths of 1024, 15 progressive steps, block size $t$1, LoRA rank $t$2, $t$3, dropout 0.1, training for 3 epochs per step on 4 A100 GPUs with effective batch size 32 and learning rate $t$4. Since $t$5, the final latent sequence is trained to substitute roughly 240 source tokens (Zhang et al., 26 Feb 2026).

After continuous latent memory is learned, NextMem applies 4-bit NormalFloat (NF4) quantization with a fixed codebook rather than vector quantization with a learned codebook. The stored representation is a 4-bit index matrix plus a per-feature scale vector, with the scales stored in FP8 format. The paper reports that this yields substantial storage reduction with very small performance loss, and that more aggressive alternatives such as MoE-style gating, RQ-VAE, OMP, explicit projection, and reparameterization tricks failed to preserve adequate reconstruction quality or suffered instability (Zhang et al., 26 Feb 2026).

4. Memory use at inference time

At inference time, an agent writes memory by encoding a factual passage, dialogue turn, or evidence snippet into a latent matrix $t$6, optionally quantizing it, and storing that representation. Because each memory item is stored as a compact latent sequence rather than a full paragraph, storage is denser and context pressure is reduced. New memories can be appended incrementally without retraining the base model, which the paper presents as a practical advantage over parametric memory methods (Zhang et al., 26 Feb 2026).

For reading memory, the framework has two modes. In decompression mode, relevant latent memories are decoded back into text and then supplied to an LLM for answer generation; this is the mode that emphasizes high-fidelity recovery. In direct latent mode, latent representations are injected directly into the model’s input embedding stream for generation. The reported results show that direct latent use is not NextMem’s strongest regime relative to ICAE, whereas decompression is the regime in which NextMem is most competitive (Zhang et al., 26 Feb 2026).

For retrieval, the same latent representation is also used as an index. Document latent representations are pooled into 1D embeddings, queries are encoded into latent embeddings, cosine similarity is computed, and documents are ranked accordingly. This unifies storage and retrieval within a single representation family. The paper also reports a semantic assignment analysis with a diagonal perturbation pattern, in which changing source sentence $t$7 primarily affects $t$8-aligned latent regions. This suggests a structured latent layout and implies possible future support for localized editing, though explicit editing algorithms are not introduced (Zhang et al., 26 Feb 2026).

The paper’s notions of robustness and extensibility are operational rather than purely conceptual. Robustness is assessed by perturbing latent memories with Gaussian noise $t$9: performance remains stable for moderate noise levels up to about $M^{t} = \mathrm{Encode}(s^t, M^{t-1}),$0, while degradation becomes evident at larger perturbations. Extensibility refers to the ability to add new memories without retraining and to maintain graceful degradation beyond the training substitution length. In compression-ratio experiments, the model was trained to substitute 240 tokens yet retained high semantic integrity beyond that limit (Zhang et al., 26 Feb 2026).

5. Empirical performance

The experimental evaluation spans three task families: factual reconstruction, contextual generation, and dense passage retrieval. The datasets are SQuAD, HotpotQA, RACE, LoCoMo, and LongMemEval. Reconstruction is measured with F1, ROUGE-1, ROUGE-L, METEOR, BLEU, and BertScore; contextual generation uses LLM-as-a-Judge accuracy; retrieval is reported with Hit@5, Recall@5, MRR@5, MAP@5, DCG@5, and NDCG@5. The main baselines are DeepSeek-OCR, ICAE, and DyPRAG, with raw textual memory and BGE-M3 used for selected comparisons (Zhang et al., 26 Feb 2026).

On reconstruction, NextMem is strongest. On HotpotQA, NextMem-Dense reaches F1 0.9820, ROUGE-L 0.9854, METEOR 0.9820, BLEU 0.9633, and BertScore 0.9966, versus ICAE F1 0.7890. On SQuAD, NextMem-Dense reaches F1 0.8920 versus ICAE 0.7084. On LongMemEval, NextMem-Dense reaches F1 0.9436 and ROUGE-L 0.9555. On RACE and LoCoMo, the Sparse variant is very close to, and in some cases slightly better than, the Dense variant, which supports the claim that NF4 quantization preserves competitive performance (Zhang et al., 26 Feb 2026).

On contextual generation, the results expose a design tradeoff. In Compression (Comp.) mode, where the model answers directly from latent representations, ICAE is better. For example, on HotpotQA Comp., ICAE reaches 0.8565, whereas NextMem-Dense reaches 0.5179. In DeCompression (DeComp.) mode, however, NextMem becomes highly competitive. On HotpotQA DeComp., NextMem-Sparse reaches 0.8184 and Dense 0.8072, versus ICAE 0.8229. On SQuAD, NextMem-Sparse is best at 0.7630 versus ICAE 0.7066. On LongMemEval, NextMem-Sparse reaches 0.5486 versus ICAE 0.5029. The paper interprets this as evidence that NextMem is optimized more for accurate decompression than for direct latent instruction following (Zhang et al., 26 Feb 2026).

On dense passage retrieval, NextMem substantially outperforms reconstruction-oriented baselines while remaining weaker than specialized retrievers. On HotpotQA, NextMem-Dense reaches Hit@5 0.7245, Recall@5 0.3925, MRR@5 0.5194, and MAP@5 0.2793, versus ICAE Hit@5 0.4453 and MRR@5 0.3187. On LongMemEval, NextMem-Dense reaches Hit@5 0.8220, Recall@5 0.6805, and MRR@5 0.5445. BGE remains strongest as a retrieval-only model, but it cannot reconstruct text, which is precisely the dual capability NextMem is designed to preserve (Zhang et al., 26 Feb 2026).

The ablation study on RACE identifies the critical components. Full Dense reaches F1 0.8552 and ROUGE-L 0.8580. Removing $M^{t} = \mathrm{Encode}(s^t, M^{t-1}),$1 at inference lowers F1 to 0.3799. Removing progressive latent substitution training collapses F1 to 0.0159, making it the most critical ingredient. Removing the progressive schedule lowers F1 to 0.7389. For the Sparse model, removing scaling in quantization drops F1 to 0.0309, indicating that per-dimension scaling is essential (Zhang et al., 26 Feb 2026).

6. Relation to the memory-systems literature and limitations

Within the memory-systems literature, NextMem is best understood as a latent factual memory substrate rather than a full memory operating system. It differs from conversational systems centered on future-oriented associative triggers such as T-Mem (Guo et al., 13 Jun 2026), from benchmarks that stress multi-party temporal and profile-aware retrieval such as EverMemBench (Hu et al., 1 Feb 2026), from embodied state-conditioned memory delivery systems such as MemCompiler (Ding et al., 8 May 2026), and from autonomous memory-policy training systems such as Mem-T (Yue et al., 30 Jan 2026). It also differs from hybrid personalized systems such as "AI-native Memory 2.0: Second Me" (Wei et al., 11 Mar 2025), whose L2 layer compresses personal memory into model parameters rather than a reconstructable latent sequence. This suggests that NextMem’s primary contribution is representational: it proposes a compact, reversible, incrementally extensible latent store for factual observations (Zhang et al., 26 Feb 2026).

The framework’s limitations are explicit. First, there is a tradeoff between reconstruction fidelity and direct latent usability: NextMem is strongest when latent memory is decompressed first, not when it is used directly for answer generation. Second, capacity is finite, and performance declines with longer inputs; beyond the training substitution length, case studies show semantic blending, entity substitution, and occasional hallucination. Third, while NF4 quantization works well, more aggressive sparsification methods failed. Fourth, decompression introduces a latency step before downstream inference. Fifth, retrieval based on pooled latent representations and cosine similarity remains weaker than specialized retrieval models such as BGE. Sixth, training and evaluation are concentrated on paragraph and dialogue evidence benchmarks, so transfer beyond such factual text is not deeply characterized (Zhang et al., 26 Feb 2026).

The appendix also includes a forgetting analysis in which a decay coefficient $M^{t} = \mathrm{Encode}(s^t, M^{t-1}),$2 is used to mix the latent memory with an average embedding: $M^{t} = \mathrm{Encode}(s^t, M^{t-1}),$3 Performance remains stable for small $M^{t} = \mathrm{Encode}(s^t, M^{t-1}),$4 but then drops sharply, indicating that factual specificity disappears as the representation drifts toward an average latent state. This is not presented as a core feature, but it clarifies the paper’s view that latent factual memory must preserve structure against homogenization if it is to remain reconstructable and useful (Zhang et al., 26 Feb 2026).

In the current literature, NextMem therefore occupies a specific niche: it is neither a pure retriever, nor a parameter-editing method, nor a hand-engineered memory controller. It is a latent-memory framework in which the same compact representation supports storage, reconstruction, and retrieval, with the strongest empirical support appearing in regimes where factual memory must remain reversible, dense, and incrementally extensible (Zhang et al., 26 Feb 2026).