Latent Personal Memory: Represent personal memory as dynamic soft prompts

Abstract: Personalizing LLMs requires encoding long-term, user-specific behavioral patterns in a way that is computationally efficient, scalable, and compatible with a frozen base model. We present Latent Personal Memory (LPM), a scalable framework that represents user-specific history as a compact, persistent matrix of N latent slots, that are interpretable. A shared cross-attention projection network maps these slots into dynamic, input-conditioned soft prompts that are prepended to the input of a frozen LLM. We evaluate LPM on PersonaMem v1 and LoCOMO benchmarks across Qwen3-1.7B, 4B, and 8B backbones. Results demonstrate that LPM outperforms LoRA and Prompt Tuning by up to 8.8% and 54.4% in overall accuracy respectively on PersonaMem v1, while reducing KV-cache usage by over 64x. On LoCoMo, LPM matches LoRA accuracy with 120x fewer trainable parameters. We also show that the efficiency of LPM grows with context length and outperforms full-context at 128K context length.

• The paper introduces a novel framework that compresses user histories into compact memory slots and dynamically generates soft prompts to personalize frozen LLMs.
• It leverages a shared projection network with entropy regularization to balance efficiency and accuracy across diverse personalization benchmarks.
• Experiments demonstrate significant gains with reduced KV-cache usage and latency, outperforming methods like LoRA and prompt tuning in scalability and performance.

Latent Personal Memory: Scalable LLM Personalization via Dynamic Soft Prompts

Motivation and Framework

The Latent Personal Memory (LPM) framework addresses critical constraints in LLM personalization: efficient long-term user memory storage, low computational overhead, scalability to large user populations, and compatibility with frozen LLM architectures. LPM represents user-specific histories as compact, persistent matrices of $N$ latent memory slots, with each slot serving as a learned, interpretable embedding that compresses user behavioral traces. A shared projection network performs input-conditioned cross-attention over these slots, dynamically retrieving soft prompts for each query, which are prepended to the input sequence of a frozen LLM. Unlike static prompt tuning or LoRA-based approaches, the LPM-generated soft prompts adaptively reflect both user-specific information and query context, addressing expressivity limitations inherent in prior parameter-efficient fine-tuning protocols.

Figure 1: Training and inference workflow for LPM, illustrating slot-based compression, projection network, and dynamic prompt retrieval.

Methodological Details

Each user’s latent memory is instantiated as a fixed-size matrix $M_u \in \mathbb{R}^{N \times d_\text{mem}}$. During training, user histories are segmented into context chunks, with memory slots updated via next-token prediction losses regularized by an entropy balancing term that prevents slot-attention collapse while preserving semantic concentration. The projection network, shared globally across the user pool, executes query-dependent cross-attention with input embeddings $\epsilon(x)$, producing $H$ multi-head soft prompts per query. The integration of an MLP and LayerNorm pipeline enhances prompt expressivity and stability. Model gradients with respect to projection parameters are accumulated over users and applied per epoch, while per-user slot matrices are updated iteratively to amortize computational cost.

LPM is trained exclusively from natural user histories without synthetic QA pairing or context distillation, which enables scalable deployment. Notably, all memory parameterizations remain local and user-specific, while the projection network admits federated learning.

Experimental Evaluation

LPM was evaluated on two long-context personalization benchmarks: PersonaMem v1 (multiple-choice user profiling task) and LoCOMO (generative QA over multi-session dialogue history). Benchmarks spanned Qwen3-1.7B, 4B, and 8B architectures. Baselines included full-context (entire user history prepended at inference), RAG, LightMem, LoRA, prompt tuning, and Embedding-to-Prefix methods. Metrics comprised categoric accuracy per task, KV-cache size, inference latency, peak GPU memory, and trainable parameter count.

LPM consistently outperformed LoRA (+8.8%) and prompt tuning (+54.4%) on PersonaMem v1, with strong gains in preference evolution, reasoning, and scenario generalization. Crucially, KV-cache usage was reduced by over 64$\times$ relative to full-context (3.65 GB to 56.73 MB), with 4$\times$ lower latency at comparable accuracy.

Scaling Behavior and Efficiency

LPM’s efficiency and scaling advantages are pronounced as context grows. On PersonaMem v1, LPM’s inference cost remains flat with context length, while full-context accuracy degrades substantially beyond 32K tokens; at 128K tokens, full-context trails LPM by 14.5%. LPM’s memory and latency profile is invariant to user history size, positioning it as an architecture uniquely suited for large-scale, real-world deployment where quadratic self-attention cost and context window limitations are major bottlenecks.

Figure 2: Comparative scaling behavior of full-context versus LPM on PersonaMem v1, demonstrating LPM's invariant efficiency as context grows from 32K to 128K tokens.

Cross-Backbone and Task Transfer

LPM’s superiority extends across model sizes (Qwen3-1.7B, 4B, 8B), with accuracy deltas most pronounced under constrained backbone capacity. On LoCOMO, LPM matches LoRA’s accuracy (33.25% vs. 33.1%) with 120$\times$ fewer trainable parameters (0.65M versus 77M) and KV-cache reduction from 2.9 GB to 15.3 MB. Ablation studies confirm that entropy regularization and increased memory head count are individually crucial for maintaining accuracy across question types, particularly on single-hop, multi-hop, and open-domain queries.

Interpretability of Memory Representations

Analysis of learned slot embeddings highlights LPM’s interpretability. UMAP projections demonstrate coherent, well-separated clusters per user, indicating that latent memory slots encode persistent behavioral signature beyond topical recall.

Figure 3: UMAP projection of 32 learned memory slots for PersonaMem users, with distinct clusters indicating user-specific behavioral encoding.

Qualitative rubric-based similarity studies align the geometric separation in embedding space with real behavioral differences judged by LLMs, substantiating that LPM learns rich, user-level interaction profiles rather than generic topical compression.

Preservation of General Reasoning

Despite intensive slot-based personalization, LPM preserves the general reasoning abilities of the base LLM, showing parity with core datasets such as GSM-8K and MMLU. This contrasts with LoRA, which incurs minor generalization degradation, as previously reported.

Failure Modes, Limitations, and Future Directions

LPM underperforms on exact factual recall tasks, reflecting inherent trade-offs in lossy latent compression. The dominant failure mode is loss of precise numerical or temporal information (e.g., dates, counts, strings), whereas high-level semantic content is robustly preserved. This suggests that hybrid memory architectures combining latent and text-based stores—potentially via explicit fact caches—are necessary for applications demanding high recall precision. Additional research avenues include hierarchical memory specialization (separating short-term and long-term heads) and integration with structured retrieval-augmented pipelines.

Implications for AI Research and Deployment

LPM’s approach to slot-based latent memory offers a rigorous paradigm for scalable LLM personalization with minimal inference cost, strong efficiency under growing context lengths, and interpretable user representations. By decoupling memory compression from backbone fine-tuning and leveraging federated training, LPM aligns with privacy-conscious deployments and robust generalization. Future developments may integrate slot-based latent stores with explicit retrieval or factual caches to jointly optimize expressivity and precision, advancing long-term adaptive AI systems.

Conclusion

LPM demonstrates significant improvements in task-specific personalization over prior methods, achieving superior accuracy and efficiency without compromising general LLM capabilities. Its slot-based architecture and projection network together define a scalable, interpretable, and robust protocol for personalized memory in LLMs, with implications for both practical deployment and research into memory representation and context adaptation.