How LoRA Remembers? A Parametric Memory Law for LLM Finetuning

Abstract: LLMs must continuously learn and update knowledge to remain effective in dynamic real-world environments. While Low-Rank Adaptation (LoRA) is widely used for such memory updates, existing studies mainly rely on qualitative downstream evaluations, leaving the quantitative capacity limits and underlying dynamics of exact parametric memory largely unexplored. To bridge this gap, we employ LoRA as a controlled memory capacity probe within the latent space to systematically quantify exact parametric memory. We introduce the Parametric Memory Law, a robust power law linking loss reduction Delta L to effective parameters and sequence length. At the token level, fine-grained analysis reveals a deterministic phase transition, demonstrating that a prediction probability of p > 0.5 constitutes a sufficient condition for verbatim recall under greedy decoding. Driven by these insights, we introduce MemFT, a threshold-guided optimization strategy that dynamically redistributes the training budget toward sub-threshold tokens. Empirical evaluations demonstrate that MemFT can enhance memory fidelity and efficiency. Code will be released at https://github.com/zjunlp/ParametricMemoryLaw.

• The paper introduces a power-law-based Parametric Memory Law that quantifies the reduction in cross-entropy loss due to LoRA adapters.
• It reveals deterministic token-level phase transitions that identify critical bottlenecks affecting exact recall in LLMs.
• MemFT, a threshold-guided fine-tuning protocol, outperforms standard SFT in both long-context and key-value mapping benchmarks.

Parametric Memory in LLMs: Quantifying and Optimizing LoRA Memory

Background and Motivation

LLMs depend on both non-parametric and parametric memory mechanisms to achieve persistent knowledge acquisition and retrieval. Non-parametric systems, such as Retrieval-Augmented Generation (RAG) and In-Context Learning (ICL), are fundamentally limited by context-window capacity, attention dilution, and growing computational overhead with input length [DBLP:journals/tacl/LiuLHPBPL24]. In contrast, parametric memory, notably via Low-Rank Adaptation (LoRA), enables permanent information embedding in model parameters, facilitating efficient and context-free recall.

The paper "How LoRA Remembers? A Parametric Memory Law for LLM Finetuning" (2605.30260) systematically investigates the parametric memory capacity with LoRA, introducing the Parametric Memory Law (PML) and exposing token-level deterministic phase transitions governing exact recall. It further proposes MemFT, a threshold-guided fine-tuning protocol to maximize memory efficiency and fidelity.

Figure 1: LoRA as a pluggable memory unit in the LLM's latent space, enabling explicit memory capacity quantification.

The Parametric Memory Law: Empirical Characterization

PML quantifies the exact memory gain from LoRA adapters as a power-law function:

$$\Delta \mathcal{L}(r, \ell) = C \cdot r^{\alpha} \cdot \ell^{-\beta} + b$$

where $\Delta \mathcal{L}$ is the reduction in cross-entropy loss, $r$ denotes LoRA rank (parameter budget), $\ell$ is sequence length, and $C, \alpha, \beta, b$ are fit parameters reflecting base-model and data characteristics. The law is validated through extensive sweeps on Qwen3-8B-IT and Llama3.1-8B-IT using both long-context synthetic benchmarks and dense PhoneBook key-value mapping tests.

Figure 2: Approximate log-linear dependence of $\Delta \mathcal{L}$ on rank $r$ and length $\ell$, confirming power-law scaling.

Empirical fits attain $R^2 > 0.98$ with low MAPE across semantic regimes and sequence lengths, supporting that parametric memory scaling is robust to both context structure and distributional difficulty.

Token-Level Dynamics and Deterministic Phase Transition

Global loss is insufficient to guarantee exact recall. Fine-grained analysis reveals a deterministic phase transition at $p > 0.5$ per token under greedy decoding. This threshold aligns with $\Delta \mathcal{L}$0, demarcating the guaranteed recall domain versus stochastic failure.

Sparse stubborn token positions resist improvement irrespective of rank expansion, and these bottlenecks tightly bound the observed sequence failures. Once a token's probability drops below $\Delta \mathcal{L}$1, autoregressive generation is prone to cascade collapse, rendering verbatim reconstruction unreliable.

Figure 3: Probability dynamics highlight localized “stubborn” tokens, forming bottlenecks in recall.

Figure 4: Per-position probability grid exhibiting critical bottlenecks that persist across high LoRA ranks.

MemFT: Threshold-Guided Fine-Tuning

MemFT explicitly targets these token-level bottlenecks by diverting optimization budget from “mastered” tokens to those remaining in the uncertain regime ($\Delta \mathcal{L}$2). Two variants are introduced: MemFT-OT (only-threshold, hard masking) and MemFT-SW (adaptive spatial and temporal sliding), focusing gradients on first-failure anchors and dynamically exposing complex sequences.

MemFT consistently outperforms standard SFT (token-averaged training) across both long-context and PhoneBook regimes, accelerating memory saturation and exact match rates under constrained parameter budgets.

Figure 5: Exact-match accuracy of Qwen3-8B under the Long-Context Memorization Stress Test, showing MemFT’s superiority.

Figure 6: Exact-match accuracy on the PhoneBook benchmark for both Qwen3-8B and Llama3.1-8B, demonstrating MemFT's parameter-efficient recall.

Implications and Analysis

Parametric memory law elucidates rigorous capacity-parameter trade-offs in LLMs for faithful memorization. MemFT provides a scalable, adaptive fine-tuning protocol crucial for domains requiring exact recall: personal credentials, compliance citations, medical codes, model watermarking, configuration endpoints, and security secrets.

The token-level phase transition analysis underscores the need for mechanistic memory inspection in downstream QA, editing, and persistent learning tasks, supplanting unsystematic reliance on average loss metrics.

MemFT's optimization not only improves exact memory but also enhances out-of-domain generalization, as verified in rule-learning settings. This suggests efficient memory protocols may mitigate overfitting and improve robustness.

Limitations and Future Directions

The study is restricted to 8B-scale models, and extrapolation to larger architectures requires validation. The deterministic threshold is specific to greedy decoding; stochastic decoding schemes necessitate further theoretical exploration. Broader trade-offs between parametric memory and open-ended reasoning remain an open question.

Practical deployment calls for further investigation of memory law scaling, domain adaptation, and hybrid memory integration with external systems, particularly as modular LoRA libraries become common [DBLP:conf/icml/OstapenkoSPCRCS24]. Integrating MemFT with other adapter-based memory strategies and testing on real-world continual learning workloads constitutes a promising research trajectory.

Conclusion

This paper provides a quantitative, mechanistic foundation for exact parametric memory in LLMs using LoRA adapters. The established Parametric Memory Law enables precise capacity planning; the deterministic phase transition analysis guides the design of robust fine-tuning strategies; MemFT translates these principles into actionable optimization. Together, these advances inform both theoretical understanding and practical development of persistent, high-fidelity memory in next-generation LLM systems.