MemFT: LoRA-Based Exact Memory Fine-Tuning

• MemFT is a fine-tuning strategy that modifies LoRA optimization by focusing on tokens with sub-threshold losses to drive exact recall.
• It leverages the Parametric Memory Law by relating loss reduction to adapter rank and token sequence length, enhancing memorization efficiency.
• Variants like MemFT-OT and MemFT-SW dynamically reallocate gradient budget, demonstrating superior performance in both long-context and exact match benchmarks.

Searching arXiv for the cited MemFT paper and closely related memory-system papers to ground the article. MemFT is a memorization-oriented fine-tuning strategy proposed in "How LoRA Remembers? A Parametric Memory Law for LLM Finetuning" (Xu et al., 28 May 2026) to improve exact parametric memory in LLMs, especially when memory must be stored in a limited LoRA adapter and later reproduced verbatim under greedy decoding. It is derived from two claims established in that work: first, that LoRA-based memory updates follow a global power law relating loss reduction to adapter rank and sequence length; second, that exact recall is controlled locally by a token-level threshold phenomenon. MemFT does not replace LoRA. It changes how LoRA parameters are optimized so that scarce update capacity is spent on tokens that actually determine exact recall.

1. Problem formulation and conceptual scope

The paper studies a frozen base model $f_{\theta_0}$ augmented with trainable parameter increments $\Delta\theta$, implemented via LoRA, such that for every key-value pair $(\mathbf q^{(i)}, \mathbf a^{(i)})$ in a memorization dataset,

$$f_\theta(\mathbf q^{(i)}) = \mathbf a^{(i)}, \quad \forall i \in \{1,\dots,N\},$$

with $\theta = (\theta_0,\Delta\theta)$ (Xu et al., 28 May 2026). Because the answer $\mathbf a^{(i)}$ is unavailable at inference time except through the key $\mathbf q^{(i)}$, the learned parameter increment is the sole storage medium. This isolates parametric memory from retrieval-based memory.

A defining feature of the setting is the requirement of exact memory rather than gist-like semantic retention. The model must regenerate the answer sequence token for token. The paper argues that downstream QA benchmarks can conflate memorization with reasoning, instruction following, or semantic approximation, whereas exact recall exposes whether the adapter truly stores the target content (Xu et al., 28 May 2026).

LoRA is treated not merely as a PEFT method but as a controllable probe of memory capacity. For an adapted linear layer with frozen weight $W_0$, the model adds a low-rank residual,

$h = W_0 x + BAx,$

with $r$ as the LoRA rank. In this framing, rank is the budget knob, and the trainable branch functions as a compact parametric memory module embedded in the latent space. MemFT therefore sits directly on top of standard LoRA fine-tuning: it preserves the adapter architecture and redefines the optimization target.

2. Parametric Memory Law

The theoretical backdrop for MemFT is the paper’s macroscopic scaling result, the Parametric Memory Law. Let the sequence-level answer-only cross-entropy be

$\Delta\theta$0

where $\Delta\theta$1 is the answer length measured in answer tokens only. The memory gain from fine-tuning is defined as the loss reduction

$\Delta\theta$2

Across sweeps of LoRA rank and sequence length, the paper reports that

$\Delta\theta$3

fits the data robustly (Xu et al., 28 May 2026).

In this law, $\Delta\theta$4 is the LoRA rank, $\Delta\theta$5 is the answer-only token length, $\Delta\theta$6 is memory gain, $\Delta\theta$7 is a scaling constant determined by model architecture and data distribution, $\Delta\theta$8 is the capacity exponent, $\Delta\theta$9 is the length penalty exponent, and $(\mathbf q^{(i)}, \mathbf a^{(i)})$0 is an offset term. The paper often speaks of “effective parameters” in the sense that LoRA rank serves as a monotonic stand-in for the usable trainable capacity of the adapter.

The law is established empirically rather than from first principles. The authors sweep ranks and lengths on Qwen3-8B-IT and Llama3.1-8B-IT in two benchmark families: a long-context memorization stress test based on LongBench samples with varying fractions of random-token replacement, and a short-context dense memory benchmark based on PhoneBook key-value pairs. In log-log coordinates, they observe approximate linearity and fit the law with $(\mathbf q^{(i)}, \mathbf a^{(i)})$1 in all settings, with values up to $(\mathbf q^{(i)}, \mathbf a^{(i)})$2, and low MAPE. The law remains robust across semantic-to-random content mixtures and across long- and short-context regimes (Xu et al., 28 May 2026).

This global result supplies the resource-allocation motivation for MemFT. If memory gain scales predictably with limited adapter capacity and deteriorates with sequence length, then optimization efficiency becomes decisive in budget-constrained regimes.

3. Deterministic recall threshold and token-level phase transition

The paper argues that the macroscopic law is not sufficient to explain recall fidelity. Average cross-entropy can become small while exact-match accuracy remains poor, because autoregressive generation is brittle: a small number of stubborn positions can corrupt the entire continuation (Xu et al., 28 May 2026).

Under greedy decoding, the generated token at position $(\mathbf q^{(i)}, \mathbf a^{(i)})$3 is

$(\mathbf q^{(i)}, \mathbf a^{(i)})$4

A sufficient condition for the target token $(\mathbf q^{(i)}, \mathbf a^{(i)})$5 to be selected is that its probability exceed one half: $(\mathbf q^{(i)}, \mathbf a^{(i)})$6 Equivalently, since token loss is $(\mathbf q^{(i)}, \mathbf a^{(i)})$7, the critical loss threshold is

$(\mathbf q^{(i)}, \mathbf a^{(i)})$8

The paper frames this as a deterministic phase transition. In the disordered phase, $(\mathbf q^{(i)}, \mathbf a^{(i)})$9, equivalently $$f_\theta(\mathbf q^{(i)}) = \mathbf a^{(i)}, \quad \forall i \in \{1,\dots,N\},$$0; the correct token does not hold majority mass and may be overtaken by a competing token. In the ordered phase, $$f_\theta(\mathbf q^{(i)}) = \mathbf a^{(i)}, \quad \forall i \in \{1,\dots,N\},$$1, equivalently $$f_\theta(\mathbf q^{(i)}) = \mathbf a^{(i)}, \quad \forall i \in \{1,\dots,N\},$$2; the target token necessarily has the largest probability among all vocabulary items, so greedy decoding will pick it (Xu et al., 28 May 2026).

The sufficiency argument is purely probabilistic: since the distribution over the vocabulary sums to $$f_\theta(\mathbf q^{(i)}) = \mathbf a^{(i)}, \quad \forall i \in \{1,\dots,N\},$$3, no other token can equal or exceed a token whose probability is greater than $$f_\theta(\mathbf q^{(i)}) = \mathbf a^{(i)}, \quad \forall i \in \{1,\dots,N\},$$4. The paper supports this threshold view empirically by showing that the earliest stubborn position tightly bounds the first free-running decoding failure. Specifically, the earliest stubborn position and the first failure index have Spearman correlation $$f_\theta(\mathbf q^{(i)}) = \mathbf a^{(i)}, \quad \forall i \in \{1,\dots,N\},$$5 over $$f_\theta(\mathbf q^{(i)}) = \mathbf a^{(i)}, \quad \forall i \in \{1,\dots,N\},$$6 settings. The authors also report strong localization, with certain positions accounting for a large fraction of failures (Xu et al., 28 May 2026).

This threshold phenomenon is the direct precursor to MemFT. Standard supervised fine-tuning allocates loss uniformly across all target tokens, including tokens already safely in the ordered phase. MemFT instead treats sub-threshold tokens as the true bottlenecks for verbatim recall.

4. Objective design and algorithmic variants

The generic MemFT objective is a normalized weighted token loss: $$f_\theta(\mathbf q^{(i)}) = \mathbf a^{(i)}, \quad \forall i \in \{1,\dots,N\},$$7 where $$f_\theta(\mathbf q^{(i)}) = \mathbf a^{(i)}, \quad \forall i \in \{1,\dots,N\},$$8 is the set of target token positions, $$f_\theta(\mathbf q^{(i)}) = \mathbf a^{(i)}, \quad \forall i \in \{1,\dots,N\},$$9 is the token-level cross-entropy, $\theta = (\theta_0,\Delta\theta)$0 is a dynamic weight, and $\theta = (\theta_0,\Delta\theta)$1 is a small constant for numerical stability (Xu et al., 28 May 2026). The normalization is important because the number of emphasized tokens varies across samples and over time.

The simplest instantiation is MemFT-OT, “Only Threshold,” which uses the hard mask

$\theta = (\theta_0,\Delta\theta)$2

Using $\theta = (\theta_0,\Delta\theta)$3, only tokens whose target probability is still below $\theta = (\theta_0,\Delta\theta)$4 contribute to the loss. Tokens already in the ordered phase are masked out. MemFT-OT therefore reallocates gradient budget away from already-mastered positions and toward unresolved bottlenecks, with no additional hyperparameters beyond the fixed theoretical threshold (Xu et al., 28 May 2026).

The more advanced variant is MemFT-SW, which adds adaptive sliding mechanisms. Its soft threshold uses

$\theta = (\theta_0,\Delta\theta)$5

and its intra-sample spatial sliding introduces

$\theta = (\theta_0,\Delta\theta)$6

followed by a sequence weight that emphasizes positions near the first greedy decoding error anchor $\theta = (\theta_0,\Delta\theta)$7 and downweights far downstream tokens outside an active window $\theta = (\theta_0,\Delta\theta)$8 (Xu et al., 28 May 2026). The intended effect is to focus local supervision near the earliest failure point, since later-token losses become less informative once the prefix is already corrupted.

MemFT-SW also includes an inter-batch temporal curriculum. Let training progress be $\theta = (\theta_0,\Delta\theta)$9. Early in training, only a prefix subset of batches $\mathbf a^{(i)}$0 is exposed; as $\mathbf a^{(i)}$1 increases, the visible batch window expands until the full dataset is used. In the appendix, the paper specifies exposure ratios

$\mathbf a^{(i)}$2

This is intended to prevent the model from being overwhelmed by globally difficult sequences before easier local memorization patterns stabilize (Xu et al., 28 May 2026).

Operationally, MemFT proceeds by computing token-level losses, determining which positions remain above the deterministic threshold, constructing weights by either the hard-threshold or sliding-window rule, and optimizing the normalized weighted objective. As training progresses and tokens cross below $\mathbf a^{(i)}$3, they naturally stop receiving budget in OT or receive reduced budget in SW. The optimization focus therefore shifts over time toward the remaining sub-threshold bottlenecks.

5. Experimental regimes, metrics, and reported performance

The empirical evaluation uses two controlled exact-memory benchmarks. The first is the Long-Context Memorization Stress Test, built from LongBench samples with 0% to 100% of tokens replaced by random Qwen-vocabulary tokens. The fully random setting is emphasized as a probe of pure parametric storage. The second is PhoneBook, adapted from prior parametric memorization work: each sample is a name-to-phone-number key-value pair with the context field removed so that the task is pure parameter storage rather than in-context lookup (Xu et al., 28 May 2026).

The base models are Qwen3-8B-IT and Llama3.1-8B-IT, both at the 8B scale, with the full base model frozen and only LoRA adapters trained. For the long-context stress test, LoRA is applied only to the MLP down_proj module. For PhoneBook, LoRA is applied once to the entire MLP block. Layer choices are: Qwen3-8B-Instruct layers 20 and 24 for long-context and layer 24 for PhoneBook; Llama3.1-8B-Instruct layers 18 and 20 for long-context and layer 18 for PhoneBook (Xu et al., 28 May 2026).

The paper reports answer-only metrics. For long-context tests, it uses token-level accuracy,

$\mathbf a^{(i)}$4

For PhoneBook, it uses exact match accuracy,

$\mathbf a^{(i)}$5

Long-context averages use length buckets

$\mathbf a^{(i)}$6

and PhoneBook averages use answer-only lengths

$\mathbf a^{(i)}$7

(Xu et al., 28 May 2026).

On the Long-Context Memorization Stress Test, the gain is reported as especially strong under limited rank. For Llama3.1-8B-IT, SFT token accuracy rises from 27.4 to 94.7 across ranks $\mathbf a^{(i)}$8 to $\mathbf a^{(i)}$9, while MemFT-OT reaches 100.0 at $\mathbf q^{(i)}$0, and MemFT-SW is strongest at low ranks, including 32.5 vs 27.4 at $\mathbf q^{(i)}$1 and 37.5 vs 28.5 at $\mathbf q^{(i)}$2. For Qwen3-8B-IT, SFT ranges from 17.9 to 47.7, whereas MemFT-SW reaches 74.6 at $\mathbf q^{(i)}$3 and 93.5 at $\mathbf q^{(i)}$4, and MemFT-OT reaches 100.0 at $\mathbf q^{(i)}$5 and $\mathbf q^{(i)}$6 (Xu et al., 28 May 2026).

The paper explicitly identifies a capacity-dependent regime shift. At low rank, MemFT-SW tends to be better because the sliding/window and curriculum mechanisms prioritize the most critical local bottlenecks. As rank grows, MemFT-OT accelerates more sharply and can surpass SW, apparently because once some capacity is available, the clean hard-threshold objective is very effective at driving the remaining tokens across the recall boundary (Xu et al., 28 May 2026).

On PhoneBook, MemFT-SW is generally the strongest method across budgets. For Llama3.1-8B-IT, SFT exact match goes from 0.50 to 59.3 over $\mathbf q^{(i)}$7 to $\mathbf q^{(i)}$8, while MemFT-SW rises from 1.84 to 100.0 and reaches perfect recall by $\mathbf q^{(i)}$9; MemFT-OT reaches 87.0 at $W_0$0. For Qwen3-8B-IT, SFT is already strong at larger ranks, but MemFT-SW still dominates in low-rank settings, including 8.45 vs 2.32 at $W_0$1 and 19.7 vs 17.4 at $W_0$2, while maintaining parity at high ranks (Xu et al., 28 May 2026).

A supporting “Linear Rule Learning” benchmark tests whether threshold-guided training harms broader learning. There MemFT improves both memory and generalization accuracy over SFT at most ranks, with generalization gains of $W_0$3 to $W_0$4. The paper treats that result as preliminary but consistent with the idea that focusing on hard tokens can reduce overconfidence on easy samples (Xu et al., 28 May 2026).

6. Interpretation, limitations, and relation to adjacent memory research

MemFT suggests a specific interpretation of LoRA capacity limits. Exact textual memory is constrained not only by average loss reduction but also by the need to move every critical token past a discrete dominance threshold. The Parametric Memory Law quantifies aggregate loss reduction as a function of adapter rank and sequence length; the token-level phase transition explains why aggregate loss reduction may still fail to yield exact recall. MemFT combines these perspectives by reallocating limited LoRA capacity toward tokens that block deterministic greedy recovery (Xu et al., 28 May 2026).

The scope of the method is clearly bounded. The study is limited to 8B-scale models, so the universality of the law and of MemFT’s gains at other scales is not established. The critical threshold $W_0$5 is specific to greedy decoding; under top-$W_0$6 or temperature sampling it is no longer the operative recall criterion. The work is fundamentally about exact memorization rather than broad factual updating, reasoning, or open-ended knowledge generalization. The paper is also explicit that MemFT may not help much once the task is saturated or when failures are not due to sub-threshold bottleneck tokens (Xu et al., 28 May 2026).

Adjacent arXiv work clarifies the boundaries of the concept. "MemFail: Stress-Testing Failure Modes of LLM Memory Systems" formalizes memory systems as summarization, storage, and retrieval, and argues that end-to-end QA accuracy hides whether a failure originated in write-time compression, persistence, access, or downstream reasoning (Garg et al., 26 May 2026). This suggests a complementary evaluation lens for memory-focused methods, although MemFail is a diagnostic benchmark rather than a fine-tuning strategy. By contrast, "FedMef: Towards Memory-efficient Federated Dynamic Pruning" is a memory-efficient federated dynamic pruning framework for training from scratch or full training in cross-device FL, not primarily a federated fine-tuning method, and its focus is client-side training memory rather than exact parametric recall in LoRA adapters (Huang et al., 2024).

Within that landscape, MemFT is most precisely characterized as a threshold-aware reweighting scheme for LoRA fine-tuning. Its central contribution is to convert a descriptive memory-scaling law and a deterministic token-level threshold into an optimization rule: budget should be spent on unresolved positions, not uniformly across all targets. The reported consequence is better exact memory fidelity and parameter efficiency, especially in low-budget, long-sequence, and hard-memorization regimes (Xu et al., 28 May 2026).