---
title: 'MemFT: LoRA-Based Exact Memory Fine-Tuning'
url: https://www.emergentmind.com/topics/memft
type: topic
---

# MemFT: LoRA-Based Exact Memory Fine-Tuning

Searching arXiv for the cited MemFT paper and closely related memory-system papers to ground the article.
MemFT is a memorization-oriented fine-tuning strategy proposed in "How LoRA Remembers? A Parametric Memory Law for LLM Finetuning" [2605.30260] to improve exact parametric memory in large language models, especially when memory must be stored in a limited LoRA adapter and later reproduced verbatim under greedy decoding. It is derived from two claims established in that work: first, that LoRA-based memory updates follow a global power law relating loss reduction to adapter rank and sequence length; second, that exact recall is controlled locally by a token-level threshold phenomenon. MemFT does not replace LoRA. It changes how LoRA parameters are optimized so that scarce update capacity is spent on tokens that actually determine exact recall.

## 1. Problem formulation and conceptual scope

The paper studies a frozen base model \(f_{\theta_0}\) augmented with trainable parameter increments \(\Delta\theta\), implemented via LoRA, such that for every key-value pair \((\mathbf q^{(i)}, \mathbf a^{(i)})\) in a memorization dataset,
\[
f_\theta(\mathbf q^{(i)}) = \mathbf a^{(i)}, \quad \forall i \in \{1,\dots,N\},
\]
with \(\theta = (\theta_0,\Delta\theta)\) [2605.30260]. Because the answer \(\mathbf a^{(i)}\) is unavailable at inference time except through the key \(\mathbf q^{(i)}\), the learned parameter increment is the sole storage medium. This isolates parametric memory from retrieval-based memory.

A defining feature of the setting is the requirement of exact memory rather than gist-like semantic retention. The model must regenerate the answer sequence token for token. The paper argues that downstream QA benchmarks can conflate memorization with reasoning, instruction following, or semantic approximation, whereas exact recall exposes whether the adapter truly stores the target content [2605.30260].

LoRA is treated not merely as a PEFT method but as a controllable probe of memory capacity. For an adapted linear layer with frozen weight \(W_0\), the model adds a low-rank residual,
\[
h = W_0 x + BAx,
\]
with \(r\) as the LoRA rank. In this framing, rank is the budget knob, and the trainable branch functions as a compact parametric memory module embedded in the latent space. MemFT therefore sits directly on top of standard LoRA fine-tuning: it preserves the adapter architecture and redefines the optimization target.

## 2. Parametric Memory Law

The theoretical backdrop for MemFT is the paper’s macroscopic scaling result, the Parametric Memory Law. Let the sequence-level answer-only cross-entropy be
\[
\mathcal L(\mathbf a;\theta) = \frac{1}{\ell}\sum_{t=1}^{\ell}\mathcal L_t(\theta), \qquad
\mathcal L_t(\theta) = -\log p_\theta(a_t \mid \mathbf q, a_{<t}),
\]
where \(\ell\) is the answer length measured in answer tokens only. The memory gain from fine-tuning is defined as the loss reduction
\[
\Delta \mathcal L = \mathcal L_{\text{init}} - \mathcal L_{\text{final}}.
\]
Across sweeps of LoRA rank and sequence length, the paper reports that
\[
\Delta \mathcal L(r,\ell) = C \cdot r^\alpha \cdot \ell^{-\beta} + b
\]
fits the data robustly [2605.30260].

In this law, \(r\) is the LoRA rank, \(\ell\) is the answer-only token length, \(\Delta \mathcal L\) is memory gain, \(C\) is a scaling constant determined by model architecture and data distribution, \(\alpha>0\) is the capacity exponent, \(\beta>0\) is the length penalty exponent, and \(b\) is an offset term. The paper often speaks of “effective parameters” in the sense that LoRA rank serves as a monotonic stand-in for the usable trainable capacity of the adapter.

The law is established empirically rather than from first principles. The authors sweep ranks and lengths on Qwen3-8B-IT and Llama3.1-8B-IT in two benchmark families: a long-context memorization stress test based on LongBench samples with varying fractions of random-token replacement, and a short-context dense memory benchmark based on PhoneBook key-value pairs. In log-log coordinates, they observe approximate linearity and fit the law with \(R^2 > 0.98\) in all settings, with values up to \(0.996\), and low MAPE. The law remains robust across semantic-to-random content mixtures and across long- and short-context regimes [2605.30260].

This global result supplies the resource-allocation motivation for MemFT. If memory gain scales predictably with limited adapter capacity and deteriorates with sequence length, then optimization efficiency becomes decisive in budget-constrained regimes.

## 3. Deterministic recall threshold and token-level phase transition

The paper argues that the macroscopic law is not sufficient to explain recall fidelity. Average cross-entropy can become small while exact-match accuracy remains poor, because autoregressive generation is brittle: a small number of stubborn positions can corrupt the entire continuation [2605.30260].

Under greedy decoding, the generated token at position \(t\) is
\[
\hat a_t = \arg\max_{v\in\mathcal V} p_\theta(v \mid \mathbf q, a_{<t}).
\]
A sufficient condition for the target token \(a_t\) to be selected is that its probability exceed one half:
\[
P_{\text{target}} > 0.5.
\]
Equivalently, since token loss is \(\mathcal L_t = -\log P_{\text{target}}\), the critical loss threshold is
\[
\mathcal L_{\text{crit}} = -\log(0.5) = \ln 2 \approx 0.693.
\]

The paper frames this as a deterministic phase transition. In the disordered phase, \(\mathcal L_t > \mathcal L_{\text{crit}}\), equivalently \(P_{\text{target}} < 0.5\); the correct token does not hold majority mass and may be overtaken by a competing token. In the ordered phase, \(\mathcal L_t < \mathcal L_{\text{crit}}\), equivalently \(P_{\text{target}} > 0.5\); the target token necessarily has the largest probability among all vocabulary items, so greedy decoding will pick it [2605.30260].

The sufficiency argument is purely probabilistic: since the distribution over the vocabulary sums to \(1\), no other token can equal or exceed a token whose probability is greater than \(0.5\). The paper supports this threshold view empirically by showing that the earliest stubborn position tightly bounds the first free-running decoding failure. Specifically, the earliest stubborn position and the first failure index have Spearman correlation \(\rho = 0.908\) over \(n=155\) settings. The authors also report strong localization, with certain positions accounting for a large fraction of failures [2605.30260].

This threshold phenomenon is the direct precursor to MemFT. Standard supervised fine-tuning allocates loss uniformly across all target tokens, including tokens already safely in the ordered phase. MemFT instead treats sub-threshold tokens as the true bottlenecks for verbatim recall.

## 4. Objective design and algorithmic variants

The generic MemFT objective is a normalized weighted token loss:
\[
\mathcal L_{\text{MemFT}}(\theta) =
\frac{\sum_{t\in\mathcal M} w_t\,\mathcal L_t(\theta)}
{\sum_{t\in\mathcal M} w_t + \varepsilon},
\]
where \(\mathcal M\) is the set of target token positions, \(\mathcal L_t(\theta)\) is the token-level cross-entropy, \(w_t\) is a dynamic weight, and \(\varepsilon\) is a small constant for numerical stability [2605.30260]. The normalization is important because the number of emphasized tokens varies across samples and over time.

The simplest instantiation is MemFT-OT, “Only Threshold,” which uses the hard mask
\[
w_t^{\text{TH}} = \mathbf 1\!\left[\mathcal L_t > \mathcal L_{\text{crit}}\right].
\]
Using \(\mathcal L_{\text{crit}}=\ln 2\), only tokens whose target probability is still below \(0.5\) contribute to the loss. Tokens already in the ordered phase are masked out. MemFT-OT therefore reallocates gradient budget away from already-mastered positions and toward unresolved bottlenecks, with no additional hyperparameters beyond the fixed theoretical threshold [2605.30260].

The more advanced variant is MemFT-SW, which adds adaptive sliding mechanisms. Its soft threshold uses
\[
w_t^{\text{base}} = \sigma\!\left(\kappa(\mathcal L_t - \mathcal L_{\text{crit}})\right),
\]
and its intra-sample spatial sliding introduces
\[
\phi_t = \exp(-\max(t-a_i,0)/\tau),
\]
followed by a sequence weight that emphasizes positions near the first greedy decoding error anchor \(a_i\) and downweights far downstream tokens outside an active window \(L_{\text{win}}\) [2605.30260]. The intended effect is to focus local supervision near the earliest failure point, since later-token losses become less informative once the prefix is already corrupted.

MemFT-SW also includes an inter-batch temporal curriculum. Let training progress be \(\gamma\in[0,1]\). Early in training, only a prefix subset of batches \(\mathcal B_{\text{cur}}\) is exposed; as \(\gamma\) increases, the visible batch window expands until the full dataset is used. In the appendix, the paper specifies exposure ratios
\[
[0.2,\,0.4,\,0.6,\,0.8,\,1.0].
\]
This is intended to prevent the model from being overwhelmed by globally difficult sequences before easier local memorization patterns stabilize [2605.30260].

Operationally, MemFT proceeds by computing token-level losses, determining which positions remain above the deterministic threshold, constructing weights by either the hard-threshold or sliding-window rule, and optimizing the normalized weighted objective. As training progresses and tokens cross below \(\ln 2\), they naturally stop receiving budget in OT or receive reduced budget in SW. The optimization focus therefore shifts over time toward the remaining sub-threshold bottlenecks.

## 5. Experimental regimes, metrics, and reported performance

The empirical evaluation uses two controlled exact-memory benchmarks. The first is the Long-Context Memorization Stress Test, built from LongBench samples with 0% to 100% of tokens replaced by random Qwen-vocabulary tokens. The fully random setting is emphasized as a probe of pure parametric storage. The second is PhoneBook, adapted from prior parametric memorization work: each sample is a name-to-phone-number key-value pair with the context field removed so that the task is pure parameter storage rather than in-context lookup [2605.30260].

The base models are Qwen3-8B-IT and Llama3.1-8B-IT, both at the 8B scale, with the full base model frozen and only LoRA adapters trained. For the long-context stress test, LoRA is applied only to the MLP `down_proj` module. For PhoneBook, LoRA is applied once to the entire MLP block. Layer choices are: Qwen3-8B-Instruct layers 20 and 24 for long-context and layer 24 for PhoneBook; Llama3.1-8B-Instruct layers 18 and 20 for long-context and layer 18 for PhoneBook [2605.30260].

The paper reports answer-only metrics. For long-context tests, it uses token-level accuracy,
\[
\mathrm{Acc}_{\mathrm{tok}}(\mathbf a;\theta) = \frac{1}{\ell}\sum_{t=1}^{\ell}\mathbf 1[\hat a_t=a_t].
\]
For PhoneBook, it uses exact match accuracy,
\[
\mathrm{Acc}_{\mathrm{EM}}(\mathbf a;\theta) = \mathbf 1[\forall t,\ \hat a_t=a_t].
\]
Long-context averages use length buckets
\[
\{50,100,200,500,1000,2000,3000,4000,5000,6000,7000,8000,10000\},
\]
and PhoneBook averages use answer-only lengths
\[
\{1\mathrm{k},2\mathrm{k},4\mathrm{k},8\mathrm{k},12\mathrm{k},16\mathrm{k},24\mathrm{k},32\mathrm{k}\}
\]
[2605.30260].

On the Long-Context Memorization Stress Test, the gain is reported as especially strong under limited rank. For Llama3.1-8B-IT, SFT token accuracy rises from 27.4 to 94.7 across ranks \(r_1\) to \(r_9\), while MemFT-OT reaches 100.0 at \(r_9\), and MemFT-SW is strongest at low ranks, including 32.5 vs 27.4 at \(r_1\) and 37.5 vs 28.5 at \(r_2\). For Qwen3-8B-IT, SFT ranges from 17.9 to 47.7, whereas MemFT-SW reaches 74.6 at \(r_6\) and 93.5 at \(r_7\), and MemFT-OT reaches 100.0 at \(r_8\) and \(r_9\) [2605.30260].

The paper explicitly identifies a capacity-dependent regime shift. At low rank, MemFT-SW tends to be better because the sliding/window and curriculum mechanisms prioritize the most critical local bottlenecks. As rank grows, MemFT-OT accelerates more sharply and can surpass SW, apparently because once some capacity is available, the clean hard-threshold objective is very effective at driving the remaining tokens across the recall boundary [2605.30260].

On PhoneBook, MemFT-SW is generally the strongest method across budgets. For Llama3.1-8B-IT, SFT exact match goes from 0.50 to 59.3 over \(p_1\) to \(p_7\), while MemFT-SW rises from 1.84 to 100.0 and reaches perfect recall by \(p_7\); MemFT-OT reaches 87.0 at \(p_7\). For Qwen3-8B-IT, SFT is already strong at larger ranks, but MemFT-SW still dominates in low-rank settings, including 8.45 vs 2.32 at \(p_1\) and 19.7 vs 17.4 at \(p_2\), while maintaining parity at high ranks [2605.30260].

A supporting “Linear Rule Learning” benchmark tests whether threshold-guided training harms broader learning. There MemFT improves both memory and generalization accuracy over SFT at most ranks, with generalization gains of \(7\%\) to \(15\%\). The paper treats that result as preliminary but consistent with the idea that focusing on hard tokens can reduce overconfidence on easy samples [2605.30260].

## 6. Interpretation, limitations, and relation to adjacent memory research

MemFT suggests a specific interpretation of LoRA capacity limits. Exact textual memory is constrained not only by average loss reduction but also by the need to move every critical token past a discrete dominance threshold. The Parametric Memory Law quantifies aggregate loss reduction as a function of adapter rank and sequence length; the token-level phase transition explains why aggregate loss reduction may still fail to yield exact recall. MemFT combines these perspectives by reallocating limited LoRA capacity toward tokens that block deterministic greedy recovery [2605.30260].

The scope of the method is clearly bounded. The study is limited to 8B-scale models, so the universality of the law and of MemFT’s gains at other scales is not established. The critical threshold \(p>0.5\) is specific to greedy decoding; under top-\(p\) or temperature sampling it is no longer the operative recall criterion. The work is fundamentally about exact memorization rather than broad factual updating, reasoning, or open-ended knowledge generalization. The paper is also explicit that MemFT may not help much once the task is saturated or when failures are not due to sub-threshold bottleneck tokens [2605.30260].

Adjacent arXiv work clarifies the boundaries of the concept. "MemFail: Stress-Testing Failure Modes of LLM Memory Systems" formalizes memory systems as summarization, storage, and retrieval, and argues that end-to-end QA accuracy hides whether a failure originated in write-time compression, persistence, access, or downstream reasoning [2605.26667]. This suggests a complementary evaluation lens for memory-focused methods, although MemFail is a diagnostic benchmark rather than a fine-tuning strategy. By contrast, "FedMef: Towards Memory-efficient Federated Dynamic Pruning" is a memory-efficient federated dynamic pruning framework for training from scratch or full training in cross-device FL, not primarily a federated fine-tuning method, and its focus is client-side training memory rather than exact parametric recall in LoRA adapters [2403.14737].

Within that landscape, MemFT is most precisely characterized as a threshold-aware reweighting scheme for LoRA fine-tuning. Its central contribution is to convert a descriptive memory-scaling law and a deterministic token-level threshold into an optimization rule: budget should be spent on unresolved positions, not uniformly across all targets. The reported consequence is better exact memory fidelity and parameter efficiency, especially in low-budget, long-sequence, and hard-memorization regimes [2605.30260].

Source: https://www.emergentmind.com/topics/memft