Local Support Learning

Abstract: We explore catastrophic forgetting in the context of large pre-trained models. By considering forgetting as a geometric problem in the input space of each weight matrix, we uncover a natural retention objective under which updates produced by gradient-based optimizers are suboptimal. Following this observation, we propose Local Support Learning (LSL), a general-purpose framework that augments gradient-based training for retention of prior capabilities without access to prior data. During a new learning phase, LSL pairs two components with distinct roles: a standard weight adapter, trained as usual to minimize the loss, and a gating function that enables the adapter only on input activations from its own training distribution, making the update local to that distribution. The key challenge is that this gate must route data from all learning phases while training only on data from the current one. We address this with a gate based on a Gaussian Mixture Model (GMM), whose likelihood decays rapidly away from its training data, giving it a natural tendency to stay closed on data from prior phases. We show that this post-training approach can resolve forgetting in LLMs of up to 7 billion parameters, retaining both pretrained and finetuned capabilities across multiple training phases, while being efficient in memory and compute, robust to hyperparameter choice, and showing scaling potential.

• The paper introduces Local Support Learning (LSL), a method that aims to prevent catastrophic forgetting by ensuring that updates to a model's weight matrices only affect specific regions of the activation space relevant to the current learning phase.
• The proposed method, LSL, improves performance retention in large pretrained models by modifying attribute and activation support from different learning phase tasks.
• LSL demonstrated better performance retention compared to other methods, such as LoRA, across multiple downstream tasks and model sizes, significantly improving the retention of prior capabilities across multiple learning phases.

Catastrophic forgetting in large pretrained models is usually framed as a conflict between acquiring new capabilities and preserving old ones. “Local Support Learning” (2610.02126) formulates this conflict more specifically as a problem of functional interference in the input space of individual weight matrices. The central claim is that a parameter update learned from a new distribution should modify the matrix only for activations belonging to that distribution. Conventional gradient-based updates do not satisfy this locality condition: although they are optimized using current-phase inputs, the resulting matrix update is applied to every future input. Local Support Learning (LSL) addresses this mismatch by combining ordinary adapters with per-matrix support-estimation gates.

Geometric formulation of forgetting

Consider a weight matrix $W$ receiving an activation $x$. A gradient update generated from a current training example $x^{(s)}$ has the form

$$\Delta W^{(s)} \propto \frac{\partial L}{\partial v} {x^{(s)}}^\top.$$

For another input $x$, the induced change in the matrix output is proportional to ${x^{(s)}}^\top x$. Consequently, an update that is useful for the current distribution changes the output for any prior input that is not orthogonal to the training activation. The update therefore has global functional support even when its optimization objective is local to the current dataset.

This observation applies not only to vanilla gradient descent but also to optimizers such as Adam and Muon. Rescaling or orthogonalizing the update in parameter space does not alter the fact that the resulting matrix acts on all inputs. The paper consequently argues that several existing retention methods optimize an insufficient proxy for forgetting. LoRA restricts updates to a low-rank parameter subspace, OP-LoRA suppresses components associated with dominant singular directions of the pretrained weights, and LwF constrains output changes on current-phase data. None of these mechanisms directly prevents an update from altering the matrix function outside the current data support.

The proposed retention objective is therefore functional and distributional: the update should be active only on the region of activation space occupied by the current learning phase. This is distinct from enforcing orthogonality to previous data. Orthogonal projection requires access to, or a sufficiently accurate representation of, prior activation distributions, which is particularly impractical for a model pretrained on trillions of tokens. LSL instead estimates where the current distribution lies and suppresses the update elsewhere.

Local Support Learning

LSL separates adaptation from routing. For each learning phase, it trains a conventional adapter, such as a LoRA module, to minimize the task loss. It then fits a gate to the adapter’s input activations. At inference, the adapter contributes only when the current activation is classified as belonging to the support of the distribution on which that adapter was trained. The pretrained weights remain active globally, while each adapter is locally applied.

This decomposition is important. The adapter retains the representational capacity and optimization behavior of standard parameter-efficient fine-tuning; the gate determines where the learned modification is permitted to affect the model. Training does not require the gate to be active because all training examples belong to the current phase. Instead, the adapter is trained normally, and the gate is fitted afterward from the resulting activations. This introduces a train–inference mismatch, since the gate may reject some current-phase activations at inference, but the experiments report no material degradation from this design.

The paper’s gate is based on a pair of Gaussian mixture models. A positive GMM, $\Phi_{\mathrm{pos}}$, is fitted to the current-phase activations. A negative GMM, $\Phi_{\mathrm{neg}}$, is fitted once to a small generic pretraining sample. The adapter is activated when the positive model assigns a higher likelihood than the negative model:

$$g(x) = \mathbb{I}\{\Phi_{\mathrm{pos}}(x) > \Phi_{\mathrm{neg}}(x)\}.$$

The negative model is not intended to reconstruct the original pretraining distribution. Its purpose is to provide a broad reference density that dominates away from the current distribution. This gives the gate a useful inductive bias: the positive density decays away from the observed current-phase support, while the negative density remains comparatively competitive elsewhere. The reference sample can therefore be unrelated to the original pretraining corpus. The experiments use at most one million reference tokens, reported as approximately $0.0000067\%$ of Qwen’s pretraining set.

LSL places gates separately on weight matrices rather than using one global router. Since representations change across depth, the support of a task is matrix-specific. A single early gate must make a decision using less contextual information and propagates any routing error to every adapter. Per-matrix gates make decisions using the activations available at each module and confine a routing error to that module. Their independent structure also permits substantial parallelization.

The method optionally applies temporal smoothing to token-level gate decisions. An exponential moving average suppresses isolated routing errors and exploits the fact that phase identity usually does not change at every token. The paper emphasizes that smoothing is secondary: in the principal ablation, the GMM increases retention from $x$0 to $x$1 relative to the base model, while smoothing raises it further to $x$2.

Theoretical motivation

The theoretical analysis isolates an asymmetry in gate learning. A gate must accept the current support and reject inputs outside it. Errors inside the support constitute a deficit; erroneous activation outside the support constitutes excess. The current-phase training objective directly observes the deficit but is blind to the excess, because regions with zero probability under the current distribution contribute nothing to a data-weighted loss.

This creates a fundamental problem for sufficiently expressive discriminative gates. Two gates can achieve identical current-data loss while behaving differently on regions never observed during training. An adversarial prior distribution can concentrate precisely on a region where the gate erroneously remains open, producing severe forgetting without affecting the gate’s training objective.

The paper analyzes the problem under bounded activation domains and defines the target gate as the indicator of the current distribution’s support. Under a worst-case test distribution with bounded density, the minimax error is proportional to the volume of the gate’s symmetric difference from that support. Density estimation controls this quantity because an $x$3 density error lower-bounds both excess and deficit according to the threshold used to define the support.

This analysis explains the preference for density estimators over generic classifiers. A density estimator imposes an inductive bias on off-support behavior, whereas a classifier trained against a finite negative buffer is unconstrained in regions not represented by either class. The paper gives explicit counterexamples in which a classifier achieves zero training error while accepting an arbitrarily large region outside the positive support. It also notes the cost of the GMM approach: a fixed parametric family has constant memory, but if the family is misspecified, excess can persist even with unlimited data. Thus, the theoretical argument supports GMMs conditionally rather than establishing universal support recovery.

Evaluation on post-training

The main experiments use Qwen2.5-Instruct models and compare LSL with LoRA, OP-LoRA, and LwF. The primary model is Qwen2.5-7B-Instruct. New learning phases cover cybersecurity instruction tuning, English-to-Igbo translation, and chemistry instruction tuning. Retention is measured on GSM8K, HumanEval, and IFEval, representing mathematical reasoning, code generation, and instruction following.

Across all three downstream settings, LSL reaches strong new-task performance while retaining pretrained capabilities close to the base-model level. LoRA adapts effectively but consistently incurs forgetting. OP-LoRA performs similarly to LoRA, supporting the paper’s claim that forgetting is not primarily explained by overlap between the adapter and the dominant singular subspace of the pretrained weights. The reported energy overlap of a standard LoRA update with the top singular directions is at most approximately $x$4 across the tested values of $x$5, implying that at least $x$6 of the update energy already lies outside those directions. This provides a concrete explanation for OP-LoRA’s limited advantage in the experiments.

LwF improves retention relative to the weight-based baselines but remains inferior to LSL. Its distillation term constrains the model’s outputs on finetuning inputs, not on the prior inputs whose behavior must be preserved. The result is consistent with the paper’s geometric diagnosis: preserving the base model on the current distribution does not prevent a local adapter update from changing the base function on other activation regions.

The numerical pattern is stronger in sequential learning. The model is trained on translation, chemistry, and cybersecurity in succession. LSL preserves pretrained performance throughout the three phases and also preserves the gains acquired in earlier phases. The LoRA baseline loses $x$7 of the pretrained-task performance after the first phase. Its translation performance subsequently declines by $x$8, chemistry declines by $x$9, and cybersecurity zero-shot performance falls by $x^{(s)}$0 after an intervening phase. LSL avoids these degradations, indicating that local routing protects both the initial model and previously mounted adapters.

The implication is that LSL changes the retention–plasticity trade-off rather than merely moving along it. Conventional methods must often reduce the learning rate, adapter rank, or update magnitude to limit forgetting. In the reported sweeps, LSL maintains pretrained performance across changes in learning rate, adapter rank, and batch size while still reaching the best available downstream performance. This decoupling is one of the paper’s strongest empirical claims: retention becomes largely insensitive to the hyperparameters that control adapter learning.

Scaling and systems properties

LSL is evaluated on models with 1.5B, 3B, and 7B parameters. It works at all tested scales, and retention improves with model size. LoRA also benefits from scaling, but more slowly and remains below LSL at every size. The result is compatible with the method’s dependence on the quality of learned representations: larger models may produce more separable activation distributions, making GMM support estimation easier. It does not, however, establish that the trend continues beyond 7B parameters.

The memory overhead is small relative to the model. On a Qwen2.5-7B-Instruct system, the GMMs require 13.8 MB of additional memory, or 6.9 MB per phase because the negative GMM is shared. The base model requires approximately 14 GB in bfloat16, making the gate overhead roughly three orders of magnitude smaller than the model weights. A local Union-of-Spheres estimator achieves comparable retention but requires approximately 5.67 GB because it stores representative activation samples, illustrating the practical advantage of the parametric GMM.

The measured inference latency for an LSL adapter is 497 ms per forward pass versus 411 ms for the LoRA baseline, an additional 86 ms in the reported setup. Training on one million tokens takes 13:45 with LSL, compared with 8:23 for LoRA; fitting the GMMs increases total training time by a factor of approximately $x^{(s)}$1. These measurements were not fully optimized, but they indicate that the method’s overhead is moderate rather than negligible. Its principal systems cost is not memory but the inability to merge adapters into the base weights.

The complexity analysis makes this limitation explicit. With $x^{(s)}$2 phases, inference cost grows linearly in $x^{(s)}$3, since each adapter and gate must be evaluated. The authors reduce the naive dimensional dependence using Johnson–Lindenstrauss projections for gate inputs and diagonal-covariance GMMs. The resulting memory and computation depend on the projected dimension rather than the full matrix dimension. Gates associated with matrices sharing identical inputs, such as query, key, and value projections, can also be shared, reducing the number of gates by 43% in a standard Qwen/LLaMA architecture.

Ablation evidence for locality

The ablations support locality as the causal mechanism. A Union-of-Spheres gate, which explicitly constructs a local neighborhood around current-phase activation samples, matches the GMM’s retention while using substantially more memory. A conventional MLP gate trained on the same positive and negative data performs worse. On in-distribution data, the MLP has a slight classification advantage, but its accuracy deteriorates on OOD pretraining tasks: relative to the GMM, its accuracy is lower by 20.4 percentage points on GSM8K, 14.4 points on HumanEval, and 7.2 points on IFEval. This result is important because it shows that in-domain gate classification is not the relevant criterion. The decisive property is conservative behavior outside the current support.

A single router placed after the embedding layer recovers only 50–77% of LSL’s new-task improvement and can also reduce retention, particularly on cybersecurity. This supports the use of per-matrix routing, although the comparison does not isolate every possible intermediate-layer or hierarchical routing design.

The number of GMM components has limited effect; near-optimal retention is obtained with as few as two components in the reported ablation. This suggests that the base model’s activation geometry is sufficiently structured for relatively simple density models. The conclusion should nevertheless be read in the context of the selected tasks and models, since the adequacy of a low-component GMM is representation-dependent.

Function-space measurements provide an additional verification. On current-task data, LSL and LoRA change the output distribution by similar amounts, indicating that LSL does not simply suppress adaptation. On OOD pretrained benchmarks, LSL changes the model’s output distribution substantially less than LoRA, in some cases by approximately an order of magnitude. The gates therefore implement the intended distinction: near-current support, the adapter behaves like a standard finetuning update; away from that support, the base model remains comparatively intact.

Limitations and open questions

LSL’s most direct limitation is cumulative inference cost. Adapters cannot be merged into the base weights, so latency and computation increase with the number of learning phases. The relevant count is the number of phases rather than the number of tasks, since multiple tasks can be grouped into one phase, but the method’s scalability to many phases remains untested. The experiments cover three sequential phases, not hundreds.

The theoretical analysis is formally established for a single weight matrix. The extension to a deep, multi-layer network with interdependent representations is supported empirically but not derived. In particular, later gates operate on activations affected by earlier adapters and gates, so the full network is not simply a collection of independent local-support problems.

The GMM gate also depends on the adequacy of the base model’s representation and on the suitability of a finite Gaussian mixture with the chosen covariance structure. The paper explicitly acknowledges that parametric density estimation can retain excess under model misspecification. Its practical negative GMM is only a generic reference, and the resulting gate still opens on approximately 16% of tokens from OOD pretraining tasks, reduced to 5% with temporal smoothing. The corresponding retention is high but not perfect: 96.6% without smoothing and 98.8% with it. Whether these residual activations are harmless across broader distributions is unresolved.

The experiments focus on Qwen2.5 instruction models, three post-training datasets, and parameter scales up to 7B. They do not establish performance for larger models, non-language modalities, reinforcement-learning objectives, or phase distributions with substantial overlap and rapidly shifting supports. A specific open question is whether the GMM’s local-support inductive bias remains reliable when a new phase deliberately modifies behavior on inputs that are also central to prior capabilities. In such cases, “apply the latest update on overlap” is a policy choice rather than an objective consequence, and retention may conflict intrinsically with the desired adaptation.

Conclusion

“Local Support Learning” (2610.02126) presents catastrophic forgetting as a mismatch between local training objectives and globally acting matrix updates. Its remedy is architectural rather than purely optimizational: retain standard adapters for learning capacity, but gate them according to per-matrix estimates of the current activation support. The GMM implementation achieves strong retention on models up to 7B parameters, preserves performance across multiple post-training phases, and does so with low persistent memory overhead. The empirical evidence indicates that the critical factor is conservative off-distribution routing, not merely low-rank adaptation or parameter-space proximity. The principal unresolved issues are the linear phase-dependent inference cost, the dependence on representation quality and density-model specification, and the absence of formal guarantees for the complete multilayer network.