Tuning without Forgetting (TwF) in Transformers

• TwF is a design principle that preserves prior knowledge by expanding model capacity without altering the original function at initialization.
• The transformer formulation replicates and scales MLP submodules to double hidden width, ensuring stable fine-tuning from step 0.
• Empirical results demonstrate that TwF prevents catastrophic forgetting, maintaining pretraining performance while adapting to new objectives.

Searching arXiv for the core TwF paper and a few historically related uses of the term. arXiv search query: (Adila et al., 9 Mar 2026) Tuning without Forgetting (TwF) denotes a class of adaptation strategies designed to acquire new task competence while preserving prior capabilities. In the recent transformer-centered formulation, TwF is a function-preserving expansion procedure that grows model capacity rather than overwriting pretrained parameters, so that the expanded network is mathematically identical to the original at initialization and can then be specialized by training only newly added components (Adila et al., 9 Mar 2026). Across the broader literature, the same label or closely related formulations have also been used for output distillation without old-task data, layer-wise feature preservation from a frozen sibling network, selective low-rank adaptation, thresholded local updates in physical networks, geometric subspace-constrained learning in neural ODEs, and value-only updates that preserve in-context learning (Li et al., 2016, Boschini et al., 2022, Bafghi et al., 26 Jan 2025, Chatterjee et al., 3 Dec 2025, Bayram et al., 3 Sep 2025, Lee et al., 26 Feb 2026). This diversity suggests that TwF is best understood as a design principle—retain previously learned behavior while tuning for new objectives—rather than as a single canonical algorithm.

1. Genealogy and conceptual scope

The earliest influential precursor is "Learning without Forgetting," which addressed the setting in which a new task must be added to a CNN while the original task data are unavailable (Li et al., 2016). Its mechanism was distillation: the pretrained network’s outputs on new-task images were preserved through a softened old-task loss, while a new task head was trained jointly. In that formulation, catastrophic forgetting is mitigated by constraining output behavior rather than by freezing all shared parameters.

Subsequent work broadened the scope of the idea. "Transfer without Forgetting" considered continual learning after pretraining and maintained a fixed pretrained sibling network whose internal representations were propagated to a continuously updated student through a layer-wise feature-preservation term, alongside replay-based losses (Boschini et al., 2022). In self-supervised ViTs, parameter-efficient anti-forgetting strategies included Block Expansion, which inserts identity-initialized trainable blocks on top of frozen pretrained blocks, and LoRA, which adds low-rank deltas while keeping the backbone fixed (Bafghi et al., 2024). Related parameter-freezing approaches also include adaptive weight modulation with task-specific multiplicative factors (Sverdlov et al., 2023) and selective activation of LoRA blocks through hard gating (Bafghi et al., 26 Jan 2025).

The 2026 paper "Grow, Don’t Overwrite: Fine-tuning Without Forgetting" sharpened the term in the transformer setting by making function preservation exact at initialization through explicit parameter replication and scaling inside Transformer MLPs (Adila et al., 9 Mar 2026). In parallel, the label has also been used in control-theoretic and physical-network settings: TwF in neural ODEs was formalized as motion along the tangent space of a Banach submanifold of controls that preserve previously memorized end-point constraints (Bayram et al., 3 Sep 2025), while in tunable resistor networks it referred to a thresholded local learning rule that ignores weak gradients and thereby partitions the network into weakly overlapping task-specific regions (Chatterjee et al., 3 Dec 2025). The term therefore spans multiple technical traditions, but all of them target the same stability–plasticity problem.

2. Function-preserving expansion in Transformer MLPs

In the transformer formulation, TwF operates on each Transformer MLP submodule by expanding hidden width while preserving the exact input–output map at initialization (Adila et al., 9 Mar 2026). For layer $n$, with biases omitted, the original forward pass is

$\mathrm{ReLU}(X W_n^{(1)}) W_n^{(2)},$

where

$$X \in \mathbb R^{s\times h}, \qquad W_n^{(1)}\in\mathbb R^{h\times p}, \qquad W_n^{(2)}\in\mathbb R^{p\times h}.$$

TwF expands the hidden dimension from $p$ to $kp$, with replication factor $k$ and, in practice, $k=2$. The expanded parameters are defined by horizontal replication of the up-projection and compensatory $1/k$ scaling in the down-projection:

$$W_{\mathrm{replica}}^{(1)} \coloneqq W_n^{(1)} \oplus W_n^{(1)} \oplus \cdots \oplus W_n^{(1)} \quad (k\ \text{copies}),$$

$$W_{\mathrm{replica}}^{(2)} \coloneqq \frac{1}{k} \begin{bmatrix} W_n^{(2)} \ \vdots \ W_n^{(2)} \end{bmatrix} \in \mathbb R^{kp\times h},$$

so that

$\mathrm{ReLU}(X W_n^{(1)}) W_n^{(2)},$0

$\mathrm{ReLU}(X W_n^{(1)}) W_n^{(2)},$1

At initialization, the expanded MLP computes

$\mathrm{ReLU}(X W_n^{(1)}) W_n^{(2)},$2

Since $\mathrm{ReLU}(X W_n^{(1)}) W_n^{(2)},$3 with $\mathrm{ReLU}(X W_n^{(1)}) W_n^{(2)},$4 repeated $\mathrm{ReLU}(X W_n^{(1)}) W_n^{(2)},$5 times, one obtains

$\mathrm{ReLU}(X W_n^{(1)}) W_n^{(2)},$6

Therefore, for every input $\mathrm{ReLU}(X W_n^{(1)}) W_n^{(2)},$7,

$\mathrm{ReLU}(X W_n^{(1)}) W_n^{(2)},$8

The central property is thus function-preservation by construction. The significance of this construction is architectural rather than regularization-based: pretrained knowledge is reused exactly, not approximately, at initialization. This is a stronger statement than merely constraining parameter drift or adding anti-forgetting penalties, because the model begins fine-tuning from an identical function with strictly larger capacity.

3. Training variants, modularity, and efficiency

After expansion, each MLP layer has twice, or more generally $\mathrm{ReLU}(X W_n^{(1)}) W_n^{(2)},$9 times, its original hidden width. The paper defines two fine-tuning variants (Adila et al., 9 Mar 2026). In G-Freeze (default), all original parameters $$X \in \mathbb R^{s\times h}, \qquad W_n^{(1)}\in\mathbb R^{h\times p}, \qquad W_n^{(2)}\in\mathbb R^{p\times h}.$$0 are frozen, and only the newly added blocks are trained: $$X \in \mathbb R^{s\times h}, \qquad W_n^{(1)}\in\mathbb R^{h\times p}, \qquad W_n^{(2)}\in\mathbb R^{p\times h}.$$1 and $$X \in \mathbb R^{s\times h}, \qquad W_n^{(1)}\in\mathbb R^{h\times p}, \qquad W_n^{(2)}\in\mathbb R^{p\times h}.$$2 minus their shared original slices. In G-Train (for hard reasoning tasks), every original matrix is frozen except that the entire expanded up-projection $$X \in \mathbb R^{s\times h}, \qquad W_n^{(1)}\in\mathbb R^{h\times p}, \qquad W_n^{(2)}\in\mathbb R^{p\times h}.$$3 is trained, while the down-projection $$X \in \mathbb R^{s\times h}, \qquad W_n^{(1)}\in\mathbb R^{h\times p}, \qquad W_n^{(2)}\in\mathbb R^{p\times h}.$$4 and all other $$X \in \mathbb R^{s\times h}, \qquad W_n^{(1)}\in\mathbb R^{h\times p}, \qquad W_n^{(2)}\in\mathbb R^{p\times h}.$$5 remain frozen.

Optimization is deliberately simple. The reported configuration uses Adam with learning rate $$X \in \mathbb R^{s\times h}, \qquad W_n^{(1)}\in\mathbb R^{h\times p}, \qquad W_n^{(2)}\in\mathbb R^{p\times h}.$$6, no extra stability penalty, and runs for exactly the number of steps until a vanilla fine-tuning baseline converges. Because the network is function-preserving at initialization, the method is described as guaranteeing stable training from step 0.

Its computational profile is tied to the choice of expansion. When $$X \in \mathbb R^{s\times h}, \qquad W_n^{(1)}\in\mathbb R^{h\times p}, \qquad W_n^{(2)}\in\mathbb R^{p\times h}.$$7 and every MLP is expanded, the total number of parameters increases by roughly $$X \in \mathbb R^{s\times h}, \qquad W_n^{(1)}\in\mathbb R^{h\times p}, \qquad W_n^{(2)}\in\mathbb R^{p\times h}.$$8, and only the new slices—approximately $$X \in \mathbb R^{s\times h}, \qquad W_n^{(1)}\in\mathbb R^{h\times p}, \qquad W_n^{(2)}\in\mathbb R^{p\times h}.$$9 of the original count—are trained. A further modularity result is obtained by ranking layers according to the magnitude of weight updates in a preliminary SFT run and expanding only the top-10 most active MLP layers. This reduces trainable parameters from approximately $p$0 to approximately $p$1 while still matching full-expansion performance. Since all original weights are frozen, no extra gradient or optimizer state is incurred for them, which the paper identifies as an additional memory and compute saving.

This modularity differentiates TwF from approaches that require uniform adaptation across all layers. A plausible implication is that the anti-forgetting property does not require global architectural growth; instead, it can be concentrated in the submodules that exhibit the greatest downstream adaptation pressure.

4. Evaluation protocol and empirical behavior

The main empirical study evaluates four downstream tasks spanning domain shift and uses WinoGrande as a proxy for the pre-training distribution (Adila et al., 9 Mar 2026).

Task	Downstream metric	Retention proxy
MTNT (English→French translation)	SacreBLEU	WinoGrande
SciTail (science entailment)	exact-match accuracy	WinoGrande
QASC (science QA)	exact-match accuracy	WinoGrande
MathQA (mathematical reasoning)	exact-match accuracy	WinoGrande

The reported comparison is against standard fine-tuning (SFT). On MTNT and SciTail, SFT quickly loses prior knowledge, with WinoGrande accuracy collapsing to near-0. By contrast, TwF (G-Freeze) achieves downstream performance on par with SFT while exhibiting zero or only $p$2 degradation on WinoGrande. TwF (G-Train) further boosts MathQA accuracy relative to G-Freeze while still preserving at least $p$3 of WinoGrande performance. Selective Expansion of 9–10 layers gives the same curves as expanding all layers.

These results instantiate the paper’s central claim that the trade-off between plasticity and stability can be removed by expansion rather than overwrite. In the authors’ wording, “Plasticity vs Stability is fully resolved” through replication plus scaling, because pretrained knowledge is reused without being overwritten (Adila et al., 9 Mar 2026). The empirical evidence is especially notable in the reported contrast between catastrophic collapse under SFT and near-complete retention under TwF at comparable downstream performance. At the same time, the paper explicitly notes that it did not directly compare against adapters or LoRA in these experiments, and instead describes TwF as orthogonal to PEFT methods.

5. Related formulations across architectures and learning regimes

The TwF design principle has been instantiated through distinct mechanisms in several literatures.

Formulation	Core mechanism	Representative papers
Output- or feature-preserving tuning	Distillation on old outputs or frozen-sibling features	(Li et al., 2016, Boschini et al., 2022)
Frozen-backbone parameter-efficient tuning	Selective LoRA, Block Expansion, adaptive modulation	(Bafghi et al., 26 Jan 2025, Bafghi et al., 2024, Sverdlov et al., 2023)
Constrained optimization or geometry	Loss-adaptive learning rates, tangent-space updates, value-only tuning	(Prashant et al., 19 May 2026, Bayram et al., 3 Sep 2025, Lee et al., 26 Feb 2026)

In "Learning without Forgetting," preservation is enforced at the level of old-task predictions on new-task images through a distillation loss with temperature $p$4, jointly optimized with the new-task classification loss (Li et al., 2016). "Transfer without Forgetting" extends this preservation into continual learning by combining replay-based classification, feature propagation from a frozen sibling, and an auxiliary diversity loss on attention gates (Boschini et al., 2022). These methods preserve behavior by constraining outputs or internal representations rather than by growing capacity.

Parameter-efficient variants move closer to the architectural spirit of the 2026 transformer formulation. Selective Low-Rank Adaptation activates only a sparse subset of LoRA blocks through an indicator function $p$5, with an $p$6 sparsity regularizer over block scores and reported effectiveness with as few as $p$7 of active blocks (Bafghi et al., 26 Jan 2025). In self-supervised ViTs, Block Expansion inserts zero-initialized identity blocks after groups of frozen pretrained blocks, while LoRA adapts only low-rank query and value deltas; Block Expansion is reported to preserve source-domain performance more strongly than LoRA (Bafghi et al., 2024). Adaptive weight modulation freezes the base network entirely and learns per-task multiplicative modulation parameters and task-specific batch-normalization statistics, thereby guaranteeing zero forgetting by design because the shared weights never change (Sverdlov et al., 2023).

Other TwF formulations shift the locus of control from architecture to optimization or geometry. FINCH derives a per-step upper bound on forgetting proportional to $p$8 and sets the learning rate inversely to the square root of a smoothed batch loss, thereby reducing forgetting without modifying the objective (Prashant et al., 19 May 2026). In neural ODEs, TwF becomes a continuation procedure on a finite-codimension Banach submanifold of controls preserving previous sample constraints, with updates projected onto the tangent space $p$9 (Bayram et al., 3 Sep 2025). In linear-attention models, full tuning of $kp$0 is shown to damage in-context learning, whereas value-only tuning preserves the pretrained in-context mechanism while improving zero-shot performance (Lee et al., 26 Feb 2026). In physical resistor networks, TwF takes the form of a hard-thresholded local learning rule $kp$1, which confines updates to spatially relevant regions and reduces overlap between tasks (Chatterjee et al., 3 Dec 2025).

6. Misconceptions, limitations, and open problems

A common misconception is that TwF always means exact zero forgetting. The literature supports a more differentiated view. Exact preservation is built into some formulations—for example, function-preserving initialization in Transformer MLP expansion (Adila et al., 9 Mar 2026), frozen-base modulation (Sverdlov et al., 2023), or tangent-space-constrained motion in neural ODEs (Bayram et al., 3 Sep 2025). Other methods instead reduce forgetting substantially without claiming exact preservation, such as FINCH’s $kp$2 average reduction in forgetting while matching target-task performance, or MLLM recipe adjustments that prevent forgetting under some distribution shifts but still reveal a distinct failure mode under in-distribution images with out-of-distribution text (Prashant et al., 19 May 2026, Li et al., 15 Mar 2026).

Another misconception is that catastrophic forgetting is always caused by the same mechanism. The surveyed papers point to several causes: direct overwriting of pretrained parameters in full fine-tuning (Adila et al., 9 Mar 2026), erosion of source-domain representations during continual learning after pretraining (Boschini et al., 2022), loss of in-context learning when all attention parameters are updated (Lee et al., 26 Feb 2026), and task-specific overfitting or template overfitting in multimodal LLMs (Li et al., 15 Mar 2026). This suggests that the appropriate TwF intervention depends on where interference occurs: weights, features, logits, control subspaces, or optimization dynamics.

The transformer-expansion formulation also has explicit limitations and extensions. It generalizes to any integer $kp$3, although the authors found $kp$4 empirically best (Adila et al., 9 Mar 2026). Other submodules, such as attention heads, could in principle be grown, but the reported ablations indicate that MLP expansion is the most efficient. Proposed future directions include combining function-preserving expansion with PEFT methods such as LoRA or adapters and applying the method to vision Transformers. Related lines of work expose further open questions: FINCH’s theory is stated for SGD while experiments use AdamW; its calibration drift is reduced but not eliminated (Prashant et al., 19 May 2026). In physical networks, setting the threshold $kp$5 too large prevents learning altogether (Chatterjee et al., 3 Dec 2025). In neural ODEs, the manifold picture relies on memorization, linearized controllability, and transversality assumptions (Bayram et al., 3 Sep 2025). In MLLMs, the most persistent failure required data-hybrid training with auxiliary instruction data, indicating that anti-forgetting may require data design in addition to parameter constraints (Li et al., 15 Mar 2026).

Taken together, these results locate TwF at the intersection of continual learning, PEFT, optimization, and control. The most specific 2026 transformer instantiation realizes TwF through a two-step operation—horizontal replication of up-projection weights and compensatory $kp$6 scaling in the down-projection—followed by training only new parameters (Adila et al., 9 Mar 2026). The broader research trajectory suggests that the enduring principle is not tied to one architecture: preserve the function or subspace that encodes prior competence, and allocate adaptation pressure to newly created, selectively activated, or geometrically admissible degrees of freedom rather than to the entire pretrained model.