Learning When to Adapt

Abstract: Low-rank adaptation (LoRA) is a widely used parameter-efficient fine-tuning method, yet its learned correction is static: the same low-rank update is applied to every input. This input-agnostic approach creates an inevitable compromise between adapting to the fine-tuning distribution and preserving pre-trained behavior on inputs outside that distribution, contributing to catastrophic forgetting. We introduce DISeL (Dynamic Input-Sensitive LoRA), which augments LoRA modules with lightweight input-dependent gates over individual rank-one components. The gating mechanism is designed to preserve the pre-trained model's behavior by default, while training learns to activate selected components that reduce the fine-tuning loss. DISeL adds only a small number of parameters and preserves the low-rank structure. Across RoBERTa on GLUE, and Llama and Mistral models fine-tuned for mathematical reasoning and code generation, DISeL reduces forgetting relative to LoRA and related variants while maintaining competitive fine-tuning accuracy. In addition, the learned gate activations provide an interpretable diagnostic view of which layers and rank components are most activated during fine-tuning, giving insight into where task-specific adaptation is concentrated. Code available at https://github.com/alizindari/DISeL .

• The paper introduces DIS, which extends LoRA with input-dependent gating to balance task adaptation and retention of pretrained performance.
• Using lightweight sigmoid gates, DIS selectively activates low-rank updates only for in-domain inputs, reducing catastrophic forgetting.
• Empirical results on models like RoBERTa, Llama, and Mistral demonstrate that DIS maintains near-baseline retention while enhancing target task accuracy.

Learning When to Adapt: Dynamic Input-Sensitive LoRA

Introduction and Motivation

Parameter-efficient fine-tuning (PEFT), especially Low-Rank Adaptation (LoRA), has become the de facto approach for adapting large pretrained models to downstream tasks under compute and storage constraints. However, current methods including LoRA, DoRA, and AdaLoRA rely on input-agnostic updates; the learned correction is applied indiscriminately to all inputs. This global modification leads to catastrophic forgetting: performance on pretraining-domain tasks degrades, as corrections intended for the fine-tuning distribution overwrite the pretrained mapping for all of input space.

"Learning When to Adapt" (2605.19028) systematically dissects this limitation and introduces Dynamic Input-Sensitive LoRA (DIS), where the low-rank update is modulated by per-component, input-dependent gates. These lightweight gates are designed to default to inactivity—preserving pretrained behavior—and only activate on inputs that benefit from adaptation. This paradigm not only mitigates catastrophic forgetting but also yields interpretable diagnostics on where the model is being adapted.

Theoretical Foundations

The authors formalize the adaptation-retention tradeoff in a minimal linear regression setting. Any input-agnostic correction $M$ added to the pretrained weights $W_0$ must compromise: it is only partially applied in the fine-tuning domain (leading to underfitting) and unnecessarily perturbs the mapping in the pretraining domain (causing forgetting). Theoretically, the Bayes-optimal predictor utilizes an input-dependent weighting $\pi_\mathrm{ft}(x)$—a function of the density ratio between fine-tuning and pretraining distributions—activating the correction when likely in-domain and suppressing it elsewhere.

Crucially, this motivates parameterizing adapters with input-conditional gates, capturing the desired dependency in a parameter- and compute-efficient way.

Methodology: DIS Architecture

DIS maintains the standard LoRA structure, i.e., an additive low-rank correction $A B x$. However, for DIS, each rank-one component is modulated by an input-dependent sigmoid gate:

$f(x) = A \cdot \mathrm{diag}(g(x)) \cdot B x$

with $g(x) = \sigma(W_g x + b_g)$, $A \in \mathbb{R}^{d_y \times r}$, $B \in \mathbb{R}^{r \times d_x}$, and $g(x) \in [0,1]^r$. Gates are initialized near zero, ensuring DIS starts as the frozen pretrained model and only activates adaptations when empirically warranted during fine-tuning.

Training proceeds with higher learning rates for gate parameters $(W_g, b_g)$ (to encourage plasticity) and lower rates for the update factors $W_0$0 (matching LoRA best practices). This separation enables DIS to effectively distinguish fine-tuning-domain inputs (activating the relevant ranks) from the pretraining distribution.

DIS adds negligible parameter and computational overhead relative to LoRA: $W_0$1 extra parameters per adapted layer, maintaining the low-rank structure and thus the efficiency central to PEFT.

Empirical Results

Fine-tuning and Retention Tradeoffs

Experiments are conducted on RoBERTa-base (GLUE), Llama 2-7B (math reasoning), and Mistral 7B (code generation). Each scenario measures both in-domain performance and forgetting (out-of-domain perplexity and multi-benchmark accuracy).

Key findings:

• Retention: DIS consistently preserves pretrained performance, incurring only marginal degradation in out-of-domain tasks—perplexity and average benchmark accuracy remain close to the pretrained baseline even for high-rank adapters.
• Target Task Accuracy: DIS matches or exceeds LoRA, DoRA, and AdaLoRA in fine-tuning accuracy across all settings and adapter ranks.
• Comparison to AdaLoRA: While AdaLoRA can also maintain retention in some cases, DIS avoids AdaLoRA's reliance on pruning schedules and maintains fine-tuning performance at higher ranks, where AdaLoRA can underperform.
• Conventional LoRA and DoRA: These methods display a sharp tradeoff: larger ranks improve target task accuracy but rapidly increase forgetting.

This decoupling of accuracy and retention in DIS is a direct consequence of its input-sensitive activation, which insulates pretraining-domain subspaces from unnecessary adaptation.

Dynamics and Robustness

The analysis of checkpoint-level retention reveals that LoRA exhibits monotonically increasing forgetting throughout training—particularly pronounced at high ranks—whereas DIS maintains nearly flat retention from initialization to convergence. This stability enables practitioners to use larger adapters without risk of catastrophic forgetting, unlike conventional LoRA.

Training stability is also noted: DIS regularizes the effective capacity by keeping non-essential gates closed even at suboptimal learning rates, making hyperparameter tuning less critical.

Gate Activation and Interpretability

A unique contribution of DIS is the built-in interpretability via gate activations. By inspecting the activity of each gate across layers, modules, and input domains, DIS reveals:

• When adaptation is triggered: Gates are substantially more open on fine-tuning-domain inputs (e.g., math/questions for a math-fine-tuned Llama), and remain closed for general text.
• Where adaptation occurs: The most active gates are localized in specific modules and layers (e.g., MLP up projections and mid-to-late transformer layers), identifying loci of task adaptation. In Mistral’s code setting, similar adaptation patterns are observed, with math and code sharing gate activations and general text remaining isolated.

This not only offers insight into the locus and importance of adaptation but also suggests actionable feedback for adapter/module design.

Relationship to Prior Work

DIS unifies and extends prior lines:

• LoRA Variants: Most variants (LoRA, DoRA, AdaLoRA, VeRA, AuroRA, PiSSA) maintain an input-agnostic parameterization and thus cannot fundamentally resolve the adaptation-retention tradeoff.
• Mixture and Gated Adapters: Mixture-of-LoRA approaches (e.g., MoLE, LoraHub, Gated LoRA) introduce input-conditional or dynamic routing but operate at adapter or branch granularity, significantly increasing parameter count and compute. Gated LoRA [eom2025gatedlora] introduces per-rank input-dependent gating but targets multi-task interference reduction, not catastrophic forgetting, and uses ReLU activation rather than the sigmoid near-zero initialization crucial for DIS’s default no-adaptation property.
• Continual Learning: Existing continual learning solutions rely on adapter proliferation, prompt routing, or task identity; DIS, in contrast, achieves single-adapter, continuous-domain selective adaptation and retention.

Implications and Future Research Directions

DIS demonstrates that the parameter-efficient adaptation-retention tradeoff is not fundamental but a consequence of input-agnostic update parameterization. Input-dependent gating can achieve high adaptation capacity with negligible forgetting. This facilitates robust deployment of massive pretrained models in multi-domain, high-stakes applications where catastrophic forgetting is unacceptable.

DIS's interpretability and robustness also open new directions:

• Adaptive Rank Selection: Gate statistics could drive dynamic resource allocation in future adapters.
• Explicit Retention Objectives: Combining dynamic gating with explicit out-of-domain retention regularization for even tighter control.
• Multi-task and Continual Learning: Extending the paradigm to sequence task settings with compositional or hierarchical gating.
• Adapter Merging: Developing theory and techniques for integrating learned input-dependent adapters into static weights for efficient deployment.

Limitations remain: the guarantee of task-agnostic retention is empirical, and extension to multi-task or multi-domain generalization remains to be seen.

Conclusion

"Learning When to Adapt" (2605.19028) introduces a theoretically principled and empirically validated mechanism for PEFT, resolving the adaptation-retention conflict by dynamic, input-dependent gating of low-rank adapters. This approach preserves the computational and memory efficiency of LoRA, outperforms existing PEFT methods on both accuracy and retention, and provides interpretable assessments of adaptation loci across the network. The method suggests a general principle that input-conditional adaptation is critical for scalable, robust, and interpretable specialization of large pretrained NNs—especially as models and downstream tasks continue to diversify.