Entropy-Adaptive Fine-Tuning: Resolving Confident Conflicts to Mitigate Forgetting

Abstract: Supervised Fine-Tuning (SFT) is the standard paradigm for domain adaptation, yet it frequently incurs the cost of catastrophic forgetting. In sharp contrast, on-policy Reinforcement Learning (RL) effectively preserves general capabilities. We investigate this discrepancy and identify a fundamental distributional gap: while RL aligns with the model's internal belief, SFT forces the model to fit external supervision. This mismatch often manifests as "Confident Conflicts" tokens characterized by low probability but low entropy. In these instances, the model is highly confident in its own prediction but is forced to learn a divergent ground truth, triggering destructive gradient updates. To address this, we propose Entropy-Adaptive Fine-Tuning (EAFT). Unlike methods relying solely on prediction probability, EAFT utilizes token-level entropy as a gating mechanism to distinguish between epistemic uncertainty and knowledge conflict. This allows the model to learn from uncertain samples while suppressing gradients on conflicting data. Extensive experiments on Qwen and GLM series (ranging from 4B to 32B parameters) across mathematical, medical, and agentic domains confirm our hypothesis. EAFT consistently matches the downstream performance of standard SFT while significantly mitigating the degradation of general capabilities.

• The paper introduces EAFT to suppress destructive gradients from confident conflicts, effectively mitigating catastrophic forgetting during domain adaptation.
• It employs a token-level gating mechanism using normalized top-K entropy to ensure learning focuses on uncertain tokens while preserving pre-trained knowledge.
• Empirical results across various LLM architectures show EAFT significantly reduces general performance loss compared to standard SFT, achieving an optimal adaptation trade-off.

Entropy-Adaptive Fine-Tuning (EAFT): Targeted Mitigation of Catastrophic Forgetting via Confident Conflict Suppression

Introduction: Analyzing Catastrophic Forgetting in Domain Adaptation

This work interrogates the empirical and theoretical underpinnings of catastrophic forgetting in Supervised Fine-Tuning (SFT) of LLMs. While SFT is a de facto strategy for rapid domain adaptation, it consistently induces a marked decrement in a model's general-domain competence—a phenomenon not mirrored in on-policy Reinforcement Learning (RL). Through systematic token-level analysis, the authors pinpoint the root cause: the destructive optimization driven by "Confident Conflicts," i.e., tokens where the model's predictive entropy is low (implying strong prior certainty) but the reference (ground truth) assigns low probability.

The authors formalize this by contrasting the entropy–probability joint distributions under SFT and RL rollouts. They demonstrate that SFT introduces a significant mass of tokens in the low-entropy/low-probability quadrant (Figure 1), while RL rollouts inherently stay within high-probability (aligned prior) or high-entropy (exploration/uncertainty) regions. These Confident Conflicts produce disproportionately large and destructive gradients under standard cross-entropy objectives, directly destabilizing the original knowledge representations.

Figure 1: (a) SFT enforces Confident Conflicts—cases where the model is certain but supervised to contradict itself—causing catastrophic forgetting. (b) The entropy–probability joint distribution reveals a unique cluster of low-entropy, low-probability tokens in SFT data.

Entropy-Adaptive Fine-Tuning: Methodological Formulation

Informed by their analysis, the authors propose Entropy-Adaptive Fine-Tuning (EAFT): a token-level gating mechanism that modulates the loss contribution using normalized predictive entropy. Specifically, the EAFT loss scales the classical cross-entropy term at each timestep by $\tilde{H}_t = {H_t^{\text{top-}K}}/{\ln(K)}$, where $H_t^{\text{top-}K}$ is the entropy over the top $K$ tokens and $K = 20$ is empirically shown to capture 99.9% of entropy mass while being highly computationally efficient.

The conceptual mechanism is robust: tokens where the model is already confident (low entropy) and in disagreement with supervision are strongly down-weighted, preventing excessive parameter drift, while uncertain (high-entropy) tokens receive maximal learning signal, sustaining domain adaptation efficacy.

Empirical Validation and Main Results

Comprehensive experiments were conducted on multiple LLM architectures (Qwen and GLM series, 4B to 32B parameters) and diverse domains (mathematical reasoning, biomedical QA, agentic tool use). EAFT is benchmarked against SFT, SFT$_{\mathrm{KL}}$, and leading dynamic loss weighting methods such as DFT, FLOW, TALR.

Key quantitative results:

• On all configurations, EAFT achieves competitive or superior in-domain (target) accuracy compared to SFT while drastically reducing general capability degradation.
• The forgetfulness penalty (average performance drop on general benchmarks) is minimized to the 1–2 point range, compared to 4–8 points for SFT and other baselines.
• Pareto analysis across method variants (Figure 2) demonstrates that only EAFT and its non-linear variants (e.g., EAFT$^2$, EAFT$^3$, EAFT$_\mathrm{Sigmoid}$) reach the top-right frontier: achieving maximal target task performance without incurring catastrophic forgetting.
• Masked SFT, which hard-filters Confident Conflicts, eliminates forgetting entirely but at the cost of substantial adaptation performance—highlighting the necessity of soft, entropy-aware gating for practical trade-offs.

  Figure 2: Pareto trade-off analysis showing EAFT and its variants simultaneously maximize target and general-domain performance, unlike hard-masked or naive SFT methods.

Mechanistic Interpretations

Gradient magnitude visualizations show that, under SFT, Confident Conflict tokens elicit the largest destructive gradients, directly supporting the theoretical hypothesis. EAFT selectively suppresses gradients in the low-entropy/low-probability region without sacrificing adaptation efficacy on high-entropy (valid learning) tokens.

Loss dynamics further confirm that EAFT preserves cross-entropy on low-entropy tokens (model doesn't catastrophically override priors), while high-entropy tokens are rapidly minimized, ensuring domain knowledge injection proceeds unimpeded.

(Empirical token group visualizations, e.g., Figure 3, establish that high-entropy tokens correspond to valid reasoning or abstraction, while Confident Conflicts align with specific entities/noise.)

Figure 3: Token-level visualizations show high-entropy tokens relate to reasoning, while low-entropy Confident Conflict tokens are often specific entities or noise, thus supporting targeted suppression.

Robustness and Efficiency

The entropy-gating mechanism's form is demonstrated to be hyperparameter-insensitive; polynomial and Sigmoid gates (EAFT$^2$, EAFT$_\mathrm{Sigmoid}$, etc.) all perform equivalently well, reinforcing the universality of entropy-based modulation. Only strict hard masking introduces a deleterious effect on in-domain performance.

On computational overhead, the Top-$K$ entropy approximation ($K=20$) correlates near perfectly with the exact entropy (Pearson >0.999) while incurring negligible memory cost (Figure 4).

Figure 4: Top-$K$ entropy estimation ($K=20$) is both accurate (saturating correlation) and nearly cost-free in memory/compute.

Practical and Theoretical Implications

EAFT provides a domain-agnostic, scalable mitigation for SFT-induced catastrophic forgetting, requiring no auxiliary reward models, rollouts, or reference encoders. For practical deployment, EAFT can act as a drop-in replacement to standard SFT for continual and domain-specific adaptation where preservation of general abilities is paramount. The method's learnable trade-off frontier is essential for applications where both specialization and generalization are required (e.g., medical QA, mathematical reasoning, tool-augmented agents).

Theoretically, the work formalizes Confident Conflicts as the primary vector of destructive interference in supervised LLM adaptation, and demonstrates that predictive entropy has greater explanatory and mitigating power than probability or KL-divergence weighting.

Notably, the method is not indicated where explicit prior overwrites are required (e.g., knowledge editing or counterfactual instruction)—here the suppression of adaptation signal may backfire.

Future Directions

Possible future research includes:

• Integration with uncertainty calibration to distinguish between true knowledge and confident hallucination;
• Exploring cross-modal and multilingual extensions of EAFT, where entropy signal may be less reliable;
• Adapting EAFT for settings that require selective prior overwriting, such as continual correction of outdated or biased knowledge.

Conclusion

This paper dissects the mechanistic origins of SFT-driven catastrophic forgetting and provides a principled, efficient algorithmic remedy. EAFT, by leveraging token-level predictive entropy for dynamic loss gating, achieves robust domain adaptation while preserving general capability across a wide spectrum of model architectures and application areas. Its technical elegance and empirical rigor recommend it as a foundation for the next generation of continual learning methods in LLMs.