Why Fine-Tuning Encourages Hallucinations and How to Fix It

Abstract: LLMs are prone to hallucinating factually incorrect statements. A key source of these errors is exposure to new factual information through supervised fine-tuning (SFT), which can increase hallucinations w.r.t. knowledge acquired during pre-training. In this work, we explore whether SFT-induced hallucinations can be mitigated using established tools from the continual learning literature, since they arise as a by-product of knowledge degradation during training. We propose a self-distillation-based SFT method that facilitates effective factual learning while minimizing hallucinations w.r.t. pre-existing knowledge by regularizing output-distribution drift. We also show that, in settings where new knowledge acquisition is unnecessary, suppressing factual plasticity by freezing parameter groups, can preserve task performance while reducing hallucinations. Lastly, we investigate the mechanism behind SFT-induced hallucinations through three hypotheses: capacity limitations, behavior cloning, and localized interference. Our experiments show that a main driver is interference among overlapping semantic representations, and that self-distillation succeeds by mitigating this interference.

• The paper identifies SFT-induced hallucinations as a byproduct of factual forgetting driven by representational interference from semantic overlap.
• The study benchmarks parameter freezing and self-distillation as practical methods to preserve pretrained knowledge while allowing new fact acquisition.
• Experimental results show that semantic overlap can cause up to 15% accuracy loss on known facts, which is reduced to about 3% using these mitigation strategies.

Fine-Tuning in LLMs as a Driver of Hallucinations: Mechanisms and Mitigation

Introduction

Supervised fine-tuning (SFT) is central to adapting LLMs to new domains and tasks, yet it frequently introduces factually incorrect generations—hallucinations—contradicting previously acquired (pretrained) knowledge. "Why Fine-Tuning Encourages Hallucinations and How to Fix It" (2604.15574) positions SFT-induced hallucinations as a manifestation of factual forgetting—a form of catastrophic interference characteristic of continual learning. The work presents a systematic analysis of SFT-driven knowledge updates, quantifies their impact, and traces the mechanistic origin of induced hallucinations to representational interference driven by semantic overlap between old and new facts. In addition, the study benchmarks and proposes mitigation techniques drawn from continual learning, notably self-distillation and parameter freezing, to address the stability–plasticity dilemma in factual storage.

SFT-Induced Factual Forgetting: Empirical Characterization

The paper employs controlled protocols to decouple task-format learning, factual knowledge acquisition, and factual retention. Utilizing SLiCK [gekhman2024doesfinetuningllmsnew], facts within datasets are partitioned by the LLM’s prior knowledge: HighlyKnown (consistently recalled), MaybeKnown, WeaklyKnown, and Unknown. Experiments train on mixtures of HighlyKnown (task-format only) and Unknown facts (novel knowledge injection), using a training-evaluation split enabling longitudinal quantification of "held-out" factual accuracy.

A characteristic pattern emerges: SFT on new facts rapidly boosts performance for Unknowns (demonstrating plasticity), but accuracy for held-out HighlyKnown facts consistently degrades—indicating substantial forgetting, often $>15\%$ absolute accuracy loss. Conversely, if fine-tuning is restricted to HighlyKnown facts (no new facts introduced), accuracy for held-out facts remains near pre-finetuning ceiling throughout.

Figure 1: Factual correctness on Known, Unknown, and Held-out facts during fine-tuning. Novel fact acquisition drives accuracy loss (“factual forgetting”) on previously held-out facts, distinguishing task-format adaptation from knowledge interference.

This provides robust evidence that SFT-induced hallucinations are not a generic fine-tuning artifact, but instead stem specifically from integrating new factual entries into the parametric model.

Parameter Group Freezing: Suppressing Plasticity When Stability is Desired

To dissociate factual plasticity from hallucination emergence, the paper explores selective parameter freezing, leveraging prior findings that FFN layers primarily encode factual associations [geva2021transformerfeedforwardlayerskeyvalue], while attention layers contribute to task/general behavior [dar2023analyzingtransformersembeddingspace]. If only attention parameters are updated (FFN frozen), models can still learn the required output style and QA task protocol, but fail to acquire new factual knowledge. Critically, this regime exhibits a near-elimination of forgetting: accuracy on held-out facts is nearly as high as training on HighlyKnown facts alone.

Figure 2: Across training epochs and experiment variants, factual correctness for different knowledge groupings reveals selective parameter freezing sharply limits forgetting.

This strategy has direct practical significance for SFT scenarios focused on task adaptation, privacy, or alignment—circumstances where preservation of pretrained factual knowledge is prioritized over incorporating new facts. The central finding: restricting factual plasticity significantly curtails SFT-induced hallucinations, affirming that unwanted drift is mediated by factual updates, not task-format adaptation per se.

Mitigating Forgetting During Knowledge Injection: Self-Distillation

Where new knowledge integration is indispensable, the stability–plasticity tradeoff arises: how to encode new facts with minimal harm to previously learned facts. The study adapts continual learning’s self-distillation framework [li2017learningforgetting, shenfeld2026selfdistillationenablescontinuallearning] to SFT. Here, a snapshot of the model is frozen after an initial epoch (post–task adaptation, pre-factual drift) to serve as the "teacher." Fine-tuning then proceeds under a regularized loss, penalizing KL divergence between student and teacher output distributions at each token.

Figure 3: For Llama-3.1-8B and Qwen2.5-7B, self-distillation maintains high accuracy on previously known facts while still acquiring new knowledge, representing a dual improvement over regular SFT.

The result is a dramatic reduction in hallucinations: forgetting on held-out facts drops from $\sim$15% to $\sim$3% across architectures and training conditions, while factual plasticity (learning new facts) remains near that of standard SFT. Notably, $\ell_2$ weight regularization toward the teacher parameters only reduces forgetting at the cost of suppressing factual acquisition, highlighting the need for output-level (rather than parameter-level) regularization.

Mechanistic Origin: Representational Interference from Semantic Overlap

To adjudicate between hypotheses (behavioral cloning induced by SFT, representational capacity limits, or local interference), the paper fine-tunes models on large numbers of synthetic facts, varying only the "entity key" structure: (1) semantic keys—novel names fabricated by recombining tokens from real entities, hence representationally proximal to existing entries, and (2) UUID keys—random identifiers with no overlap with the pretrained lexicon.

When new facts are injected with semantic overlap (i.e., keys similar to existing ones), forgetting on held-out facts increases sharply with the number of new facts. In contrast, when keys are UUIDs—even for up to $10^6$ new entries—forgetting remains at or below $4\%$, despite full factual acquisition. Formally, the degradation in held-out accuracy scales with representational overlap, not simply with fact count or generic task difficulty.

Figure 4: Construction of synthetic entities—either semantically overlapping with real entities or orthogonal (UUID)—allows direct manipulation of representational interference.

Further, analysis of representational drift and geometric structure demonstrates that SFT on semantically overlapping keys causes reorganization and translation of entity representations, directly supporting a structural interference account. Self-distillation, in contrast, sharply limits such drift and preserves both factual retention and acquisition rates.

Figure 5: Across hidden-state and output space drift metrics, standard SFT produces substantial representational reorganization for semantically overlapping new facts; self-distillation preserves the geometry of pretrained entity representations.

Discussion and Implications

The results suggest that SFT-induced hallucinations are a tractable, predictable by-product of representational interference among semantically similar facts—a mechanistic diagnosis that challenges explanations relying on global capacity constraints or SFT-driven behavior cloning. Accordingly, techniques targeting interference at the representation or output level (self-distillation, strategic module freezing) can significantly attenuate these side effects without fundamentally impeding either knowledge acquisition or task-format adaptation.

Practical implications are immediate: for SFT scenarios requiring high reliability with respect to pretrained knowledge, freezing fact-storage modules (e.g., FFN) or deploying self-distillation offers substantial hallucination reduction at minimal to no cost in desired task performance. The findings expand the scope of continual learning literature to direct applications in LLM fine-tuning pipelines and suggest that representation-level interference analysis should play a central role in SFT protocol design.

Theoretically, this prompts a reframing of post-pretraining fine-tuning: behavioral side effects observed during SFT (e.g., hallucinations) are best understood, and remediated, through the lens of selective interference—the local dynamics of distributed knowledge storage—rather than as emergent and irreducible tradeoffs of large-scale optimization.

Conclusion

This study offers a rigorous, mechanistic account of why SFT increases hallucination rates in LLMs, tying the phenomenon to catastrophic factual forgetting and, more specifically, representational interference among semantically similar facts. Mitigation is accessible: factual plasticity can be selectively suppressed (via parameter freezing), or new knowledge can be integrated with minimal forgetting (via output-level regularization such as self-distillation). Fine-tuning protocols that engage this structural understanding effectively decouple knowledge acquisition and hallucination rates, charting a path toward robust, reliable LLM deployment in continual and task-adaptive scenarios.

Figure 6: Schematic: SFT-induced hallucinations can be conceptualized as factual forgetting due to representational interference in parameter space; parameter freezing and self-distillation provide two orthogonal mitigation axes.

The unified perspective is that SFT-induced hallucinations are a local and preventable outcome of how new knowledge is encoded and distributed in pretrained parameter space, rather than an unavoidable side effect of downstream adaptation.

---

References

Key cited works relevant to mechanisms and mitigation include [li2017learningforgetting], [shenfeld2026selfdistillationenablescontinuallearning], [geva2021transformerfeedforwardlayerskeyvalue], [dar2023analyzingtransformersembeddingspace], [zhu2025teachlargemultimodalmodels], [gekhman2024doesfinetuningllmsnew]. See (2604.15574) for full experimental procedures and additional analyses.