Countering Catastrophic Forgetting of Large Language Models for Better Instruction Following via Weight-Space Model Merging

Abstract: LLMs have been adopted in the medical domain for clinical documentation to reduce clinician burden. However, studies have reported that LLMs often "forget" a significant amount of instruction-following ability when fine-tuned using a task-specific medical dataset, a critical challenge in adopting general-purpose LLMs for clinical applications. This study presents a model merging framework to efficiently adapt general-purpose LLMs to the medical domain by countering this forgetting issue. By merging a clinical foundation model (GatorTronLlama) with a general instruct model (Llama-3.1-8B-Instruct) via interpolation-based merge methods, we seek to derive a domain-adapted model with strong performance on clinical tasks while retaining instruction-following ability. Comprehensive evaluation across medical benchmarks and five clinical generation tasks (e.g., radiology and discharge summarization) shows that merged models can effectively mitigate catastrophic forgetting, preserve clinical domain expertise, and retain instruction-following ability. In addition, our model merging strategies demonstrate training efficiency, achieving performance on par with fully fine-tuned baselines under severely constrained supervision (e.g., 64-shot vs. 256-shot). Consequently, weight-space merging constitutes a highly scalable solution for adapting open-source LLMs to clinical applications, facilitating broader deployment in resource-constrained healthcare environments.

• The paper demonstrates that weight-space model merging via SLERP effectively mitigates catastrophic forgetting, balancing clinical expertise with instruction adherence.
• The study shows that SLERP-merged models achieve high performance in under an hour on a single GPU, reducing dependency on additional labeled data.
• The findings indicate improved downstream task performance in both zero-shot and few-shot settings, highlighting the method's resource and label efficiency.

Countering Catastrophic Forgetting in Clinical LLMs via Weight-Space Model Merging

Introduction

The proliferation of LLMs for clinical documentation automation has exposed limitations in domain adaptation protocols that rely on supervised fine-tuning (SFT) or continuous pretraining (CPT). Notably, catastrophic forgetting—the loss of instruction-following capability during domain adaptation—poses an obstacle for LLM deployment in high-constraint medical environments. The paper "Countering Catastrophic Forgetting of LLMs for Better Instruction Following via Weight-Space Model Merging" (2604.01538) addresses this by evaluating parameter-space model merging as a non-gradient-based alternative for mitigating forgetting while enhancing instruction adherence in medical LLMs.

Model Merging for Dual Capability Retention

The study proposes merging a clinically adapted LLM (GatorTronLlama) and a general instruction-tuned LLM (Llama-3.1-8B-Instruct) using both Linear Interpolation and Spherical Linear Interpolation (SLERP) of weights. Model merging is explicitly positioned as a Pareto optimization problem where the goal is to maximize clinical proficiency and preserve instruction-following capabilities without additional gradient-based training or labeled data.

Evaluation protocols include two axes: (1) a series of standard medical knowledge benchmarks (MMLU medical subsets, MedMCQA, MedQA, PubMedQA) and (2) the IFEval instruction-following benchmark. The merged checkpoints are further used as initialization for downstream supervised fine-tuning in multiple clinical generation tasks (radiology/discharge summarization, problem list generation, dialogue-to-note generation), with both zero-shot and few-shot SFT settings.

Key Findings and Numerical Results

• Resource Efficiency: Model merging generates multiple high-performing checkpoints in less than an hour on a single GPU, as opposed to >18h of gradient-based SFT on a clinical instruction dataset. No additional labels or RLHF data are required.
• Zero-Shot Performance:
  • GatorTronLlama outperforms Llama-3.1-8B-Instruct in medical benchmarks (Medical Avg: 0.6896 vs. 0.6845), but instruction following is severely degraded (IFEval: 0.2244 vs. 0.5253).
  • SLERP-merged models, at an optimal interpolation weight ($t=0.4$), boost clinical score to 0.7039 (a 1.43% increase over GatorTronLlama) while recovering 98.3% of the instruction-following capability of Llama-3.1-8B-Instruct. This demonstrates that model merging achieves a superior trade-off than either original model.
  • Performance trajectories are non-monotonic w.r.t. the interpolation ratio, indicating a nontrivial interaction between general and domain-specific weight spaces. Excessive interpolation toward the instruction model leads to performance degradation in specialized domains.
• Downstream Task Performance:
  • In full-data SFT, SLERP-merged checkpoints outperform both clinical and instruction-tuned baselines on all five evaluated tasks. For instance, SLERP attains a post-SFT composite score of 0.2950 (MIMIC-BHC), 0.4099 (ACI-Bench), and 0.6471 (IU-Xray), consistently leading across tasks.
  • For few-shot learning, SLERP merged models exhibit superior sample efficiency. On radiology summarization (IU-Xray), 64-shot SLERP (score 0.5167) matches the 256-shot clinical foundation performance, and at 512 shots, SLERP achieves the highest score (0.5281).
  • Similar trends are observed in problem list generation, with SLERP (64-shot: 0.2587) surpassing the 512-shot baseline model (0.2581), demonstrating a reduction in label complexity to reach competitive clinical generation capability.

Theoretical and Practical Implications

The findings substantiate that weight-space interpolation—especially via SLERP—effectively mitigates catastrophic forgetting induced by task-specific SFT in the clinical domain. The resulting model does not merely average the representational spaces but exploits a non-monotonic, Pareto-optimal region that enables the transfer of instruction adherence without diluting specialized clinical knowledge.

Practically, this enables scalable adoption of open-source LLMs in clinical production settings with minimal resource expenditure and without reliance on large, expensive, domain-specific instruction datasets or RLHF. This is particularly significant for applications in healthcare systems where computational resources, labeled data, and expert RLHF datasets are limited or unavailable.

Theoretically, the observed non-monotonic performance as a function of interpolation weight suggests complex interactions within parameter space, warranting further exploration into geometric and initialization-dependent factors in weight-space merging. The established protocol provides a blueprint for similar interventions in other high-stakes or data-scarce domains.

Limitations and Future Directions

While extensive, the evaluation is restricted to automatic metrics; clinical deployment mandates human adjudication for accuracy and minimization of hallucination risks. Additionally, grid search over interpolation weights is coarse; future work should investigate denser or adaptive strategies and extension to larger model scales. Further, the impact of merging multiple models or more complex weight-space conflict resolution schemes remains unexplored.

Conclusion

This study demonstrates that model merging via interpolation in weight-space constitutes a scalable, resource-efficient, and effective strategy for maintaining instruction-following capability in domain-adapted LLMs, mitigating catastrophic forgetting in clinical applications. These results imply direct applicability for healthcare LLM deployment under resource-constraint and instruction-data scarcity. The work further motivates more granular investigation into the structure of LLM weight space and transferability properties, with implications far beyond the medical domain.