Is Biomedical Specialization Still Worth It? Insights from Domain-Adaptive Language Modelling with a New French Health Corpus

Abstract: LLMs have demonstrated remarkable capabilities across diverse domains, yet their adaptation to specialized fields remains challenging, particularly for non-English languages. This study investigates domain-adaptive pre-training (DAPT) as a strategy for specializing small to mid-sized LLMs in the French biomedical domain through continued pre-training. We address two key research questions: the viability of specialized continued pre-training for domain adaptation and the relationship between domain-specific performance gains and general capability degradation. Our contributions include the release of a fully open-licensed French biomedical corpus suitable for commercial and open-source applications, the training and release of specialized French biomedical LLMs, and novel insights for DAPT implementation. Our methodology encompasses the collection and refinement of high-quality French biomedical texts, the exploration of causal language modeling approaches using DAPT, and conducting extensive comparative evaluations. Our results cast doubt on the efficacy of DAPT, in contrast to previous works, but we highlight its viability in smaller-scale, resource-constrained scenarios under the right conditions. Findings in this paper further suggest that model merging post-DAPT is essential to mitigate generalization trade-offs, and in some cases even improves performance on specialized tasks at which the DAPT was directed.

• The paper demonstrates minimal aggregate gains from domain-adaptive pre-training in large French biomedical models, with notable benefits only for smaller setups.
• The study introduces the PARCOMED corpus, a fully open, compliant French medical text collection that enhances reproducibility and accessibility in medical NLP.
• The paper shows that SLERP merging effectively balances specialization and generalization, mitigating performance loss in general tasks observed after domain-specific training.

Domain-Adaptive Pre-Training for French Biomedical LLMs: Critical Evaluation and Corpus Creation

Introduction

The paper "Is Biomedical Specialization Still Worth It? Insights from Domain-Adaptive Language Modelling with a New French Health Corpus" (2604.06903) critically examines the efficacy and necessity of domain-adaptive pre-training (DAPT) for LLMs in the French biomedical domain. The study is situated at the intersection of resource scarcity for specialized models in non-English languages and the rising prevalence of foundation models trained on vast general corpora. The authors construct and release the PARCOMED corpus—a fully open and commercially licensed French biomedical text collection—and systematically evaluate the practical outcomes of domain specialization, continued pre-training on Qwen3 models, and subsequent model merging via SLERP interpolation.

PARCOMED Corpus: Design and Implications

The PARCOMED corpus aggregates 892k documents, predominantly scientific articles, regulatory documents, clinical case reports, and medication leaflets. Its uniqueness lies in scrupulous licensing, permitting both research and commercial downstream use, which addresses pressing issues of reproducibility, accessibility, and legal compliance in French medical NLP. The corpus curation process involved thorough text normalization and document-level organization, ensuring data quality for robust model training and public benchmarks. This contribution stands as a foundational resource for subsequent work in French medical language modeling with open, compliant datasets.

Experimental Protocol: DAPT and Model Merging

Domain specialization in the study is pursued via continued pre-training (PDAPT) of four Qwen3 base models (0.6B, 1.7B, 4B, 8B) using causal language modeling objectives. All models were trained on the tokenized PARCOMED corpus for 4,320 update steps (~2.53 epochs), with regular checkpointing for performance tracking. The evaluation harness included multi-domain QA benchmarks (MMLU, MMLU-Pro-X), in both French and English, comprising medical and general subject tasks, providing granular analysis of specialization vs. generalization trade-offs.

A pivotal methodological step is the application of Spherical Linear Interpolation (SLERP) for merging checkpoints post-DAPT with base models. This technique is motivated by prior evidence that merging can recover generalization capacity lost to over-specialization and in some instances enhance domain-specific performance even beyond the specialized checkpoint.

Empirical Results: Specialization vs. Generalization

The core findings reveal minimal aggregate impact of PDAPT, especially as model size increases. Performance improvements in French medical tasks (FR-MEDICAL) are isolated primarily to the smallest model, with only marginal and non-monotonic gains in larger variants. Notably, base models (especially Qwen3-8B-Base) already outperform several domain-specialized alternatives and general competitors across medical QA tasks.

Figure 1: Progression of evaluation scores on the four task groups over PDAPT training.

DAPT results in statistically significant decreases in general-domain tasks, underscoring the risk of over-specialization. However, when SLERP merging is applied, no statistically significant decreases remain in any task group, while specialized medical performance persists, particularly for small and mid-sized models. This demonstrates that model merging is not merely a tuning step but an essential safeguard against the domain-generalization trade-off.

Figure 2: Comparison of group-level averages for base vs. specialized models and SLERP merge; confidence intervals are shown.

Aggregate evidence shows that the benefits of DAPT are diminishing for highly capable base models, contrary to accepted wisdom in prior literature citing robust gains for domain adaptation [gururangan_dont_2020]. The current findings align with recent critical reassessments of medical specialization in LLMs [jeong_medical_2024], especially in resource-constrained and low-resource language settings.

Numerical and Contradictory Claims

Quantitative evaluation emphasizes the following:

• 56 task/domain comparisons show only 8 statistically significant improvements; only 1 is negative for FR-MEDICAL.
• No significant generalized performance losses post-merging, while medical specialization improvements are retained.
• Larger models benefit less from DAPT, confirming that high-capacity generalist LLMs already subsume much of the medical knowledge that specialization attempts to impart.
• SLERP merging can sometimes result in superior performance compared to both the base and specialized checkpoints, contradicting naive assumptions about linear trade-off between specialization and generalization.

Practical and Theoretical Implications

The implications are manifold:

• Corpus Release: The PARCOMED corpus and the Qwen3-PDAPT models provide reproducible benchmarks and practical tools for French medical NLP, opening avenues for commercial applications.
• Model Adaptation Strategies: DAPT is shown to be best utilized in resource-constrained or small-scale setups and for subdomain-focused applications. For broad-domain tasks and larger models, model merging is imperative.
• Evaluation Paradigm Shift: The results call for a reevaluation of the standardized domain adaptation pipeline, suggesting that targeted, granular specialization and iterative merging may yield more substantial gains than traditional broad DAPT.
• Future Directions: Theoretical extensions include exploring granular-topic DAPT, supervised fine-tuning overlays, alternative merging techniques (e.g., DARE, TIES [yadav_ties_2023; ueda2026merging]), and expanding evaluations beyond academic QA to real-world clinical document processing.

Conclusion

This paper provides rigorous evidence that domain-adaptive pre-training offers diminishing returns for large, generalist LLMs, especially in the French biomedical domain. The introduction of the PARCOMED corpus and Qwen3-PDAPT models fills a substantial gap in openly-available, legally compliant resources for medical NLP in French. Model merging emerges as a necessary intervention to reconcile domain specialization with generalization, particularly important for small and mid-sized models. The study’s detailed evaluations inform practical deployment choices and encourage further research into more granular specialization, alternative adaptation strategies, and comprehensive real-world evaluation settings.

See original paper for full citations.