Luth: Efficient French Specialization for Small Language Models and Cross-Lingual Transfer

Abstract: The landscape of LLMs remains predominantly English-centric, resulting in a significant performance gap for other major languages, such as French, especially in the context of Small LLMs (SLMs). Existing multilingual models demonstrate considerably lower performance in French compared to English, and research on efficient adaptation methods for French remains limited. To address this, we introduce \textbf{Luth}, a family of French-specialized SLMs: through targeted post-training on curated, high-quality French data, our models outperform all open-source counterparts of comparable size on multiple French benchmarks while retaining their original English capabilities. We further show that strategic model merging enhances performance in both languages, establishing Luth as a new state of the art for French SLMs and a robust baseline for future French-language research.

• The paper demonstrates that specialized fine-tuning using the Luth-SFT dataset boosts French benchmark performance by up to +11.26% in small language models.
• The methodology combines high-quality data curation, advanced translation, and model merging (using LERP and SLERP) to optimize French while retaining English proficiency.
• The approach provides an efficient training pipeline for resource-constrained SLMs, paving the way for scalable language specialization and cross-lingual transfer.

Luth: Efficient French Specialization for Small LLMs and Cross-Lingual Transfer

Motivation and Context

The predominance of English-centric data and model development in LLMs has resulted in significant performance disparities for other major languages, notably French. This gap is especially pronounced in Small LLMs (SLMs), which are increasingly relevant for resource-constrained deployments and edge applications. Existing multilingual models, while broad in coverage, typically underperform on individual languages due to lack of targeted optimization. The Luth project addresses this deficiency by introducing a family of French-specialized SLMs, leveraging high-quality post-training data and model merging techniques to achieve state-of-the-art results in French while maintaining robust cross-lingual capabilities.

Luth-SFT Dataset Construction

A central contribution of the work is the Luth-SFT dataset, comprising 570k French instruction-response pairs (338M tokens). The construction pipeline integrates several stages:

• Data Collection: French samples are extracted from multilingual datasets (AYA, Smoltalk2, CroissantLLM) using langdetect for efficient language isolation.
• Data Translation and Generation: High-quality English instruction datasets (Tülu 3, OpenHermes) are translated into French using GPT-4o and Qwen3 32B. Responses are generated natively in French, not simply translated, to ensure idiomaticity and domain relevance.
• Filtering: A two-stage pipeline enforces strict linguistic and content criteria, removing programming/tool-calling content, logical errors, and grammatical issues.
• Scholar Subset: To address the scarcity of scientific French data, over 14,000 official exam PDFs (Baccalauréat, CPGE) are parsed and enriched, yielding 30,300 samples with detailed solutions across mathematics, physics, engineering, biology, and social sciences.

This dataset is notable for its emphasis on non-synthetic, expert-authored content, particularly in scientific domains, and its rigorous filtering for linguistic and instructional quality.

Model Training and Specialization

Luth models are based on two SLM families: Qwen3 (0.6B, 1.7B) and LFM2 (350M, 700M, 1.2B). Selection was based on baseline performance in French and English across math, general knowledge, and instruction-following tasks. Full fine-tuning (not LoRA) is performed on the Luth-SFT dataset using Axolotl, with FlashAttention and sequence packing to maximize GPU utilization (single NVIDIA H100, 80GB VRAM, 3 epochs, max sequence length 16,384). Hyperparameters are tuned per model, with AdamW optimizers and cosine schedulers.

Fine-tuning on Luth-SFT infuses models with domain-specific French vocabulary, improved reasoning, and general knowledge, but can slightly degrade non-French performance due to catastrophic forgetting.

Model Merging for Cross-Lingual Retention

To mitigate performance loss in English and enhance overall capabilities, model merging is employed using MergeKit. Both linear interpolation (LERP) and spherical linear interpolation (SLERP) are evaluated:

• LERP: $w = (1-\alpha) w_0 + \alpha w_1$
• SLERP: $$w = \frac{\sin((1-\alpha)\theta)}{\sin(\theta)} w_0 + \frac{\sin(\alpha \theta)}{\sin(\theta)} w_1$$, $\theta = \arccos(w_0 \cdot w_1)$

Empirical results indicate that simple merging methods (LERP, SLERP) yield stable improvements, with optimal coefficients selected per model (e.g., SLERP 0.7 for Luth-0.6B-Instruct). Merging recovers lost English performance and further boosts French results, confirming prior findings that merged models can outperform their components.

Evaluation Methodology

Evaluation is conducted on six benchmarks in both French and English: IFEval, Math500, GPQA-Diamond, MMLU, Arc-Challenge, and HellaSwag. LightEval is extended for French support and reasoning mode control. All results are averaged over three runs (temperature=0), with standardized system prompts.

Luth models consistently outperform all open-source SLMs of comparable size in French, with absolute average improvements up to +11.26% across benchmarks. English performance is preserved or slightly improved (+0.76% to +3.20%), indicating effective cross-lingual transfer. The approach is reproducible and transparent, with open-source code and data.

Implementation Details

Training Pipeline

• Frameworks: Axolotl for fine-tuning, FlashAttention for memory efficiency, MergeKit for model merging.
• Hardware: Single NVIDIA H100 (80GB VRAM).
• Hyperparameters: Model-specific, e.g., Qwen3-0.6B uses LR $2 \times 10^{-5}$, batch size 6, gradient accumulation 4, cosine scheduler, bfloat16 precision.
• Sequence Packing: Maximizes GPU utilization for long-context training.
• Filtering and Data Quality: Automated and LLM-assisted pipelines for linguistic/content validation.

Model Merging

• Method Selection: Empirical evaluation of LERP and SLERP, with coefficient tuning for stability.
• Framework: MergeKit, supporting multiple merging strategies.
• Catastrophic Forgetting Mitigation: Merging base and fine-tuned models preserves cross-lingual skills.

Benchmarking

• Framework: LightEval, extended for French and reasoning mode toggling.
• Metrics: Pass@1, averaged over three runs.
• Prompt Engineering: Standardized system prompts for consistency.

Numerical Results and Claims

• French SLMs: Luth models set new state-of-the-art results for SLMs (<2B params) on six French benchmarks.
• Cross-Lingual Transfer: Fine-tuning on French data and subsequent model merging not only preserves but can enhance English performance.
• Efficiency: The methodology is computationally tractable (single H100, 3 epochs), reproducible, and extensible to other languages.
• Data Quality: The Luth-SFT dataset, especially the Scholar subset, provides high-quality, non-synthetic French scientific content, addressing a major gap in existing resources.

Limitations and Future Directions

• Benchmark Coverage: Evaluation is limited to selected benchmarks; broader coverage is needed for comprehensive assessment.
• Language Retention: Stability in languages other than English and French is not fully evaluated.
• Model Scale: The approach is demonstrated only for SLMs (<2B params); scalability to larger architectures remains to be validated.
• Capability Scope: Luth-SFT does not cover tool use or code generation, which are increasingly important in LLM applications.

Future work should extend the methodology to larger models and additional languages, incorporate broader capability coverage (e.g., tool use, code generation), and further investigate optimal merging strategies.

Conclusion

Luth demonstrates that targeted post-training on high-quality, language-specific data, combined with strategic model merging, enables SLMs to achieve state-of-the-art performance in French without sacrificing cross-lingual capabilities. The approach is efficient, reproducible, and generalizable, providing a robust baseline for future research in language specialization and cross-lingual transfer for SLMs. The results suggest that similar strategies could be applied to other mid- and low-resource languages and larger model architectures, with potential for substantial gains in multilingual NLP.