Conversational Domain Adaptation of IndicTrans2 across 21 Indic Languages via Experience Replay and Model Soups

Abstract: IndicTrans2 is the strongest open English to Indic translation system, but like most systems it is trained on general text and tends to sound stiff on casual, conversational input. We adapt IndicTrans2-1B to conversational register across all 21 Indic languages using only public data (OpenSubtitles, BPCC-H-Daily, Tatoeba). Plain fine-tuning improves conversational chrF but forgets the general domain (it drops 3.9 chrF on FLORES for Hindi). Mixing general data back into training (experience replay) and then averaging the fine-tuned weights with the base (model souping) removes that trade-off: the resulting model beats IndicTrans2-1B on conversational chrF in every one of the 21 languages (mean +6.2) while matching it on FLORES (mean change -0.17, all within 0.7 chrF). Paired bootstrap tests confirm the conversational gains are significant (p <= 0.004) and that FLORES is not significantly degraded. We are deliberate about scope: these are chrF gains, and a blind human plus multi-model LLM check does not confirm them as a perceived quality improvement, so we treat the conversational gain as largely a register match to the references rather than proof of better translation. The techniques are not new; the contribution is the honest, end-to-end study in the Indic conversational setting.

• The paper introduces conversational domain adaptation of IndicTrans2 using experience replay and model soups to enhance translation quality across 21 Indic languages.
• It mitigates catastrophic forgetting by mixing general and conversational data, resulting in significant chrF2 gains while preserving general-domain performance.
• Empirical evaluations indicate robust, statistically significant improvements with noted limitations in low-resource languages, setting a reproducible baseline for future studies.

Conversational Adaptation of IndicTrans2: Empirical Assessment Across 21 Indic Languages

Overview

The paper "Conversational Domain Adaptation of IndicTrans2 across 21 Indic Languages via Experience Replay and Model Soups" (2606.29024) provides a rigorous empirical study on adapting the multilingual NMT model IndicTrans2-1B to produce translations that better match the conversational register across 21 Indic languages. Relying solely on public conversational corpora, the study systematically applies and ablates domain adaptation techniques—specifically, experience replay and weight averaging via model soups. The primary metric examined is chrF2, with both conversational and general-domain performance measured, and paired significance tests provided.

Methods: Adaptation Strategies and Evaluation

The base model is ai4bharat/indictrans2-en-indic-1B (1.1B parameters), trained on the general-domain corpus BPCC-H-Wiki. For conversational adaptation, the paper utilizes three primary strategies:

1. Plain Fine-tuning (FT): Direct adaptation using only conversational data, such as OpenSubtitles, BPCC-H-Daily, and Tatoeba, identified and deduplicated, leading to around 294k conversational pairs.
2. Experience Replay (Mixed FT): Augmenting conversational adaptation by mixing general-domain data into training at a 1:1 ratio, mitigating catastrophic forgetting.
3. Model Soups: Weight-wise averaging of the base and fine-tuned models, sweeping the interpolation coefficient to maximize the conversational gain while maintaining generality.

Performance is quantitatively assessed using chrF2 on both conversational test sets and the FLORES-200 dev set. Qualitative evaluation includes human and LLM judgments on a subset.

Findings: Effects Across Domains and Languages

Catastrophic Forgetting and Its Mitigation

Plain fine-tuning in the conversational domain sharply increases conversational chrF2 but results in a notable decrease in FLORES general-domain performance, signifying catastrophic forgetting. Experience replay substantially abates this, and model souping (with an optimal $\alpha$) fully restores general-domain performance while preserving much, though not all, conversational gain.

Figure 1: On Hindi, plain fine-tuning lifts conversational chrF but drops FLORES below the base. Replay keeps most of the general quality, and averaging with the base restores FLORES while staying ahead on conversation.

Pan-Language Outcomes

In the aggregate across all 21 languages, the mixed model soup configuration consistently increases conversational chrF2 (mean +6.2), with general-domain chrF2 changes tightly constrained ($-0.17$ on average, all changes within a 0.7 band). For all languages, the adapted model produces output measurably closer to conversational references, a finding highly significant for languages with robust test sets such as Hindi, Tamil, and Malayalam.

Figure 2: Conversational chrF gain for all 21 languages. Blue marks the five languages with a hard subtitle test set; these are the gains to trust. The large grey gains come from easy in-domain test sets.

Figure 3: Each point is a language. Every language gains on conversation (all points above zero) while general quality stays inside a 0.7 chrF band around the base.

Trustworthy gains are associated with languages for which subtitle-style test sets are available; the largest apparent gains in some languages are known to be inflated by alignment between training/evaluation and the formulaic nature of their conversational references.

Statistical Significance

Bootstrap resampling confirms that conversational gains are highly significant (p ≤ 0.004) for languages with strict subtitle-based evaluation. FLORES score differences are statistically non-significant, indicating general quality is preserved. Notably, in certain cases (e.g., Malayalam), there is a statistically significant improvement in FLORES after adaptation, though mean changes remain small.

Analysis and Limitations

While the adaptation protocol is effective for achieving register-matching as measured by chrF2, critical caveats are reported:

• The chrF2 metric is sensitive to surface overlap and style, potentially inflating perceived gains when the reference shifts style but semantic quality may not improve.
• A qualitative human and LLM evaluation on Hindi does not confirm that these register-matching gains translate to actual improved translation quality.
• Gains are artifactually large in languages where conversational test sets mirror the training distribution.
• Languages with limited base model performance (e.g., Santali, Sanskrit, Kashmiri, Bodo) are bottlenecked by available parallel data, and adaptation yields marginal quality improvement.

The findings thus underscore the necessity for future controlled, multi-annotator human studies to substantiate any claims of translation quality improvement beyond metric alignment.

Practical and Theoretical Implications

This study demonstrates the practical feasibility of conversational domain adaptation for high-parameter, multilingual NMT models even in highly imbalanced, low-resource multilingual regimes using only public data. The established recipe—combining experience replay with model souping—effectively prevents catastrophic forgetting. However, the work reinvigorates awareness of the limitations of automatic metrics (chrF2, BLEU) in high-variance domains such as conversation, echoing broader discussions on metric-humanness correspondence in NMT.

Theoretically, these results reinforce the view that register adaptation does not equate to semantic quality improvement and demand future research integrating semantic, pragmatic, and human-centered metrics. On the system side, extending this adaptation approach to Indic-to-English and Indic-to-Indic translation remains an open technical problem, given the likely asymmetric effects of catastrophic forgetting across translation directions and resource imbalances.

Conclusion

Adaptation of IndicTrans2-1B using replay and model souping yields significant, statistically robust increases in conversational chrF2 across 21 Indic languages, with no detectable degradation in general-domain (FLORES) quality. Despite these metric gains, current evidence does not establish a human-perceived quality improvement. The methods are directly reproducible, and all code and models are public, providing a baseline for further empirical studies into conversational NMT in low-resource settings. The critical challenge ahead is to reliably link metric improvements to genuine enhancements in user experience and translation adequacy.