ReCoVeR: Reducing Language Confusion

• The paper introduces a method for extracting language-specific steering vectors that enable precise control over the output language in LLMs.
• ReCoVeR uses both arithmetic (unsupervised) and learnable interventions to reduce language confusion while maintaining or enhancing task performance across benchmarks.
• The approach scales efficiently by adding new languages through simple averaging, making it ideal for multilingual applications like chatbots, translation, and summarization.

ReCoVeR (REducing language COnfusion in VEctor Representations) is a methodology designed to mitigate the pervasive phenomenon of "language confusion" in multilingual LLMs, where generated text may inadvertently switch to a language other than that specified by the user or the context. ReCoVeR operates by extracting and leveraging language-specific "steering vectors" from the model's continuous internal representations. These vectors allow precise control over the output language during inference, reducing undesired language mixing while maintaining task performance, even in complex cross-lingual scenarios. The core innovations and extensive evaluation of the approach are detailed in "ReCoVeR the Target Language: Language Steering without Sacrificing Task Performance" (Sterz et al., 18 Sep 2025).

1. Motivation and Problem Definition

Language confusion in multilingual LLMs manifests as partial or complete output in an unintended language, regardless of explicit user requests or the prompt language. This problem is prominent in both monolingual settings (where the prompt and answer should match languages) and cross-lingual setups (where the answer language differs from the prompt language). It undermines the reliability of LLMs for multilingual tasks, including chatbots, question answering, and summarization.

Traditional language steering methods often degrade task performance or require extensive retraining whenever new languages are added, making them impractical for expanding language support. ReCoVeR aims to resolve these limitations by introducing a lightweight, scalable approach to extracting and applying language-specific control within the LLM architecture.

2. Extraction and Isolation of Language Steering Vectors

ReCoVeR isolates the language-specific signal from hidden representations using multi-parallel corpora (such as FLORES-200). For each language $l$ in a set $L$, the method computes layer-wise language vectors $v_l^{(i)}$ by averaging the hidden states $h_p^{(i)}$ over all positions $p$ and samples $x$ in language $l$: $$v_l^{(i)} = \frac{1}{|D_l|}\sum_{x \in D_l}\sum_p h_p^{(i)}$$ To separate language from content, a language-agnostic content vector is derived: $c^{(i)} = \frac{1}{|L|}\sum_{l \in L} v_l^{(i)}$ The language-specific steering vector for language $l$ in layer $i$ is then obtained by subtracting the content vector from the raw average: $r_l^{(i)} = v_l^{(i)} - c^{(i)}$ These steering vectors encapsulate the distinctive features that drive the model to behave as a specific language, thus providing a direct, interpretable means of intervention at each transformer layer.

3. Steering Mechanisms: Unsupervised and Learnable Interventions

ReCoVeR supports two primary steering mechanisms:

A. Arithmetic (Unsupervised) Steering

• Monolingual Language Control (Mono-LC): The hidden state is modified by adding the L2-normalized, scaled steering vector of the target language:

$$\hat{h}^{(i)} = h^{(i)} + \alpha \cdot \frac{r_{\text{target}}^{(i)}}{\|r_{\text{target}}^{(i)}\|}$$

where $\alpha$ is a scaling hyperparameter.

• Cross-Lingual Language Control (Cross-LC): To convert a prompt in a source language into a response in a target language, the hidden state is adjusted using the normalized difference of target and source steering vectors:

$$\hat{h}^{(i)} = h^{(i)} + \alpha \cdot \frac{r_{\text{target}}^{(i)} - r_{\text{source}}^{(i)}}{\|r_{\text{target}}^{(i)} - r_{\text{source}}^{(i)}\|}$$

B. Learnable Steering Function (ReCoVeR+)

• For additional flexibility and sample efficiency, a low-rank, trainable intervention layer is introduced:

$$\hat{h}^{(i)} = h^{(i)} + A \cdot B\left[h^{(i)}; r_{\text{target}}^{(i)}; r_{\text{source}}^{(i)}\right]$$

where $A$ and $B$ are trainable matrices, and $[\cdot]$ denotes concatenation. $B$ is typically initialized as zero to ensure model stability before training.

An implementation detail includes restoring the norm of the hidden state after steering to prevent model collapse, maintaining activation consistency.

4. Benchmarking and Evaluation Methodology

ReCoVeR is rigorously evaluated across three multilingual benchmarks:

• Language Confusion Benchmark (LCB): Assesses the proportion of output lines (Line-Level Pass Rate, LPR) and words (Word-Level Pass Rate, WPR) in the specified target language, for both monolingual and cross-lingual tasks.
• MultiQ: A multi-parallel question answering dataset spanning dozens of languages, measuring both language fidelity and QA accuracy.
• CrossSum: Used for cross-lingual summarization over hundreds of language pairs.

The language steering interventions are empirically shown to:

• Substantially increase both LPR and WPR metrics in outputs across 18 languages.
• Retain or improve task performance metrics such as QA accuracy.
• Scale gracefully: additional language vectors can be added without recomputing existing ones.

Below is a summary table of the principal evaluation dimensions:

Benchmark	Metric(s)	Impact of ReCoVeR
Language Confusion	LPR, WPR	Reduced language confusion
MultiQ (QA)	Accuracy, Fidelity	Maintained/improved task performance
CrossSum (Summarization)	Language Fidelity	Significant language steering gains

5. Comparison to Earlier Language Steering Approaches

Prior steering methods—such as Language Steering Interventions (LSI)—had notable limitations:

• Required recomputation of steering vectors upon addition of new languages due to mutually dependent representations (often via classification probes).
• Sometimes resulted in decreased task accuracy when maximizing language fidelity.
• Were not uniformly applicable to both monolingual and cross-lingual scenarios.

ReCoVeR directly addresses these drawbacks:

• Language representations are computed independently via averaging, facilitating post-hoc addition of languages.
• Both unsupervised and trainable steering interventions robustly preserve (and occasionally improve) downstream accuracies.
• The arithmetic and learnable fusion strategies generalize across monolingual and cross-lingual cases, matching or surpassing baseline performance.

6. Practical Implications and Scalability

ReCoVeR's lightweight intervention scheme has substantial implications for practical systems:

• In multilingual chatbots and assistants, ReCoVeR ensures the model remains within the requested language, eliminating disruptive unintended language switches.
• For machine translation and cross-lingual summarization, the approach enforces output in the correct language, even when language markers are ambiguous or absent in the prompt.
• The method's scalability—adding new languages with only simple averaging—eliminates the need for costly retraining or probe recomputation with corpus expansion.
• In low-resource scenarios, ReCoVeR's cross-lingual steering can be applied even without extensive language-specific data, as the intervention is constructed from parallel corpora irrespective of task supervision.

7. Future Directions

The paper suggests several avenues for further research:

• Expanding ReCoVeR to finer-grained steering, such as dialect, register, or domain adaptation, by isolating corresponding steering vectors.
• Integrating steering mechanisms into other architectural components, such as decoding strategies or output formatting layers.
• Extending the methodology to hybrid cross-modal systems and code-mixed scenarios where language boundaries are fluid.

Further study may address optimal strategies for combining unsupervised and trainable intervention layers and exploring the interaction between steering vectors and other representational axes (e.g., style, topic).

---

In conclusion, ReCoVeR is an explicit, corpus-driven and highly modular approach for reducing language confusion in multilingual LLMs. By isolating and deploying language-specific steering vectors at the hidden representation level, it enhances output language fidelity in diverse settings—monolingual, cross-lingual, and low-resource—while preserving or improving task accuracy compared to baseline and earlier steering methods (Sterz et al., 18 Sep 2025). This methodology is suited to scalable and real-world deployments where robustness against language confusion is critical.