Language-Specific Steering Vectors

• Language-specific steering vectors are interventions that adjust hidden activations in LLMs to enforce correct target languages without retraining.
• They are derived using unsupervised methods on parallel corpora, computing average language representations and subtracting a universal content baseline.
• Empirical evaluations show significant reductions in language confusion with maintained or improved accuracy in tasks like QA and summarization across 18 languages.

Language-specific steering vectors are interventions in LLMs designed to causally modulate the output language of the model by manipulating internal representations. These vectors are typically derived without model retraining, operate at the level of hidden activations, and aim to directly address language confusion—the phenomenon where multilingual LLMs generate responses in an unintended language even when a target language is requested. The ReCoVeR approach exemplifies the current state of the art in isolating, constructing, and applying such steering vectors to robustly control output language across numerous languages and evaluation tasks (Sterz et al., 18 Sep 2025).

1. Motivation and Definition

Language-specific steering vectors are constructed to mitigate pervasive language confusion in multilingual LLMs, which manifests as inconsistent or incorrect output language selection in both monolingual and cross-lingual settings. As large LLMs acquire broader multilingual capacity, their internal representations increasingly conflate content and language signals. This leads to undesirable behaviors such as answering in the prompt’s language rather than the requested output language, or abrupt language switching within outputs. Language-specific steering vectors are designed to disentangle and selectively amplify the signal corresponding to the desired language—effectively “nudging” latent representations during inference to favor the target language without degrading semantic fidelity or core task performance.

Within ReCoVeR, a language-specific steering vector $r_\ell^{(i)}$ for language $\ell$ at layer $i$ is defined as the difference between the mean hidden representation of that language and a layerwise, language-agnostic content vector: $r_\ell^{(i)} = v_\ell^{(i)} - c^{(i)}$ where

$$v_\ell^{(i)} = \frac{1}{|D_\ell|} \sum_{x \in D_\ell} \sum_p h_p^{(i)}$$

and

$$c^{(i)} = \frac{1}{|L|} \sum_{\ell \in L} v_\ell^{(i)}$$

Here $D_\ell$ denotes the set of parallel corpus examples for language $\ell$, and $h_p^{(i)}$ is the activation at position $p$ in layer $i$.

2. Construction and Implementation

The derivation of language-specific steering vectors in ReCoVeR is unsupervised and exploits multi-parallel corpora such as FLORES-200. The process involves:

• Computing $v_\ell^{(i)}$, the average hidden state per layer for each target language.
• Averaging across all languages at that layer to obtain $c^{(i)}$, the layer’s content-agnostic mean.
• Subtracting $c^{(i)}$ from $v_\ell^{(i)}$ to obtain $r_\ell^{(i)}$, which is then L2-normalized for numerical stability.

During inference, these vectors are integrated as residual additions into the activations:

• Monolingual steering: $$h'^{(i)} = h^{(i)} + \alpha \cdot \frac{r_\mathrm{target}^{(i)}}{\|r_\mathrm{target}^{(i)}\|}$$
• Cross-lingual steering: $$h'^{(i)} = h^{(i)} + \alpha \cdot \frac{(r_\mathrm{target}^{(i)} - r_\mathrm{source}^{(i)})}{\|r_\mathrm{target}^{(i)} - r_\mathrm{source}^{(i)}\|}$$

The parameter $\alpha$ controls the steering strength. For improved expressivity, ReCoVeR+ introduces a trainable, low-rank intervention replacing the fixed addition: $$h'^{(i)} = h^{(i)} + A \cdot B\left([h^{(i)}; r_\mathrm{target}^{(i)}; r_\mathrm{source}^{(i)}]\right)$$ where $A$ and $B$ are learned matrices and $[\cdot;\cdot;\cdot]$ indicates concatenation.

A principal advantage is that new vectors can be added for additional languages without recomputing all previous vectors, due to the language-agnostic content baseline.

3. Evaluation and Results

Key benchmarks for evaluating language-specific steering vectors in ReCoVeR include:

• LCB (Language Confusion Benchmark): Assesses both line-level and word-level language accuracy.
• MultiQ: Provides QA tasks spanning 137 languages, measuring both correct language usage and answer accuracy.
• CrossSum: Consists of summarization tasks over more than 1,500 language pairs, allowing detailed examination of cross-lingual steering.

Empirical findings show:

• Significant reduction of language confusion when steering vectors are applied, in both monolingual and cross-lingual configurations.
• Retention or improvement of task performance (QA, summarization accuracy) compared to previous methods that often degrade these metrics.
• For instance, in CrossSum, ReCoVeR yields up to +50 percentage point improvements over baselines such as LSI for certain language pairs (see Table 1 in (Sterz et al., 18 Sep 2025)).
• The learned steering function (ReCoVeR+) can further boost accuracies, especially on models like Gemma, with gains up to +9pp over unsupervised arithmetic steering, while maintaining generalization to languages not seen in training.

These outcomes are robust across evaluation on 18 languages and three major LLMs, including Llama and Qwen 2.5.

4. Practical Implications and Scalability

The ability to deploy language-specific steering vectors at inference-time, with minimal per-sample compute overhead and no model or corpus retraining, enables a range of practical applications:

• Maintaining language compliance in chatbots, QA, translation, and summarization systems without introducing disruptive translation artifacts or costly prompt engineering.
• Fine-grained control for code-switching or enforcing user-specified output language regardless of prompt or corpus language distribution.
• Supports updating deployed systems to new languages with composable vector additions, as language vectors are constructed independently.

Because steering vectors are computed by simple corpus means and subtractions (or a compact, learnable function in ReCoVeR+), the approach remains practical as language coverage scales, and the code and data are public for broad experimentation.

5. Limitations and Future Research

The ReCoVeR framework acknowledges several limitations and directions:

• Incorporation of additional linguistic features, such as syntactic roles (from multilingual dependency parsers) or typological encodings (from databases like URIEL), may yield more precise or transferable language-specific steering vectors.
• The methodology currently relies on access to reliable parallel corpora for mean vector computation; minimum corpus requirements and performance on low-resource languages require further paper.
• While fixed and low-rank interventions are computationally efficient and empirically robust, further architectural refinements or learned, context-dependent steering routines may enhance applicability for highly code-switched or morphologically complex languages.
• Evaluation to date is limited to major open LLM backbones; transferability to other architectures or to non-text languages (such as programming languages) presents an open area (Sharma et al., 23 Jun 2025, Gurgurov et al., 30 Jul 2025).

6. Comparative Context and Broader Impact

Language-specific steering vectors, as realized in ReCoVeR, contrast with prior forms of language control—prompt engineering, explicit translation, or fine-tuning—by disentangling and manipulating purely latent, model-internal representations. Unlike some earlier methods that degrade downstream accuracy, ReCoVeR sustains or even enhances core task metrics while enforcing language constraints. The modular nature of the approach permits post-hoc adaptation and integration with future control methods, offering a foundation for ongoing research into causal and interpretable multilingual LLM interventions.

Approach	Corpus Required	Intervention	Task Fidelity	Generalization
ReCoVeR	Parallel, moderate	Vector add, learn	High	Across 18 languages
LSI	Parallel	Layer replacement	Modest-High	Lower (QA degraded)
Prompting	No	Prompt edit	Variable	High, less control
Fine-tuning	Task-specific	Weights update	High	Model-altering

This framework highlights the potential of explicit, language-specific steering vectors for scalable, interpretable, and task-preserving multilingual LLM control, with code and data publicly available for research and deployment (Sterz et al., 18 Sep 2025).