Code-Switching Reveals Language Anchoring in Multilingual LLMs

Abstract: Multilingual LLMs (MLLMs) are increasingly expected to handle Code-Switched (CS) inputs, yet mixing languages frequently degrades performance relative to source- or target-language monolingual counterparts. To understand this degradation, we use grammar-forced CS as a controlled diagnostic setting for locating CS representations relative to their source and target counterparts. We introduce Anchor Bias, a geometric measure that quantifies language anchoring, whether a CS hidden state aligns closer to its source or target language counterpart. Across diverse MLLMs, Anchor Bias reveals a consistent grammar-frame effect: source-framed CS stays source-anchored, whereas target-framed CS shifts target-ward and shows larger Question Answering (QA) degradation. Motivated by this representational pattern, we propose CANVAS (Contextual Anchor-based Neural Vector Alignment Steering), an inference-time intervention that extracts a source-side canvas from the input and softly steers target-language hidden states toward the source anchor during prefill. CANVAS consistently recovers QA F1 across MLLMs and CS conditions, showing that internal anchoring signals provide an actionable target for mitigating CS inference failures.

• The paper introduces anchor bias, a geometric metric based on cosine similarities that quantifies the shift in hidden representations during code-switching.
• It presents CANVAS, an inference-time intervention that dynamically steers token hidden states to mitigate QA degradation in target-framed code-switched inputs.
• Empirical evaluations across diverse MLLMs reveal consistent source-anchoring patterns and demonstrate significant QA improvements, highlighting practical mitigation for multilingual deployments.

Code-Switching Reveals Language Anchoring in Multilingual LLMs

Introduction and Motivation

Multilingual LLMs (MLLMs) are increasingly used in environments where code-switching (CS)—the interleaving of multiple languages within a single interaction—is routine. The paper explores why code-switched inputs frequently result in performance degradation compared to monolingual peers, specifically in the context of code-switched question-answering. Unlike prior work that treats CS as a form of lexical noise, the authors treat CS as a grammatically-structured linguistic phenomenon and introduce a controlled diagnostic setting leveraging grammar-forced CS pairs. This framework enables explicit contrast between source-framed and target-framed code-switching, mapped onto the internal representation space of MLLMs.

Figure 1: Locating code-switched inputs in the multilingual representation space. Source- and target-framed CS variants preserve different grammatical structures for the same information need, plotted alongside their matched monolingual anchors.

Anchor Bias: Geometric Measure for Language Anchoring

The paper introduces "Anchor Bias", a geometric metric quantifying the proximity of the code-switched hidden state to either the source or target language anchor. The anchor bias is computed as the difference in cosine similarity of the code-switched representation to source and target monolingual references. To ensure validity across diverse models with varying hidden-state anisotropies, similarity normalization is performed for robust cross-model comparison.

Empirical analysis across MLLMs (Aya-Expanse-8B, Qwen3.5-{4B, 9B, 27B}, Llama3.1-70B, Mixtral-8x7B, Phi-3.5-MoE, among others) reveals a consistent pattern: source-framed CS prompts are closer to the source anchor, while target-framed CS prompts move sharply toward the target anchor. The degree of this geometric shift correlates with observed QA degradation—target-framed CS produces larger drops in F1 score.

Figure 2: Per-layer anchor bias across model depth. Raw $\mathrm{AB}_\ell$ on question-content tokens, averaged across examples, plotted against layer index for each model.

Critically, this anchor bias is robust across transformer depth: source-framed CS trajectories remain source-anchored in ~76% of layers, while target-framed CS is anchored in only ~36% of layers.

CANVAS – Contextual Anchor-based Neural Vector Alignment Steering

Motivated by the frame-dependent anchoring phenomenon, the authors propose CANVAS, an inference-time intervention targeting internal representation dynamics. CANVAS decomposes the input at the token level (using lightweight language tagging), computes layerwise source and target anchor vectors, and dynamically steers target-language token hidden states toward the source canvas during prefill at upper layers where language anchoring is most pronounced.

The intervention strength $\alpha$ is adaptively set per instance, decreasing for source-heavy inputs and increasing for target-heavy ones, based on a context-sensitive alignment score.

Figure 3: Latent displacement along the SRC--TGT axis. CANVAS projects token states onto a 2D subspace where the horizontal axis captures the SRC--TGT direction, arrows show directional shift induced by CANVAS.

Figure 4: Adaptive intervention strength. CANVAS assigns larger interpolation strength alpha to more target-heavy inputs, measured by lower source-token ratio rho.

Empirical evaluation on multiple MLLMs and CS conditions demonstrates that CANVAS consistently improves QA F1—especially for target-framed CS where anchor bias is strongest and base QA degradation is largest.

MoE Routing Behavior and Representation Alignment

For mixture-of-experts (MoE) models, the authors complement representation-level analysis with routing diagnostics. Expert routing profiles derived from token-wise expert selection show that grammar-framed CS inputs induce routing behavior aligning with their respective language anchors. There is strong positive correlation ($\rho=+0.703$) between routing bias and anchor bias, reinforcing the claim that sparse expert allocation mirrors representation-space anchoring.

Figure 5: Routing bias tracks representation anchor bias. Each point visualizes SRC-vs-TGT expert-routing bias (horizontal axis) and representation anchor bias (vertical axis) for example-level CS inputs.

Model-wise and Condition-wise CANVAS Movement Diagnostics

Model-specific analyses confirm the intervention’s effect: CANVAS shifts code-switched representations toward the source anchor in every evaluated architecture and CS condition. The signed projection ratio of the CANVAS-induced displacement remains positive for $>90\%$ of examples across models, with substantial vector traversal along the SRC--TGT axis.

Figure 6: Source-ward representation movement. Signed projection ratio eta of CANVAS displacement onto the SRC--TGT axis ($\eta>0$).

Figure 7: Model-wise CANVAS representation movement. Arrows connect base and CANVAS centroids for grammar-forced CS conditions; CANVAS induces consistent source-directed shifts across diverse models.

Generalization and Extensions

The CANVAS mechanism generalizes to Spanish as source language, argues against an English-centric explanation, and produces similar improvements when applied to multi-turn CS dialog (PingPong benchmark) and retrieval-augmented QA. The source-ward steering is effective for both bilingual and trilingual CS combinations as well as non-QA tasks like topic classification.

Strong Numerical Results and Claims

• CANVAS consistently increases QA F1 across all evaluated MLLMs and CS conditions, with the most substantial improvements in target-framed CS.
• Across conditions and models, source-framed CS remains source-anchored (anchor bias $>0$), while target-framed CS is sharply target-anchored (anchor bias $<0$), producing a mean gap of $+0.566$ in upper-layer anchor bias.
• MoE models’ expert routing profiles strongly align with representation anchor bias, confirming convergent signal across model families.
• Statistically significant improvements are obtained in CANVAS vs. base CS answering, with robust FDR-controlled positive paired lifts across models and conditions.
• Frontier models (DeepSeek-V3.2, Gemini 2.5, GPT-5) also exhibit code-switching degradation and source-language preference, indicating that scaling alone does not eliminate the anchoring behavior.

Practical and Theoretical Implications

The findings expose internal structural biases in MLLMs with respect to code-switched input processing. Representations are not balanced bilingual spaces but are dynamically anchored to the grammatical frame, raising questions on universality and robustness in multilingual deployments. CANVAS demonstrates that internal anchoring signals are actionable, enabling lightweight, inference-time mitigation without parameter updates. The geometric approach provides new diagnostic axes for probing multilingual model internal structure and suggests steering-based interventions as promising tools for addressing CS-induced degradation.

From a theoretical perspective, the results reinforce the notion that LLMs develop structural linguistic representations aligned to dominant grammatical frames, not arbitrary lexical mixing. The practical implication is that cross-lingual steering can be implemented efficiently, generalizes to non-English sources, and transfers to multi-turn and RAG tasks.

Speculation on Future Developments

Activation steering applied to cross-lingual phenomena may be extended to multi-anchor scenarios, supporting more than two languages and capturing richer discourse-level CS dynamics. Representation-level alignment could be harnessed for broader tasks beyond QA, including generation and contextual reasoning in multilingual dialog agents. Improvements in token-level language detection and anchor vector construction may further enhance adaptive steering interventions. Given the limitations noted regarding compression of multilingual dynamics to a source--target axis, future work should aim to model more granular syntactic, semantic, and pragmatic cues in CS processing.

Conclusion

This work establishes layerwise geometric anchoring as a mechanism underlying code-switched performance degradation in MLLMs. By diagnosing anchor bias and introducing CANVAS as a runtime representation correction, the paper demonstrates both actionable mitigation and new probes into internal language dynamics. The results bear significant implications for robust multilingual model deployment and future activation steering research (2606.19668).