Multilingual Latent Reasoning in LRMs

• Multilingual latent reasoning refers to LRMs’ capability to process inputs, generate reasoning steps, and mix languages within a single chain-of-thought.
• LRMs typically pivot to a high-resource language like English, revealing performance disparities when reasoning in low-resource languages.
• Empirical metrics such as mixing entropy and latent reasoning scores are used to quantify internal language transfer, alignment, and accuracy trade-offs.

Multilingual latent reasoning in large reasoning models (LRMs) refers to the phenomenon where models process inputs, generate reasoning steps, and reach conclusions across multiple languages—sometimes within a single chain-of-thought (CoT)—in a manner that reveals nontrivial internal preferences, transfer mechanisms, and performance limitations. This field has recently seen rapid development, with research converging on several core principles: most LRMs reason internally in a high-resource "pivot" language (typically English), even when prompted and required to respond in a different language; their ability to represent and utilize multilingual representations in latent space is highly dependent on both model scale and pretraining distribution; and language-mixing, or code-switching, emerges as a potentially strategic, rather than incidental, mode of internal reasoning. Systematic study of these mechanisms has enabled the design of new training paradigms, reward functions, and architectural modules that explicitly align, constrain, or monitor the latent reasoning processes, with the goal of closing performance gaps and enhancing interpretability across languages.

1. Formalization and Metrics of Multilingual Latent Reasoning

Multilingual latent reasoning is most commonly formalized in the context of CoT-capable LRMs. Let a reasoning trace $T$ consist of tokens $t_1,\ldots,t_n$ produced during the CoT. Detect the language $L_i \in \mathcal{L}$ of each segment using a classifier (e.g., fastText), and compute the discrete usage distribution $$p_l = \frac{\sum_{i=1}^{n} 1[L_i = l]}{\sum_{l'}\sum_{i=1}^{n} 1[L_i = l']}$$ over $\mathcal{L}$. Language mixing occurs when $\exists\,l_1\neq l_2 : p_{l_1}>0, p_{l_2}>0$. The degree of mixing is quantified by entropy $H(T) = -\sum_{l\in\mathcal{L}} p_l \log p_l$, with $H(T)=0$ for pure traces and higher $H(T)$ for balanced mixing (Wang et al., 20 May 2025).

Latent reasoning is operationalized by probing hidden states at intermediate steps—well before the explicit answer is verbalized—using logit lens projections to measure the "readiness" of the model to produce the correct answer token; metrics include stepwise pass@$k$, area under the truncation accuracy curve (AUTC), and a faithfulness-weighted latent reasoning score (LRS) (Liu et al., 6 Jan 2026).

Consistency, compliance, and faithfulness are evaluated via: (1) compliance rate (fraction of CoT sentences in the expected language); (2) substitution consistency (how well the trace in language $\ell_1$ transfers to $\ell_2$); (3) drop in accuracy upon CoT truncation/error injection, to assess whether the CoT is used as support or merely as rationalization (Zhao et al., 10 Oct 2025).

2. Empirical Patterns: Language Mixing, Resource Effects, and Performance

Language mixing is prevalent when models are prompted in languages with under-represented scripts (e.g., Arabic, Hindi, Japanese); CoT steps then increasingly blend Latin (English) or Han (Chinese) tokens (Wang et al., 20 May 2025). The mixing entropy $H(T)$ correlates with both task difficulty and subject area: harder puzzles and STEM subjects induce more mixing, e.g., DeepSeek-R1-70B yields $H(\text{STEM}) \approx 0.28$ vs $H(\text{Humanities}) \approx 0.10$.

Performance drops sharply in low-resource languages when models are forced to reason monolingually; in reasoning tasks, "prefill English" outperforms "prefill native" by up to 31.5 percentage points in Swahili (MATH-500), with similar performance gaps on MMMLU and other general knowledge tasks (Tam et al., 23 May 2025). Conversely, on cultural or behavioral benchmarks, native-language reasoning is sometimes beneficial.

Large reasoning models consistently default to reasoning in high-resource "hub" languages in latent space, regardless of the input or prompt language, as measured by the hubness index ($H(M;\text{EN}) \approx 0.75-0.90$) for major LRMs (Tam et al., 23 May 2025).

3. Mechanisms: Latent Space Alignment, Transfer Neurons, and Pivoting

Representational analyses reveal that models process multilingual inputs through a pipeline: initial layers map input into a shared semantic latent space (typically English-centric), intermediate layers perform reasoning, and output layers translate the result back into the target language (Tezuka et al., 21 Sep 2025). The "Transfer Neurons Hypothesis" posits that specific neurons in the MLP module act as encoders and decoders, transferring the hidden state across language-specific and shared latent spaces; empirical ablation of these neurons sharply impairs accuracy and disrupts the expected latent transitions (Tezuka et al., 21 Sep 2025).

English-pivoted CoT training, where input is in a low-resource language but reasoning is explicitly performed (and learned) in English, robustly aligns the model's latent space and improves performance on low-resource tasks (+28.3 pp on Irish AIME2024 relative to native CoT-only training) (Tran et al., 2 Apr 2025). In this regime, representation retrieval and alignment experiments confirm that cross-language pairs (e.g., Irish/English) remain tightly clustered throughout reasoning, so long as the CoT remains in the English latent manifold (Tran et al., 2 Apr 2025).

4. Control and Optimization of Reasoning Language

Constrained decoding (script control) can exploit a model’s latent preference for specific writing systems. Masking logits during reasoning so that only tokens in the preferred script (e.g., Latin or Han) are generated improves accuracy in under-resourced languages—up to +115% relative gain for Hindi prompt with Latin script vs. native script (Wang et al., 20 May 2025). Language-level constrained decoding and targeted representation engineering (e.g., via auxiliary losses to bias internal states) are proposed for even finer control.

Probe-guided or selective translation, where understanding failures are detected by probes on hidden states (e.g., mmBERT, Prober MLP), allows for efficient hybrid systems: only 20% of inputs are translated to English while recovering 85% of the accuracy gap, substantially increasing efficiency over full-translation baselines (Kang et al., 31 Oct 2025).

Language mixing, when appropriately guided (via lightweight probes trained to predict the benefit/harm of language switches), can yield further accuracy improvements: on MATH500 (Chinese prompts), unconstrained mixing outperforms monolingual decoding by +5.6 pp, and adaptively controlled mixing increases accuracy by up to +6.25 pp (Li et al., 21 Jul 2025). This demonstrates that code-switching is a strategic, model-internal behavior, not accidental noise.

5. Training and Reward Design: RL with Verifiable or Semantic Rewards

Reinforcement learning with verifiable rewards (RLVR), where task reward reflects only final answer correctness, induces language mixing and cross-lingual collapse—the gradual drift of CoT reasoning into the dominant pretraining language (Park et al., 6 Jun 2025). This collapse is rapid, especially in low-resource languages (−97.6 pp in Ukrainian within 250 updates), and is largely irreversible; mitigation via language-consistency reward (fraction of target-language tokens) is effective but imposes a 5–10 pp cost in accuracy (Park et al., 6 Jun 2025).

More advanced semantic reward schemes, such as pivot-based reinforcement learning with semantically verifiable rewards (PB-RLSVR), leverage a high-resource "pivot" model to generate reference traces against which multilingual model outputs are aligned using a combination of answer precision (COMET), multilingual/translated embedding similarity, and format compliance (Faisal et al., 29 Sep 2025). PB-RLSVR increases multilingual reasoning scores by 10.2–16.4% and shrinks the English–non-English gap from ~14 pp to ~2 pp. This continuous, dense reward directly regularizes models towards semantic equivalence with the high-resource pivot.

Consistency-enhanced reinforcement-based techniques (e.g., M-Thinker) employ hard language-consistency penalties and cross-lingual thinking alignment (CTA) rewards measured by LLM judges that compare CoT step alignment between languages. On MMATH and PolyMath, this approach produces near-100% language-consistent chains and large gains in accuracy, even for out-of-domain languages (Zhang et al., 8 Oct 2025).

6. Task-Specific Phenomena and Limitations

Latent reasoning, as measured by stepwise answer formation and logit lens probes, is strong in high-resource languages for relatively easy problems (MGSM: AUTC$_1$≈0.62 for English/Chinese, AUTC$_1$≈0.22 for Swahili/Telugu), but collapses in low-resource languages and very hard tasks (AIME: LRS$_1$<$0.05$ uniformly) (Liu et al., 6 Jan 2026). Cosine similarities of layer-wise hidden states confirm that correct predictions in high-resource languages remain tightly aligned to English latent traces, whereas mid/low resource languages sometimes reside in intermediate or alternative subspaces; correct instances for these cases align with English only at the moment of correct prediction (Liu et al., 6 Jan 2026).

Faithfulness of CoT traces is highly variable: in high-resource languages, truncating the final third of the CoT drops accuracy by 30–40 pp, but in low-resource languages the drop is often negligible, indicating the traces are less functionally used (Zhao et al., 10 Oct 2025). Error injection further confirms stronger surface copying in low-resource languages.

7. Strategies for Equitable and Efficient Multilingual Reasoning

The dominant trend is explicit alignment of internal reasoning pipelines to a high-resource latent manifold (almost always English), but recent work explores several remedies:

• Reward shaping to enforce or regularize language consistency, at the expense of some accuracy (Park et al., 6 Jun 2025, Zhang et al., 8 Oct 2025).
• Data augmentation and instruction tuning on high-quality CoT exemplars in low-resource languages (Tam et al., 23 May 2025, Zhao et al., 10 Oct 2025).
• Language-adaptive decoding—dynamically prefilling anchor tokens or adapters to maintain performance parity across languages (Tam et al., 23 May 2025).
• Architectures that expose the language switching decision as a controllable policy output, enabling the model to "choose" its latent reasoning mode (Li et al., 21 Jul 2025).
• Transfer via minimal parameter bridges (LangBridge) between a multilingual encoder and frozen high-quality English reasoning model, yielding strong zero-shot gains with only English training data, driven by language-agnostic prompt embeddings (Yoon et al., 2024).
• Exemplar-enhanced RL, where a high-resource LLM provides both reference traces and quality judgments, scaling deep-reasoning translation ability to 90+ directions using efficient reward propagation (Wang et al., 19 May 2025).

High-level, the field continues to grapple with the optimal balance between efficiency, accuracy, faithfulness, and language equity. Scaling high-quality multilingual CoT data, designing hybrid reward signals, and developing more interpretable latent state monitoring are active areas of ongoing research.

---

References:

• "Language Mixing in Reasoning LLMs: Patterns, Impact, and Internal Causes" (Wang et al., 20 May 2025)
• "Large Reasoning Models Are (Not Yet) Multilingual Latent Reasoners" (Liu et al., 6 Jan 2026)
• "Why Do Multilingual Reasoning Gaps Emerge in Reasoning LLMs?" (Kang et al., 31 Oct 2025)
• "EfficientXLang: Towards Improving Token Efficiency Through Cross-Lingual Reasoning" (Ahuja et al., 30 Jun 2025)
• "Scaling Test-time Compute for Low-resource Languages: Multilingual Reasoning in LLMs" (Tran et al., 2 Apr 2025)
• "The Transfer Neurons Hypothesis: An Underlying Mechanism for Language Latent Space Transitions in Multilingual LLMs" (Tezuka et al., 21 Sep 2025)
• "Can LLMs reason over extended multilingual contexts?" (Hengle et al., 17 Apr 2025)
• "Cross-lingual Collapse: How Language-Centric Foundation Models Shape Reasoning in LLMs" (Park et al., 6 Jun 2025)
• "A Comprehensive Evaluation of Multilingual Chain-of-Thought Reasoning" (Zhao et al., 10 Oct 2025)
• "Aligning Multilingual Reasoning with Verifiable Semantics from a High-Resource Expert Model" (Faisal et al., 29 Sep 2025)
• "Think Natively: Unlocking Multilingual Reasoning with Consistency-Enhanced Reinforcement Learning" (Zhang et al., 8 Oct 2025)
• "The Impact of Language Mixing on Bilingual LLM Reasoning" (Li et al., 21 Jul 2025)
• "Language Matters: How Do Multilingual Input and Reasoning Paths Affect Large Reasoning Models?" (Tam et al., 23 May 2025)
• "LangBridge: Multilingual Reasoning Without Multilingual Supervision" (Yoon et al., 2024)
• "ExTrans: Multilingual Deep Reasoning Translation via Exemplar-Enhanced Reinforcement Learning" (Wang et al., 19 May 2025)