Multilingual Reasoning Gym Overview

Updated 4 July 2026

Multilingual Reasoning Gym is a benchmark suite that procedurally generates verifiable reasoning problems in multiple languages under controlled conditions.
It offers parallel, language-aligned instances with adjustable difficulty to diagnose language-specific reasoning failures.
The framework incorporates interventions like translation and language mixing to train models for robust multilingual reasoning and cultural sensitivity.

Searching arXiv for papers on multilingual reasoning gym and related multilingual reasoning benchmarks/methods. Multilingual Reasoning Gym denotes both a specific procedural benchmark suite and a broader research program for evaluating, diagnosing, and improving reasoning in many languages under controlled conditions. In its concrete form, it is an extension of Reasoning Gym that procedurally generates verifiable reasoning problems across 14 languages, with 94 tasks, parallel instance generation, adjustable difficulty, and direct usability for Reinforcement Learning with Verifiable Rewards (RLVR) (Dobler et al., 11 Mar 2026). In the wider literature, the term also functions as a blueprint for multilingual mathematical reasoning, controlled reasoning-language experiments, bilingual logical puzzles, abstract grouping games, linguistic olympiad tasks, and compact cross-lingual audits of accuracy, safety, cultural sensitivity, and inference cost (Ko et al., 5 Jan 2025, Tam et al., 23 May 2025).

1. Scope, definition, and benchmark families

The core idea of a multilingual reasoning gym is to move beyond single-language accuracy reporting and to provide parallel, language-controlled environments in which the same underlying problem can be posed across languages, answered with verifiable criteria, and analyzed for language-specific failure modes. In the procedural formulation, the underlying problem state is held fixed while only the rendered surface form changes by language, making cross-lingual comparisons structurally aligned (Dobler et al., 11 Mar 2026). In the diagnostic and training-oriented formulations, the gym additionally exposes interventions on input language, reasoning language, translation pathways, answer language, and chain-of-thought structure (Ko et al., 5 Jan 2025, Kang et al., 31 Oct 2025).

A large part of the literature uses the “gym” notion to unify otherwise separate assets: benchmarks, data-generation pipelines, intervention mechanisms, and evaluation dashboards. HRM8K is presented as a diagnostic gym for Korean mathematical reasoning (Ko et al., 5 Jan 2025). “Language Matters” frames multilingual input and reasoning-path control as a multilingual reasoning gym across MMMLU, MATH-500, CulturalBench-Hard, and LMSYS-Toxic (Tam et al., 23 May 2025). MultiZebraLogic is described as a multilingual gym for logical reasoning across nine Germanic languages (Bruun et al., 5 Nov 2025). Yi-Sang and Language-Mixed CoT are proposed as a foundation for a gym for language-specific reasoning in Korean (Son et al., 5 Oct 2025). mmPISA-bench treats compact, official human translations and matched machine translations as a reusable multilingual reasoning gym across 43 languages (Sapenov et al., 5 Jun 2026).

Component	Languages / scale	Primary focus
Multilingual Reasoning Gym	94 tasks, 14 languages	Procedural generation, verifiable rewards, RLVR (Dobler et al., 11 Mar 2026)
HRM8K	8,011 English–Korean parallel math problems	Comprehension vs. reasoning bottlenecks (Ko et al., 5 Jan 2025)
MMATH	374 problems, 10 languages, 3,740 examples	Complex multilingual mathematical reasoning and off-target generation (Luo et al., 25 May 2025)
MultiZebraLogic	2,048 puzzles per language in nine Germanic languages	Logical reasoning with clue types and red herrings (Bruun et al., 5 Nov 2025)
mmPISA-bench	25 questions, 43 languages, 2,150 prompts	Controlled multilingual reasoning, translation type, and cost (Sapenov et al., 5 Jun 2026)
GlobalGroup	Five languages plus English translations	Abstract word-grouping reasoning and modality bias (Guerra-Solano et al., 15 Oct 2025)

This family resemblance suggests that the field treats “gym” less as a single dataset than as a standardized experimental regime: aligned multilingual instances, explicit control of reasoning conditions, and metrics that isolate reasoning quality from translation artifacts.

2. Procedural architecture and verifiable-reward design

The formal Multilingual Reasoning Gym extends the original English-only Reasoning Gym by keeping the generator and verifier language-independent while localizing the rendering templates. Its architecture comprises a task set $T = \{t_1,\dots,t_{|T|}\}$ , a language set $L = \{\ell_1,\dots,\ell_{|L|}\}$ , a generator function $R_t$ for each task, a verifier $V_t(x,\hat y)$ , and a natural-language rendering template $\psi_{t,\ell}$ for each task-language pair (Dobler et al., 11 Mar 2026). The instance mapping is defined as

$f: T \times L \times Seed \to I,$

where aligned seeds make problems parallel across languages. Given task $t$ , language $\ell$ , and seed $s$ , the generator samples the abstract instance $x$ , renders the prompt $L = \{\ell_1,\dots,\ell_{|L|}\}$ 0, computes the ground-truth answer, and yields a fully instantiated problem (Dobler et al., 11 Mar 2026).

The design goal is unlimited instance generation. Because the surface form is template-based, the same seed can be used to generate parallel prompts in English, German, Chinese, Thai, Bengali, Swahili, and the other supported languages while preserving the exact logic of $L = \{\ell_1,\dots,\ell_{|L|}\}$ 1 and $L = \{\ell_1,\dots,\ell_{|L|}\}$ 2 (Dobler et al., 11 Mar 2026). This is central for cross-lingual generalization studies: it prevents problem-selection effects from masquerading as language effects.

Translation and validation are handled as a pipeline rather than as an afterthought. In the 14-language Multilingual Reasoning Gym, 10 of the original 104 tasks are omitted because they depend on English lexicon or ASCII-art conventions, and six tasks that embed English data are retained with a note that the data remains English (Dobler et al., 11 Mar 2026). Draft translations are produced with Claude Sonnet 4, refined for slot consistency and terminology, and manually validated by two annotators per language for the 10 high-resource languages. For Bengali, Telugu, and Swahili, the system relies solely on the LLM pipeline to maximize coverage (Dobler et al., 11 Mar 2026).

Difficulty is an explicit control parameter. Each task defines a continuous $L = \{\ell_1,\dots,\ell_{|L|}\}$ 3 that scales internal size or structure via a task-specific difficulty function $L = \{\ell_1,\dots,\ell_{|L|}\}$ 4; the reported operating points are $L = \{\ell_1,\dots,\ell_{|L|}\}$ 5 for easy and $L = \{\ell_1,\dots,\ell_{|L|}\}$ 6 for hard (Dobler et al., 11 Mar 2026). Empirically, moving from $L = \{\ell_1,\dots,\ell_{|L|}\}$ 7 to $L = \{\ell_1,\dots,\ell_{|L|}\}$ 8 can reduce model performance by up to 15–30 percentage points, which validates procedural generation as a controllable stress-testing mechanism rather than a static translated benchmark (Dobler et al., 11 Mar 2026).

The same framework is directly usable for RLVR. Each episode is a single problem instance whose state is the rendered prompt, whose action space is token sequences, and whose reward is binary:

$L = \{\ell_1,\dots,\ell_{|L|}\}$ 9

with $R_t$ 0 given by the verifier (Dobler et al., 11 Mar 2026). Because the reward is deterministic and verifiable, the framework can function both as an evaluation suite and as a training environment.

3. Diagnostic use: separating language understanding from reasoning

A defining contribution of multilingual reasoning gyms is their use as diagnostic instruments. HRM8K was built precisely to test whether multilingual performance gaps in Korean mathematical reasoning arise from weak reasoning or from failures in understanding the Korean input (Ko et al., 5 Jan 2025). The benchmark contains 8,011 fully parallel English–Korean math problems: 1,428 Korean School Math problems and 6,583 translated prior-set problems from GSM8K, MATH, Omni-MATH, and MMMLU (Ko et al., 5 Jan 2025).

The central intervention compares three prompting paths under the factorization

$R_t$ 1

where $R_t$ 2 is the question language and $R_t$ 3 the chain-of-thought language (Ko et al., 5 Jan 2025). The three tested modes are K2K, K2E, and E2E. Across Qwen2.5 and Llama-3, the average gain from K2K to K2E is only +1%, whereas the gain from K2E to E2E is about +10%, with the excerpted table reporting +11% for E2E relative to K2K (Ko et al., 5 Jan 2025). The reported conclusion is that the primary bottleneck is comprehension of non-English inputs rather than the reasoning mechanism itself.

This diagnosis generalizes in a related but broader form in “Language Matters,” which evaluates eight languages—English, Chinese, Spanish, Russian, Japanese, Korean, Telugu, and Swahili—and explicitly steers the reasoning language by prefilling a language-specific anchor phrase after the > token (Tam et al., 23 May 2025). The reported base behavior is a “hub tendency”: models default internally to high-resource hub languages, primarily English or Chinese, regardless of the input language (Tam et al., 23 May 2025). On MATH-500, English-prefilled reasoning outperforms native-language reasoning for all reported non-English languages, with absolute gains ranging from +1.9 points for Chinese to +31.5 points for Swahili (Tam et al., 23 May 2025). On MMMLU, the same English-hub advantage persists, with gains up to +13.6 points for Swahili (Tam et al., 23 May 2025).

A further diagnostic layer appears in the residual-based gap attribution framework of “Why Do Multilingual Reasoning Gaps Emerge in Reasoning LLMs?” which decomposes multilingual gaps into understanding, answer extraction, and residual factors via stage-wise interventions and a two-player Shapley decomposition (Kang et al., 31 Oct 2025). The same work defines understanding failure labels and compares detection methods, reporting on Qwen3-4B and Polymath-Low that supervised detectors perform best: the mmBERT detector reaches balanced accuracy about 85.2%, F1 about 65.9%, and PR-AUC about 72.6%, while a hidden-state prober reaches balanced accuracy about 85.5%, F1 about 63.7%, and PR-AUC about 75.7% (Kang et al., 31 Oct 2025). This pushes the gym concept beyond benchmarking toward causal diagnosis of multilingual errors.

A common misconception is that multilingual reasoning gaps are simply evidence of universally weaker reasoning in non-English languages. The cited studies do not support that interpretation. Their reported evidence instead isolates understanding failures and reasoning-language routing as major contributors to the gap (Ko et al., 5 Jan 2025, Kang et al., 31 Oct 2025).

4. Training and intervention regimes

Once diagnostic evidence identified understanding and routing as major bottlenecks, several works converted the gym into a training and intervention platform. UST—“Understand, Solve, and Translate”—is the most explicit example. It fine-tunes a model to parse Korean in English, solve in English, and translate the solution back to Korean in a single pass (Ko et al., 5 Jan 2025). The three stages are: an English understanding stage with bullet-point context, an English solution stage with full chain-of-thought and numeric answer, and a Korean translation stage (Ko et al., 5 Jan 2025). Training uses standard autoregressive cross-entropy over the concatenated stages:

$R_t$ 4

The synthetic training set contains approximately 130K high-quality three-stage examples (Ko et al., 5 Jan 2025).

On HRM8K, UST fine-tuned on Qwen2.5-7B-Instruct reaches 50.43% accuracy, compared with 39.52% for K2K prompting, 41.29% for K2E prompting, and 51.12% for E2E prompting (Ko et al., 5 Jan 2025). The reported gains are +10.91% absolute over K2K and a narrowing of the E2E–K2K gap from 11.60% to 0.69% (Ko et al., 5 Jan 2025). The same work reports that UST consumes about 66% of the tokens of a prompting-based multi-step inference version and is preferred by humans in 87.3% of head-to-head comparisons (Ko et al., 5 Jan 2025).

Selective Translation addresses the same diagnosis from a different angle. Rather than always translating, it detects understanding failure and translates only when needed (Kang et al., 31 Oct 2025). The decision rule generates a partial reasoning trace, computes a detector score, and prepends an English understanding prefix only if the input is predicted to be “not understood” (Kang et al., 31 Oct 2025). On Polymath-Low with Qwen3-4B, the reported base average across non-English languages is 81.1%, selective translation reaches 88.0% with translation used on about 19.3% of inputs, and full translation reaches 89.4% with 100% translation usage (Kang et al., 31 Oct 2025). On MMLU-ProX-Lite, the corresponding averages are 72.7% for base, 74.3% for selective translation, and 76.5% for full translation, with translation used on about 20.8% of inputs under selective translation (Kang et al., 31 Oct 2025).

Language-Mixed CoT extends the intervention space by alternating English and Korean within the reasoning trace. Its formal constraint requires the Korean-character ratio

$R_t$ 5

to satisfy $R_t$ 6 (Son et al., 5 Oct 2025). The reported rationale is that English segments carry most logical scaffolding while Korean segments preserve terminology, quotations, and cultural context (Son et al., 5 Oct 2025). Yi-Sang curates 5.79M native-Korean prompts, 3.7M long reasoning traces, and a 260K high-yield subset; training on this data yields an average improvement of +18.6 points across nine benchmarks for 4B–35B models, while KO-REAson-35B reaches an overall average score of $R_t$ 7 and ranks first on 5/9 benchmarks (Son et al., 5 Oct 2025).

MMATH shows a related trade-off in complex multilingual mathematics. It reports an off-target generation problem in which reasoning-trained models often think or answer in English or Chinese even when prompted in another language (Luo et al., 25 May 2025). On a fine-tuned Qwen2.5-32B-Instruct, EN-Think—native-language question and answer, but English chain-of-thought—achieves 66.72% average accuracy with 97.61% Answering LCR, outperforming Native-Think at 61.46% and EN-SFT at 62.38% (Luo et al., 25 May 2025). This suggests that “reason in English, answer in target” is not merely a prompting heuristic but a trainable regime with measurable accuracy–consistency trade-offs.

5. Task diversification beyond mathematical reasoning

Although mathematical reasoning has been the dominant proving ground, the multilingual reasoning gym has diversified into logical, linguistic, educational, and abstract-reasoning tasks. MultiZebraLogic generates zebra puzzles in nine Germanic languages—English, Danish, Swedish, Norwegian Bokmål, Norwegian Nynorsk, Faroese, Icelandic, German, and Dutch—with two grid sizes, 14 clue types, and eight red-herring types (Bruun et al., 5 Nov 2025). Its generation engine samples a random solution matrix, adds clues until the solution is unique, prunes unnecessary clues, injects exactly $R_t$ 8 red herrings, and shuffles clue order (Bruun et al., 5 Nov 2025). It reports that going from zero to five red herrings on 4×5 puzzles reduces o3-mini puzzle-level accuracy by $R_t$ 9, while a single red herring has only a marginal effect of about $V_t(x,\hat y)$ 0 (Bruun et al., 5 Nov 2025).

mmPISA-bench shifts the focus to compact, high-quality multilingual reasoning derived from official PISA items (Sapenov et al., 5 Jun 2026). It contains 25 multiple-choice questions—14 reading items and 11 mathematics items—with official human translations and matched machine-translated counterparts in 43 languages, for 2,150 prompts in total (Sapenov et al., 5 Jun 2026). It evaluates two proprietary LLMs across five reasoning-effort configurations, with 107,500 API calls overall (Sapenov et al., 5 Jun 2026). At high effort on human translations, Claude reaches 96.6% and GPT reaches 95.7% (Sapenov et al., 5 Jun 2026). The benchmark is particularly notable for showing that machine-translated questions do not degrade accuracy relative to official human translations, and that some languages are simultaneously more expensive and less accurate in token-cost terms (Sapenov et al., 5 Jun 2026).

GlobalGroup addresses abstract reasoning rather than formulaic or knowledge-heavy tasks. Inspired by the New York Times “Connections,” it asks models to partition a word pool into topic-linked groups across English, Spanish, Chinese, Hindi, and Arabic, with native-language and English-translation versions for each non-English language (Guerra-Solano et al., 15 Oct 2025). It defines three explicit difficulty metrics—group count, semantic cohesion via ARI, and word-overlap score—and combines them into an integrated difficulty metric $V_t(x,\hat y)$ 1 whose Spearman correlation with F1 is reported as $V_t(x,\hat y)$ 2 (Guerra-Solano et al., 15 Oct 2025). The cross-lingual findings show a broad English-modality advantage for many models, but also exceptions: for GPT-4, Chinese native F1 is 0.971 versus 0.935 for zh-en; for GPT-3.5, Chinese native F1 is 0.927 versus 0.836 for zh-en (Guerra-Solano et al., 15 Oct 2025).

Linguistic puzzle work supplies another branch of the gym. “Inductive Linguistic Reasoning with LLMs” organizes modeLing and LINGOLY through a two-stage analogical prompting pipeline in which one model generates auxiliary demonstrations from typologically related languages and a second model performs deduction (Ramji et al., 2024). On modeLing, analogical prompting improves GPT-4o from 59.2% to 66.9% exact match and Llama-3.1-405B from 65.8% to 71.7% (Ramji et al., 2024). On LINGOLY, the same strategy improves performance across all problem types and difficulty levels, including a reported jump on Round 2 Rosetta problems from 12% to 41% (Ramji et al., 2024).

This task diversification matters because it prevents the gym from collapsing into a math-only evaluation stack. A plausible implication is that multilingual reasoning should be treated as a family of capabilities whose bottlenecks differ across mathematical, logical, cultural, linguistic, and abstract tasks.

6. Biases, trade-offs, and open research questions

The literature converges on a strong but qualified finding: multilingual reasoning systems exhibit hub-language bias, yet English is not uniformly optimal for every objective. In “Language Matters,” input-language reasoning degrades performance on reasoning tasks but can benefit cultural tasks, while safety behavior is language-specific (Tam et al., 23 May 2025). On CulturalBench-Hard, native-language reasoning yields modest gains in some regions, specifically +1.0 points in South Europe and +2.9 points in Oceania (Tam et al., 23 May 2025). On LMSYS-Toxic, forcing English reasoning slightly increases toxicity acceptance in several non-English languages, including Japanese and Korean, but makes Russian safer (Tam et al., 23 May 2025). The resulting picture is not a simple “English always wins” rule, but a multi-objective trade-off among reasoning accuracy, cultural appropriateness, and safety.

A second tension concerns language mixing. “The Impact of Language Mixing on Bilingual LLM Reasoning” studies Chinese-English bilingual models and reports that RLVR is the training stage that induces language mixing (Li et al., 21 Jul 2025). On math reasoning tasks, enforcing monolingual decoding reduces accuracy by 5.6 percentage points, while a lightweight probe used to guide switching increases accuracy by up to 6.25 percentage points (Li et al., 21 Jul 2025). This complicates the common assumption that language consistency is always the desired inference-time constraint. In some settings, code-switching is reported as a strategic reasoning behavior rather than a defect (Li et al., 21 Jul 2025).

A third issue is answer selection and aggregation. “Could Thinking Multilingually Empower LLM Reasoning?” reports that multilingual reasoning has a higher upper bound than English-only reasoning: on GPQA, multilingual Acc@17 reaches 74.3% for Qwen2.5-72B, 73.9% for LLaMA3.1-70B, and 80.1% for R1-Distill-LLaMA-70B, whereas English accuracies are around 45.0%, 38.0%, and 51.6%, respectively (Gao et al., 16 Apr 2025). Yet the same work finds that Vote@k and Judge@k fail to recover this upper bound because correct answers may remain in the minority and judges exhibit language bias (Gao et al., 16 Apr 2025). This suggests that a multilingual reasoning gym must evaluate not only per-language solving but also the aggregation mechanisms used when multiple reasoning paths are available.

A fourth issue is efficiency and fairness. mmPISA-bench reports negative correlations between cost and accuracy across languages, with Thai costing about 2.7 times more than English for Claude and incurring a 5.1-point accuracy drop, and Greek costing about 1.8 times English for GPT with an accuracy drop of about 5.3 points (Sapenov et al., 5 Jun 2026). This extends the gym concept from pure reasoning accuracy to tokenizer- and cost-aware auditing.

Several misconceptions can therefore be rejected on the basis of current evidence. Multilingual reasoning gym research is not simply translated benchmarking: procedural generation, verifiers, and RLVR make it an environment class rather than just a corpus (Dobler et al., 11 Mar 2026). The multilingual gap is not reported as a pure reasoning deficit: multiple studies trace it primarily to language understanding and routing failures (Ko et al., 5 Jan 2025, Kang et al., 31 Oct 2025). Nor does the literature support a universal monolingual target-language discipline: English-anchored reasoning, selective translation, and language mixing all improve performance in specific regimes, while native-language reasoning can still help on cultural tasks and sometimes on language-specific benchmarks (Tam et al., 23 May 2025, Li et al., 21 Jul 2025, Guerra-Solano et al., 15 Oct 2025).

Taken together, the field’s current definition of a multilingual reasoning gym is best understood as a controlled multilingual laboratory. Its canonical components are aligned multilingual instances, explicit control over input and reasoning language, verifiable or structured scoring, intervention mechanisms such as translation or language mixing, and reporting that spans accuracy, robustness, cultural sensitivity, safety, language consistency, and cost. The concrete 14-language procedural framework provides the most general implementation of this idea (Dobler et al., 11 Mar 2026), while the surrounding literature supplies specialized gyms for mathematical reasoning, logical puzzles, linguistic inference, abstract grouping, and compact cross-lingual auditing (Ko et al., 5 Jan 2025, Bruun et al., 5 Nov 2025, Sapenov et al., 5 Jun 2026).