MultiNRC: Multilingual Reasoning Challenge

• MultiNRC is a comprehensive evaluation suite designed to assess LLMs' native reasoning capabilities with culturally and linguistically authentic challenges.
• It employs a two-stage native review, template-first annotation, and bilingual alignment to capture language-specific phenomena across multiple reasoning categories.
• Empirical findings reveal significant performance gaps across linguistic, cultural, and mathematical tasks, highlighting challenges in cross-lingual consistency.

The Multilingual Native Reasoning Challenge (MultiNRC) is a rigorous evaluation suite designed to assess the capability of LLMs to perform authentic, culturally and linguistically grounded reasoning in multiple languages. Unlike prior benchmarks relying predominantly on translation from English sources, MultiNRC prioritizes native-authored content, cultural specificity, and diagnostic analysis of reasoning phenomena unique to non-English contexts. It is motivated by the need to characterize both strengths and failures of LLMs in domains where language, local knowledge, and formal reasoning interact tightly.

1. Motivation and Design Principles

Existing multilingual reasoning benchmarks suffer from two major limitations: (1) cultural/linguistic bias due to simple translation of English tasks, and (2) lack of native-speaker authenticity, preventing coverage of phenomena such as language-specific wordplay, morphological agreement, or culturally embedded logic (Fabbri et al., 23 Jul 2025). MultiNRC addresses these deficiencies by assembling more than 1,000 problems authored in French, Spanish, and Chinese by native speakers, each designed to demand multi-step culturally or linguistically grounded reasoning. Filtering ensures that only questions unsolvable by a majority of state-of-the-art LLMs are retained, guaranteeing challenge and relevance.

Core design tenets include:

• Coverage of reasoning types (language-specific grammar, wordplay/riddles, cultural/tradition, and math with cultural relevance)
• Emphasis on linguistic/cultural authenticity, with two-stage native review for quality.
• Provision of English-equivalent translations for selected categories, enabling direct comparison between performance in native and pivoted (English) contexts.

2. Benchmark Construction and Task Taxonomy

Native Authoring Pipeline

Problems are natively authored and reviewed by expert annotators, with prompt selection intentionally demanding at least one reasoning step anchored in local language or culture. To ensure scalability and extensibility, a template-first methodology is recommended (Elsetohy et al., 11 Feb 2026), where language-agnostic skeletons span major reasoning and cultural types. This facilitates systematic coverage, bilingual alignment, and instance generation for both multiple-choice and true/false modalities.

Reasoning Categories

Each MultiNRC item is assigned to one (and only one) of the following categories:

Category	Description	Example Phenomena
Linguistic Reasoning	Grammar, morphology, language-specific logic	French plural gender, Spanish agreement
Wordplay & Riddles	Puns, homophones, culturally bound riddles	French “marre tôt” ≈ “marteau”
Cultural/Tradition	Multi-step logic relying on festivals, customs, timelines	Planning around Día de la Candelaria
Math w/ Culture	Numeric/computational reasoning using local units/systems	French viager, Spanish “codo de ribera”

Compositional metadata enables fine-grained stratification for analysis and extension to additional languages, scripts, and reasoning modes (Elsetohy et al., 11 Feb 2026).

3. Evaluation Methodology and Metrics

Model Evaluation

MultiNRC mandates testing via both direct answer and chain-of-thought (CoT) prompting regimes. Fourteen top-performing LLMs spanning all major families have been systematically evaluated (Fabbri et al., 23 Jul 2025). Metrics adopted include:

• Accuracy: $$\text{Accuracy} = \frac{\,\#\,\text{correct answers}}{\,\#\,\text{total questions}} \times 100\%$$
• Language fidelity: Fraction of CoTs or answers produced solely in the input language (Wang et al., 25 Apr 2025).
• Cross-lingual gap: $$\Delta_{\text{Eng-Orig}} = \text{Acc}_{\text{Eng}} - \text{Acc}_{\text{Orig}}$$, quantifying the delta between native and English-equivalent settings.
• Category-specific breakdowns: Average accuracy per reasoning type and per language, with confidence intervals estimated by bootstrap.

For composite formats (multiple-choice + paired true/false), paired accuracy is computed to surface overestimation of verification abilities from single-item scores (Elsetohy et al., 11 Feb 2026).

Consistency and Self-Assessment

Benchmarks such as “The Riddle of Reflection” (M et al., 2 Nov 2025) supplement MultiNRC with meta-evaluation of self-awareness: models are tasked with both solving riddles and evaluating their own answers, revealing mismatches between generation accuracy and error identification rates.

4. Empirical Results and Key Findings

Across the main MultiNRC set (Fabbri et al., 23 Jul 2025), no model exceeds 50% accuracy. For the full 1,055-question evaluation, leading models such as o3-pro achieve 49.0 % ± 3.0, with the distribution of performance by category and language summarized as follows:

Category	French	Spanish	Chinese
Linguistic	36.1 %	32.5 %	35.7 %
Wordplay	35.6 %	25.3 %	24.7 %
Cultural	31.7 %	26.2 %	36.9 %
Math	29.8 %	18.7 %	23.3 %

• Wordplay and math are systematically the most challenging.
• Spanish is notably difficult in math and wordplay.
• Model strengths are not uniform: o3-pro, for example, excels on French wordplay, while Gemini-2.5 is strongest in Spanish math.

For categories allowing direct English-equivalent translations, there is a significant “English boost” in math (up to +19.1 % for Spanish) but not in cultural tasks, indicating persistent knowledge and representation gaps around local customs.

Cross-Language Consistency

Analyses of equivalence and inheritance in reasoning (Arora et al., 2024) reveal high rates of answer conflict (up to 57.5%) and inheritance violation (up to 37.2%) among SOTA LLMs, especially in less-represented languages. The introduction of compositional representations (CoRe) reduces such conflicts, demonstrating the importance of shared, language-agnostic embeddings.

Reasoning Structure and Latent Variable Control

Layer-wise analysis of multilingual LLMs demonstrates a language-agnostic “reasoning core” situated in the mid-stack transformer layers (Lasbordes et al., 26 May 2026). Language-specific features are handled predominantly in input and output layers, a finding leveraged by the “Layer Swap” technique: exchanging mid-layers from an English specialist into a native LLM closes up to 100% of the native-pivoted reasoning gap without compromising language fidelity of the output CoT.

Explicit control of output language notably affects performance: instructing the model to think and answer in English maximizes math performance in PolyMath (Wang et al., 25 Apr 2025). Conversely, forcing reasoning in the native language—absent explicit multilingual reasoning supervision—degrades accuracy, especially in low-resource languages.

5. Methodological and Architectural Innovations

Layer Swap and Reasoning Core Alignment

The Layer Swap method involves replacing a contiguous mid-stack layer window (e.g., layers 13–22) from an English specialist into a native LLM specialist (Lasbordes et al., 26 May 2026). This transfer preserves chain-of-thought target-language fidelity (≥ 99%), while fractionally or wholly bridging the native vs. English-pivot gap on complex multi-domain benchmarks.

Latent-Variable RL (polyGRPO)

Treatment of language as a latent variable for reasoning structure underpins the polyGRPO reinforcement learning framework (Wu et al., 23 Apr 2026). By sampling rollouts in both constrained (specified language) and unconstrained (free language choice) modes, polyGRPO treats language-switching as an exploration mechanism. This approach yields state-of-the-art improvements (+6–8 pp absolute accuracy) in both English and multilingual settings, with particularly strong gains on non-English, under-resourced languages.

Template-First and Bilingual Grounding

Template-first annotation and scenario-aligned bilingual authoring, as piloted in Macaron (Elsetohy et al., 11 Feb 2026), provides scalable mechanisms for extending MultiNRC. Each question template is annotated with reasoning type and cultural aspect, ensuring controlled difficulty and robust generalization analysis across 20+ languages and scripts.

6. Diagnostic Analyses and Recommendations

Systematic error analyses reveal:

• Persistent gaps in handling culturally specific units, idioms, and wordplay (e.g., Spanish “codo de ribera” misretrieved as a volume unit, French homophonic puns).
• Difficulty with cross-lingual consistency, with baseline LLMs producing semantically conflicting answers for parallel questions in different languages (Arora et al., 2024).
• Inverse correlation between model generation accuracy and error self-identification, with higher-accuracy models exhibiting overconfidence but low self-awareness (TNR ≈ 4–7%), and weaker models being more self-cautious (TNR ≈ 42%) (M et al., 2 Nov 2025).

General recommendations include:

• Matched token budgets and supervision regimes for cross-lingual specialist training (Lasbordes et al., 26 May 2026).
• Design benchmarks with per-layer and per-task analysis, leveraging architectural modularity.
• Integrate both direct-answer and chain-of-thought evaluation, with focus on explicit reasoning in both native and pivot languages (Wang et al., 25 Apr 2025).
• Include self-critique assessment (e.g., two-stage answer + evaluation) to detect overconfidence and reliability issues (M et al., 2 Nov 2025).

7. Future Directions and Open Research Problems

While MultiNRC establishes a robust framework for evaluating native multilingual reasoning, open research challenges remain:

• Extending coverage to low-resource tongues and regional dialects, leveraging bilingual scenario alignment and self-generated polyglot preference data for bootstrapping (Wu et al., 23 Apr 2026).
• Exploring mechanisms for output-language control that “unlock” latent reasoning structure without sacrificing linguistic fidelity or correctness.
• Incorporating architectural advances, such as compositional token representations and modular transfer of reasoning cores, to democratize performance gains across diverse linguistic scenarios.
• Developing meta-evaluation protocols quantifying both structural consistency and model self-awareness across reasoning types and languages.
• Investigating methods that disentangle high-level reasoning abstraction from surface language tokens, enabling generalizable, robust multilingual reasoning capabilities.

MultiNRC serves as the basis for progress in quantifying, analyzing, and ultimately narrowing the gap in natively grounded, multilingual reasoning, setting a high bar for evaluation and insight into the core mechanisms underlying generalization and fairness in multilingual LLMs (Fabbri et al., 23 Jul 2025, Lasbordes et al., 26 May 2026, Wang et al., 25 Apr 2025, Elsetohy et al., 11 Feb 2026, Wu et al., 23 Apr 2026, Arora et al., 2024, M et al., 2 Nov 2025).