mSCoRe: Multilingual Commonsense Benchmark
- mSCoRe is a benchmark for evaluating multilingual and culturally grounded commonsense reasoning using a detailed reasoning-skill taxonomy.
- It employs a structured data synthesis pipeline and complexity scaling to test performance across languages and social contexts.
- Empirical results highlight model performance drops with increased task complexity and the benefits of explicit skill-structured prompting.
mSCoRe is a benchmark for evaluating multilingual and culturally grounded commonsense reasoning in LLMs. Its title expands the acronym as “a ultilingual and Scalable Benchmark for kill-based mmonsense asoning,” and the benchmark is designed around three components: a reasoning-skill taxonomy, a data synthesis pipeline, and a complexity scaling framework. It is intended not only to measure answer accuracy, but also to analyze which reasoning skills models use, how those skills vary across languages and social contexts, and how performance changes as task complexity increases (Ngo et al., 13 Aug 2025).
1. Conceptual scope and motivation
mSCoRe was introduced against a specific evaluation gap in LLM research: existing commonsense benchmarks are often monolingual or English-centric, multilingual variants frequently rely on direct translation, and most prior resources emphasize final accuracy rather than reasoning-process analysis. The benchmark therefore targets three limitations at once: multilingual commonsense coverage, culturally grounded social commonsense, and fine-grained skill-level analysis of reasoning traces (Ngo et al., 13 Aug 2025).
The benchmark separates two related but distinct evaluation settings. The first is general commonsense reasoning, implemented as mSCoRe-G. The second is social commonsense reasoning, implemented as mSCoRe-S. For mSCoRe-G, multilinguality is literal: the benchmark covers English, German, French, Chinese, and Japanese. For mSCoRe-S, multilinguality is expressed through culturally diverse social scenarios derived from CultureBank examples sourced from TikTok and Reddit. This design makes mSCoRe unusual among commonsense benchmarks because it combines cross-lingual evaluation with culturally specific social reasoning rather than treating multilinguality as translation alone (Ngo et al., 13 Aug 2025).
A central premise of the benchmark is that commonsense reasoning should be decomposed into minimal interpretable units. The paper defines an Atomic Reasoning Step as “an indivisible unit of reasoning that predominantly utilizes one reasoning skill,” and states that an optimal reasoning path uses the minimum number of such steps necessary to eliminate answer options. This process view turns mSCoRe from a standard multiple-choice QA set into an evaluation framework for reasoning structure (Ngo et al., 13 Aug 2025).
2. Reasoning-skill taxonomy
The reasoning-skill taxonomy is the benchmark’s main analytical device. It is organized hierarchically into three major categories and ten specific skills. Each atomic reasoning step in a reference reasoning path is labeled with exactly one skill, enabling step-level analysis rather than only end-task scoring (Ngo et al., 13 Aug 2025).
| Category | Skills |
|---|---|
| Logical Reasoning | Inductive, Deductive, Abductive |
| Contextual Reasoning | Analogical, Counterfactual, Probabilistic, Temporal, Spatial |
| Social and Ethical Reasoning | Social, Moral |
The paper defines these skills in compact operational terms. Inductive reasoning draws general conclusions from specific observations; deductive reasoning derives specific conclusions from general premises; abductive reasoning forms hypotheses to explain observations. Analogical reasoning draws parallels between similar situations, counterfactual reasoning considers alternative scenarios that did not happen, probabilistic reasoning handles inference under uncertainty, temporal reasoning tracks sequences and durations of events, and spatial reasoning concerns visualization and manipulation in space. Social reasoning concerns social interactions and norms, while moral reasoning concerns what is right or wrong based on ethical principles (Ngo et al., 13 Aug 2025).
The benchmark’s structured annotation schema encodes this taxonomy directly. Each reasoning step contains "reasoning_skill", "reasoning", "eliminated_options", and "possible_options". This means that evaluation can examine not only whether a model selected the correct option, but also whether it used an appropriate reasoning skill sequence and whether its elimination path resembles the reference process. This suggests a shift from outcome-only evaluation toward process-sensitive commonsense evaluation, though the benchmark itself still reports answer accuracy as the primary aggregate metric (Ngo et al., 13 Aug 2025).
3. Benchmark construction and data synthesis
mSCoRe is not generated from scratch. To preserve answer semantics and reduce label noise, it begins from human-rooted seed datasets. mSCoRe-G uses mCSQA as its seed source, while mSCoRe-S uses CultureBank. The construction pipeline then applies a four-step generation procedure to selected seed instances (Ngo et al., 13 Aug 2025).
The first step is Seed Data Filtering. A general LLM-judge, specifically Flow Judge, scores candidate seed examples. For mSCoRe-G, the criteria are Commonsense-ness, Complexity, and Expandability. For mSCoRe-S, the same criteria are used, with multiculture-ness added. The second step is Structured Reasoning Generation, in which an LLM generates a commonsense context and a structured reasoning process. For mSCoRe-S, the model first generates the QA pair itself from CultureBank descriptors before generating structured reasoning. The third step is Data Complexity Scaling, which expands the base instance through context expansion, option adjustment, and reasoning refinement. The fourth step is Commonsense Implicitation, which hides much of the explicit support by fusing context into the question, thereby forcing greater reliance on internal commonsense knowledge (Ngo et al., 13 Aug 2025).
The released benchmark contains 5,600 instances total. Of these, 4,000 belong to mSCoRe-G and 1,600 to mSCoRe-S. For mSCoRe-G, there are 200 examples per language, each expanded from level 0 to level 3, yielding 800 examples per language across five languages. For mSCoRe-S, there are 200 examples for each source, TikTok and Reddit, also expanded across level 0 to level 3, yielding the total of 1,600 social-commonsense instances (Ngo et al., 13 Aug 2025).
4. Task format and complexity scaling
All mSCoRe tasks are formulated as multiple-choice QA. Each instance includes a commonsense context, a commonsense question, answer options, a correct answer represented as a letter-text pair, and a structured reasoning process composed of labeled atomic steps. This standardized representation allows the benchmark to evaluate both answer selection and reasoning-process structure within a uniform schema (Ngo et al., 13 Aug 2025).
Complexity is increased through three explicit mechanisms: context expansion, option adjustment, and commonsense implicitation. Context expansion adds more background or situational detail; option adjustment revises existing options and introduces an additional plausible but incorrect option; reasoning refinement adds an additional atomic reasoning step required to eliminate the new distractor. The key design principle is that each new level should require one more reasoning step. The benchmark releases levels L0 through L3, and the paper also reports analysis extending to L6 for English general and social subsets (Ngo et al., 13 Aug 2025).
The benchmark’s representative examples illustrate this scaling logic. In a general-commonsense item asking “What is the best way to experience a live performance?”, the correct answer remains “go to theatre” across levels, but higher levels add distractors that require additional social and analogical reasoning beyond the initial deductive and temporal reasoning. In a social-commonsense example derived from CultureBank concerning German school swimming, the correct answer is “Participate in compulsory swimming education that includes accommodations for diverse cultural practices,” and higher levels add abductive, analogical, and probabilistic demands. The semantic target is thus preserved while the elimination path becomes longer and more skill-diverse (Ngo et al., 13 Aug 2025).
5. Experimental evaluation and empirical findings
The paper evaluates eight state-of-the-art LLMs: GPT-4o, o1, o1-mini, LLaMA-3.3-70B, LLaMA-3.1-8B, R1-70B, R1-8B, and Aya-32B. All models were prompted with a consistent prompt that included the reasoning-skill taxonomy and step-by-step instructions to produce a reasoning process before answering. The primary reported metric is accuracy on multiple-choice questions; the paper also analyzes reasoning traces, skill distributions, and the average number of reasoning steps (Ngo et al., 13 Aug 2025).
The main empirical pattern is that mSCoRe remains challenging as complexity increases. On mSCoRe-G, average accuracy from L0 to L3 drops for all reported models, including GPT-4o: 79.2 → 74.9 → 73.1 → 69.5, o1: 76.6 → 71.3 → 68.5 → 65.3, and LLaMA-3.3-70B: 78.9 → 75.9 → 70.2 → 70.4. On mSCoRe-S, LLaMA-3.3-70B is the strongest model, with 81.8 → 75.8 → 76.8 → 74.8, while o1 reports 73.3 → 70.0 → 65.5 → 65.3 and GPT-4o reports 73.0 → 68.0 → 65.5 → 66.5. The weakest model is LLaMA-3.1-8B, which performs poorly in both settings (Ngo et al., 13 Aug 2025).
The extended scaling experiments to L6 show that degradation continues beyond the released L3 regime. For English general commonsense, GPT-4o drops from 80.5 at L0 to 68.0 at L6, o1 from 82.5 to 67.0, LLaMA-3.3-70B from 78.5 to 69.0, and Aya-32B from 77.5 to 54.5. For social commonsense, LLaMA-3.3-70B drops from 81.8 to 73.3, while Aya-32B drops from 69.5 to 54.0. The paper notes that the largest drop occurs between L0 and L2, after which degradation slows and begins to plateau (Ngo et al., 13 Aug 2025).
The reasoning-process analyses are as important as the accuracy tables. In general commonsense, reference solutions diversify into more contextual reasoning, especially analogical and probabilistic reasoning, as complexity rises, whereas o1 remains heavily dependent on deductive reasoning. In social commonsense, reference solutions use a more balanced mixture of categories and increasingly rely on social/ethical reasoning, but o1 still over-relies on logical reasoning. The paper also reports that reference paths increase roughly linearly with complexity, GPT-4o’s reasoning length also increases, but o1 stays nearly constant at around 3 steps regardless of complexity. The benchmark thus exposes not only answer failures but also failures of reasoning-skill adaptation and reasoning-depth scaling (Ngo et al., 13 Aug 2025).
An additional experimental result concerns prompting. The paper compares standard CoT, Logical-only, General three-category, and the proposed fine-grained taxonomy prompts for o1 and o1-mini. For o1, the proposed taxonomy is best overall. One reported example is social L3, where o1 with the proposed taxonomy reaches 65.3, compared with 40.3 for cot-o1, 58.3 for logical-o1, and 48.3 for general-o1. This indicates that explicit skill-structured prompting improves performance within the benchmark’s own evaluation setting (Ngo et al., 13 Aug 2025).
6. Interpretation, limitations, and disambiguation
The paper’s broader interpretation is that reasoning-reinforced training does not automatically improve multilingual commonsense reasoning. It reports that o1 is best in English, but lags behind GPT-4o and LLaMA-3.3-70B in several other languages, and suggests that training optimized for math- and coding-style reasoning may not transfer cleanly to multilingual and culturally nuanced commonsense. At the same time, it also reports that reasoning reinforcement helps smaller models, since R1-8B behaves much better than LLaMA-3.1-8B. This suggests that the effects of reasoning-focused training are model-scale dependent within this evaluation setting (Ngo et al., 13 Aug 2025).
The benchmark also has explicit limitations. The mSCoRe-G language set contains only medium- to high-resource languages, so multilingual gaps may be underestimated. Complexity scaling begins to plateau at higher levels, suggesting that the multiple-choice format may constrain how far difficulty can be pushed. The paper further notes that some benchmark creation used GPT-4o, so strong GPT-4o performance may partly reflect benchmark-generation artifacts. These are benchmark-specific caveats rather than generic criticisms of commonsense evaluation (Ngo et al., 13 Aug 2025).
The name mSCoRe is distinct from several similarly named arXiv methods and benchmarks. It is not MSCoRe, the benchmark for Multi-Stage Collaborative Reasoning in LLM Agents (Lei et al., 22 Sep 2025); not MSD-Score, the Multi-Scale Distributional Scoring metric for reference-free image caption evaluation (Kan et al., 7 May 2026); not MoEScore, the text-audio relevance evaluator for text-to-audio systems (Sun et al., 11 Jan 2026); not the undeveloped “MSG score” concept in multi-scene video generation (Yoon et al., 2024); and not GoF-MSCORE, the network goodness-of-fit metric for the block-model family (Jin et al., 12 Feb 2025). It is also distinct from MSQs, the Multidimensional Service Quality Scoring System (Lai, 2022), and from SCORE, the semi-supervised EHR representation-learning framework later applied to multiple sclerosis phenotyping (Wang et al., 27 May 2025). Within the provided literature, the exact string mSCoRe corresponds specifically to the multilingual and scalable benchmark for skill-based commonsense reasoning (Ngo et al., 13 Aug 2025).