MME-SCI: Multimodal Science Benchmark

• MME-SCI is a comprehensive benchmark designed to evaluate multilingual and multimodal scientific reasoning in high-school level subjects using 1,019 curated QA pairs and 63 fine-grained knowledge points.
• It combines text-only, image-only, and image-text hybrid evaluation modes, providing a systematic diagnostic tool for assessing model performance across multiple languages and modalities.
• The construction involves rigorous multi-stage quality control, translation into five languages, and careful curation from mock exam papers to ensure reliability and diagnostic depth.

MME-SCI is a comprehensive and challenging scientific benchmark for multimodal LLMs (MLLMs) designed to evaluate scientific reasoning under multiple languages, multiple input modalities, and fine-grained scientific knowledge categories. It contains 1,019 manually curated question-answer pairs spanning mathematics, physics, chemistry, and biology; supports Chinese, English, French, Spanish, and Japanese; and evaluates models in text-only, image-only, and image-text hybrid modes (Ruan et al., 19 Aug 2025). Despite its scientific orientation, it is a Chinese high-school-level benchmark rather than a graduate-level literature benchmark; graduate- to PhD-level scientific figure understanding is instead the focus of MMSci (Li et al., 2024).

1. Definition and scope

MME-SCI was proposed to address three gaps in prior scientific multimodal benchmarks: insufficient multilingual evaluation, inadequate modality coverage, and lack of fine-grained scientific knowledge annotations. Its stated goal is to provide a systematic, difficult, and diagnostic benchmark for scientific reasoning capabilities of MLLMs across languages, modalities, and knowledge categories, and to break the performance saturation observed on earlier benchmarks such as MMMU and AI2D (Ruan et al., 19 Aug 2025).

Dimension	Coverage
Question-answer pairs	1,019
Subjects	Mathematics, physics, chemistry, biology
Languages	Chinese, English, French, Spanish, Japanese
Evaluation modes	text-only, image-only, image-text hybrid
Fine-grained knowledge points	63

The benchmark’s subject structure comprises 12 knowledge points in mathematics, 22 in physics, 6 in chemistry, and 23 in biology, for a total of 63 fine-grained concepts. It emphasizes recency as a contamination-control measure: 83.3% of questions are from 2025, 16.2% from 2024, and 0.5% from before 2024. In the Chinese source set $\boldsymbol{\mathrm{D_{zh}}}$, 805 questions require image-based understanding and 214 are text-only; screenshots of all questions are then created to form $\boldsymbol{\mathrm{D_{zh}^{img}}}$ for image-only evaluation (Ruan et al., 19 Aug 2025).

A central distinguishing claim is that MME-SCI is the only benchmark in its comparison table marked positive on all four dimensions of multilinguality, comprehensive modality coverage, multidisciplinary scope, and fine-grained knowledge points. This places it as a diagnostic benchmark rather than a single-mode VQA set or a coarse subject-level science test (Ruan et al., 19 Aug 2025).

2. Construction pipeline and curation

The construction pipeline consists of sample filtering, data digitization, language transformation, and post-auditing, although the figure caption describes it as “three stages.” Approximately 300 person-days were spent on problem selection, digitization, language conversion, and post-verification (Ruan et al., 19 Aug 2025).

The benchmark is built from mock exam papers of high-school science subjects rather than official Gaokao papers. The rationale is contamination control: Gaokao papers have wide dissemination and low accessibility barriers, and are therefore more likely to have appeared in MLLM training corpora. Sample filtering was performed by three evaluation volunteers—two senior undergraduates and one graduate student—all ranked within the top 0.1% in average score in China’s Gaokao. These volunteers solved questions from mock exam papers and filtered out questions they answered incorrectly or found confusing (Ruan et al., 19 Aug 2025).

After difficulty-based manual selection, five annotators used GPT-4o and OCR tools to extract questions and answers and convert them into JSON format, producing $\boldsymbol{\mathrm{D_{zh}}}$. Screenshots of all questions were then taken to create $\boldsymbol{\mathrm{D_{zh}^{img}}}$. The Chinese dataset was translated into English, French, Spanish, and Japanese to form $\boldsymbol{\mathrm{D_{en}}}$, $\boldsymbol{\mathrm{D_{fr}}}$, $\boldsymbol{\mathrm{D_{es}}}$, and $\boldsymbol{\mathrm{D_{ja}}}$ (Ruan et al., 19 Aug 2025).

Quality control is explicitly multi-stage. A final post-audit used three reviewers to cross-validate OCR results in $\boldsymbol{\mathrm{D_{zh}}}$, screenshot integrity in $\boldsymbol{\mathrm{D_{zh}^{img}}}$, and language conversion results. If one reviewer identified an error, the other two conducted secondary verification; if any reviewer then identified an error, a revision was made. The benchmark’s answers are deliberately restricted to single-choice, multiple-choice, and fill-in-the-blank formats so that correctness remains concise and verifiable (Ruan et al., 19 Aug 2025).

3. Task structure, notation, and evaluation protocol

MME-SCI defines three evaluation modes. Text-only mode measures language comprehension abilities. Image-only mode measures visual semantic parsing capabilities, especially for diagrams, geometry figures, circuits, symbolic notation, charts, and screenshot-formatted exam questions. Image-text hybrid mode measures integration of visual information and textual logic. The benchmark’s multimodal design is therefore not limited to standard image-text VQA, but explicitly targets full-modal reasoning evaluation (Ruan et al., 19 Aug 2025).

The dataset notation used in the paper is $\boldsymbol{\mathrm{D_{zh}^{img}}}$0, $\boldsymbol{\mathrm{D_{zh}^{img}}}$1, $\boldsymbol{\mathrm{D_{zh}^{img}}}$2, $\boldsymbol{\mathrm{D_{zh}^{img}}}$3, $\boldsymbol{\mathrm{D_{zh}^{img}}}$4, and $\boldsymbol{\mathrm{D_{zh}^{img}}}$5. The reported metric is accuracy, given as subject-wise accuracy and average accuracy. The paper adopts the “LLM-as-a-Judge” paradigm and introduces different evaluation templates for various languages, but it does not provide explicit mathematical formulas for scoring, aggregation, or benchmark definition beyond percentage accuracy reporting (Ruan et al., 19 Aug 2025).

This absence of explicit scoring equations is one of the benchmark’s notable formal features. The paper conceptually defines text-only, image-only, and image-text hybrid modes, but its main quantitative table does not provide a full separate breakdown isolating text-only versus image-text hybrid scores independently. Instead, it reports detailed subject scores for $\boldsymbol{\mathrm{D_{zh}^{img}}}$6 and $\boldsymbol{\mathrm{D_{zh}^{img}}}$7, and average scores only for the translated datasets. Standardized decoding uses a maximum number of new tokens of 8,192 and temperature $\boldsymbol{\mathrm{D_{zh}^{img}}}$8, and all experiments were conducted on 8×H20 GPUs (Ruan et al., 19 Aug 2025).

4. Models evaluated and quantitative difficulty

The benchmark evaluates 20 MLLMs in total: 16 open-source and 4 closed-source. The open-source set includes Llava-OneVision-7B and -72B, Qwen2.5VL-3B, -7B, and -72B, Qwen2.5-Omni-7B, Ovis2-8B, -16B, and -34B, InternVL3-2B, -8B, and -78B, Kimi-VL-A3B-Instruct and -Thinking, Phi-4-multimodal-instruct, and Skywork-R1V-38B. The closed-source models are Claude-4-Sonnet, Doubao-1.5-thinking-vision-pro, Doubao-Seed-1.6, and o4-mini-20250416 (Ruan et al., 19 Aug 2025).

Model	$\boldsymbol{\mathrm{D_{zh}^{img}}}$9 AVG.	$\boldsymbol{\mathrm{D_{zh}}}$0 AVG.
Qwen2.5VL-72B	22.08	19.43
Doubao-Seed-1.6	38.86	41.32
o4-mini-20250416	37.00	35.62

The benchmark is difficult for all evaluated systems. Among large open-source models, Qwen2.5VL-72B is the strongest in the main table, but it reaches only 22.08 average accuracy on $\boldsymbol{\mathrm{D_{zh}}}$1 and 19.43 on $\boldsymbol{\mathrm{D_{zh}}}$2. Closed-source models are substantially stronger: Doubao-Seed-1.6 obtains 38.86 on $\boldsymbol{\mathrm{D_{zh}}}$3, 41.32 on $\boldsymbol{\mathrm{D_{zh}}}$4, and multilingual averages between 36.41 and 40.14; o4-mini reaches 37.00 and 35.62 on the two Chinese settings, with multilingual averages between 29.54 and 33.17 (Ruan et al., 19 Aug 2025).

The paper explicitly highlights the open-versus-closed gap. The most powerful open-source models show an average accuracy reduction of 13.94% across six scenarios relative to closed-source systems, and in the $\boldsymbol{\mathrm{D_{zh}}}$5 scenario the closed-source models achieve a gain of 125.84% over the Large group. Image-only evaluation is especially revealing: the abstract reports that o4-mini achieves only 52.11% in mathematics, 24.73% in physics, 36.57% in chemistry, and 29.80% in biology under image-only evaluation, which the paper uses as evidence of difficulty even for strong systems (Ruan et al., 19 Aug 2025).

5. Diagnostic findings

The benchmark’s analysis is designed to expose where models fail, not merely to rank them. At subject level, models generally perform better in chemistry and biology, while mathematics is harder and physics is hardest. The authors attribute physics difficulty to complex reasoning combined with the need to understand real-world physical laws (Ruan et al., 19 Aug 2025).

Cross-lingual consistency is particularly weak. Out of 1,019 total questions, even the best model, Doubao-Seed-1.6, achieves only 13.84% linguistically consistent correct responses across all five languages, while Phi-4 achieves only 0.59%. This suggests that moderate average multilingual scores do not imply stable cross-language reasoning, and that language-dependent pattern matching remains substantial (Ruan et al., 19 Aug 2025).

Fine-grained annotations reveal subject-internal asymmetries that aggregate scores would obscure. On the “Magnetic Field” knowledge point, o4-mini answers only 5 out of 33 correctly. In chemistry, the same model reaches 80.00% on its strongest knowledge point but only 31.25% on “Fundamentals of Organic Chemistry.” Error analysis on Doubao-Seed-1.6 classifies failures into visual perception errors, text comprehension errors, knowledge deficiency, calculation errors, and reasoning process errors; reasoning process errors are the most frequent at 49.05%, whereas calculation errors account for only 5.03% (Ruan et al., 19 Aug 2025).

Several comparative analyses reinforce the benchmark’s diagnostic intent. Reasoning-oriented variants perform better: Kimi-VL-A3B-Thinking exceeds Kimi-VL-A3B-Instruct by 48.54% on $\boldsymbol{\mathrm{D_{zh}}}$6 and by 72.88% on image-only evaluation, and Skywork-R1V-38B outperforms Ovis2-34B by 4.09% average accuracy across six scenarios. By contrast, Any2Any expansion can degrade visual reasoning: Qwen2.5-Omni-7B drops by 4.71% on $\boldsymbol{\mathrm{D_{zh}}}$7 relative to Qwen2.5VL-7B. The paper also reports that supplying knowledge-point descriptions can slightly improve weaker models; for example, detailed descriptions generated by Doubao-Seed-1.6 raise Qwen2.5VL-7B from 15.21% to 15.90% on $\boldsymbol{\mathrm{D_{zh}}}$8, and Qwen2.5VL-72B from 22.08% to 22.50% (Ruan et al., 19 Aug 2025).

6. Benchmark position, interpretation, and limitations

Within the benchmark taxonomy used by its authors, MME-SCI is presented as the only listed benchmark that is simultaneously multilingual, full-modality, multidisciplinary, and annotated with fine-grained knowledge points. It is compared against GAOKAO-Bench, MathVerse, MATH-Vision, MMMU, EMMA, GeoSense, PhyX, and VisioMath, and is explicitly intended to provide higher difficulty and stronger diagnostic resolution than those predecessors (Ruan et al., 19 Aug 2025).

A plausible implication is that MME-SCI occupies a niche complementary to both MME and MMSci rather than replacing either. MME is a general comprehensive MLLM evaluation benchmark organized around perception and cognition with manually designed paired yes/no instructions and ACC/ACC+ scoring (Fu et al., 2023). MMSci, by contrast, evaluates grounded interpretation of authentic peer-reviewed scientific figures across 72 disciplines at graduate- to PhD-level difficulty (Li et al., 2024). MME-SCI differs from both by centering multilingual scientific reasoning at the Chinese high-school level, combining text-only, image-only, and image-text hybrid modes with 63 fine-grained knowledge points (Ruan et al., 19 Aug 2025).

The benchmark’s limitations are also clear. Its scope is broad but remains high-school-level rather than university or graduate science. Its source construction is Chinese-centered, with later translation into four additional languages. The main text does not provide explicit formulas for the judge or scoring pipeline. Image-only results may partly reflect OCR and screenshot handling quality rather than only scientific reasoning. Finally, question formats are constrained to single-choice, multiple-choice, and fill-in-the-blank, so the benchmark does not cover open-ended proof-style reasoning or research-literature interpretation (Ruan et al., 19 Aug 2025).

In that form, MME-SCI functions less as a single leaderboard and more as a structured probe of multilingual scientific multimodal reasoning. Its main significance lies in showing that current MLLMs remain far from saturation when evaluated jointly on modality robustness, cross-lingual consistency, and concept-level scientific understanding.