AstroMMBench: Astronomical Multimodal Benchmark

• AstroMMBench is a specialized benchmark assessing MLLMs' ability to interpret real astronomical images using domain-specific multiple-choice questions across six astrophysics subfields.
• It was created through a rigorous pipeline that combines automated caption refinement, model-based filtering, and expert review of questions sourced from 3,592 arXiv astro-ph preprints.
• Empirical results show top models like Ovis2-34B achieving 70.53% accuracy, indicating both progress and persistent challenges in scientific visual reasoning.

AstroMMBench is a benchmark for evaluating multimodal LLMs (MLLMs) on astronomical image understanding. Introduced as the first comprehensive benchmark designed for this purpose, it comprises 621 multiple-choice questions across six astrophysical subfields, each paired with a real astronomical image and four answer options, and it was curated and reviewed by 15 domain experts. The benchmark was created to address a gap left by general multimodal benchmarks, which do not capture the complexity and domain specificity of astronomical data. Its initial evaluation covered 25 MLLMs, including 22 open-source and 3 closed-source systems, and identified Ovis2-34B as the highest-scoring model with 70.53% overall accuracy (Shi et al., 29 Sep 2025).

1. Definition and domain scope

AstroMMBench targets a specific failure mode of general-purpose multimodal evaluation: strong generic vision-language performance does not, by itself, establish competence in astronomical image interpretation. The benchmark therefore operationalizes astronomy-specific multimodal assessment through a controlled multiple-choice setting in which the model must identify the correct answer from options A, B, C, and D using both the image and the associated scientific context (Shi et al., 29 Sep 2025).

The benchmark spans six major subfields of astrophysics: Astrophysics of Galaxies (GA), Cosmology and Nongalactic Astrophysics (CO), Earth and Planetary Astrophysics (EP), High Energy Astrophysical Phenomena (HE), Instrumentation and Methods for Astrophysics (IM), and Solar and Stellar Astrophysics (SR). The number of questions per subfield ranges from 87 to 111, which was intended to provide broad topical coverage while maintaining balance across the astronomical research landscape. The questions were also curated to span a range of difficulties, with analysis indicating that most items cluster in medium difficulty rather than at the extremes (Shi et al., 29 Sep 2025).

This scope is significant because astronomical figures are heterogeneous: they include observational imagery, plots, maps, simulations, and instrument-related visualizations. AstroMMBench is designed so that successful performance requires genuine image analysis and advanced astronomical knowledge rather than superficial pattern matching or text-only elimination strategies. A plausible implication is that the benchmark probes both domain knowledge and scientific visual grounding, rather than treating astronomy as a purely textual specialization.

2. Corpus assembly and curation pipeline

AstroMMBench was built from contemporary arXiv astronomy literature. Images and captions were extracted from 3,592 arXiv astro-ph preprints submitted between January 1, 2024, and July 31, 2024, yielding 19,299 initial image-text pairs. This choice ties the benchmark to current scientific communication practices and to the figure types actually used in research papers (Shi et al., 29 Sep 2025).

The generation pipeline had multiple stages. First, LLaMA3.3-70B-Instruct rewrote noisy captions and surrounding context into uniform scholarly image descriptions. Next, InternVL2.5-78B generated the multiple-choice questions from the cleaned descriptions and the images. The prompts were explicitly designed to require genuine image analysis and advanced astronomical knowledge, and they included the construction of plausible distractors and explanations. This design is central to the benchmark’s intent: it attempts to suppress shortcuts that would allow a model to answer from textual priors alone (Shi et al., 29 Sep 2025).

A model-based filtering phase followed. Five strong LLMs answered each automatically generated question five times. Questions that were easily answerable via text, without real visual analysis, were removed. More specifically, questions answered correctly by at least two models through text-only reasoning were filtered out, reducing the pool to 9,677 visually grounded questions. This stage functions as an adversarial screen against leakage from captions or obvious verbal cues.

Expert review then determined the final benchmark composition. Fifteen domain experts, each with at least a Master’s degree in astronomy or a related field, independently reviewed 1,800 randomly selected questions within their subfields. They evaluated alignment between image and question, completeness and context, answer accuracy and uniqueness, and the necessity for domain-specific knowledge. The final release contains 621 questions judged to be high-quality, challenging, and scientifically rigorous. Answer options were randomized to avoid labeling bias (Shi et al., 29 Sep 2025).

3. Task structure and content characteristics

Each AstroMMBench item consists of a real astronomical image and a four-option multiple-choice question with a single correct answer. The benchmark is therefore a specialized multimodal visual question answering setting, but constrained to a discrete-choice protocol that permits standardized large-scale scoring. The reliance on real figures from astro-ph preprints is a defining property: the visual input is not synthetic and is not restricted to natural-image analogues (Shi et al., 29 Sep 2025).

The content profile varies across subfields. Questions in galaxies and Earth/planetary astrophysics span morphological analysis, atmospheric dynamics, and orbital data. Instrumentation and solar/stellar astrophysics often involve interpreting standard astronomical plots and identifying known objects. Cosmology and high-energy astrophysics are described as especially challenging because they require deep understanding of theoretical or abstract concepts, interpretation of unusual visualizations such as cosmological simulations and X-ray maps, and advanced multi-step reasoning across fragmented information (Shi et al., 29 Sep 2025).

Appendix examples further illustrate the benchmark’s technical depth. They include items based on actual arXiv images and plots, requiring interpretation of periodograms, instrument transmissivity, and cosmological models. Some examples also require understanding LaTeX expressions embedded in captions, including quantities such as $A_V$, $v_f$, and $\tau_s$. This indicates that the benchmark is not limited to object recognition; it also tests whether models can use notation-bearing scientific figures in context.

4. Evaluation protocol and measurement

The initial evaluation covered 25 MLLMs: 22 open-source models and 3 closed-source models. The tested systems were chosen to represent the strongest publicly available and proprietary models as of mid-2024. Evaluation used VLMEvalKit, an open-source toolkit for standardized MLLM benchmarking, and all tests were conducted on 8 NVIDIA A100 GPUs (Shi et al., 29 Sep 2025).

The primary metric is accuracy, defined as the proportion of correctly answered questions. A response counts as correct only if the model’s selected letter matches the answer key. Because many MLLMs produce verbose or natural-language outputs rather than a bare option label, DeepSeek-V3 was used to extract the intended answer from the response and map it to A, B, C, or D. This detail matters methodologically: the scoring protocol distinguishes answer selection from explanation style and reduces formatting variance as a confound (Shi et al., 29 Sep 2025).

AstroMMBench was also compared with general-purpose multimodal evaluation through correlation against OpenCompass multimodal leaderboard scores. The relationship was quantified using Pearson’s correlation coefficient,

$$r = \frac{\sum (X - \bar{X})(Y - \bar{Y})}{\sqrt{\sum (X - \bar{X})^2 \sum (Y - \bar{Y})^2}}$$

where $X$ is the OpenCompass score and $Y$ is the AstroMMBench score for each model. This procedure does not merely report benchmark performance; it examines whether general multimodal strength transfers to astronomy-specific multimodal reasoning (Shi et al., 29 Sep 2025).

5. Empirical results and subfield behavior

The headline result of the initial benchmark study is that Ovis2-34B, an open-source model, achieved the highest overall accuracy at 70.53%, outperforming all evaluated alternatives, including strong closed-source systems. The leading models reported in the study are as follows (Shi et al., 29 Sep 2025).

Model	Overall accuracy
Ovis2-34B	70.53%
ChatGPT-4o	69.07%
Doubao-1.5-Vision-Pro	68.12%
InternVL3-38B	67.63%
Qwen2.5-VL-72B	67.47%

All evaluated models performed significantly above random guess, which is 25% in a four-option setting, but there remained a large gap to perfect accuracy. The benchmark therefore distinguishes among high-performing models while still leaving substantial headroom. This suggests that astronomical image understanding remains unresolved even for frontier MLLMs (Shi et al., 29 Sep 2025).

Performance varied across the six astrophysical subfields. The paper reports that instrumentation and methods for astrophysics and solar/stellar astrophysics achieved the highest accuracies, while cosmology and high-energy astrophysics were the most difficult overall. The identified causes include the need to reason about abstract concepts, unusual scientific visualizations, and fragmented evidence. Galaxy and Earth/planetary questions occupied an intermediate range of difficulty (Shi et al., 29 Sep 2025).

For Ovis2-34B specifically, the reported subfield accuracies were 68.04 in galaxies, 67.57 in cosmology, 68.57 in Earth/planetary astrophysics, 72.73 in high-energy astrophysics, 78.16 in instrumentation, and 69.37 in solar/stellar astrophysics. The paper also notes that radar and difficulty-distribution plots reveal substantial subfield-specific weaknesses even for top-performing systems. A plausible implication is that overall accuracy alone compresses distinct failure modes: a model that is strong on standard plots or familiar object classes may still be weak on abstract or atypical astrophysical representations.

6. Position within astronomy benchmark literature

AstroMMBench belongs to a broader movement toward astronomy-specific evaluation, but it occupies a distinct niche. AstroMLab 1 introduced a text-only astronomy benchmark comprising 4,425 multiple-choice questions curated from 885 Annual Review of Astronomy and Astrophysics articles, enabling evaluation of LLMs on domain knowledge, topic variation, and calibration rather than on image understanding (Ting et al., 2024). AstroMLab 2 used the same MCQ setting to analyze continual pretraining and instruction fine-tuning in specialized astronomy models such as AstroLLaMA-3-8B and AstroLLaMA-2-70B, showing that smaller specialized models can underperform their base models while 70B-scale continual pretraining can improve next-token performance (Pan et al., 2024). AstroAlertBench subsequently extended astronomy-specific multimodal evaluation to 1,500 real-world ZTF alerts, emphasizing a three-stage chain of metadata grounding, scientific reasoning, and hierarchical classification, together with explicit analysis of honesty and calibration (Chen et al., 7 May 2026).

Within that landscape, AstroMMBench is differentiated by its focus on astronomical image understanding using figures extracted from current arXiv preprints. It is neither a text-only graduate-level knowledge benchmark nor an operational alert-triage benchmark. Instead, it evaluates whether MLLMs can interpret the visual forms in which astronomy is routinely communicated in the literature. This suggests that AstroMMBench fills a methodological gap between textual astronomy exams and downstream observatory workflows.

The benchmark’s release under a CC BY 4.0 license, together with its modular design for future updates using new arXiv papers, is intended to support open, fair, and reproducible evaluation while mitigating data leakage as models evolve (Shi et al., 29 Sep 2025).

7. Limitations and prospective extensions

The initial release of AstroMMBench focuses on multiple-choice visual question answering. The paper explicitly identifies possible future expansions to open-ended, multi-step, or predictive reasoning tasks. This limitation is substantive rather than incidental: multiple-choice accuracy is efficient and standardized, but it does not exhaust the spectrum of scientific reasoning behaviors relevant to astronomy (Shi et al., 29 Sep 2025).

A second limitation is the cost of expert review. Because 15 domain experts were involved in validating quality and relevance, rapid expansion is constrained by human curation overhead. Ongoing work is described as aiming to automate and scale quality curation. A plausible implication is that the long-term success of the benchmark family will depend on maintaining expert-level rigor while reducing marginal annotation cost (Shi et al., 29 Sep 2025).

A common misconception is that strong performance on generic multimodal leaderboards is sufficient evidence of scientific readiness. AstroMMBench was explicitly designed to test that assumption through its comparison with OpenCompass scores, while astronomy-specific text benchmarking has already shown that models with similar general benchmark performance can diverge substantially on astronomy tasks (Shi et al., 29 Sep 2025, Ting et al., 2024). The benchmark therefore functions not only as a leaderboard but also as an instrument for separating general multimodal competence from specialized scientific visual reasoning.