AstroVLBench: Multimodal Astronomy Benchmark

Updated 5 July 2026

AstroVLBench is a comprehensive multimodal astronomy benchmark that assesses scientific reasoning across diverse data modalities such as optical imaging, radio maps, and spectral data.
It extends earlier benchmarks by integrating multiple evaluation regimes, including visual Q&A, notebook execution, and end-to-end observational tasks verified by astronomy experts.
Empirical findings highlight modality-dependent performance gaps, underscoring the need for domain-adaptive pre-training and iterative refinement in model development.

Searching arXiv for AstroVLBench and related benchmarks to ground the article in the relevant papers. AstroVLBench is a multimodal astronomy benchmark lineage centered on evaluating whether contemporary language and vision-LLMs can perform scientifically meaningful astronomical reasoning across heterogeneous data representations. In the materials associated with this term, two closely related usages appear. First, AstroMMBench defines a benchmark for astronomical image understanding by multimodal LLMs (MLLMs), with its design explicitly described as a foundation for a broader AstroVLBench (Shi et al., 29 Sep 2025). Second, AstroVisBench is described as a code benchmark for scientific computing and visualization in astronomy and is presented in its detailed description as “AstroVLBench,” emphasizing notebook-based workflow execution and visualization assessment (Joseph et al., 26 May 2025). A later paper formalizes AstroVLBench as a “comprehensive benchmark” of over 4,100 expert-verified instances spanning five observational astronomy tasks and multiple modalities, including optical imaging, radio interferometric imaging, spectral energy distributions, time-domain light curves, and optical spectroscopy (Ren et al., 27 Apr 2026). Taken together, these works define AstroVLBench as an evolving benchmark paradigm for astronomy-specific multimodal evaluation, extending from image-based multiple-choice question answering to end-to-end computational workflows and finally to systematic, modality-diverse observational reasoning.

1. Origins and conceptual scope

AstroMMBench was conceived as “the first dedicated multimodal benchmark for probing the astronomical image interpretation capabilities of modern vision-LLMs,” and its “design, scope, and evaluation protocols can serve as an exemplar for a broader AstroVLBench” (Shi et al., 29 Sep 2025). In this formulation, AstroVLBench is not initially a single fixed dataset, but a generalization of domain-specific multimodal benchmarking in astronomy to “other data modalities and richer question types.”

AstroVisBench advances that generalization along a different axis. It is defined as “the first publicly released benchmark that tests an LLM’s ability to implement real astronomy research workflows—spanning catalog queries, image-and-spectrum processing, scientific analysis, and domain-specific visualization—inside a Jupyter notebook environment” (Joseph et al., 26 May 2025). Its detailed description explicitly labels AstroVisBench as “a.k.a. ‘AstroVLBench’,” indicating that the term was also used for a benchmark of scientific computing and visualization rather than only visual question answering.

The 2026 AstroVLBench paper consolidates the term into a single end-to-end observational benchmark. It describes AstroVLBench as “the first end-to-end, multi-modal benchmark to probe whether today’s most advanced vision-LLMs (VLMs) can reliably perform core tasks in observational astronomy,” comprising $N = 4\,133$ expert-verified examples across five tasks (Ren et al., 27 Apr 2026). This suggests a maturation of the benchmark concept from prototype and benchmark family into a standardized, task-structured evaluation suite.

A plausible implication is that “AstroVLBench” names both a concrete benchmark introduced in 2026 and a broader benchmarking agenda already anticipated by AstroMMBench and AstroVisBench. The shared motivation across all three works is the inadequacy of general multimodal benchmarks for astronomy-specific reasoning, scientific visualization, and domain-constrained interpretation.

2. Benchmark lineage and constituent formulations

The benchmark lineage can be organized into three related formulations, each emphasizing a distinct evaluation regime.

Benchmark	Primary emphasis	Core format
AstroMMBench	Astronomical image understanding	621 four-choice multiple-choice questions
AstroVisBench	Scientific computing and visualization in astronomy	Jupyter notebook tasks with validators
AstroVLBench	Observational astronomical reasoning across modalities	4,133 expert-verified classification instances

AstroMMBench assembles “621 four-choice, multiple-choice questions drawn from real research figures in the ‘Astrophysics’ (astro-ph) category on arXiv (January–July 2024)” and distributes them across six core subfields: Astrophysics of Galaxies, Cosmology and Nongalactic Astrophysics, Earth and Planetary Astrophysics, High-Energy Astrophysical Phenomena, Instrumentation and Methods, and Solar and Stellar Astrophysics (Shi et al., 29 Sep 2025). Its explicit target is astronomical image understanding.

AstroVisBench instead decomposes “each end-to-end notebook into a sequence of atomic ‘tasks,’ each evaluated separately.” These tasks are divided into processing tasks, denoted $t_{\mathrm{process}}$ , and visualization tasks, denoted $t_{\mathrm{visualize}}$ (Joseph et al., 26 May 2025). The benchmark evaluates code generation for astronomy pipelines as well as the scientific fidelity of resulting figures.

The later AstroVLBench is task-based in a different way. It spans five observational astronomy tasks: QSO host-galaxy classification, Fanaroff–Riley radio-galaxy morphological classification, multi-wavelength SED classification, time-domain light-curve classification, and hierarchical spectral interpretation (Ren et al., 27 Apr 2026). Its scope is therefore broader in modality than AstroMMBench and more focused on direct scientific inference than AstroVisBench’s notebook-execution setting.

This progression is methodologically significant. AstroMMBench isolates visual-physical synthesis on research figures; AstroVisBench isolates executable research workflows and visualization conventions; AstroVLBench formalizes modality-dependent observational reasoning. This suggests a benchmark family with complementary operating points rather than a single homogeneous resource.

3. Data construction and task design

AstroMMBench uses a two-stage, model-assisted curation pipeline. It begins by harvesting “3,592 TeX submissions,” extracting “19,299 image–caption–context triples,” and then standardizing captions and snippets with “LLaMA 3.3-70B-Instruct.” InternVL 2.5-78B then produces “an initial question plus three distractors” (Shi et al., 29 Sep 2025). To suppress “verbal shortcut” items, five smaller open-source LLMs attempt each question five times, and any item “answered correctly by two or more of these models is discarded.” A panel of 15 astronomy experts then reviews 1,800 remaining questions according to “image-question alignment, context completeness, uniqueness of the correct answer, and necessity of domain knowledge,” yielding the final 621 items. Correct options are “deliberately re-assigned to A–D with uniform frequency to remove positional bias” (Shi et al., 29 Sep 2025).

AstroVisBench derives its tasks from “110 Jupyter notebooks curated by NOIRLab (Astro Data Lab) and STScI, intended as tutorials for graduate students and researchers” (Joseph et al., 26 May 2025). Its data include “SDSS catalogs, Gaia star catalogs, HST and ground-based FITS images, Kepler light curves, synthetic model outputs.” The preprocessing pipeline consists of setup, data retrieval, data cleaning, and “Pickling Key Products” for validation. In this benchmark, the task design preserves notebook state by providing “context cells,” a natural-language query, the model’s “single code cell answer,” and “a validator that checks the result” (Joseph et al., 26 May 2025).

The 2026 AstroVLBench uses “real survey data” and defines each task as a scientifically meaningful classification problem (Ren et al., 27 Apr 2026). The task inventory is highly explicit:

Task	Input representation	Question
Task 1	RGB composite of Hyper Suprime-Cam $g,r,i$ cutouts	AGN point-source PSF vs. extended stellar bulge light
Task 2	FIRST and NVSS radio maps	FR I vs. FR II morphology
Task 3	Log–log SED plot	Type 1 AGN vs. Type 2 AGN vs. galaxy
Task 4	Multi-band light-curve plots	Five-class variability classification
Task 5	Smoothed DESI DR1 optical spectra	Hierarchical spectral interpretation

Task 1 has $N = 557$ and asks whether the nucleus is dominated by a “seeing-limited point-source PSF” or “extended stellar bulge light.” Task 2 uses radio maps from FIRST at “5″ resolution” and NVSS at “45″ resolution,” with $N = 833 \times 2$ . Task 3 uses a broad-band SED spanning optical, Euclid, AKARI, and WISE bands, with $N = 168$ . Task 4 uses LSST-like simulated multi-band light curves with $N = 142$ . Task 5 uses DESI DR1 spectra for three subquestions: H $\alpha$ + H $\beta$ detection ( $t_{\mathrm{process}}$ 0), broad-line AGN identification ( $t_{\mathrm{process}}$ 1), and BPT classification ( $t_{\mathrm{process}}$ 2) (Ren et al., 27 Apr 2026).

In aggregate, these benchmark designs demonstrate three distinct curation strategies: literature-figure extraction and expert vetting, notebook-task decomposition with executable validation, and observational task construction over survey-native modalities.

4. Evaluation protocols and metrics

AstroMMBench evaluates models “via VLMEvalKit on exact-match accuracy” (Shi et al., 29 Sep 2025). The overall accuracy is defined as

$t_{\mathrm{process}}$ 3

with $t_{\mathrm{process}}$ 4. Per-domain accuracy is defined as

$t_{\mathrm{process}}$ 5

To compare benchmark performance with general multimodal capability, AstroMMBench computes a Pearson correlation coefficient between each model’s AstroMMBench score and OpenCompass leaderboard score, obtaining $t_{\mathrm{process}}$ 6 (Shi et al., 29 Sep 2025). The benchmark also measures difficulty post hoc by the number of models out of 25 answering each item correctly, with most items in the “medium-difficulty regime (5–15 models correct).”

AstroVisBench uses a more heterogeneous metric suite because it evaluates both code and plots. For processing tasks, it reports “Execution Success Rate” and “Variable Inspection Score (VIscore),” where

$t_{\mathrm{process}}$ 7

For visualization tasks, it again reports execution success, “VisFail%,” and a three-level error taxonomy: “No Error (1),” “Minor Error (2),” and “Major Error (3)” (Joseph et al., 26 May 2025). The benchmark also evaluates an automatic visualization judge using “Spearman $t_{\mathrm{process}}$ 8,” with the judge compared against expert labels on a subset of tasks.

The 2026 AstroVLBench adopts conventional classification metrics. It reports “classification accuracy and, where relevant, per-class recall,” with all confidence intervals given as “95 % bootstrap (10 000 resamples)” (Ren et al., 27 Apr 2026). Accuracy is defined as

$t_{\mathrm{process}}$ 9

and precision, recall, and $t_{\mathrm{visualize}}$ 0 are also specified. This benchmark therefore emphasizes statistical reliability and classwise behavior in addition to point estimates.

A common feature across all three formulations is that they attempt to separate superficial benchmark performance from domain-grounded competence. AstroMMBench filters out language-only shortcuts, AstroVisBench validates intermediate computational artifacts and assesses scientific plot quality, and AstroVLBench distinguishes final-label accuracy from reasoning quality.

5. Empirical results and performance patterns

AstroMMBench evaluates “25 diverse MLLMs, including 22 open-source and 3 closed-source models” (Shi et al., 29 Sep 2025). The top three are “Ovis2-34B (open-source): 70.53%,” “ChatGPT-4o (closed-source): 69.07%,” and “Doubao-1.5-Vision-Pro (closed-source): 68.12%.” Performance varies substantially by subfield. Models “uniformly do best in IM and SR” and “struggle in CO and HE,” with InternVL3-38B scoring “80.5% in IM” but “53.2% in CO” (Shi et al., 29 Sep 2025). Smaller architectures such as Gemma3-4B and InternVL3-1B remain near “45%–50% overall,” while larger open models narrow the gap to proprietary systems.

AstroVisBench reports results over “432 tasks” in both the processing and visualization stages (Joseph et al., 26 May 2025). In the processing stage, “code crash rates ranged from ∼31% (best: Gemini 2.5 Pro) up to ∼54% (GPT-4o),” and VIscore ranged from “0.480 (GPT-4o) up to 0.694 (o3-mini).” In the visualization stage, crash rates are “29–61%,” while “VisFail% (no or >1 figure) is small (2–9%), but model-judged Major Errors account for 15–31% of tasks.” “No Error% ranges only 8–16%,” and “even the strongest model (Gemini 2.5 Pro) achieves only ~16% perfectly correct visualizations” (Joseph et al., 26 May 2025).

The later AstroVLBench evaluates six frontier VLMs in a “zero-shot guided setting (temperature = 0 for deterministic outputs)” (Ren et al., 27 Apr 2026). Gemini 3 Pro is the most consistently strong model across tasks, but leadership remains task-dependent. On Task 1, Gemini 3 Pro reaches $t_{\mathrm{visualize}}$ 1 accuracy with 95% CI $t_{\mathrm{visualize}}$ 2. On Task 2, it attains $t_{\mathrm{visualize}}$ 3 on FIRST and $t_{\mathrm{visualize}}$ 4 on NVSS. On Task 3, it reaches $t_{\mathrm{visualize}}$ 5, with per-class recall showing severe asymmetry: “Galaxy $t_{\mathrm{visualize}}$ 6, Type 1 AGN $t_{\mathrm{visualize}}$ 7, Type 2 AGN $t_{\mathrm{visualize}}$ 8.” On Task 4, Gemini 3 Pro attains $t_{\mathrm{visualize}}$ 9. For Task 5, Claude 4.5 leads Q1 H $g,r,i$ 0+H $g,r,i$ 1 detection at $g,r,i$ 2, while Gemini 3 Pro leads Q2 broad-line AGN identification at $g,r,i$ 3 and Q3 BPT classification at $g,r,i$ 4 (Ren et al., 27 Apr 2026).

These results support two recurrent observations stated in the sources. First, performance is highly modality- and task-dependent. Second, general-purpose multimodal or LLMs remain substantially limited on astronomy-specific workloads, especially when the task requires precise scientific conventions, physically grounded interpretation, or numerically delicate classification boundaries.

6. Mechanistic findings, limitations, and future directions

AstroMMBench identifies specific difficulty patterns. Current MLLMs face “persistent hurdles in interpreting high-energy phenomena and cosmological simulations, where multi-step reasoning over statistical maps, X-ray emission contours, or cosmographic fits is required,” whereas “standard plot types (light curves, histograms, transmissivity curves) and familiar stellar or instrument imagery are well handled” (Shi et al., 29 Sep 2025). The paper states that future MLLMs will need “domain-adaptive pre-training on astronomy-specific visual data and jargon,” “enhanced reasoning modules capable of chaining across multiple figure panels and textual annotations,” and “integration of symbolic and physical priors.”

AstroVisBench reaches a related conclusion from the perspective of code and figure generation. It finds that “broad-sweeping domain APIs (matplotlib, numpy) are easier,” while “niche calls (ADQL, astropy.cosmology, healpy.mollview) stump models” (Joseph et al., 26 May 2025). Common plotting errors include “wrong axis orientation,” “poor choice of color scales,” “cut-off data ranges,” and “missing legends or labels.” The benchmark also notes that models “rarely chain iterative refinements,” instead producing a single figure rather than the human pattern of repeated adjustment and rerun.

The 2026 AstroVLBench adds mechanistic ablations. For Gemini 3 Pro, prompt guidance materially changes performance. On Task 1, “Unguided = 0.641; Phenomenological = 0.697; Physical = 0.745,” while on Task 4, “Unguided = 0.528; Phenomenological = 0.655; Physical = 0.613” (Ren et al., 27 Apr 2026). The paper interprets this as evidence that “describing ‘what to look for’ sharpens visual focus,” whereas explaining “why it matters” yields more balanced decisions and may improve robustness. Representation also matters: on Task 4, replacing plots with an “ordered-table” improves Gemini 3 Pro from $g,r,i$ 5 to $g,r,i$ 6, while on Task 3 the table representation reduces performance from $g,r,i$ 7 to $g,r,i$ 8 (Ren et al., 27 Apr 2026).

A further limitation concerns explanation fidelity. AstroVLBench distinguishes “Right-Answer–Right-Reason” from “Right-Answer–Wrong-Reason,” documenting cases in which models produce correct predictions with “physically inaccurate justification” (Ren et al., 27 Apr 2026). The paper explicitly concludes that “accuracy alone is insufficient for scientific trust.”

Across the benchmark lineage, several limitations are stated directly. AstroMMBench notes its “exclusive reliance on multiple-choice VQA” and “a modest total size (621 items)” (Shi et al., 29 Sep 2025). AstroVisBench emphasizes the large gap between current models and “reliable astronomy research assistants” (Joseph et al., 26 May 2025). AstroVLBench recommends limiting “zero-shot reliance to spatial morphology tasks,” while asserting that for “SEDs, light curves, and diagnostic lines, domain-specialized models or fine-tuned VLMs remain essential” (Ren et al., 27 Apr 2026).

The future directions also align. AstroMMBench proposes that AstroVLBench should incorporate “spectral data matrices, time-domain light-curve streams, radio interferometric visibilities, and even simulation volumetric maps,” expand into specialized subfields such as “gravitational waves” and “astrochemistry,” and introduce “interactive, multi-turn dialogues” as well as open-ended reasoning and generative tasks (Shi et al., 29 Sep 2025). AstroVisBench proposes “real observational uncertainties and multi-band data,” “end-to-end hypothesis testing,” and “human-in-the-loop evaluation” (Joseph et al., 26 May 2025). AstroVLBench recommends providing structured numerical tables when feasible, using physical grounding prompts, and validating reasoning chains via expert review or automated consistency checks (Ren et al., 27 Apr 2026).

Taken together, these works characterize AstroVLBench not merely as a dataset but as a research program in astronomy-specific multimodal evaluation. Its central premise is that trustworthy scientific deployment requires simultaneous assessment of representation choice, physical grounding, reasoning fidelity, executable workflow competence, and modality-aware generalization.