AstroBench: AI Benchmarking in Astronomy

• AstroBench is a comprehensive suite of domain-specific benchmarks designed to evaluate large language models and multimodal AI on tasks like image interpretation, scientific reasoning, and code visualization.
• It employs expert-validated datasets and rigorous evaluation protocols—including accuracy, confidence calibration, and error taxonomy—to systematically expose model deficits in specialized astronomical workflows.
• Key recommendations include integrating astrophysical retrieval modules, fine-tuning on domain-specific code, and continuously updating datasets to enhance AI performance in scientific research.

AstroBench refers collectively to a set of domain-specific benchmarking protocols for evaluating LLMs and multimodal LLMs (MLLMs) on real astronomical tasks, including knowledge recall, scientific reasoning, image interpretation, event classification, scientific code synthesis, and visualization. Developed to address the unique challenges posed by specialized astronomical data and workflows, key variants under the AstroBench umbrella include AstroMMBench, AstroAlertBench, AstroVisBench, and the original AstroBench question set. These benchmarks are curated and validated by expert astronomers and are now central to the evaluation and advancement of LLMs in astronomy, exposing both general and fine-grained deficits overlooked by standard benchmarks.

1. Genesis and Motivation

Astronomy’s demand for highly specialized knowledge, visual interpretation, and code-driven data analysis makes general-purpose AI benchmarks insufficient for scientific evaluation. Standard LLM and MLLM evaluations, focused on general tasks or synthetic benchmarks, fail to cover unique modalities such as astronomical imagery, instrumentation plots, event alerts, and code-based scientific workflows. AstroBench benchmarks were designed to:

• Rigorously test recall, reasoning, and code synthesis in real astronomical contexts.
• Evaluate tasks at a level relevant to PhD-level research and daily scientific workflows.
• Establish objective, expert-validated standards for AI evaluation in astronomy.
• Identify domain-specific failure cases that hamper the applicability of generic AI systems.

2. Benchmark Variants and Dataset Construction

AstroMMBench

AstroMMBench (commonly referred to as “AstroBench”) is the first large-scale, domain-specific benchmark for probing MLLMs using real astronomical images and expert-curated multiple-choice questions. Its dataset consists of 621 MCQs distributed across six core astrophysical subfields (GA: 97, CO: 111, EP: 105, HE: 110, IM: 87, SR: 111). Question types include image-based identification, plot and curve interpretation, conceptual reasoning, quantitative inference, and multi-step chain reasoning.

Data were extracted (19,299 image–text pairs) from 3,592 arXiv astro-ph articles published Jan–Jul 2024. LLaMA3.3-70B-Instruct assisted in rewriting captions, InternVL2.5-78B generated MCQs, and multi-model filtering retained only challenging items. Final curation and answer validation were performed by 15 domain experts, each holding at least an M.S. in astronomy, resulting in a rigorously filtered pool with balanced answer keys (Shi et al., 29 Sep 2025).

Original AstroBench

The original AstroBench, as described by the AstroMLab group, is a community-driven, astronomy-specific question set constructed from 4,425 MCQs automatically generated by prompting Gemini-1.5-Pro with 885 articles from the Annual Review of Astronomy and Astrophysics (1963–2023). Questions are labeled by cognitive skill and subfield following arXiv astro-ph taxonomy. The curation involved both automatic protocols and spot checks by experts, though estimated ambiguity rates remain a few percent. This benchmark is designed explicitly to evaluate recall and reasoning at the PhD level using chain-of-thought prompting and explanation (Ting et al., 2024).

AstroAlertBench

AstroAlertBench (sometimes called “AstroBench” in informal usage) is a multimodal benchmark aimed at evaluating LLMs on the interpretation and classification of real-time astronomical event alerts, using a structured three-stage chain: metadata grounding, scientific rationale, and hierarchical classification (artifact/reality, origin type, astrophysical subtype). The dataset comprises 1,500 Zwicky Transient Facility alerts, balanced across five classes, with each item combining image montages and raw metadata for comprehensive context. This benchmark is further distinguished by its metrics for “honesty” (self-reported reasoning confidence and calibration) and its inclusion of a human-in-the-loop protocol via a Zooniverse project for reference accuracy (Chen et al., 7 May 2026).

AstroVisBench

AstroVisBench tests LLMs on scientific computing and visualization workflows relevant to astronomy. It consists of 864 tasks derived from 110 real Jupyter notebooks, parsed into setup, processing (e.g., data ingestion, cleaning, measurement, fitting), and visualization (e.g., color–magnitude diagrams, light curves, sky coordinate plots). Evaluation relies both on execution-based scoring for processing tasks (variable inspection) and a two-phase error taxonomy for visualization, employing both expert human annotation and a vision-capable LLM-as-judge pipeline. The benchmark exposes task-specific failure modes in LLM-generated code and plots (Joseph et al., 26 May 2025).

Benchmark Variant	Domain Focus	Input Modality
AstroMMBench	Image QA	Astronomical images
AstroBench (orig.)	QA	Text MCQs
AstroAlertBench	Event review	Images + metadata
AstroVisBench	Code/vis	Notebooks, code, figures

3. Evaluation Protocols and Metrics

AstroBench variants employ a range of rigorous evaluation metrics calibrated for domain-specific reasoning and scientific correctness.

Multiple Choice QA: Accuracy is measured as

$$\mathrm{Accuracy} = \frac{N_\text{correct}}{N_\text{total}}$$

with subfield-resolved performance reported in AstroMMBench, and Wilson Score intervals for uncertainty in the original AstroBench (Shi et al., 29 Sep 2025, Ting et al., 2024). For ranking, a Pearson correlation coefficient $r$ is measured versus general multimodal evaluation scores.

Confidence Calibration: Top models report confidence as $\max p(A \ldots D)/(\sum p)$, with calibration curves and bin-wise empirical accuracy. Pearson $r$ between confidence and actual correctness, and mean absolute calibration offsets, are explicitly calculated (Ting et al., 2024).

Classification & Reasoning (AstroAlertBench):

• End-to-end classification accuracy per stage and overall five-class accuracy.
• “Honesty” metrics: mean self-reasoning score (MSRS), self pass rate, calibration gap ($\Delta_\mathrm{cal}$), and Pearson $r$ between self-rated confidence and empirical correctness.
• “Second-rollout” behavior tracks self-correction rates (Chen et al., 7 May 2026).

Scientific Code and Visualization (AstroVisBench):

• Processing: Variable Inspection Score (VIscore) and crash fraction.
• Visualization: Error categories (No Error, Minor Error, Major Error), precision, recall, F1. LLM-as-judge metrics are validated against expert consensus (Spearman ρ ≈ 0.82) (Joseph et al., 26 May 2025).

4. Key Results Across Benchmarks

AstroMMBench revealed that open-source Ovis2-34B achieved 70.5% accuracy, exceeding leading closed-source models such as ChatGPT-4o and Doubao-1.5-vision-pro. Subfield-wise, models performed best on Instrumentation and Methods (IM, >78%), with the lowest performance in Cosmology (CO) and High-Energy (HE), indicating the sustained difficulty of abstract theoretical and multi-step reasoning tasks.

Original AstroBench established that top proprietary models like Claude-3.5-Sonnet reach 85.0% overall accuracy, with open-weights models (LLaMA-3-70B, Qwen-2-72B) now within 4–5 percentage points of the best proprietary models. Accuracy gains now scale logarithmically with inference cost, with a 10× cost increase yielding ≈3.5 percentage-point performance improvement. Confidence calibration for leading models is now approaching deployment thresholds, with r > 0.9 between self-assessed and empirical accuracy (Ting et al., 2024).

AstroAlertBench found that accuracy in the critical fine-grained classification stage ranged from 32% to 61%, with overall best end-to-end scores of 60.6% (Claude Opus 4.7 think). Models exhibit high accuracy in metadata grounding, but confusion between AGN and variable stars is systematic, highlighting the need for temporal context and external catalogues. “Reasoning-enabled” models gain +7–12 pp accuracy, but population-level analyses reveal that more capable models are systematically more self-critical in their reasoning scores (Chen et al., 7 May 2026).

AstroVisBench demonstrated that best-in-class models have high crash rates (up to 64%) on code execution and struggle to produce correct scientific visualizations (only ≈16% “No Error” rate). Hallucinated file paths, incorrect use of astronomy APIs, and non-standard plotting conventions are common pitfalls. Even with a validated LLM-judge evaluating figures, models fall short on both correctness and adherence to domain visual standards (Joseph et al., 26 May 2025).

5. Analytical Insights and Failure Modes

AstroBench benchmarks expose multiple domain-driven and architectural limits:

• Abstract conceptual reasoning and multi-hop inference remain limiting in CO and HE questions (Shi et al., 29 Sep 2025).
• Historical and technical subfields (exoplanets, instrumentation, stellar evolution) pose persistent deficits, especially for models not extensively trained on specialist English corpora (Ting et al., 2024).
• Visualization errors in scientific code outputs frequently result from misunderstanding conventions (e.g., axis inversion for magnitudes, coordinate ordering), lack of robust handling for domain-specific database queries, and missing iterative code refinement steps (Joseph et al., 26 May 2025).
• In event classification, all models exhibit AGN-variable star confusion, and some are “blind” to specific classes (e.g., Gemini on asteroids). Honesty metrics indicate that model self-assessment, while improved at the aggregate, remains noisy at the instance level (Chen et al., 7 May 2026).

6. Impact, Extensions, and Recommendations

AstroBench has catalyzed the use of expert-guided, execution-based, and multimodal evaluation protocols in astronomy AI research. The convergence of open-weight and proprietary models, logarithmic scaling of cost–accuracy tradeoffs, and the affirmation that chain-of-thought reasoning boosts performance are all evidenced in these benchmarks.

Principal recommendations include:

• Integrating explicit astrophysical retrieval modules and curated knowledge bases to improve theoretical inference and subfield coverage.
• Expanding to open-ended, multi-step, and explanation-style tasks for deeper assessment of scientific reasoning and not just answer selection (Shi et al., 29 Sep 2025).
• Incorporating temporal context and light-curve data in event classification, and supporting external catalogue queries (Chen et al., 7 May 2026).
• Fine-tuning on domain code repositories and interactive tool-augmented protocols (e.g., for ADQL, Astropy, visualization) to ameliorate code execution and plotting performance (Joseph et al., 26 May 2025).
• Maintaining and periodically refreshing datasets to avoid saturation and support discovery-driven science as LLMs are deployed at scale.

7. Outlook

With the sustained progress documented across AstroBench, domain-specific benchmarks have become indispensable for illuminating advances and gaps in AI models’ scientific capabilities. AstroBench and its derivatives serve as definitive yardsticks for evaluating LLMs’ readiness for integration into astronomical research, supporting not only quantitative comparisons but also the principled development of new architectures, training regimes, and agent-based workflows for the data-intensive era of astronomy (Shi et al., 29 Sep 2025, Ting et al., 2024, Chen et al., 7 May 2026, Joseph et al., 26 May 2025).