AstroMLab-1 Benchmark

• AstroMLab-1 Benchmark is an astronomy-specific multiple-choice benchmark designed to assess factual recall, calibration, and cost efficiency in astrophysics and cosmology.
• It comprises 4,425 rigorously generated MCQs covering subfields such as stellar structure, galaxy formation, cosmology, and astronomical instrumentation.
• Comparative analysis shows domain-specialized LLMs achieving up to 86.2% accuracy while significantly reducing inference costs relative to larger general-purpose models.

AstroMLab-1 is an astronomy-specific multiple-choice benchmark designed to rigorously evaluate the factual knowledge, recall, and practical utility of LLMs in astronomy, astrophysics, cosmology, and astronomical instrumentation. Its design and widespread adoption have established it as the de facto standard for cross-model comparison and ablation studies in domain-specialized LLMs applied to astronomical research and education (Haan et al., 2024, Ting et al., 2024, Haan et al., 23 May 2025).

1. Construction and Scope

AstroMLab-1 comprises 4,425 multiple-choice questions (MCQs), each with four or five answer options and exactly one correct answer. Questions were generated using large-context LLM prompting applied to the full text of Annual Review of Astronomy and Astrophysics (ARAA) chapters (spanning 2007–2023), but the source papers employed to produce the questions were withheld from LLM training and fine-tuning corpora. This exclusion protocol ensures that high benchmark scores reflect genuine generalization rather than verbatim recall of training content (Haan et al., 2024, Haan et al., 23 May 2025).

Questions cover all major subfields of astronomy:

• Astronomy and astrophysics (stellar structure, galaxy formation)
• Cosmology (large-scale structure, early universe physics)
• Astronomical instrumentation and methods (detector principles, telescope optics)
• Related subtopics including exoplanetary science and high-energy astrophysics

Post-hoc, questions have been classified into six topical domains (e.g., Solar & Stellar, Earth & Planetary, Instrumentation) and stratified by empirical difficulty, based on professional astronomer performance labels (“easy,” “medium,” “hard”)—though the released MCQ set contains only aggregated category distributions (Haan et al., 23 May 2025, Ting et al., 2024).

2. Evaluation Methodology and Metrics

Benchmarked models receive each question as a standalone prompt, restricted from accessing reference articles at inference time. The primary metric is accuracy, expressed as

$$\mathrm{Accuracy} = \frac{1}{N} \sum_{i=1}^N \mathbf{1}[y_i = \hat y_i ]$$

where $y_i$ is the ground-truth answer and $\hat y_i$ is the model prediction (Haan et al., 2024, Haan et al., 23 May 2025).

Additional evaluation metrics include:

• Wilson-score confidence intervals: For estimated accuracy $\hat p$ and $N$ questions, the binomial interval at significance $z$:

$$\mathrm{CI}_{\rm Wilson} = \frac{ \hat p + \frac{z^2}{2N} \pm z\,\sqrt{ \frac{\hat p (1-\hat p)}{N} + \frac{z^2}{4 N^2} } }{ 1+ \frac{z^2}{N} }$$

• Expected Calibration Error (ECE) and Brier Score: Calibration metrics assess the agreement between model confidence and correctness. ECE, for $M$ bins $B_m$, is computed as

$$\mathrm{ECE} = \sum_{m=1}^{M} \frac{|B_m|}{N} \big|\, \mathrm{acc}(B_m) - \mathrm{conf}(B_m) \big|$$

• Pearson correlation coefficient between answer confidences and empirical accuracy

All evaluation is performed under macro-averaging, assigning equal weight to each question.

3. Dataset Composition and Workflow

AstroMLab-1 questions are curated as follows:

• MCQ generation: Review articles are OCR-processed and parsed with high-context LLMs (e.g., Gemini-1.5-Pro) using prompt engineering to extract challenging, generalizable MCQs from each article, typically producing five questions per source. Generated questions are checked for technical precision, clarity, and relevance (Ting et al., 2024).
• Human expert verification: Domain experts vet a subset of questions for accuracy and difficulty, and performance of professionals (~68% average accuracy under time constraints) serves as a reference baseline.
• Difficulty stratification: Post-hoc labels are assigned (easy: >80% expert accuracy; medium: 60–80%; hard: <60%), but only aggregated category-level statistics are currently released (Haan et al., 23 May 2025).
• Content withholding: All text from ARAA review chapters used to generate benchmark items is excluded from all LLM training and fine-tuning, enforcing unseen generalization.

4. Model Performance and Comparative Analysis

AstroMLab-1 enables comprehensive comparison of both proprietary and open-weight LLMs. Key results include:

• State-of-the-art results:
  • Proprietary models (Claude-3.5-Sonnet: 85.0%, GPT-4o: 80.4%) and the best open-weight models (LLaMA-3-70B: 80.6%) now achieve performance exceeding unaided professional astronomers working at speed (Haan et al., 2024, Ting et al., 2024).
  • Domain-specialized models significantly outperform general-purpose models at equal or lower parameter count. For example, AstroSage-Llama-3.1-8B achieves 80.9% accuracy—an 8-point gain over its Llama-3.1-8B base, equaling GPT-4o and 70B–90B parameter models at a fraction of the cost (Haan et al., 2024).
  • Among 8B-parameter checkpoints, no other model exceeds 75% accuracy.
  • The introduction of AstroSage-70B raises performance to 86.2%, surpassing both open-weight and proprietary models of higher cost and parameter scale (Haan et al., 23 May 2025).
• Subfield variation: Accuracy ranges from ~78% (solar/stellar, instrumentation) to >85% (galaxies) for top models. Lower performance is observed in subfields with sparser training data (e.g., exoplanets, instrumentation) and in high-complexity multi-step reasoning queries (Ting et al., 2024).
• Calibration: Recent proprietary and specialized models show strong calibration (Pearson $y_i$0), essential for research deployment. Most models are slightly underconfident.
• Cost-efficiency: Empirically, a ~3.5 percentage point accuracy gain is achieved for every tenfold increase in cost. Domain-specialized models such as AstroSage-8B provide flagship-level accuracy at less than 0.1% of the inference cost of large proprietary or open-weight 50B+ parameter models (Haan et al., 2024).

5. Design Strengths and Limitations

Strengths

1. Rigorous domain coverage: Questions cover all major astronomical subfields, with balanced topical representation and alignment to developments in the field.
2. Human verification: Quality assurance with professional astronomers ensures high technical validity for MCQs (Haan et al., 2024, Haan et al., 23 May 2025).
3. Generalization enforcement: Withholding of all generator-source ARAA articles from model training ensures true knowledge retrieval and synthesis.
4. Calibration and uncertainty quantification: The benchmark employs robust confidence and accuracy assessment, facilitating model trustworthiness analysis.

Limitations

1. Declarative focus: AstroMLab-1 evaluates factual recall and closed-form MCQ decision-making; it does not probe open-ended, generative, or agentic task capabilities (Ting et al., 2024).
2. Absence of detailed per-domain statistics: Public releases do not provide item-level or subdomain-stratified accuracy, limiting some fine-grained analyses (Haan et al., 2024).
3. Instruction-following: Smaller or less-supervised models remain less fluent in pure instruction-following, as shown by IF-EVAL benchmarks (Haan et al., 2024).
4. Lack of real-world workflow simulation: No coding, literature search, or end-to-end agent operations are included.

6. Implications and Future Directions

AstroMLab-1 has demonstrated the feasibility and value of domain-specialized benchmarking in astronomy and serves as a template for scientific LLM evaluation. The central finding is that targeted continued pre-training and domain-specific supervised fine-tuning enable mid-scale models (8B–70B) to surpass, at lower cost, the performance of much larger general-purpose LLMs (Haan et al., 2024, Haan et al., 23 May 2025). This suggests a paradigm shift in scientific natural language AI, emphasizing specialization over scale for research tasks.

Limitations in multi-step and problem-solving reasoning on MCQs signal that further evolution of the benchmark—in particular, integration of open-ended, generative, and code-based tasks—is critical to assessing “System 2” capabilities in LLMs for scientific research (Haan et al., 23 May 2025).

Future work will extend AstroMLab-1 to fully human-curated MCQs, stratified difficulty tiers, free-form and coding questions, and real-world data analysis agent simulations. This progression is anticipated to clarify both the boundaries and the emerging strengths of astronomy-focused LLMs as research partners (Ting et al., 2024, Haan et al., 23 May 2025).

7. Comparative Tabulation

The table below summarizes the key performance metrics and inference cost for high-performing models on AstroMLab-1 (Haan et al., 2024, Ting et al., 2024, Haan et al., 23 May 2025):

Model	Accuracy (%)	API Cost (rel.)
AstroSage-70B	86.2	1×
Claude-3.5-Sonnet	85.0	20×
GPT-4o	80.4	12×
LLaMA-3-70B	80.6	~0.05×
AstroSage-8B	80.9	~0.001×

This comparison reflects both the current state of the art and the transformative cost-efficiency gains from targeted, domain-specialized pre-training. Deployment at scale—such as inference on survey-scale astronomical catalogs—becomes feasible when leveraging models that attain high accuracy at low resource utilization.

---

AstroMLab-1 is now foundational to benchmarking and progress-tracking in the application of LLMs to astronomical domains, demonstrating that field-specific ML benchmarks are indispensable as scientific applications of large-scale LLMs accelerate (Haan et al., 2024, Ting et al., 2024, Haan et al., 23 May 2025).