A systematic evaluation of vision-language models for observational astronomical reasoning tasks

Abstract: Vision-LLMs (VLMs) are increasingly proposed as general-purpose tools for scientific data interpretation, yet their reliability on real astronomical observations across diverse modalities remains untested. We present AstroVLBench, a comprehensive benchmark comprising over 4,100 expert-verified instances across five tasks spanning optical imaging, radio interferometry, multi-wavelength photometry, time-domain light curves, and optical spectroscopy. Evaluating six frontier models, we find that performance is strongly modality-dependent: while one model (Gemini 3 Pro) emerges as the most consistently capable across tasks, task-specific strengths vary, and all models substantially underperform domain-specialized methods. Mechanistic ablations reveal that performance depends not only on directing attention to salient visual features but also on grounding those features in physical knowledge. Phenomenological prompts describing what to look for improve accuracy by sharpening model focus, but physical prompts explaining why those features matter perform better overall and yield more balanced classifications with reduced class-specific bias. Consistent with this picture, presenting the underlying one-dimensional measurements directly as numerical tables instead of rendered plots yields up to 13 percentage points improvement. Reasoning quality analysis further demonstrates that, without explicit physical grounding, models may reach correct predictions from phenomenologically plausible cues while providing physically imprecise justifications, establishing that accuracy alone is insufficient for trustworthy scientific deployment. These findings provide the first systematic, multi-modal baselines for VLMs in observational astronomy and identify the specific representation, grounding, and reasoning bottlenecks where current models fail.

• The paper introduces AstroVLBench, a benchmark for assessing vision-language models' capability to perform astronomical reasoning tasks using diverse scientific data modalities.
• It explores performance across five modalities, highlighting Gemini 3 Pro as the most consistent model, excelling in radio morphology and optical imaging tasks.
• Findings reveal gaps in model performance, showing bias and mode collapse, particularly in classification tasks and light curve interpretation, advocating for future representation learning advancements.

Systematic Evaluation of Frontier Vision-LLMs in Observational Astronomy with AstroVLBench

Introduction

The evaluation of foundation and multi-modal vision-LLMs (VLMs) on complex, heterogeneous scientific data has immediate consequences for the deployment of AI in research. This paper establishes AstroVLBench, the first comprehensive, expert-verified benchmark for assessing the capacity of six frontier VLMs—GPT-5.2, Claude Opus 4.5, Gemini 3 Pro, Grok-4, Qwen3-235B, Intern-S1-Pro—to execute authentic astronomical reasoning tasks across five distinct modalities: optical imaging, radio interferometry, spectral energy distribution (SED) analysis, time-domain photometric classification, and optical spectroscopy (2604.24589). The authors provide over 4,100 real, modality-diverse test cases targeting core physical distinctions in extragalactic astronomy, with a focus on AGN identification and related phenomena as a unifying cross-modal theme.

Benchmark Structure and Modality-Specific Tasks

AstroVLBench is constructed to dissect both perceptual and conceptual AI reasoning in astronomy. Each task aligns with a distinct data representation: (1) QSO host galaxy discrimination in optical images, (2) Fanaroff–Riley radio morphology in interferometric images, (3) AGN and galaxy classification from multi-wavelength SEDs, (4) five-way classification of time-domain light curves (AGN, SNIa, TDE, RRL, Mira), and (5) hierarchical interpretation of DESI optical spectra (emission line detection, BLAGN identification, BPT classification).

Figure 1: Overview of the five AstroVLBench task modalities, dataset class distributions, and a comparative radar plot of VLM model accuracy across tasks.

Quantitative Model Evaluation

Striking modality-dependent performance characteristics are demonstrated in baseline (guided prompt) evaluations. Gemini 3 Pro emerges as the most consistent, leading four out of five tasks, with notable accuracy on radio morphology (FIRST: 84.6%, NVSS: 55.3%), optical imaging (74.5%), and BPT spectral diagnostics (55.8%) (Figure 2). However, on tasks involving subtle photometric distinctions (e.g., SED shape; Task 3) or numerical temporal reasoning (Task 4), all models are outperformed by specialist astronomical algorithms and show class-specific bias and mode collapse. For instance, on the SED classification task, models achieve high galaxy recall but fail catastrophically for AGN, and on time-domain tasks, only Gemini 3 Pro surpasses 60% accuracy, with others defaulting to single-class predictions.

Figure 2: Model accuracy with 95% bootstrap confidence intervals across all five AstroVLBench modalities.

Representation and Prompt Mechanism Ablations

To elucidate the mechanistic sources of model success and failure, the paper performs systematic ablations using Gemini 3 Pro:

• Prompt Guidance: Physical prompts (those which explain why features matter) consistently yield more balanced performance and mitigate class-specific bias compared to phenomenological (describing what to look for) or unguided settings. For optical imaging (Task 1), physical prompting results in 74.5% accuracy, with stronger class balance versus phenomenological (69.7%) and unguided (64.1%) modes. For light curve classification (Task 4), phenomenological prompting slightly overtakes physical, but gains are concentrated in classes with overtly described observational artifacts (e.g., RRL variability).
• Data Representation: For the interpretation of time-series data, direct access to structured numerical tables (as opposed to rendered plots) produces up to 13 percentage point accuracy improvements. This gain is task-dependent: SED classification realizes minimal benefit, but light curve interpretation is significantly enhanced, indicating that model bottlenecks often lie in the fidelity of data structure rather than visual feature localization (Figure 3).

  Figure 3: Impact of prompt guidance and input representation (image-based vs. numerical table) on Gemini 3 Pro classification accuracy in selected tasks.

Failure Modes and Modal Reasoning Dissection

Numerical and qualitative analysis of confusion matrices reveals that performance gaps differ by modality. For example, in high-resolution radio morphology, Gemini 3 Pro displays the best precision/recall trade-off, but all VLMs degrade sharply on lower-resolution images where relevant morphological structure is lost (Figures 7 and 8). SED and spectral tasks expose a strong tendency toward conservative bias—defaults to “galaxy” or “star-forming” in ambiguous regimes—indicative of a failure to extract subtle accretion or excitation signatures.

Confusion matrices for each modality show that:

• Intern-S1-Pro and Grok-4 exhibit significant mode collapse depending on the task (e.g., predicting nearly all AGN or all galaxies in Task 1).
• Only Gemini 3 Pro largely avoids this pitfall in BLAGN and BPT spectral subtasks, preserving sensitivity to less common classes (Figures 12–14).

  Figure 4: Binary AGN/Galaxy confusion matrices for Task 1, demonstrating class biases and mode collapse in most models except Gemini 3 Pro.

Few-Shot Instruction and Visual Exemplars

The effect of few-shot visual exemplar prompting is systematically evaluated. Improvements are modest and highly task-dependent: emission line detection in spectra (Task 5, Q1) benefits with a +5% accuracy increase, reflecting the low variance, cue-focused nature of the task. However, radio morphology and BLAGN identification see negligible gains. This suggests that illustrative few-shot context is insufficient when relevant features are inherently diverse or continuous across classes (Figure 5).

Figure 5: Accuracy comparison for zero-shot and few-shot (exemplar-augmented) guidance in radio morphology and emission line detection tasks.

Logical Validity and the Right-Answer-Wrong-Reason Phenomenon

Qualitative reasoning chain analysis demonstrates that the decoupling of prediction accuracy and sound physical justification is pervasive. Models, including Gemini 3 Pro, can return correct labels while providing physically invalid or imprecise rationales, particularly for spectrum-based tasks. For example, even when emission lines are correctly identified, non-Gemini models frequently confuse rest-frame and observed wavelengths or cite the wrong lines, indicating reliance on phenomenological pattern matching rather than true physical understanding (Figure 6).

Figure 6: Hierarchical spectral reasoning chains from Gemini 3 Pro, illustrating valid physical logic; competitor models often fail at this level despite correct final labels.

Implications and Prospects for AI in Scientific Discovery

The results suggest that, while state-of-the-art VLMs encode general scientific knowledge and are competitive at broad classification tasks, they lack the precision, robustness, and logical reliability required for unsupervised scientific discovery. Performance ceilings are sharply modality-dependent, with substantial gaps relative to domain-specialist pipelines—especially for tasks rooted in precise quantitative or temporal relationships rather than gross morphological cues.

For practical deployment in astronomy, several axes require focused progress:

• Representation Learning: Enabling models to ingest and reason directly over structured, high-granularity numeric arrays—rather than through secondary visual encodings—will be necessary for science-quality inference in time-domain astrophysics and spectroscopy.
• Prompting and Trustworthiness: Prompts must support physical grounding rather than merely feature focus. Evaluation protocols should rate not only label accuracy but also the logical and physical validity of motivational chains to avoid “right-answer-wrong-reason” failures that undermine scientific trust.
• Domain Adaptation: Purpose-built adapters or retraining on instrument-specific distributions could close the remaining gap with specialist algorithms, but general-purpose VLMs alone are not currently competitive for precision applications.

Conclusion

AstroVLBench establishes the first rigorous, multi-modal baseline for VLM performance on authentic astronomical reasoning. The results reveal substantial challenges in generalizing across modalities, maintaining physically valid reasoning, and handling structured scientific data representations. The field must address these bottlenecks through methodological advances in prompt engineering, data structuring, and model training strategies before VLMs can be reliably trusted in high-stakes scientific workflows.

• "A systematic evaluation of vision-LLMs for observational astronomical reasoning tasks" (2604.24589)