Stargazer: A Scalable Model-Fitting Benchmark Environment for AI Agents under Astrophysical Constraints

Abstract: The rise of autonomous AI agents suggests that dynamic benchmark environments with built-in feedback on scientifically grounded tasks are needed to evaluate the capabilities of these agents in research work. We introduce Stargazer, a scalable environment for evaluating AI agents on dynamic, iterative physics-grounded model-fitting tasks using inference on radial-velocity (RV) time series data. Stargazer comprises 120 tasks across three difficulty tiers, including 20 real archival cases, covering diverse scenarios ranging from high-SNR single-planet systems to complex multi-planetary configurations requiring involved low-SNR analysis. Our evaluation of eight frontier agents reveals a gap between numerical optimization and adherence to physical constraints: although agents often achieve a good statistical fit, they frequently fail to recover correct physical system parameters, a limitation that persists even when agents are equipped with vanilla skills. Furthermore, increasing test-time compute yields only marginal gains, with excessive token usage often reflecting recursive failure loops rather than meaningful exploration. Stargazer presents an opportunity to train, evaluate, scaffold, and scale strategies on a model-fitting problem of practical research relevance today. Our methodology to design a simulation-driven environment for AI agents presumably generalizes to many other model-fitting problems across scientific domains. Source code and the project website are available at https://github.com/Gudmorning2025/Stargazer and https://gudmorning2025.github.io/Stargazer, respectively.

• The paper demonstrates that statistical optimization can produce good fits while failing to extract valid physical parameters in exoplanet detection.
• It employs a scalable, seed-driven synthetic pipeline paired with anonymized real RV data to challenge agents across varying task complexities.
• Empirical results reveal that increased computational resources lead to recursive failure loops, underscoring the need for feedback-driven hypothesis revision.

Stargazer: Evaluating AI Agents on Physics-Grounded Model Fitting in Exoplanet Discovery

Introduction

The "Stargazer: A Scalable Model-Fitting Benchmark Environment for AI Agents under Astrophysical Constraints" (2604.15664) study presents a benchmark tailored to expose the limitations of autonomous AI agents on the end-to-end, iterative model-fitting pipeline for exoplanet detection and characterization with radial-velocity (RV) time series data. Unlike prior agentic and LLM benchmarks that are predominantly static, retrieval-oriented, or limited to symbolic problem domains, Stargazer operationalizes tasks with realistic astrophysical complexity, requiring agents to synthesize data-driven inference with physical constraints and provide scientifically interpretable results. The benchmark is meticulously engineered to reveal a fundamental dissociation between statistical goodness-of-fit and the ability to extract valid physical system parameters, a challenge that encapsulates a central barrier for deploying AI in high-fidelity scientific workflows.

Figure 1: The Stargazer environment: tasks span synthetic and archival RV data, agents run the full periodogram-to-Keplerian workflow, but models often fit the data statistically while failing to recover correct orbital parameters.

Benchmark Architecture and Task Design

Stargazer includes 120 discrete tasks stratified by physical difficulty (20 Easy, 40 Medium, 40 Hard, 20 real-data) with complexity driven by signal-to-noise ratio, planetary multiplicity, orbital resonance, period coverage, observation count, and correlated stellar noise. The synthetic pipeline is seed-driven and infinitely scalable, supporting reproducibility and continual escalation of task complexity as model capabilities improve. Real RV measurements from the literature are anonymized and domain-randomized to mitigate data contamination and ensure that agents operate only on empirical time series, not prior system metadata.

Tasks require agents to execute the complete researcher workflow—periodogram analysis, iterative Keplerian fitting, residual diagnostic, model selection, and result submission—with physical model parameters (orbital period $P$, semi-amplitude $K$, eccentricity $e$, argument of periastron $\omega$, mean longitude $l$). Test-time resource constraints are imposed that mimic practical computational limits encountered in actual research settings.

Figure 2: Stargazer's agentic loop: task synthesis from real and synthetic data, agents receive feedback (BIC, RMS, Match, Count) and iteratively refine their submissions.

Evaluation Metrics and Criteria

A submission is deemed successful if, and only if, it simultaneously passes four stringent criteria:

• Statistical Fit:
  • RMS residuals must not exceed $1.5\times$ the median observational uncertainty.
  • $\Delta$BIC per-point must be positive (penalizing over-parameterized models).
• Physical Validity:
  • Match score ($S_\mathrm{match}$, Hungarian assignment on Keplerian orbits) must surpass 0.8.
  • The submitted planet count must exactly equal the true count.

The conjoined gate on fit and physical recovery sharply discriminates between spurious statistical solutions and genuine physical inference, disallowing degenerate fits that do not reconstruct the underlying planetary system.

Figure 3: Match score distribution and pass rates as a function of threshold—model rankings are robust to the match threshold, indicating the discriminative sharpness of the physical criteria.

Empirical Findings

Baseline and LLM Agent Performance

Classical analytical pipelines (Lomb-Scargle plus multi-start Kepler fitting) achieve up to 95% pass rate on Easy tasks, but degrade catastrophically on Hard tasks, especially for systems with multiple planets or strong aliasing. Sophisticated nested sampling pipelines exhibit the same brittleness. Among the eight LLM agents benchmarked, including contemporary GPT and Gemini variants:

• No agent achieves nonzero pass rate on any real-data task, despite these systems being solved by humans and data provenance verification.
• On synthetic data, agents reach 70–80% on Easy, 17–35% on Medium, and $\leq6\%$ on Hard. Most agents default to statistical optimization that fails to generalize to the correct physical model in noisier, more entangled scenarios.
• Statistical fit rates (BIC/RMS) remain high across the board, while physical recovery (match and count) collapses as difficulty increases.

  Figure 4: Decomposition of success rates: statistical criteria remain robust, but physical criterion pass rates plummet as task difficulty escalates, indicating a reasoning bottleneck.

Resource Utilization and Scaling

Contrary to expectations, increasing the computational or token budget at test-time does not translate to improved pass rates; agents simply engage in recursive failure loops, resubmitting the wrong hypothesis. Pass@3 analysis (multiple independent episode attempts) demonstrates that apparent performance gains arise from stochasticity, not from any meaningful exploration of the hypothesis space.

Skills Injection

Providing procedural skill summaries, distilled from successful episodes, boosts efficiency (e.g., more tasks reach submission within budget) but has minimal impact on physical recovery rates in hard tasks. For harder regimes, blindly following template workflows can actually degrade residual analysis and hypothesis escalation, underscoring the necessity of adaptive, evidence-driven reasoning rather than rote protocol execution.

Case Studies

Exemplar success and failure trajectories demonstrate the core behavioral pathology: successful agents update their hypothesis when fit diagnostics (e.g., low match despite good RMS) are inconsistent and escalate model complexity, while failed agents overfit a single local minimum, resubmitting the same (often aliased) solution and exhausting budget with no new exploration.

Figure 5: RV fits for case studies: successful agent recovers both planets with high orbital fidelity; failed agent locks onto alias periods, producing low statistical error but poor physical match on a three-planet system.

Theoretical and Practical Implications

Stargazer's construction and results reveal the persistent limitation that numerical optimization—maximizing a likelihood, minimizing a loss—does not equate to successful scientific discovery in the presence of physical degeneracies and model misspecification. This distinction is crucial for the credible deployment of AI in scientific domains:

• Methodological Validity: Reliance on statistical metrics alone is insufficient for evaluating the scientific competence of AI agents. Future benchmarks must operationalize physically grounded and domain-meaningful criteria.
• Simulation-to-Real Transfer: Observed failures on real data, despite strong synthetic performance, highlight an urgent need to address the sim-to-real gap when training agentic RL models for scientific domains.
• Skills Transfer and Workflow RL: Pure protocol distillation or skills injection does not resolve model selection failure modes in complex multi-planet or low-SNR systems. Multi-level, feedback-based workflow reinforcement (with explicit triggers for hypothesis revision) is needed.
• Resource Efficiency: Increasing computational expenditure is not a solution to fundamental search and reasoning pathologies; models must efficiently allocate their hypothesis exploration and treat feedback as informative signals requiring model escalation.

  Figure 6: Pearson correlation analysis: SNR and planet multiplicity are the dominant negative predictors of task success, revealing the physical axes of reasoning breakdown.

Future Directions

The Stargazer environment and its evaluation methodology provide a scalable, physics-grounded protocol for RL fine-tuning, hierarchical workflow learning, and curriculum design for scientific AI agents. Immediate advancements should focus on integrating structured model search, diagnostic-driven hypothesis revision, and robust handling of ambiguous, noisy real-world datasets. Progress should be evaluated on simultaneous physical recovery and statistical fit, particularly on real astrophysical systems with known solutions but challenging data quality.

The authors' methodology is generically extensible to other domains where physical interpretability and iterative hypothesis testing are central—suggesting broader implications for automated discovery in, e.g., chemistry, climate modeling, and precision biology.

Conclusion

Stargazer exposes a sharp boundary in agentic AI competence: strong statistical fits do not guarantee valid physical inference. The benchmark's stratification, resource constraints, and real-data regime significantly raise the bar for evaluating scientific workflows in AI agents. Results demonstrate a pronounced reasoning deficit that must be addressed for agent-assisted or autonomous scientific research to become credible—demanding investment in robust, feedback-driven, and physics-aware AI design.