AstroReason-Bench: Evaluating Unified Agentic Planning across Heterogeneous Space Planning Problems

Abstract: Recent advances in agentic LLMs have positioned them as generalist planners capable of reasoning and acting across diverse tasks. However, existing agent benchmarks largely focus on symbolic or weakly grounded environments, leaving their performance in physics-constrained real-world domains underexplored. We introduce AstroReason-Bench, a comprehensive benchmark for evaluating agentic planning in Space Planning Problems (SPP), a family of high-stakes problems with heterogeneous objectives, strict physical constraints, and long-horizon decision-making. AstroReason-Bench integrates multiple scheduling regimes, including ground station communication and agile Earth observation, and provides a unified agent-oriented interaction protocol. Evaluating on a range of state-of-the-art open- and closed-source agentic LLM systems, we find that current agents substantially underperform specialized solvers, highlighting key limitations of generalist planning under realistic constraints. AstroReason-Bench offers a challenging and diagnostic testbed for future agentic research.

• The paper introduces a unified evaluation benchmark that tests LLM-based agentic planning across diverse, physics-constrained space tasks.
• The paper compares zero-shot LLM agents with specialized solvers over five canonical space planning problems, highlighting significant performance gaps.
• The paper reveals both emergent reasoning strengths and critical limitations in resource management, suggesting hybrid planning and active exploration for future research.

Evaluating Unified Agentic Planning for Space: Insights from AstroReason-Bench

Motivation and Problem Setting

Agentic LLM systems have become viable candidates for generalist planners, purportedly capable of spanning diverse tasks by leveraging language-mediated reasoning, tool use, and long-horizon decision-making. Existing evaluation protocols, however, rarely challenge these systems with domains exhibiting tightly coupled physical and combinatorial constraints. AstroReason-Bench directly fills this gap by introducing a suite of Space Planning Problems (SPPs)—including ground station communication, agile Earth observation, regional coverage, stereo imaging, and low-latency backhaul scheduling—characterized by nontrivial physics, differentiated resource/kinematic boundaries, and heterogeneous objectives.

AstroReason-Bench's critical innovation is its unified, agent-centric protocol that exposes these task environments to LLM-based agents through well-structured interfaces and simulation backends. The explicit comparison between zero-shot LLM-based agents and specialized solvers unveils the limits of current agentic planning capabilities in operationally challenging, high-stakes engineering domains.

Figure 1: The transition from isolated, domain-specific algorithms to a unified agentic paradigm allowing coordinated scheduling with a single central agent and cross-domain toolkit.

Environment and Interface Architecture

The architecture underpinning AstroReason-Bench is modularized into four primary layers. The Physics Layer delivers stateless, high-fidelity simulations utilizing the SGP4 orbit propagator, a geometric slew model for agile maneuvers, and strict models for energy and data storage resources. The Scenario Layer maintains experiment-specific state, including persistent catalogs and action logs, ensuring scenario integrity via atomic operations. The Interface Layer exposes MCP-based semantic tools (for JSON-level feedback and constraint diagnostics) and a Python API (enabling scripting and batched API queries), bridging the gap between the agent and the environment. The uppermost Cognitive Layer runs the LLM agent, typically using a ReAct loop.

Figure 2: The environment is hierarchically structured, isolating physics from scenario state, and providing agents with programmatic and semantic access for robust planning.

Benchmark Suite Composition

AstroReason-Bench operationalizes five canonical satellite planning problems:

• SatNet/DSN Scheduling: Resource allocation for competing ground station requests, evaluated with unsatisfied request ratios.
• Revisit Optimization: Minimizing revisit gaps for dynamic targets, blending maintenance of periodicity and storage/downlink constraints.
• Regional Coverage: Polygonal area mapping, requiring decomposition of complex regions into observation strips with careful kinematic/resource planning.
• Stereo Imaging: Planning observation pairs with specified geometric and temporal separation, crucial for 3D reconstruction under tight constraints.
• Latency Optimization: Multi-hop network topologies for persistent, low-latency ground connectivity, intersecting ISL and observation objectives.

Each environment is procedurally generated from real-world satellite constellations and target databases. Physical models, scenario complexity (e.g., resource contention, temporal horizon), and mission objectives are algorithmically specified to guarantee diversity and realism; scenarios are serialized in standard formats for reproducibility.

Experimental Protocol and Results

Model and Baseline Selection

Evaluation comprises both open and closed-source state-of-the-art LLM systems (e.g., Claude Sonnet 4.5, Gemini 3 Flash, DeepSeek V3.2, Qwen3, Kat Coder Pro, Nex N1), all deployed with agentic scaffolding enabling tool use, simulation queries, and program synthesis. These agents operate entirely zero-shot, with limited agent-environment interaction budgets and resource-constrained compute configurations.

Baselines include prior published results for SatNet (greedy, MILP, RL/PPO) and domain-specialized heuristics or metaheuristics (greedy, simulated annealing) for the novel benchmarks—implemented for canonical comparison rather than optimized performance ceilings.

Summary of Quantitative Findings

On traditional DSN scheduling, agentic LLMs consistently outperform trivial heuristics but remain significantly behind specialized combinatorial solvers (e.g., MILP achieves $U_{rms} = 0.30$ vs best LLM at $U_{rms} \approx 0.53$). In revisit optimization, simulated annealing is superior to any LLM (best LLM at $M_{gap}=18.83$h), but LLMs exhibit nontrivial resource coordination relative to greedy approaches. Regional coverage exposes fundamental agent weaknesses—most agents fail to exploit ground track information, with even the best achieving only 0.11 coverage ratio. In stereo imaging, classical constructive heuristics universally fail, but LLM agents achieve up to 0.18 coverage by explicit doublet reasoning. For the latency benchmark, almost all agents fail to construct multi-hop networks except Kat Coder Pro, which realizes a nonzero mean availability and the lowest latency by explicitly organizing relay chains.

Figure 3: The multi-hop ISL relay constructed by Kat Coder Pro demonstrates correct spatial reasoning, in contrast to the single-hop misconception prevalent among other agents.

Figure 4: In regional mapping, agentic reasoning about near-polar orbits in 'plan mode' enables correctly-aligned observation strips, enhancing coverage over the default, orientation-agnostic strategy.

Analysis of Agentic Strengths and Failure Modes

LLM agents demonstrate the ability to reason over compound, non-local constraints and exhibit emergent skills (e.g., recognizing stereo pairs, constructing relay chains). However, their planning suffers from severe action bias and limited exploration—they frequently fail to leverage environment querying tools effectively, preferring recall and fixed heuristics over situational analysis. Resource lifecycle coordination (especially for storage and downlink management) remains a critical bottleneck. Case study analysis further reveals that pure RAG augmentation brings marginal improvement unless coupled with explicit planning phases.

Implications and Future Research Directions

AstroReason-Bench demonstrates that current agentic LLMs, even with advanced tool use and simulation access, remain substantially inferior to specialized solvers in tasks necessitating exhaustive search, long-horizon coordination, or rigorous resource management. The ability of some agents to outperform baselines in highly compound tasks (e.g., stereo, multi-hop relays) suggests that hybrid agent-optimization approaches, closed-loop exploration, and structured planning could yield measurable gains. For progress, research must address: (1) stronger integration of active learning and exploration, (2) incorporation of formal planning phases into agent loops, (3) better handling of temporal and resource coupling, and (4) systematic exposure to domain-specific priors via structured knowledge injection.

AstroReason-Bench positions itself as a rigorous evaluation suite, grounding agentic AI research in domains where real-world physical validity, operational reliability, and adaptability are essential. The architecture and open-source framework allow for methodologically robust extensions to deeper architectural design, lifelong mission planning, and mixed-initiative human-agent collaboration in space systems.

Conclusion

AstroReason-Bench operationalizes a critical step toward evaluating and improving generalist agentic planners in physics-constrained, heterogeneous task domains. The benchmark's integrated environments and diagnostic scenarios reveal both the adaptability and the profound limitations of current LLM agents, laying a foundation for future systems that must unify reasoning, simulation-driven exploration, and robust optimization to be viable in operationally critical AI for space missions (2601.11354).