GeoNatureAgent Benchmark: Benchmarking LLM Agents for Environmental Geospatial Analysis Across Frontier and Open-Weight Foundation Models

Abstract: Environmental scientists spend disproportionate effort on data wrangling rather than analysis, and AI agents that automate geospatial workflows remain unvalidated: no benchmark evaluates agents operating through structured tool calling against real APIs. We introduce the GeoNatureAgent Benchmark, the first benchmark for environmental analysis agents that operate via structured tool calls to a production-style geospatial API. It comprises 93 tasks across 18 categories, covering municipality analysis, multi-turn conversation, spatial reasoning, cross-indicator synthesis, error handling and recovery, ranking, comparison, multilingual understanding, habitat analysis, and task rejection. Tasks are evaluated against an open, self-hostable API serving three environmental indicators across Spain and Portugal via sixteen tools. We evaluate seven LLMs (Claude Sonnet 4, DeepSeek V3.2, GLM-5, Gemini 2.5 Pro, Qwen3-235B, GPT-OSS-120B, Llama 4 Scout) under three temperature-1.0 seeds, reporting capability and per-case cost as orthogonal axes. We find: (1) Claude Sonnet 4 leads at 60.8% +/- 0.8%, followed by DeepSeek V3.2 at 56.3% +/- 3.1%, with no other model above 51%; (2) the cost-accuracy Pareto frontier is occupied mostly by open-weight models, with DeepSeek V3.2 offering 93% of Claude's capability at 11x lower cost ($0.011/case); (3) comparison tasks remain universally unsolved (0% on close-value comparisons), exposing systematic reasoning limits; and (4) structured tool calling against a real API is more discriminative than general-purpose GIS benchmarks, with accuracies 25-35 points lower. We further show extensibility by integrating BigEarthNet V2 land cover for Portugal alongside Spanish CO2 and erosion indicators. The benchmark, harness, and self-hostable API are publicly available.

• The paper introduces a rigorous benchmark evaluating LLM agents with structured API tool calls to assess geospatial analysis in realistic, multi-turn workflows.
• It details a methodology using 93 tasks across 18 categories and precise binary scoring, exposing strengths and deficiencies in spatial reasoning and numeric comparison.
• The study highlights cost-performance trade-offs, showing open-weight models like DeepSeek V3.2 achieve near-proprietary accuracy at significantly lower costs.

GeoNatureAgent Benchmark: A Rigorous Evaluation of LLM Agents for Environmental Geospatial Analysis

Introduction

GeoNatureAgent Benchmark represents a substantive contribution to the evaluation of LLM agent capabilities in the specialized context of environmental geospatial analysis. Distinct from general GIS agent benchmarks, GeoNatureAgent Benchmark targets workflow automation through structured tool invocation against a production-grade cloud geospatial API, embodying scenarios reflective of real-world requirements in environmental data management, cross-indicator synthesis, and spatial reasoning.

Figure 1: GeoNatureAgent Benchmark system architecture. The LLM agent interacts with a production API serving COG data; the eval harness logs all tool calls and scores responses.

The motivation is drawn from persistent barriers in environmental informatics, where domain experts face steep learning curves in orchestrating multi-indicator spatial workflows. The benchmark consists of 93 tasks across 18 categories, stressing agent-level tool orchestration rather than mere pixel- or code-level accuracy. This enables the quantification of LLM proficiency under realistic, API-mediated conditions, a departure from prevailing LLM-as-judge or code-generation-centric protocols.

Benchmark Design and Evaluation Methodology

GeoNatureAgent Benchmark operationalizes environmental geospatial analysis as a sequence of multi-turn, tool-mediated interactions. The agent ecosystem exposes a suite of 16 tools—12 core domain operations and 4 GIS primitives—over an open, self-hostable FastAPI interface. The agent loop employs a ReAct paradigm, iteratively selecting tools based on observation, action, and reasoning phases.

The evaluation corpus spans cost-constrained, multi-lingual, and error-handling cases involving:

• CO$_2$ absorption suitability (Spain)
• Gully erosion probability (Spain, Portugal)
• BigEarthNet V2 land cover (Portugal)

Binary pass/fail scoring is enforced, with up to 8 checks per case (tool call recall, keyword inclusion/exclusion, numeric accuracy, action history, chart generation) (Figure 2). Partial-credit metrics (tool F1, check score) are computed for diagnostic resolution. No LLM-as-judge is utilized; all scoring is mechanistic and reproducible.

Figure 2: Scoring pipeline. Each case is evaluated by up to eight checks; binary pass requires all to pass. Partial-credit metrics (check score, tool F1, keyword coverage) are always computed.

Seven contemporary LLMs are evaluated under matched infrastructure, using consistent zero-shot instructions and temperature=1.0 sampling to assess agent reliability under non-determinism.

Results: Capability, Cost, and Agent Behavior

The main result set reveals a significant capability and cost spread across models (Figure 3). Claude Sonnet 4 exhibits maximal accuracy (60.8% ± 0.8%), trailed closely by DeepSeek V3.2 (56.3% ± 3.1%). The remainder—GLM-5, Gemini 2.5 Pro, Qwen3-235B, GPT-OSS-120B, and Llama 4 Scout—cluster distinctly lower, and none exceed 51%.

Figure 3: GeoNatureAgent Benchmark v5 leaderboard. Accuracy (%) with total cost in parentheses. Dashed line at 50%.

Cost-accuracy analysis is particularly revealing: open-weight models Llama 4 Scout, Qwen3-235B, and DeepSeek V3.2, alongside Claude, comprise the Pareto frontier (Figure 4). Notably, DeepSeek V3.2 achieves 93% of Claude’s accuracy at 11× lower cost ($0.011$/case), indicating strong viability for cost-sensitive deployment. Gemini 2.5 Pro and GPT-OSS-120B are strictly Pareto-dominated.

Figure 4: Cost-accuracy trade-off. Bubble size is proportional to total tokens; the Pareto frontier (dashed line) runs Scout, Qwen3-235B, DeepSeek V3.2, Claude Sonnet 4. Three of the four frontier models are open-weight.

Token consumption does not correlate with accuracy (Figure 5), reinforcing that performance is not trivially improved by larger contexts or generation volume.

Figure 5: Token consumption vs. accuracy. Token volume is a poor predictor of performance.

Partial-credit measurements demonstrate that a majority of failures are “near-misses”—incorrect in one check but otherwise sound (Figure 6). This underscores opportunities for architectural or prompt-engineering mitigation.

Figure 6: Binary accuracy vs partial-credit check score. The gap indicates many ``near-miss'' failures.

Task and Error Analysis

Granular analysis reveals persistent agent weaknesses in close-value comparison (0% success rate across all models), ranking, and multi-step numeric reasoning. All models hallucinate directional differences when presented with nearly identical values—a systematic LLM deficiency rather than isolated misrouting. Conversely, memory, multilingual understanding, and basic tool recall tasks are comparatively tractable (mean 66–82%).

Category-level accuracy divergence reveals that Claude Sonnet 4 attains maximal performance on habitat analysis (100%), multi-turn memory (100%), and cross-indicator synthesis (94%), but fails universally on small-margin comparison. DeepSeek V3.2 and GLM-5 display strengths in single analysis, spatial reasoning, and ranking. No single model is optimal across all task classes (Figure 7, Figure 8).

Figure 7: Category difficulty — mean accuracy across all seven models, sorted by difficulty.

Figure 8: Accuracy by category across the seven models. The close-value comparison sub-case defeats every model; at the category level only DeepSeek~V3.2 (33%) and GPT-OSS-120B (50%) exceed zero.

Tool selection, rather than model parameter count, drives per-category gains. The overall best LLM remains well below general GIS benchmarks; GeoNatureAgent’s top score (60.8%) is 25–35 points lower than GeoBenchX or GeoJSON Agents.

Implications and Benchmark Advances

GeoNatureAgent Benchmark demonstrates that environmental geospatial workflows pose substantially increased challenges relative to generic GIS evaluation sets. Key differences include:

• Necessity for geospatial knowledge (legal criteria, cross-indicator synthesis)
• Tool chains involving real API interaction and multi-step reasoning
• Significant penalties for hallucination or misapplication of non-analyzable layers
• Strict binary scoring with no LLM-as-judge leniency

The structured tool-calling paradigm reduces execution failures prevalent in free-form code generation (as seen in UnivEARTH, with 58% code failure rates), justifying production preference for API-oriented designs. Open-weight models are now not only competitive, but cost-dominant at equivalent or near-equivalent capability, warranting reevaluation of proprietary/closed-model economics for operational deployments in earth observation.

GeoNatureAgent is further validated as domain-agnostic by seamless integration of BigEarthNet V2, expanding the data and country domain with minimal framework adjustment.

Theoretical and Practical Outlook

Practically, this research enables more discriminative and realistic benchmarking of geospatial LLM agents in environmental informatics. Theoretically, it establishes a baseline that is both more robust (no LLM-as-judge) and more reflective of production workflows in scientific or regulated contexts.

Anticipated future developments include:

• Expansion to additional indicators (e.g. NDVI, precipitation, biodiversity) and geographies
• Prompt ablation studies and advanced multi-agent decompositions (e.g., Planner-Worker)
• Dedicated error mitigation for hard categories (ranking, margin comparison)
• Further investigation of partial-credit failures for model refinement

The cost-performance findings presage a shift towards cost-efficient open LLMs and the use of domain-specific benchmarking to capture operational challenges not represented by general LLM test sets.

Conclusion

GeoNatureAgent Benchmark fills a critical gap for LLM evaluation in environmental geospatial analysis, exposing current model limitations in real-world tool orchestration even among the most advanced closed and open-weight LLMs. The results demonstrate that cost-optimized open LLMs are viable alternatives to proprietary agents, but that foundational challenges in spatial and numerical reasoning remain unsolved. The benchmark, codebase, and API ecosystem are open-sourced, positioning GeoNatureAgent as the reference suite for future developments in agentic environmental AI evaluation (2606.12821).