GeoNatureAgent Benchmark

Updated 6 July 2026

GeoNatureAgent Benchmark is a specialized evaluation framework for environmental analysis agents that interact through structured tool calls to a production-grade geospatial API.
It integrates 16 tools over 93 tasks across 18 categories, assessing model accuracy and cost efficiency in handling real environmental data from Spain and Portugal.
Experimental results highlight Claude Sonnet 4’s leading 60.8% accuracy while exposing consistent challenges in near-equality comparisons and grounded reasoning.

GeoNatureAgent Benchmark is a benchmark for evaluating environmental analysis agents that operate through structured tool calls to a production-style geospatial API rather than through free-form code generation or text-only reasoning. It is defined as “the first benchmark for environmental analysis agents that operate via structured tool calls to a production-style geospatial API,” and it targets whether an agent can translate natural-language environmental queries into correct API interactions, interpret returned spatial data, and produce correct answers under realistic workflow constraints. The benchmark comprises 93 tasks across 18 categories, spans Spain and Portugal, exposes 16 tools over three environmental indicators, and evaluates both capability and cost as orthogonal axes. In the reported experiments, Claude Sonnet 4 leads at $60.8\% \pm 0.8\%$ , DeepSeek V3.2 follows at $56.3\% \pm 3.1\%$ , and comparison tasks remain a systematic failure mode, including $0\%$ accuracy on close-value comparisons (Diaz-Ireland et al., 11 Jun 2026).

1. Definition, motivation, and scope

GeoNatureAgent Benchmark is motivated by a specific operational gap in environmental geospatial analysis: environmental scientists spend disproportionate effort on data wrangling, dataset discovery, coordinate reprojection, debugging remote-sensing API interactions, and assembling multi-step analysis pipelines rather than on substantive environmental reasoning. The benchmark is designed to test whether LLM agents can reduce that burden under conditions closer to deployment than static GIS question answering or code synthesis. Its stated target is not generic geospatial competence, but environmental analysis via structured tool calling against a real API (Diaz-Ireland et al., 11 Jun 2026).

The benchmark’s core claim of novelty is tied to three design choices taken together: environmental workflow coverage, structured tool calling, and real API interaction. The paper argues that prior work did not jointly evaluate environmental workflows, structured API use, multi-turn interaction, multilingual inputs, cross-indicator synthesis, task rejection, and strict non-LLM scoring. It also argues that code-generation-centered evaluation is poorly aligned with production deployment, noting that 58% of LLM-generated Earth Engine code fails to execute. This positions GeoNatureAgent Benchmark as an evaluation of agent behavior under operational constraints rather than as an evaluation of code fluency alone (Diaz-Ireland et al., 11 Jun 2026).

Its scope is intentionally narrower than “all GeoAI” and more specific than general GIS-assistant evaluation. The benchmark covers municipality analysis, spatial reasoning, cross-indicator synthesis, error handling and recovery, ranking, comparison, multilingual understanding, habitat analysis, temporal change, deep-dive profiling, and explicit rejection of unsolvable tasks. The paper further argues that this structured-tool, API-centered setting is more discriminative than earlier general GIS benchmarks, with accuracies roughly 25–35 percentage points lower than those reported on easier general-purpose geospatial evaluations (Diaz-Ireland et al., 11 Jun 2026).

2. API architecture, tools, and environmental data substrate

The benchmark is built around an open, self-hostable geospatial API implemented as a FastAPI service. The API is described as production-style and cloud-backed, and the benchmark architecture treats the agent as operating in a ReAct-style loop: receive query, select tools, execute calls, inspect responses, and then generate a final response. The harness normalizes heterogeneous model outputs into a common tool-call schema, including an AST-based parser for Llama 4 Scout, whose tool calls are emitted as Python-style function invocations rather than JSON (Diaz-Ireland et al., 11 Jun 2026).

The tool surface contains 16 tools: 12 principal operations and 4 auxiliary tools. The principal tools are lookup_province, lookup_municipality, analyze_area, analyze_multi_layer, compare_areas, find_top_n, toggle_layer, generate_chart, create_buffer, select_spatial_rel, get_centroids, and reject_task. The four auxiliary tools are list_layers, get_legend, get_layer_bounds, and query_erosion_stats. The first eight principal tools are domain-specific, while the last four are adapted from GeoBenchX. This design combines high-level environmental operations with a small set of lower-level spatial primitives (Diaz-Ireland et al., 11 Jun 2026).

The API serves Cloud Optimized GeoTIFFs for raster indicators and pre-computed JSON statistics for land-cover distributions. It exposes three environmental indicators and seven additional display-only layers that are intentionally not available for statistical analysis, creating explicit hallucination traps.

Indicator	Scope and type	Serving format
CO $_2$ absorption suitability	Spain; categorical; five MITECO spatial criteria; classes “Not Eligible”, “Eligible with Conditions”, “Eligible”	COG
Gully erosion probability	Europe; continuous; derived from LUCAS 2022 soil survey; range 0–100%	COG
BigEarthNet V2 land cover	Portugal; categorical; 75k+ Portuguese patches aggregated by 9 mainland districts into a 7-class distribution	pre-computed JSON

The BigEarthNet V2 integration functions both as a new data source and as an extensibility demonstration. The benchmark aggregates 75k+ Portuguese Sentinel-2 patches into seven land-cover classes—Coniferous Forest, Broadleaf Forest, Shrubland, Grassland, Sparse Vegetation, Water, and Urban—and makes them queryable through the same analyze_area interface used elsewhere. The benchmark, harness, and API are reported as publicly available and self-hostable, with software and dataset artifacts archived on Zenodo (Diaz-Ireland et al., 11 Jun 2026).

3. Task construction, taxonomy, and benchmark instances

GeoNatureAgent Benchmark contains 93 tasks across 18 categories, with a difficulty distribution of 19% easy, 45% medium, and 36% hard. Each task specifies a natural-language query, optional multi-turn history, expected tool calls, must-contain strings, must-not-contain strings, maximum rounds, cost budget, and domain-expert ground truth. The benchmark also includes deliberately unsolvable tasks that require graceful rejection rather than hallucinated analysis, including prompts about non-existent municipalities, unsupported indicators, and display-only layers (Diaz-Ireland et al., 11 Jun 2026).

The 18 categories, their counts, and their intended test focus are as follows.

Category	Tasks	Test focus
Tool selection	21	correct tool choice
Cross-indicator	8	CO $_2$ × erosion × land cover synthesis
Deep dive	6	multi-tool environmental profiling
Interpretation	7	reasoning over analysis results
Error handling	6	non-existent entities, invalid indicators
Habitat analysis	7	BigEarthNet V2 land cover
Language	6	Galician, Basque inputs
Municipality	4	municipality-level analysis, disambiguation
Memory (multi-turn)	6	follow-up questions, context retention
Spatial reasoning	4	coastal, meseta, community knowledge
Multi-muni ranking	3	multi-municipality comparisons
Threshold	3	numeric threshold queries
Temporal change	1	cross-country temporal context
Error recovery	3	graceful fallback when data is unavailable
Ranking	2	top- $N$ and bottom- $N$ province queries
Comparison	2	close-value and directional comparisons
Single analysis	2	basic single-province analysis
Province aggregation	2	aggregate statistics across provinces

A notable subset of the benchmark is the BigEarthNet-specific task family. The paper reports 20 purpose-built BigEarthNet benchmark cases across seven patterns: single-district land-cover breakdown (3), cross-indicator Portugal–Spain synthesis (3), multi-turn recall from session history (2), chart generation (2), legend or layer discovery (2), deep-dive multi-indicator profile (4), and two-district comparison (4). Portuguese land-cover data also appears in four additional tasks elsewhere in the benchmark, yielding 24 BigEarthNet-related cases in total (Diaz-Ireland et al., 11 Jun 2026).

This task construction makes the benchmark distinctively heterogeneous while preserving a single interaction model. Some categories probe tool orchestration directly; others probe semantic interpretation of returned statistics; others test grounded refusal behavior. A plausible implication is that the benchmark is designed less as a uniform skill test than as a structured decomposition of environmental analysis agency into operational sub-capabilities (Diaz-Ireland et al., 11 Jun 2026).

4. Evaluation protocol, scoring logic, and execution controls

All models are evaluated under a fixed protocol: the v3 system prompt, zero-shot ReAct-style instruction, max_turns = 10, max_tokens = 4096, and temperature = 1.0. The paper states that temperature 1.0 was chosen deliberately to expose run-to-run variance and stress-test tool reliability under realistic sampling noise rather than to optimize best-case deterministic performance. Prompting was held constant across models: no few-shot prompting, no chain-of-thought variants, and no self-reflection variants (Diaz-Ireland et al., 11 Jun 2026).

The evaluation uses three seeds $\{42, 1337, 2024\}$ for models accessed through Vertex MaaS endpoints. Claude Sonnet 4, accessed through the Anthropic Messages API, does not expose deterministic seeding, so the paper reports two independent temperature-1.0 samples under otherwise identical settings. Experiments were run sequentially inside a single Cloud Run Job to keep infrastructure conditions fixed, and result rows include seed, case_id, and experiment_id for traceability (Diaz-Ireland et al., 11 Jun 2026).

Each task may activate up to eight checks:

expected_tools
expected_actions
must_contain
must_not_contain
numeric_accuracy
chart_generated
max_rounds
max_cost_usd

The pass criterion is deliberately strict: “binary capability pass requires every applicable check except max_cost_usd to pass.” Thus capability and cost are reported separately. For expected_tools, recall must equal 1.0, but extra tools do not fail the case. Tool F1 is computed but not used as a gate. For numeric_accuracy, the default tolerance is $\pm 2$ percentage points, and the numeric parser searches a 120-character window after the label for the first float-percent pattern. Per-case budgets default to \$0.10 in the v5 task set, but cost overruns do not invalidate capability pass/fail (Diaz-Ireland et al., 11 Jun 2026).

The benchmark also reports continuous diagnostics: check score, keyword coverage, tool F1, and cost utilisation. When a task fails, the first failing check is assigned according to a priority order: tool_missing → chart_missing → wrong_data → rounds_exceeded → keyword_missing → forbidden_keyword. This ordering is intended to make tool failures the most visible diagnostic signal. The scoring design is therefore mechanistic and non-LLM-based, contrasting with judge-driven semantic equivalence schemes (Diaz-Ireland et al., 11 Jun 2026).

5. Experimental results and empirical behavior

The main leaderboard on 93 tasks shows a relatively compressed capability range among the top models and a wide cost range across the full set.

Model	Accuracy	\$ / case
Claude Sonnet 4	$60.8\% \pm 0.8\%$	\$0.127
DeepSeek V3.2	$56.3\% \pm 3.1\%$ 0	\$0.011
GLM-5	$56.3\% \pm 3.1\%$ 1	\$0.038
Gemini 2.5 Pro	$56.3\% \pm 3.1\%$ 2	\$0.052
Qwen3-235B	$56.3\% \pm 3.1\%$ 3	\$0.010
GPT-OSS-120B	$56.3\% \pm 3.1\%$ 4	\$0.089
Llama 4 Scout	$56.3\% \pm 3.1\%$ 5	\$0.003

The paper emphasizes four headline findings. First, Claude Sonnet 4 leads overall, but the reported gap to DeepSeek V3.2 is only 4.5 percentage points; the Welch comparison yields $56.3\% \pm 3.1\%$ 6, $56.3\% \pm 3.1\%$ 7, and $56.3\% \pm 3.1\%$ 8, so the top-two separation is not decisive at the available sample size. Second, the Pareto frontier is dominated by open-weight systems at the low-cost end: Llama 4 Scout, Qwen3-235B, DeepSeek V3.2, and Claude Sonnet 4 define the frontier. Third, DeepSeek V3.2 delivers 93% of Claude’s capability at 11.6× lower per-case cost. Fourth, no model exceeds 65%, which the paper takes as evidence that environmental geospatial tool orchestration remains unsolved (Diaz-Ireland et al., 11 Jun 2026).

Category-level results reveal that benchmark difficulty is not uniform. The easiest categories are memory/multi-turn (mean 82%), habitat analysis (mean 71%), and language (mean 66%). Claude Sonnet 4 reaches 100% on habitat analysis, memory, and municipality tasks, and 94% on cross-indicator synthesis. GLM-5 is the strongest model on ranking at 67%. DeepSeek V3.2 shows the most balanced profile and exceeds 70% on cross-indicator, memory, multi-municipality ranking, single analysis, spatial reasoning, threshold, and municipality tasks. By contrast, tool selection is one of the most discriminative categories, and comparison is the hardest (Diaz-Ireland et al., 11 Jun 2026).

The benchmark’s most widely cited failure case is close-value comparison. In case V5_21, all model-seed trials fail: Murcia is reported at 68.5% and Córdoba at 68.4%, yet every system verbalizes a directional difference rather than near-equality. The models typically call the correct compare_areas tool and receive the correct numeric output, but fail in the final reasoning step. The second comparison case, V5_22, is easier when the directional difference is larger: DeepSeek V3.2 succeeds in 2 of 3 seeds, GPT-OSS-120B succeeds in 3 of 3, and the remaining five models fail. The paper interprets this as a systematic weakness in near-equality judgment rather than a universal inability to compare magnitudes (Diaz-Ireland et al., 11 Jun 2026).

Ranking is a second major failure mode. Category-level ranking performance ranges from 0% to 67%, with typical errors including wrong tool choice—such as using analyze_area instead of find_top_n—and confabulated rankings based on model priors rather than API results. This supports the paper’s broader claim that structured tool access does not by itself guarantee grounded reasoning (Diaz-Ireland et al., 11 Jun 2026).

6. Position in the benchmark literature, significance, and limitations

GeoNatureAgent Benchmark sits at a specific point within the emerging literature on geospatial and Earth-science agents. GeoAnalystBench evaluates Python-based geospatial workflow and code generation with static workflow-validity, semantic-similarity, and CodeBLEU-style metrics, but it does not use a live tool API and does not center environmental workflows (Zhang et al., 7 Sep 2025). GeoBenchX evaluates multistep GIS-assistant behavior over local files and general-purpose GIS tools with an LLM-as-Judge framework and explicit solvable-versus-unsolvable tasks, but it is not environmental-analysis-specific and does not use metric-based mechanistic scoring over a production-style API (Krechetova et al., 23 Mar 2025). GeoAgentBench later advances dynamic execution with a sandbox of 117 atomic GIS tools and parameter-sensitive scoring, making runtime feedback first-class (Yu et al., 15 Apr 2026). TerraBench similarly argues for executable environmental workflows, process-level tool-use metrics, and tolerance-aware numeric scoring over heterogeneous Earth-system data (Nguyen et al., 11 Jun 2026). GeoDisaster extends the trajectory toward operational EO/GIS reasoning with executable workflows, deterministic checks, and step-level failure analysis in disaster settings (Hasan et al., 15 Jun 2026).

Against that background, GeoNatureAgent Benchmark’s distinct contribution is narrower but sharply defined: environmental analysis agents interacting through structured tool calls to a self-hostable API over Spain and Portugal, with multilingual prompts, explicit rejection behavior, chart generation, cost reporting, and a strict non-LLM scoring regime. The paper’s claim that its accuracies are 25–35 percentage points lower than those on easier general GIS benchmarks suggests that benchmark choice materially changes the apparent state of the field (Diaz-Ireland et al., 11 Jun 2026).

The benchmark also has clear limitations. The paper explicitly notes that strict binary pass/fail can understate real capability because many failures are near-misses. All evaluated systems use a single-agent ReAct loop; no Planner-Worker or multi-agent baselines are included. Coverage is limited to three indicators and two countries, with BigEarthNet land-cover statistics aggregated over nine mainland Portuguese districts. Variance estimates for Claude Sonnet 4 rest on only two samples. The paper reports no prompt ablations beyond the fixed zero-shot setting and no experiments with few-shot prompting, chain-of-thought variants, or self-reflection (Diaz-Ireland et al., 11 Jun 2026).

Future work proposed in the paper includes expansion to additional countries and indicators, especially NDVI, precipitation, and biodiversity indices; prompt ablations across zero-shot, CoT, and few-shot settings; multi-agent comparisons, particularly Planner-Worker architectures; and dedicated mitigation strategies for the comparison category. These directions suggest that the benchmark is intended not only as a leaderboard but also as a reusable evaluation harness for environmental agent design (Diaz-Ireland et al., 11 Jun 2026).