EVGeoQA: Benchmarking LLMs on Dynamic, Multi-Objective Geo-Spatial Exploration

Abstract: While LLMs demonstrate remarkable reasoning capabilities, their potential for purpose-driven exploration in dynamic geo-spatial environments remains under-investigated. Existing Geo-Spatial Question Answering (GSQA) benchmarks predominantly focus on static retrieval, failing to capture the complexity of real-world planning that involves dynamic user locations and compound constraints. To bridge this gap, we introduce EVGeoQA, a novel benchmark built upon Electric Vehicle (EV) charging scenarios that features a distinct location-anchored and dual-objective design. Specifically, each query in EVGeoQA is explicitly bound to a user's real-time coordinate and integrates the dual objectives of a charging necessity and a co-located activity preference. To systematically assess models in such complex settings, we further propose GeoRover, a general evaluation framework based on a tool-augmented agent architecture to evaluate the LLMs' capacity for dynamic, multi-objective exploration. Our experiments reveal that while LLMs successfully utilize tools to address sub-tasks, they struggle with long-range spatial exploration. Notably, we observe an emergent capability: LLMs can summarize historical exploration trajectories to enhance exploration efficiency. These findings establish EVGeoQA as a challenging testbed for future geo-spatial intelligence. The dataset and prompts are available at https://github.com/Hapluckyy/EVGeoQA/.

• The paper introduces EVGeoQA, benchmarking LLMs on dynamic, multi-objective geo-spatial reasoning for EV charging and POI selection.
• It presents a novel dataset using softmax-weighted K-Means for realistic user location synthesis and dual-objective query generation.
• The study shows that explicit tool-augmented and meta-reasoning strategies improve LLM performance in long-horizon spatial planning.

EVGeoQA: A Benchmark for LLMs on Dynamic, Multi-Objective Geo-Spatial Exploration

Motivation and Problem Formulation

The paper addresses the fundamental limitations of current Geo-Spatial Question Answering (GSQA) benchmarks, which focus primarily on static fact retrieval and do not encapsulate the complexity of real-world, purpose-driven spatial planning constrained by user mobility, real-time location contexts, and multi-objective requirements. The paradigm shift advocated here involves benchmarking LLMs in dynamic, multi-objective geo-spatial reasoning, specifically in the electric vehicle (EV) charging context—a prototypical scenario requiring users to identify an optimal charging station that also satisfies a co-located activity preference (e.g., charging while dining). An illustrative instance is shown below.

Figure 1: An illustrative EVGeoQA query. Identifying the optimal target requires combining semantic understanding with real-time location and POI information.

Dataset Construction and Novelty

Diverse Urban Coverage and Multi-Source Location Synthesis

EVGeoQA encompasses three Chinese cities—Hangzhou, Qingdao, and Linyi—that represent a broad spectrum of urban scales and economic profiles. The spatial distribution of queries and Points of Interest (POIs) is visualized to emphasize both high-density urban and sparser peripheral contexts.

Figure 2: (a) Distribution of query-anchored locations in Qingdao, showing density along populated areas and roads. (b) Surrounding POI diversity and non-uniformity around charging stations.

To remedy the bias of synthetic or random coordinate sampling, user query anchors are generated via a softmax-weighted K-Means fusion of population heatmaps and road network features. This strategy produces location distributions closely approximating realistic urban user mobility.

Figure 3: Illustration of the Multi-Source Fusion strategy for user location synthesis.

Dual-Objective Query Generation

Unlike previous GSQA datasets restricted to static topological queries, EVGeoQA links each QA pair to a realistic user coordinate and a bipolar objective: (1) mandatory charging, and (2) a secondary, intent-driven activity satisfied by nearby POIs. Template-based initial seed queries are paraphrased with instruction-tuned LLMs to ensure both coverage and linguistic diversity, mapping static POI labels to functional user intentions. Ground-truth answers are exhaustively annotated with semantic matching and vehicle-distance prioritization, allowing for multiple valid solutions—more closely reflecting actual user constraints and planning under non-determinism.

Figure 4: Spatial distribution of charging stations across the three representative cities with a zoom-in on city centers to highlight density variation.

The GeoRover Evaluation Framework

To operationalize this benchmark and enable dynamic spatial exploration rather than single-shot retrieval, the paper introduces GeoRover: a tool-augmented agent framework. The agent orchestrates four atomic geo-spatial tools—SearchStations (localized charging station lookup), SearchPOIs (nearby activity filtering), ChangeLocation (arbitrary spatial navigation), and CalculateDistance (vehicle path metric)—enabling iterative, partial-observable search across complex city-scale environments.

The framework is architecturally agnostic, supports model-agnostic tool invocation, and leverages Few-Shot and Chain-of-Thought (CoT) prompting to encourage deeper reasoning.

Figure 5: GeoRover Framework Overview. The agent leverages interactive tools and trajectory synthesis for dual-constraint geo-spatial exploration.

Experimental Analysis

LLM Performance and Model Scaling

A suite of LLMs of varying capacity and architecture are evaluated, including Qwen3 (8B/30B), Qwen2.5-72B, GPT-OSS, and Gemini-2.5-Pro. The principal findings are as follows:

• Performance degrades rapidly with increased spatial exploration depth. For example, Qwen2.5-72B's Hits@2 drops from 0.6379 ($<10$ km) to 0.4878 (unconstrained range) in Hangzhou, highlighting a pronounced deficiency in long-horizon planning.
• Larger models exhibit greater robustness to search range expansion, consistent with scaling laws observed in other domains, but the overall accuracy in unconstrained long-range scenarios remains limited.
• LLMs equipped with explicit "Thinking" (i.e., a reflective search strategy leveraging trajectory summarization) outperform standard, reactive-only variants. This is particularly evident via increased ChangeLocation tool usage, leading to deeper environment exploration.

Tool Usage and Exploration Patterns

ChangeLocation, the tool enabling spatial navigation, is invoked only sparingly (on average 3–5 times in long-range scenarios), symptomatic of a "laziness" bottleneck: models abort search prematurely, offering hallucinated or suboptimal answers. There is clear quantitative correlation between increased tool use and improved recommendation accuracy.

Error Diagnosis

Error attribution demonstrates that the predominant failure mode is insufficient exploration, with models terminating search before satisfying all constraints. A secondary major source is "Lost in the Middle" context conflation—attribute errors arising from inability to synthesize large or heterogeneous spatial contexts. Argument-related tool misuse (malformed queries) is also observed, suggesting significant room for improvement in agentic LLM interaction with geo-spatial APIs.

Figure 6: Distribution of error causes in Linyi. Insufficient exploration and factual conflation dominate failure cases.

Qualitative Insights: Exploration Trajectories

In the qualitative case study below, a Gemini-2.5-Pro agent demonstrates emergent path planning: after local search, it executes informed long-range moves based on historical failure, avoids redundant queries, and ultimately converges on a final solution by synthesizing all observations—though still missing closer optimal stations at times, underscoring the need for further advances in global optimality and exploration efficiency.

Figure 7: Case Study—multi-step spatial exploration and reasoning trajectory for a dual-objective query.

Implications and Directions for Future Research

The introduction of EVGeoQA and the GeoRover framework exposes the status quo of LLMs as spatial reasoning agents: current models are limited in their ability to plan over large city-scale environments under partial observability and multi-objective constraints. The negative results—particularly model "laziness" and context-conflation—imply that scaling model size alone is insufficient for truly generalizable spatial reasoning. The observed gains from explicit "Thinking" highlight the importance of meta-reasoning and trajectory summarization, suggesting future research should pursue agent architectures that optimize exploration–exploitation strategies, memory-aware reasoning, and explicit coordination between action and synthesis steps in geo-spatial contexts.

Practically, the dataset and framework are immediately extensible to domains such as delivery routing, trip planning in smart cities, and embodied spatial assistants, while the current focus on Chinese urban centers and constraint to Chinese-language queries is a notable limitation for global generalizability.

Conclusion

EVGeoQA establishes a new standard for evaluating LLMs in dynamic, multi-objective geo-spatial reasoning, rigorously emphasizing contextual planning, active environment interaction, and dual-objective optimization over static retrieval. The combination of realistic data generation, tool-augmented agent evaluation, and scalable benchmark design surfaces actionable strengths and weaknesses of current LLMs. The framework and error analyses motivate future work in long-horizon agentic reasoning, context synthesis, and robust tool use, advancing the development of spatially intelligent autonomous systems.

Reference: "EVGeoQA: Benchmarking LLMs on Dynamic, Multi-Objective Geo-Spatial Exploration" (2604.07070)