LocalSearchBench: Benchmarking Agentic Search in Real-World Local Life Services (2512.07436v1)

Abstract: Recent advances in large reasoning models (LRMs) have enabled agentic search systems to perform complex multi-step reasoning across multiple sources. However, most studies focus on general information retrieval and rarely explores vertical domains with unique challenges. In this work, we focus on local life services and introduce LocalSearchBench, which encompass diverse and complex business scenarios. Real-world queries in this domain are often ambiguous and require multi-hop reasoning across merchants and products, remaining challenging and not fully addressed. As the first comprehensive benchmark for agentic search in local life services, LocalSearchBench includes over 150,000 high-quality entries from various cities and business types. We construct 300 multi-hop QA tasks based on real user queries, challenging agents to understand questions and retrieve information in multiple steps. We also developed LocalPlayground, a unified environment integrating multiple tools for agent interaction. Experiments show that even state-of-the-art LRMs struggle on LocalSearchBench: the best model (DeepSeek-V3.1) achieves only 34.34% correctness, and most models have issues with completeness (average 77.33%) and faithfulness (average 61.99%). This highlights the need for specialized benchmarks and domain-specific agent training in local life services. Code, Benchmark, and Leaderboard are available at localsearchbench.github.io.

• The paper introduces LocalSearchBench, a benchmark evaluating agentic search in local services using multi-hop reasoning and retrieval techniques.
• The paper details the construction of a high-fidelity local merchant database and the LocalRAG system for efficient, context-specific query processing.
• The paper finds that even state-of-the-art large reasoning models achieve only ~34% correctness, highlighting challenges in complex multi-hop reasoning and entity retrieval.

LocalSearchBench: Evaluating Agentic Search in Real-World Local Life Services

Introduction

The paper "LocalSearchBench: Benchmarking Agentic Search in Real-World Local Life Services" (2512.07436) addresses the gap in evaluating large reasoning models (LRMs) and agentic search systems within vertical domains, specifically real-world local life services. Existing benchmarks predominantly cover general web or open-domain search and lack the complexity, ambiguity, and multi-hop requirements that characterize local services such as dining, healthcare, travel, and shopping. The authors introduce LocalSearchBench, a comprehensive benchmark grounded in real user scenarios, and LocalPlayground, a unified agentic evaluation environment, to assess model competence in multi-hop reasoning, information retrieval, and decision making in this challenging domain.

Figure 1: A conceptual illustration of general multi-hop QA and local life service QA, highlighting the increased complexity and context specificity in multi-hop scenarios.

Construction of LocalSearchBench

Local Merchant Database

The foundation of LocalSearchBench is a large-scale, high-fidelity local merchant database. Data is sourced from Platform M, resulting in an initial pool of over 260,000 real-world merchant records spanning January to December 2025 within Shanghai, Beijing, and Guangzhou. The database ensures multi-scenario and multi-city coverage, capturing regional and categorical diversity across six verticals: dining, lifestyle, shopping, accommodation, healthcare, and tourism.

Data augmentation is performed via an LLM-driven agentic pipeline, expanding sparse records into dense merchant profiles along 29 standardized fields encompassing business details, location intelligence, services, contacts, and promotions. Privacy rewriting and rigorous validation by LLM-as-a-Judge frameworks ensure both data anonymization and high-quality, logically consistent augmentation.

Figure 2: Workflow diagram for constructing the local merchant database, illustrating data augmentation, anonymization, and validation.

The geographic distributions reflect actual urban clustering and economic patterns. Figures for city- and category-level distributions demonstrate the benchmark’s representativeness of real-world local life ecosystems.

Figure 3: Word cloud for Shanghai, visualizing the density and variety of merchant and landmark data.

Figure 4: Distribution of merchant records across cities, confirming data balance and diversity.

Figure 5: Pie chart showing the breakdown of service categories for Shanghai, ensuring representativeness in each city.

Retrieval-Augmented Generation (LocalRAG)

To facilitate high-performance LRM retrieval, the paper introduces LocalRAG, a retrieval-augmented generation system tailored for the local merchant database. The pipeline involves vector embedding of natural language merchant profiles, fast nearest neighbor search using dense retrieval, and a reranking stage using specialized LLM components. This architecture efficiently surfaces relevant entities given semantically rich and context-dependent user queries.

Figure 6: End-to-end workflow for the LocalRAG system, combining vector search and neural reranking.

Multi-hop QA Task Construction

A unique contribution is the creation of 300 multi-hop QA tasks, which reflect composite requirements, schedule planning, service bundling, and multi-constraint merchant recommendations. Tasks are annotated at distinct intelligence levels (L3: composite/multi-turn, L4: expert personalization), often requiring 3-5 reasoning hops. Annotators ground these with actual city-specific details and tool requirements (web search vs. LocalRAG).

Figure 7: Workflow diagram for constructing multi-hop QA tasks, detailing seed question expansion, instantiation, and validation.

Distributions of intelligence levels and hop counts are measured to ensure coverage of real-world complexity.

Figure 8: Analysis of intelligence level distribution across the QA corpus, confirming task diversity and challenge range.

Figure 9: Distribution of intelligence level annotations for QA tasks.

LocalPlayground: Agentic Search Evaluation Framework

LocalPlayground provides an integrated evaluation environment for agentic search, coupling a Search Agent (supporting both LocalRAG and web search tool calls per step) and a Validation Agent (LLM-as-a-Judge). The search agent autonomously decomposes queries, invokes retrieval tools, and iterates multi-hop reasoning over up to N=5 rounds, a configuration determined optimal in sensitivity analysis.

Evaluation spans seven dimensions: correctness, completeness, fluency, faithfulness, safety, average tool calls, and average rounds. Validation is carried out using state-of-the-art LLMs independent from any tested model, with result aggregation over randomized, anonymized trial runs.

Figure 10: Modular overview of the LocalPlayground framework, highlighting agentic orchestration, tool interfaces, and validation components.

Benchmarking and Model Results

Experiments encompass ten LRMs, both closed- and open-source, including GPT-4.1, Gemini-2.5-Pro, DeepSeek-V3.1, GLM-4.5, Qwen variants, and others. LocalPlayground is shown to require coordinated use of web search and domain-specific retrieval for success on local service tasks.

Key findings include:

• Correctness remains low: The best-performing model (DeepSeek-V3.1) achieves only 34.34% correctness on multi-hop QA; most models are below this, indicating the unsolved nature of complex local search.
• Web search vs. faithfulness: Incorporating web search improves correctness (avg +4.4pp) and completeness (avg +4.0pp) but decreases faithfulness (avg −3.6pp), as external noise and off-domain content lead to hallucinations.
• Efficiency and completeness: Closed-source models achieve higher completeness and efficiency (fewer rounds, fewer tool calls) than open-source counterparts, though the correctness gap is narrower.
• Optimal agent configuration: Allowing up to 5 rounds yields the best correctness and efficiency trade-off; increasing rounds further leads to noise saturation and marginal or negative gains.

Practical and Theoretical Implications

The LocalSearchBench infrastructure exposes the inability of current LRMs—even state-of-the-art, deeply tuned architectures—to generalize multi-hop reasoning and precise entity retrieval in rich, ambiguous, and tightly coupled, domain-specific environments. This domain cannot be addressed by generic web data and tool-use training alone. Instead, the findings argue for:

• Research focus on domain-adapted agent training, incorporating vertical-specific ontology, lexicon, and reasoning templates.
• Development of new architectures for robust multi-source grounding, noise filtering, and user intent disambiguation.
• Emphasis on privacy-safe, high-density data augmentation and publishing protocols, as demonstrated in LocalSearchBench.

As interest grows in agentic AI for e-commerce, urban infrastructure, and personalized recommendation, further expansion beyond China-specific city contexts, support for real-time dynamic state, and integration with public APIs is necessary. Open-sourcing both benchmark and evaluation code lays the foundation for collaborative advancements in robust, trustworthy local service agents.

Figure 11: Real-world data category distributions underpinning the benchmark’s representativeness and transferability.

Conclusion

LocalSearchBench constitutes the first comprehensive, multi-hop agentic search benchmark for real-world local life services. Through extensive data curation, anonymization, agentic augmentation, and dual-tool evaluation, the benchmark sets a new standard for evaluating real-world LRM agentic search capabilities. Results showed that current LRMs exhibit fundamental limitations (with top correctness at only 34%) in multi-hop reasoning, information synthesis, and domain adaptation. Going forward, these benchmarks will direct the development of LRMs, agentic tool frameworks, and domain-specific transfer methodologies to address the complex demands of local life service scenarios and catalyze advances in verticalized AI search.