LoHoSearch: Benchmarking Long-Horizon Search Agents Beyond the Human Difficulty Ceiling

Abstract: Search agent benchmarks exemplified by BrowseComp have rapidly saturated over the past year, with the strongest models surpassing 90% accuracy. Since these benchmarks are predominantly human-authored, annotators lack a global perspective on entity statistics and cannot systematically maximize search space size and structural complexity. This creates a difficulty ceiling that is hard to break. To address this, we introduce LoHoSearch (Long-Horizon Search Agents), a challenging benchmark comprising 544 human-verified questions across 11 domains. LoHoSearch is constructed via an automated pipeline built upon a knowledge graph covering over 7 million Wikipedia entities, which selects relations with large search spaces and assembles them into structurally complex questions with KG-verified unique answers. Our evaluation demonstrates that even the strongest model achieves only 34.74% accuracy, and existing context management strategies (best +6.8%) yield far smaller gains than on prior benchmarks. LoHoSearch provides a more demanding standard for evaluating long-horizon reasoning and context management in search agents.

• The paper introduces an automated, knowledge-graph-driven pipeline that systematically increases search space and structural complexity in long-horizon search tasks.
• It employs tree-structured and graph-structured subgraph sampling with multi-stage QA generation and rigorous human review to ensure high evaluation standards.
• Empirical results reveal that even SOTA models, like GPT-5.5 at 34.7% accuracy, struggle with the benchmark, emphasizing limits in long-horizon reasoning.

LoHoSearch: Advancing Benchmarking for Long-Horizon Search Agents

Motivation and Background

Benchmarks like BrowseComp have provided robust evaluation settings for search agents. However, an inherent limitation of such human-authored datasets is a "difficulty ceiling": annotators, lacking a global entity perspective, overwhelmingly select popular, well-connected entities and relations. As a result, state-of-the-art (SOTA) models have rapidly approached and surpassed 90% accuracy, saturating these benchmarks and sharply diminishing their discriminative utility. Moreover, the scale, diversity, and structural complexity of these benchmarks remain fundamentally restricted by manual construction.

LoHoSearch addresses these core issues with an automated knowledge-graph-driven pipeline that systematically maximizes search space size and structural complexity. The result is a challenging, verifiably rigorous evaluation set that extends far beyond the ceiling imposed by human annotation and tests both long-horizon reasoning and robust context management.

Benchmark Construction Pipeline

LoHoSearch begins with the construction of a massive knowledge graph from the full English Wikipedia, representing over 7 million entities and 265 million directed edges, with entity type assignments and popularity statistics derived from Wikidata P31 classes and in-degree counts.

The pipeline proceeds through four major algorithmic stages:

1. Subgraph Sampling: Two structural paradigms are employed:
   • Tree-structured subgraphs emphasize maximized candidate search spaces at each relational edge, requiring layered intersection constraints to guarantee answer uniqueness only when all relations are combined.
   • Graph-structured subgraphs incorporate rich cyclic dependencies and cross-constraints, maximizing non-decomposability and requiring sophisticated global constraint satisfaction.
2. QA Generation and Verification: For each subgraph, entity relations are obfuscated via LLM-extracted natural-language descriptions, strongly filtered to ensure non-triviality and non-retrievability by direct search or naive LLM inference. A multi-stage verification regime—automatic and human-in-the-loop—guarantees rigorous coverage, uniqueness, and correctness of each QA pair.

   Figure 1: Overview of the LoHoSearch pipeline.

3. Post-filtering and Human Review: Each question passes redundancy pruning, manual review for logical coherence, language fluency, and information redundancy. Additionally, exhaustive automated agent-based answer search further enforces answer uniqueness.

   Figure 2: Domain distribution of LoHoSearch. The dataset consists of 544 samples spanning 11 categories.

LoHoSearch ultimately comprises 544 human-verified questions across 11 knowledge domains, with balanced coverage and systematically increased structural difficulty relative to prior benchmarks.

Benchmark Characteristics and Difficulty Analysis

A key innovation of LoHoSearch is the systematic control of search space size per constraint and structural complexity at the single-question level. Empirical analysis reveals:

• Entity Popularity: Hidden entities that must be inferred are consistently of lower popularity in LoHoSearch than in prior benchmarks.
• Search Space: Even under matched popularity, the candidate sets satisfying LoHoSearch's relational constraints are far larger, reflecting the intentional global maximization of difficulty.
• Structural Complexity: Graph-structured questions, which include cycles and cross-links, are significantly harder—DeepSeek-V4-Flash achieves only 8.01% accuracy on these versus 11.89% for tree-structured forms.

  Figure 3: Analysis of hidden entity popularity and search space size. (a) LoHoSearch produces significantly lower popularity hidden entities. (b) For fixed popularity, LoHoSearch yields far larger search spaces per relation than BrowseComp.

  

  Figure 4: Distribution of the number of tool calls (correct answers) for BrowseComp and LoHoSearch. LoHoSearch requires significantly more tool calls to reach correct answers compared to BrowseComp.

The number of tool calls required to solve LoHoSearch examples is far greater—mean (median) of 61 (59) for correct solutions, versus 35 (26) for BrowseComp—highlighting the demand for extended multi-step reasoning.

Empirical Results and Model Performance

Comprehensive evaluation with leading closed and open models demonstrates the elevated challenge of LoHoSearch:

• Strongest Model Ceiling: GPT-5.5 achieves 34.7% accuracy; all other models score below 16%. This sharply contrasts with their performance on human-authored benchmarks, thus decisively breaking the ceiling present in datasets like BrowseComp.
• Context Management Limitations: Strategies such as context summarization, trajectory discarding, and verification modules provide only incremental gains (up to +6.8%), far below the improvements observed in prior datasets.

  Figure 5: Illustration of pass@N and three answer aggregation strategies (majority voting, weighted voting, and best-of-N) under parallel sampling.

Parallel sampling with answer aggregation methods reveals headroom for score improvements, but the upper bound remains low given the base difficulty of each instance.

Implications and Future Directions

Practically, LoHoSearch delineates a new standard for long-horizon search agent evaluation, especially in settings involving open web tools, complex context management, and combinatorial constraint satisfaction. The automated pipeline demonstrates a scalable methodology for dataset generation that adapts alongside advancing model capabilities, providing a path to continuous benchmark renewal.

Theoretically, the dataset reveals persistent limitations in current LLM-based agents’ ability to solve large search spaces and high-complexity questions, even as context windows and retrieval tools expand. The limited impact of context management ablations underscores the need for more fundamental advances in memory, planning, and reasoning over extended horizons.

The approach also formalizes a methodology—verifiable uniqueness, global structure statistics, and adversarial automated filtering—that can be extended to new languages and domains, and adapted for evaluating other forms of knowledge-intensive, multi-hop reasoning.

Conclusion

LoHoSearch represents a principled advance in search agent benchmarking, systematically exceeding the human-authored difficulty ceiling via a scalable, knowledge-graph-centric pipeline. The limited success of current agentic models on this benchmark makes it a discriminative platform for research on long-horizon reasoning, context management, and the next generation of search-capable AI systems.