AgentSearchBench: A Benchmark for AI Agent Search in the Wild

Abstract: The rapid growth of AI agent ecosystems is transforming how complex tasks are delegated and executed, creating a new challenge of identifying suitable agents for a given task. Unlike traditional tools, agent capabilities are often compositional and execution-dependent, making them difficult to assess from textual descriptions alone. However, existing research and benchmarks typically assume well-specified functionalities, controlled candidate pools, or only executable task queries, leaving realistic agent search scenarios insufficiently studied. We introduce AgentSearchBench, a large-scale benchmark for agent search in the wild, built from nearly 10,000 real-world agents across multiple providers. The benchmark formalizes agent search as retrieval and reranking problems under both executable task queries and high-level task descriptions, and evaluates relevance using execution-grounded performance signals. Experiments reveal a consistent gap between semantic similarity and actual agent performance, exposing the limitations of description-based retrieval and reranking methods. We further show that lightweight behavioral signals, including execution-aware probing, can substantially improve ranking quality, highlighting the importance of incorporating execution signals into agent discovery. Our code is available at https://github.com/Bingo-W/AgentSearchBench.

• The paper presents a novel benchmark that formalizes agent discovery as a retrieval and reranking problem based on execution outcomes.
• It aggregates nearly 10,000 real-world agents and standardizes heterogeneous metadata for performance-based evaluation.
• Experimental results reveal a significant semantic–performance gap, emphasizing the need for execution-aware behavioral signals.

AgentSearchBench: A Performance-Grounded Benchmark for AI Agent Search

Motivation and Problem Formalization

The proliferation of ecosystem-scale AI agents has introduced a critical selection challenge: identifying agents with compositional, execution-dependent capabilities for arbitrary tasks. Existing benchmarks largely assume well-specified functionalities or controlled candidate pools, and restrict evaluation to executable queries, which is misaligned with open-world agent search scenarios. AgentSearchBench (2604.22436) formalizes agent discovery as a retrieval and reranking task under both executable queries and high-level descriptions, grounding relevance in execution-based outcomes rather than semantic similarity or static documentation.

Figure 1: Task and relevance label generation pipeline for AgentSearchBench, producing execution-grounded task queries, descriptions, and composite relevance via agent execution.

Benchmark Construction and Schema

AgentSearchBench aggregates nearly 10,000 real-world agents from multiple platforms (GPT Store, Google Cloud Marketplace, AgentAI Platform) and standardizes their heterogeneous metadata, capabilities, usage guidance, and constraints into a unified schema. Task queries are synthesized from documentation and clustered for semantic alignment; multi-agent queries are composed from entailed subtasks. High-level task descriptions abstract clusters of queries, each description linked to candidate queries via rubric-based, aspect-wise relevance scoring. All relevance signals are derived from agent execution results evaluated on a 5-point LLM-as-judge scoring protocol, yielding binary or graded labels that capture the misalignment between documentation and observed performance.

Figure 2: Overview statistics—agent/task/description counts, executions, and candidate set sizes highlight benchmark scale and diversity.

Diversity and Challenge Analysis

The benchmark exhibits broad, long-tailed distributions across both agent functionality and task types, with substantial overlap and partial coverage in capabilities. Each query typically matches multiple relevant agents, and task descriptions are associated with composite query sets (average 10 queries per description). Score entropy across agents for each query is high, indicating nontrivial reranking difficulty due to agent functional variance and documentation-performance misalignment.

Figure 3: Distribution of relevant agents per query, illustrating nontrivial difficulty in agent selection and the need for fine-grained reranking.

Retrieval and Reranking Evaluation

AgentSearchBench evaluates multiple families of retrieval (sparse, dense, tool-specific, decoder-only) and reranking (cross-encoder, tool-specific, decoder-only, LLM-based) models under execution-grounded relevance. Tool-aware retrievers (e.g., ToolRet, Tool-Embed) dominate on executable queries, while dense retrievers (e.g., BGE-Large v1.5) lead on high-level task descriptions. However, completeness metrics and NDCG on descriptions remain low, especially compared to executable queries. Strong numerical results for top models indicate that retrieval is largely coarse and misses comprehensive capability, especially under abstract queries.

Figure 4: Accumulated agent scores (golden ranking) for 2452 single-agent task queries, illustrating persistent gaps between surface-level matching and execution-grounded ranking.

LLM-based and decoder-only rerankers improve ordering under high-level descriptions but still yield limited completeness—i.e., the best agents remain distributed lower in the ranking, evidence of semantic–performance gap.

Semantic–Performance Gap and Behavioral Signals

Empirical analyses substantiate the semantic–performance gap: methods relying on static documentation or semantic similarity fail to capture execution-level agent competence, particularly for abstract and composite tasks. Indexing agent usage examples, a form of developer-provided behavioral evidence, measurably improves ranking across most retrievers.

Explicit execution-aware probing—generating probing queries, executing candidates, and extracting probe score variance—provides valuable behavioral differentiation, yielding consistent NDCG gains for reranking when probe variance is high. This demonstrates that lightweight, scalable execution signals (probing) are an essential complement to description-based search.

Figure 5: NDCG@5 comparison between realistic and synthetic single-agent task queries; absolute values drop in realistic settings, but relative rank ordering holds.

Figure 6: NDCG@5 win rate versus probe score variance; probing yield is most effective under medium/high score variance, validating behavioral probing’s discriminative power.

Benchmark Validation and Implications

AgentSearchBench is validated against external benchmarks (Last Exam, Finance Agent Benchmark), exhibiting consistent ranking trends and performance differentials. LLM-as-judge relevant signals are shown to correlate strongly with human expert judgments (Cohen's kappa $\kappa=0.93$, accuracy $96.7\%$) for large-scale execution-based annotation. Practical implications are clear: agent discovery systems that rely on static metadata or semantic retrieval are fundamentally constrained; execution-aware behavioral signals (including probing) must be integrated to advance ranking fidelity and completeness, especially in open, heterogeneous agent ecosystems.

Theoretical and Practical Impact

The benchmark establishes an execution-centered paradigm for agent search, where retrieval and ranking are conditional on behavioral capability, not documentation. It enables reproducible evaluation over realistic, large-scale candidate pools, multi-modal queries, and variable task granularity. Future AI developments are expected to leverage behavioral signal integration, active probing, and continual adaptation—moving toward self-evolving agent selection and orchestration. The formalization here supports further work in data-driven agent composition, scalable orchestration, and metric-driven improvement of agentic search in practical deployments.

Conclusion

AgentSearchBench (2604.22436) reframes agent search as a retrieval and reranking problem under execution-dependent uncertainty, exposes a substantial semantic–performance gap, and demonstrates the importance of behavioral signals for accurate agent selection. As agent ecosystems scale, benchmarks like AgentSearchBench will become foundational for developing robust, performance-grounded AI agent discovery systems, accelerating advances in agent orchestration and compositional capability assessment.