Agentic Search Queryset (ASQ) Overview

• The paper captures full behavioral traces of agentic search attempts by logging intermediate queries, retrieved documents, and reflective outputs to understand search dynamics.
• ASQ formalizes an agentic run as a sequence of frames detailing query reformulation, retrieval ordering, and decision logging with preserved temporal order.
• The dataset supports retrieval optimization and behavioral analysis by exposing unique agent-specific patterns, challenging standard IR assumptions and methods.

Agentic Search Queryset (ASQ) is a dataset and collection methodology for studying how agentic retrieval-augmented systems search, reformulate queries, consume retrieved evidence, and produce answers. It was introduced to capture the full behavioral trace of an agentic search attempt—reasoning-induced queries, retrieved documents, thoughts, refinement outputs, and final answers—rather than only the terminal answer quality. ASQ is motivated by the claim that information retrieval has been historically human-centred, while modern search traffic increasingly includes autonomous agents issuing synthetic queries on behalf of users; the dataset is therefore intended to support both retrieval optimization and behavioral analysis under this emerging mixed human–agent regime (Pezzuti et al., 19 Feb 2026).

1. Historical motivation and the shift from human to agent query streams

ASQ is rooted in a diagnosis of structural misalignment between conventional IR assumptions and agentic search. The underlying paper models the search ecosystem as two interleaved streams: a fraction $\alpha$ of queries is human-generated, forming an organic query stream, while the remaining fraction $(1-\alpha)$ is issued by agents as a synthetic query stream. Its argument is that much of IR has implicitly assumed $\alpha=1$, whereas in practice that assumption is weakening as automated systems increasingly search on behalf of users (Pezzuti et al., 19 Feb 2026).

The need for ASQ follows from several claimed differences between agentic and human search. Agents can generate substantially higher query volumes, often as part of reasoning-induced search; their workloads may be bursty and less predictable than human traffic; their query style differs linguistically from human formulations; and standard optimization techniques such as query preprocessing, caching, and relevance modeling may no longer behave as expected. The paper further stresses that existing synthetic-query resources are typically generated to resemble human queries, which is not the same as observing real agent behavior during search. ASQ is therefore designed to record what agents actually do, rather than what researchers assume they should do (Pezzuti et al., 19 Feb 2026).

This positions ASQ as an infrastructural rather than purely task-level benchmark. Its purpose is not principally to decide which agent answers best on a fixed set of end tasks, but to expose the mechanics of query reformulation, retrieval interaction, and iterative search behavior that arise when a LLM is allowed to search autonomously.

2. Formalization of agentic runs and trace collection

ASQ represents search as an agentic run, or arun, defined as the sequence of actions taken by an agent to answer an initial query $q_0$. The paper formalizes a single trace element as a frame

$f=(q,\mathcal{R}_{q},\mathcal{D}),$

where $q$ is an intermediate query generated by the agent, $\mathcal{R}_{q}$ is the ranked list of retrieved documents, and $\mathcal{D}$ is the list of natural-language descriptions produced by reflective operations. A complete trace for agent $A$ on initial query $q_0$ is written as

$(1-\alpha)$0

with $(1-\alpha)$1 denoting the final answer and $(1-\alpha)$2 the trace length. Incomplete traces are allowed, so an arun may terminate without an answer because of errors or early stopping (Pezzuti et al., 19 Feb 2026).

A central methodological feature is that ASQ is collected by intercepting retrieval calls during decoding rather than reconstructing traces post hoc. This preserves temporal order and retains incomplete trajectories. The logging pipeline parses agent-specific control tags with regular expressions: search tags extract synthetic queries, think tags extract thoughts, refine tags record refinement outputs for AutoRefine, and answer tags extract final answers. Retrieved document identifiers are intercepted before the documents are wrapped and passed back to the generator, so the released data stores document IDs rather than full retrieved text, partly to reduce storage (Pezzuti et al., 19 Feb 2026).

The paper emphasizes that the collection procedure does not prompt agents to decompose or reformulate queries, does not alter their decisions, and does not filter or post-process traces. This design is intended to preserve the integrity of agent behavior as observed in the wild execution of an agentic system.

3. Dataset composition: query sources, agents, and retrieval settings

ASQ is built from three benchmark query collections chosen to vary query type, domain, and document type. HotpotQA uses multi-faceted, in-domain queries over Wikipedia articles; Researchy Questions contains open-ended research-style questions over long web documents; and MS MARCO dev provides factoid queries over short web passages. All three were selected in part because they have publicly available relevance judgments, which the paper describes as useful for both optimization and evaluation (Pezzuti et al., 19 Feb 2026).

Source dataset	Queries	Characterization
HotpotQA (HQA, test split)	7,405	Multi-faceted, in-domain, average length $(1-\alpha)$3
MS MARCO dev (MSM)	101,093	Factoid, out-of-domain, average length $(1-\alpha)$4
Researchy Questions (RQ)	96,448	Researchy, out-of-domain, average length $(1-\alpha)$5

The agent side comprises three configurations: Search-R1 with Qwen2.5-3B, Search-R1 with Qwen2.5-7B, and AutoRefine with Qwen2.5-3B-Base. The paper explicitly excludes R1-Searcher because it is outdated, ParallelSearch because it is not publicly available, and commercial agents because they do not expose the intermediate data needed for trace logging (Pezzuti et al., 19 Feb 2026).

Two retrieval pipelines are used: BM25 only, and BM25 followed by MonoElectra reranking. In the reranked configuration, MonoElectra is applied to the top 1000 BM25 results, and the number of documents fed to the generator is fixed to $(1-\alpha)$6. For Researchy Questions, reranking is skipped because it is too slow and does not affect trace dynamics. These choices make ASQ a cross-product of agent architecture, model scale, source corpus, and retrieval configuration rather than a single static trace dump (Pezzuti et al., 19 Feb 2026).

4. Storage format, scale, and extensibility

ASQ is organized in a trace-centric format. Each trace is stored in its own directory, with separate TSV files for different artifact types. The released structure contains four per-trace files indexed by the original query ID: answers, synthetic queries, thoughts, and ranked lists. Every frame carries an iteration ID, and the chronological sequence of frames is preserved explicitly (Pezzuti et al., 19 Feb 2026).

The dataset is quantitatively large enough to expose substantial variation in trace length and search behavior. The paper reports, for example, 18.4k search calls for Qwen-7B on HotpotQA with average trace length $(1-\alpha)$7 and maximum trace length 7, and 192.3k search calls for Qwen-7B on MS MARCO with average trace length $(1-\alpha)$8 and maximum trace length 8. AutoRefine on HotpotQA produces 7.3k search calls with average trace length $(1-\alpha)$9 and maximum 6, while AutoRefine on MS MARCO reaches 56.9k search calls with average trace length $\alpha=1$0 and maximum 77. Under BM25-only on Researchy Questions, the reported tails are longer: Qwen-7B reaches 294.3k search calls with average trace length $\alpha=1$1 and maximum 85, ხოლო Qwen-3B reaches 4.9k search calls with average trace length $\alpha=1$2 and maximum 186 (Pezzuti et al., 19 Feb 2026).

These statistics are significant because ASQ is not restricted to short, successful traces. It records long and sometimes pathological trajectories, including duplication, stalling, and early termination. This makes it suitable for studying convergence behavior, search inefficiency, and query-loop phenomena.

The accompanying toolkit is a second major contribution. The paper states that ASQ is released with public code intended to let researchers extend the framework to new agents, new retrievers, and new datasets. Reproducibility, interoperability, and extensibility are explicit design goals, reflecting the likelihood that any fixed trace collection will age quickly as agentic search systems change (Pezzuti et al., 19 Feb 2026).

5. Behavioral findings from ASQ

ASQ is accompanied by a behavioral analysis that compares agent query reformulation patterns with human session behavior. The paper uses a Markov-style representation with states $\alpha=1$3, $\alpha=1$4, $\alpha=1$5, $\alpha=1$6, $\alpha=1$7, $\alpha=1$8, and $\alpha=1$9, together with transition probabilities and termination probabilities $q_0$0. Here $q_0$1 denotes expansion, $q_0$2 shortening, $q_0$3 exact resubmission, $q_0$4 duplication of an earlier query, and $q_0$5 other reformulation (Pezzuti et al., 19 Feb 2026).

The main qualitative result is that human search is characterized by frequent $q_0$6, $q_0$7, and $q_0$8, whereas agents transition more often to $q_0$9, $f=(q,\mathcal{R}_{q},\mathcal{D}),$0, and $f=(q,\mathcal{R}_{q},\mathcal{D}),$1. Agents exhibit substantially more duplication and looping than humans. The paper also notes that $f=(q,\mathcal{R}_{q},\mathcal{D}),$2 and $f=(q,\mathcal{R}_{q},\mathcal{D}),$3 appear mostly in early iterations, while $f=(q,\mathcal{R}_{q},\mathcal{D}),$4 tends to arise later; in-domain queries show lower duplication rates than out-of-domain ones; and the learned reformulation pattern appears to be shaped more by agent capacity and optimization strategy than by retriever quality (Pezzuti et al., 19 Feb 2026).

These observations have direct systems implications. Exact-match caching may be less effective under agentic traffic; semantic caching may be more promising because subqueries from the same original intent often have high Jaccard similarity; spelling correction and other human-oriented query preprocessing may offer limited benefit because synthetic queries contain fewer spelling errors; and query-understanding models trained on human logs may transfer poorly. The paper goes further and questions whether long-standing IR assumptions such as the Probability Ranking Principle and standard relevance notions remain adequate once autonomous agents become a substantial query-producing population (Pezzuti et al., 19 Feb 2026).

ASQ thereby transforms agentic search from a downstream application issue into an IR workload phenomenon. It suggests that autonomous querying changes not only interface design, but also index management, cache policy, and evaluation methodology.

6. Position within agentic search research and stated limitations

ASQ occupies a distinctive place among agentic search benchmarks because its primary object is behavior, not just answer scoring. Other contemporaneous resources evaluate different slices of the problem: HotelQuEST emphasizes the quality–efficiency tradeoff and underspecified user preferences in hotel search (Hadad et al., 27 Feb 2026); ScholarQuest studies taxonomy-guided agentic academic paper search with intent-level recall and efficiency measures (Pan et al., 18 Jun 2026); Mind2Web 2 evaluates long-horizon web browsing and synthesis via task-specific judge agents (Gou et al., 26 Jun 2025); and RAVine combines attributable nugget-based report evaluation with process-level search and fetch metrics in a static web environment (Xu et al., 22 Jul 2025). ASQ differs in being constructed around full traces of reasoning-induced queries, retrieved rankings, and reflective outputs rather than around final report quality or task completion alone.

This distinction makes ASQ particularly relevant to research agendas that require intermediate supervision or trajectory analysis. A plausible implication is that datasets of this kind are especially useful wherever retrieval must be aligned with evolving agent-generated queries rather than only with the original user request. That implication is consistent with later work such as Agentic-R, which trains retrievers on evolving agent queries and evaluates passage utility through both local query relevance and global answer correctness (Liu et al., 17 Jan 2026).

The ASQ paper is explicit about its limitations. The dataset may become stale as new agents appear; inference is expensive, which restricted the number of included agents, corpora, and retrieval configurations; benchmark queries are used because public production logs are unavailable, so ASQ does not capture real traffic frequency distributions or the long tail of search demand; and the release does not provide relevance judgments for subqueries, limiting direct evaluation of intermediate retrieval quality (Pezzuti et al., 19 Feb 2026).

Despite these constraints, ASQ is presented as the first dataset for studying agentic search behavior as a first-class IR phenomenon. Its broader significance lies in making the synthetic query stream observable: not as isolated prompts, but as ordered search traces that expose how agents search, loop, stop, and fail.