- The paper establishes a benchmark that rigorously evaluates LLMs on tasks requiring complex state configuration for evidence extraction.
- The paper introduces a comprehensive task design with 100 curated tasks across 12 data ecosystems, highlighting failures in state management and retrieval-scope drift.
- The paper reports that current systems achieve at most 66.18% Item-F1 and 43.37% row-level accuracy, emphasizing the need for improved interface-level state tracking.
SGR-BENCH: A Benchmark for State-Gated Web Retrieval Tasks
Motivation and Context
The SGR-BENCH benchmark addresses a previously undercharacterized class of retrieval tasks—state-gated retrieval (SGR)—where access to answer-bearing evidence on specialized data-retrieval websites is contingent upon establishing the correct site-specific retrieval state (i.e., configuring complex filters, hierarchical views, or scope controls). Unlike prevailing benchmarks that focus on source discovery, shallow extraction, or browser-grounded task execution, SGR-BENCH measures an agent’s ability to both identify a suitable source and manipulate the website’s retrieval interface to make the required evidence accessible and extract structured outputs.
The introduction of SGR-BENCH fills a notable gap in the evaluation of agentic LLM systems and web-search agents. Most prior benchmarks (e.g., BrowseComp, DeepSearchQA, WebArena) focus on aspects such as source traversal, document reading, or action sequencing, but fail to sufficiently stress the explicit need for manipulating retrieval state—a reality in many professional or domain-specific data portals.
SGR-BENCH comprises 100 expert-curated tasks across six source families and 12 public data ecosystems, such as regulatory resources, environmental monitoring, scholarly archives, and life-sciences databases. All tasks are constructed such that the reference answer is not exposed under default website states but only becomes retrievable after the agent configures domain-appropriate filters, sorts, and scopes.
Tasks are formulated in both constraint-guided and goal-oriented variants, enabling controlled study of the effects of explicit versus implicit procedural guidance. Critically, the benchmark is designed to evaluate whether an agent can independently discover the target website, manipulate its controls (filtering, hierarchy, scoping), and extract all required information in a schema-compliant table.
The construction pipeline is rigorous: it combines LLM-assisted annotation with multi-round expert validation to ensure domain specificity, task answer uniqueness, ground-truth stability, shortcut resistance, and logical dependency across retrieval steps. This results in a collection of tasks specifically difficult for current agentic approaches, with validation steps designed to filter out cases solvable by direct or shallow search.
Evaluation Protocols and Metrics
Evaluation in SGR-BENCH follows a strict protocol with canonicalization for output normalization (handling format, capitalization, abbreviations, etc.) and structured alignment for fields and rows. The primary metrics are:
- Item-level F1: Fraction of individual, aligned field slots that are correct.
- Row-level F1: Fraction of rows where all schema fields are correct.
- Pairwise Order Accuracy (P.O.A.): Agreement in ordering between predicted and reference outputs, following pairwise rank-agreement.
This approach robustly decouples field-level extraction quality from fully correct row assembly and order preservation, which are crucial for practical structured retrieval scenarios.
Empirical Findings
The main experimental results reveal severely limited SGR competence in current production agents. Across eight CLI-based agentic LLM systems and three commercial search-agent products, the highest overall Item-F1 achieved is 66.18% (GPT-5.5), while row-level F1 is markedly lower (maximum 43.37%). Commercial solutions, including Google Search AI Mode and Gemini Deep Research, consistently underperform open tool-using LLMs, with some Item-F1 scores below 15%.
Detailed trajectory audits show that the central bottleneck is not source discovery (i.e., finding the target website), but failure to correctly configure and maintain the retrieval state required to access the needed evidence. Specifically:
- Error distribution: Retrieval-scope drift (37.2%) and criterion mismatch (27.6%) collectively account for 64.7% of failed trajectories, with final answer formatting responsible in only 10.3%.
- Row versus item-level accuracy: Consistently, models recover individual field values more often than complete, correctly-scoped rows, indicating that errors in scope configuration lead to “local” plausibility but “global” inaccuracy in output.
- Source-family difficulty: Scholarly archives and life-science resources yield higher Item-F1 than regulatory or official statistics sources. Harder cases invariably require alignment of multiple hierarchical controls (e.g., jurisdiction, reporting period, population group).
Further, the constraint-guided form (with explicit procedural hints) leads to slight Item-F1 gains but does not fundamentally resolve scope maintenance issues. The distinction between item- and row-level metrics is also diagnostic for distinguishing error types: criterion mismatch tends to preserve field recall but breaks at the row aggregation stage, while retrieval-scope drift manifests as uniform degradation across metrics.
Implications and Future Directions
SGR-BENCH’s empirical analysis underscores a critical limitation in present-day web agents: the inability to preserve and propagate correct retrieval context across dependent operations. Maintaining state over multi-step interactions with complex websites is a non-trivial “interface-level” reasoning challenge. This property is not sufficiently addressed by models optimized on open-domain QA, shallow document retrieval, or even multi-hop search benchmarks.
Practically, these findings highlight challenges for deploying LLM-based agents in real-world data retrieval or decision-support settings in domains such as government, regulatory compliance, or biomedical curation. Theoretical implications concern the need for research on techniques for context-coupled reasoning, persistent state tracking, and multi-variable control alignment across retrieval steps. There is an evident need for agentic architectures that couple navigation (action selection at the interface level), internal state-tracking, and evidence extraction under explicit state-dependency constraints.
SGR-BENCH also exposes gaps in current evaluation methodology: answer-only scoring is insufficient when agents can extract plausible, contextually-invalid evidence. For future system development, the benchmark motivates co-supervised approaches where navigation choices, control states, and extraction correctness are trained and evaluated jointly.
From a data engineering perspective, the rigorous construction process underlying SGR-BENCH sets a methodological precedent for future benchmarks that require tightly controlled, “shortcut-proof” tasks with precise error localization.
Conclusion
SGR-BENCH establishes structured, state-gated web retrieval as a challenging evaluation axis for agentic LLMs. The results demonstrate that even the most advanced systems struggle to translate partial evidence access into reliably scoped, structured answers, primarily due to failures in within-site state management rather than source discovery or evidence composition per se. Addressing these shortcomings requires a shift in both benchmarking and algorithmic focus—towards retrieval-state preservation, context-aware interface manipulation, and integrated supervision of navigation and extraction. The benchmark provides a crucial foundation for advancing research in specialized web retrieval, paving the way for more robust, interface-aware AI agents.
Citation: "SGR-Bench: Benchmarking Search Agents on State-Gated Retrieval" (2605.22219)
