SeekBench: Evaluating Epistemic Competence in LLM Agents

• SeekBench is a benchmark designed to evaluate the epistemic competence of information-seeking LLM agents by analyzing intermediate reasoning traces rather than final answers.
• It assesses three competencies—grounded reasoning, recovery from weak evidence, and calibration of answer decisions—using expert-annotated multi-step traces.
• Employing 190 annotated traces and over 1,800 response steps, SeekBench reveals that final accuracy can mask important failures in adaptive search and reasoning quality.

Searching arXiv for SeekBench and closely related benchmark papers to ground the article in current literature. to=arxiv_search.search ฝ่ายขายรายการjson code to=arxiv_search.search 天天彩票怎么 code to=arxiv_search.search 久久免费热在线精品{"query":"SeekBench epistemic competence information-seeking agents", "max_results": 5} SeekBench is a benchmark for evaluating the epistemic competence of information-seeking LLM agents through step-level analysis of their response traces rather than final-answer accuracy alone. It was introduced as the first benchmark to assess whether search agents (1) generate reasoning steps grounded in observed evidence, (2) adaptively reformulate searches to recover from low-quality results, and (3) are properly calibrated in deciding when the current evidence is sufficient for answering. The benchmark contains 190 expert-annotated traces with over 1,800 response steps, and its experimental study also uses a larger corpus of 28,493 traces and 283,950 steps across seven QA benchmarks and multiple agent models (Shao et al., 26 Sep 2025).

1. Conceptual scope and motivation

SeekBench was proposed in response to a limitation in standard QA evaluation: final-answer metrics such as exact match or F1 are too coarse for multi-step search agents. In the setting targeted by the benchmark, an agent alternates between reasoning, searching, and answering. A correct final answer can therefore conceal several epistemically distinct failure modes, including unsupported intermediate reasoning, repeated unhelpful searches, or premature answering without sufficient support. SeekBench reframes evaluation around the question of whether the path to an answer is justified and adaptive, not only whether the answer is correct (Shao et al., 26 Sep 2025).

This orientation makes SeekBench a process-level benchmark rather than an answer-only benchmark. The benchmark’s central argument is that modern search agents, especially RL-trained ones, should be assessed as multi-step systems whose internal traces can be inspected for epistemic quality. In that sense, SeekBench operationalizes epistemic competence as a property of trace behavior under evidence acquisition, rather than as a scalar property of final output.

A common misconception addressed by the benchmark is that improved answer accuracy necessarily implies better reasoning quality. SeekBench was designed precisely because this implication does not hold in general. Its trace-level analysis aims to expose distinctions that answer-only evaluation suppresses, including differences between grounding, recovery, and calibration.

2. Benchmark construction and annotation schema

SeekBench consists of 190 expert-annotated traces in its core annotation set and over 1,800 response steps. The paper also reports a larger evaluation corpus of 28,493 traces and 283,950 steps gathered across seven QA benchmarks and multiple agent models. The annotation scheme was developed iteratively from trace inspection and content-analysis principles. An initial schema of 12 candidate annotation fields was refined to 8 well-defined features after three rounds of expert annotation and agreement analysis (Shao et al., 26 Sep 2025).

Three expert annotators independently coded the 190 traces. Low-agreement fields were either pruned or merged. The resulting schema achieved overall human agreement: Cohen’s $\kappa = 0.811$. The study also evaluated LLM judges against the human annotations, reporting GPT-4.1: $\kappa = 0.693$, GPT-4.1-mini: $\kappa = 0.731$, and GPT-5: $\kappa = 0.754$ (Shao et al., 26 Sep 2025).

The final schema distinguishes both the functional type of a step and its epistemic quality. For reasoning steps, the main functional types are InformationSynthesis, PlanFormation, and StateAssessment. For search steps, the benchmark marks InitialQuery, RefinedQuery, FollowUpQuery, and RepeatQuery. Retrieved evidence is also labeled for whether it is Sufficient or Insufficient, and Clear or Unclear. This dual annotation makes the benchmark suitable for analyzing not only what an agent does at each turn, but whether the step is epistemically appropriate given the evidence available at that point.

3. Trace formalism and evidence state

SeekBench models an agent’s multi-turn output trace as

$$\mathcal{T} = \langle \tau_1, \tau_2, \ldots, \tau_T \rangle.$$

Each non-final turn is typically represented as

$\tau_t = \langle r_t, s_t, e_t \rangle,$

where $r_t$ denotes reasoning, $s_t$ search, and $e_t$ retrieved evidence. The final turn is represented as

$\tau_T = \langle r_T, a_T \rangle,$

with $\kappa = 0.693$0 the final answer (Shao et al., 26 Sep 2025).

A central formal object is the evidence state. For each turn, $\kappa = 0.693$1 indicates whether evidence is clear and $\kappa = 0.693$2 whether it is sufficient, with

$\kappa = 0.693$3

SeekBench then defines

$\kappa = 0.693$4

This yields a three-level evidence state: $\kappa = 0.693$5 for poor evidence, $\kappa = 0.693$6 for partial evidence, and $\kappa = 0.693$7 for good evidence. The benchmark explicitly treats this evidence-state variable as the backbone of all three evaluated competencies: groundedness, recovery, and calibration (Shao et al., 26 Sep 2025).

This formalization is significant because it ties step quality to the evidence actually observed at the same turn. Groundedness is not assessed abstractly, but relative to retrieved evidence. Recovery is not defined as eventual correctness alone, but as movement from weak evidence states toward good evidence. Calibration is not a generic confidence estimate, but the adequacy of the decision to answer conditional on the evidence state.

4. The three evaluated competencies

SeekBench organizes epistemic competence into three dimensions with distinct formalizations and empirical signatures (Shao et al., 26 Sep 2025).

Competency	Operationalization	Reported pattern
Groundedness	Binary step label $\kappa = 0.693$8; Reasoning Quality Index (RQI)	Few-shot outperformed RL-trained agents; PlanFormation and StateAssessment were weakest
Recovery	$\kappa = 0.693$9; Evidence Recovery Function (ERF); query-type analysis	RefinedQuery and FollowUpQuery were most effective; RepeatQuery had minimal benefit
Calibration	Calibration Error (CE) against ideal answering policy $\kappa = 0.731$0	RL-trained agents had lower CE than base or few-shot variants

Grounding of reasoning

For each reasoning step, SeekBench assigns a binary grounding label

$\kappa = 0.731$1

where $\kappa = 0.731$2 means the reasoning is supported by evidence and $\kappa = 0.731$3 means it is not. The primary metric is the Reasoning Quality Index (RQI), defined at trace level and then aggregated at model level. The benchmark also defines type-specific groundedness scores for $\kappa = 0.731$4, corresponding to InformationSynthesis, PlanFormation, and StateAssessment (Shao et al., 26 Sep 2025).

The reported findings show that few-shot prompting outperformed RL-trained agents in RQI. Among reasoning subskills, PlanFormation and StateAssessment were the weakest, while InformationSynthesis was the strongest. The benchmark also decomposes groundedness by evidence state, allowing analysis of whether reasoning becomes better grounded when the evidence improves.

Recovery from weak search results

Recovery is the ability to escape low-evidence states by changing search behavior after poor results. SeekBench defines the first turn at which a trace reaches good evidence or produces a correct answer as

$\kappa = 0.731$5

From this it constructs the Evidence Recovery Function (ERF), the cumulative fraction of traces that have recovered by turn $\kappa = 0.731$6 (Shao et al., 26 Sep 2025).

To interpret recovery behavior, the benchmark classifies search steps as InitialQuery, RefinedQuery, FollowUpQuery, and RepeatQuery. The study finds that Refined and Follow-up strategies are the most effective for recovery, whereas Repeat queries provide minimal benefit. It further uses Kaplan–Meier survival analysis to handle variable-length traces and right-censoring. The strongest recovery performance was reported for ASearcher, which also had the best overall F1.

Calibration of the answer decision

The third competency concerns whether an agent knows when it has enough evidence to answer. SeekBench defines the answer decision at turn $\kappa = 0.731$7 as

$\kappa = 0.731$8

and studies the conditional answering rate $\kappa = 0.731$9. The ideal policy is

$\kappa = 0.754$0

meaning that an agent should answer if and only if evidence is good. The calibration metric is Calibration Error (CE), with a perfectly calibrated agent satisfying

$\kappa = 0.754$1

SeekBench interprets calibration errors as either overconfidence—answering when evidence is poor—or overcautiousness—failing to answer when evidence is good (Shao et al., 26 Sep 2025).

The paper reports that RL training improves calibration: RL-trained agents had lower CE than base or few-shot variants. However, this improvement did not carry over to grounded reasoning. One of the benchmark’s main substantive claims is therefore that calibration and groundedness are distinct competencies and should not be conflated.

5. Evaluation setting and empirical findings

SeekBench evaluates several LLM search agents, including the Qwen-2.5-7B-Instruct base model, a few-shot prompted variant, and the RL-trained agents Search-R1, ReSearch, ASearcher, and DeepResearcher. These agents were tested on seven QA benchmarks: NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, MuSiQue, and Bamboogle (Shao et al., 26 Sep 2025).

The test sets were sanitized to remove two classes of cases: ambiguous or unanswerable questions where speculative answers could get credit, and data-contamination cases where the answer seemed memorized rather than retrieved. For large-scale annotation, the study used GPT-4.1-mini as an LLM judge, validated against the human-labeled subset.

The empirical conclusions are structured around a tradeoff. First, RL training improves final-answer accuracy and calibration, but not grounded reasoning. Second, reasoning quality remains weak, especially for planning and state assessment. Third, recovery behavior matters independently of final accuracy, and is strongly linked to adaptive search strategies. Fourth, accuracy-only evaluation can misrepresent agent capabilities and hide specialization (Shao et al., 26 Sep 2025).

The paper highlights several concrete capability distinctions that are invisible under answer-only metrics. Search-R1 is described as a strong synthesizer. ASearcher is better at recovery. Base models can sometimes generate better reasoning than their final accuracy suggests. This suggests that multiple agent behaviors contribute differently to overall performance, and that a single answer score cannot capture their distribution across competencies. A plausible implication is that modular or hybrid agent designs could combine these strengths more effectively than monolithic optimization against final-answer reward alone.

6. Relation to adjacent benchmarks and broader significance

SeekBench belongs to a broader family of benchmarks that evaluate intermediate process quality rather than only terminal outputs, but its emphasis is specifically on epistemic competence in open-domain information-seeking agents. In this respect it differs from ScenarioBench, which is a policy-grounded, trace-aware benchmark for Text-to-SQL and retrieval-augmented generation in compliance contexts. ScenarioBench binds each decision to a falsifiable clause-ID witness trace under strict grounding and no-peek rules, whereas SeekBench analyzes whether reasoning, search reformulation, and answer timing are epistemically appropriate in agent traces (Atf et al., 29 Sep 2025).

SeekBench is also distinct from seqBench, a tunable benchmark for sequential reasoning in LLMs. seqBench studies logical depth, backtracking count, and noise ratio in synthetic pathfinding tasks on 2D grids, whereas SeekBench focuses on information-seeking traces, evidence states, and step-level epistemic behavior in QA agents (Ramezanali et al., 21 Sep 2025).

These contrasts clarify SeekBench’s specific contribution. It is not primarily a benchmark for compliance auditing, nor a benchmark for abstract sequential reasoning limits. Its target object is the information-seeking agent that must alternate between evidence collection and answer production under uncertainty. A common misconception is that such agents can be adequately characterized by end-task accuracy alone. SeekBench argues instead that groundedness, recovery, and calibration are separate axes of competence, and that the interaction among them is central to evaluating agentic QA systems (Shao et al., 26 Sep 2025).

In summary, SeekBench operationalizes the proposition that the quality of an information-seeking agent should be measured not only by whether it answers correctly, but by whether it reasons from observed evidence, adapts when evidence is poor, and answers only when the available support warrants doing so. Its significance lies in making these properties measurable at the level of response traces rather than final outputs.