SearchDecision-Bench: Schema-Conditioned Search Agent Decisions

Updated 7 October 2026

SearchDecision-Bench is a framework that treats key search decisions as schema-conditioned probabilistic tasks involving finite options, and processes related to search agent states.
The framework evaluates various decision-making tasks within search agents: routing, relevance, sufficiency, navigation, verification, and claim verification, expediting search through finite-option results that facilitate speed and ease in determining the direct next-step actions.
SearchEvaluation-Bench sets clear, actionable steps in assessing each search-related decision based on clear rules, coupled with concrete data and decision guidelines that make the benchmarking process seamless and practical.

SearchDecision-Bench is a benchmark for evaluating short, schema-conditioned decisions made inside search agents. It treats routing, query rewriting, relevance assessment, evidence sufficiency, navigation, and claim verification as finite-option probabilistic decisions rather than as open-ended text-generation tasks. A benchmark instance supplies a search state, a decision question, an output type, and legal options; the evaluated system returns an option and a probability distribution over those options. SearchDecision-Bench is introduced together with SearchJev, a System-1 decision model that performs direct option scoring, while a slower System-2 model retains planning, query generation, and answer composition (Cao et al., 4 Oct 2026).

1. Scope and conceptual framework

SearchDecision-Bench addresses the repeated local judgments that occur during agentic search. These judgments include determining whether a passage is relevant, whether current evidence is sufficient, whether a query should be rewritten, which source or sub-query to pursue, and whether retrieved evidence supports a claim. The benchmark is not an end-to-end answer benchmark: it evaluates intermediate decisions that may subsequently control a search trajectory.

Its central abstraction is a schema-conditioned decision. At search step $t$ , the state is represented as

$s_t=(q,H_t,C_t),$

where $q$ is the user request, $H_t$ is the trajectory history containing prior queries, opened documents, and intermediate notes, and $C_t$ is a candidate set such as documents, links, or sources. A decision schema specifies a named field, a natural-language question, an output type, and a finite set of legal options. The decision function is therefore

$(s,f,\mathcal V(f))\longrightarrow v\in\mathcal V(f),\quad p_f(v),$

where $p_f(v)$ is the predicted probability assigned to legal option $v$ .

The benchmark uses three output types:

Output type	Function	Examples
Choice	Select one categorical option	intent, next action, candidate query
Score	Assign an ordinal or graded judgment	retrieval need, source authority
Noul	Binary judgment	true or false

The term “noul” denotes binary typed decisions. The finite-option formulation makes decision quality, calibration, latency, and downstream delegation measurable under a common interface. It also separates decision making from natural-language generation: a model need not generate a JSON object or an explanation merely to select an option.

SearchDecision-Bench is methodologically related to broader work on structured selection and decision evaluation. “General Decision Models: Benchmarking and Insights Beyond Jev” defines a similar state–instruction–criteria formulation and emphasizes that accurate local decisions do not necessarily compose into reliable long-horizon systems (Duan et al., 2 Oct 2026). “Benchmarking Candidate Coverage in Typed Decision Models” further distinguishes selecting a correct offered candidate from recognizing that the reference candidate is absent (Lu et al., 2 Oct 2026). These issues are relevant extensions of SearchDecision-Bench, although candidate coverage is not one of its six principal decision families.

2. Decision families

SearchDecision-Bench unifies six families of search-agent decisions.

Routing

Routing determines what type of search or query-processing operation is appropriate. Tasks include intent classification, retrieval-need estimation, and query-clarity assessment. Retrieval need may be categorized as no retrieval, retrieval of one fact, retrieval of several facts, or multi-source research. Query clarity concerns which information is missing from an unclear request, such as time, location, object, aspect, or another specification.

Routing decisions affect the subsequent computational path. A system may decide that a request can be answered without retrieval, requires a single lookup, or demands multi-source investigation.

Rewriting

Rewriting evaluates whether a query or reformulation preserves the original information need. Representative tasks include query equivalence and conversational rewrite equivalence. These are generally binary decisions, with legal options True and False.

The benchmark treats rewriting as a decision rather than as generation quality. A generated reformulation may be fluent yet fail to preserve the intended query semantics; SearchDecision-Bench evaluates the semantic relation between the original and rewritten forms.

Relevance

Relevance evaluates retrieved material or source quality. It includes binary or graded document relevance, document usefulness for a current sub-query, and source-authority judgments with levels such as untrusted, user-generated, professional, and authoritative.

Relevance is the benchmark family most directly associated with candidate ranking. It is evaluated primarily with NDCG@10 rather than only classification accuracy, reflecting the graded nature of some relevance judgments.

Sufficiency

Sufficiency determines whether the evidence currently available is adequate to answer the original question. The canonical decision is whether the evidence covers all facts needed to answer.

This differs from relevance. A document can be relevant without making the evidence set sufficient. In multi-hop tasks, sufficiency requires coverage of every required fact or hop, not merely the presence of one useful passage.

Navigation selects the next search action. Tasks include choosing among search, reading a page, and answering; selecting the next unresolved sub-query; and assessing whether a candidate query targets the next unsolved step.

Navigation therefore evaluates trajectory control rather than isolated document assessment. Its decisions can change the future state $H_t$ , influence retrieval cost, and determine whether the agent reaches sufficient evidence.

Verification

Verification evaluates whether evidence supports a claim. The principal options are supports, refutes, and insufficient. Evidence detection is a related binary task concerning whether a sentence constitutes evidence for a claim.

Verification is used after or alongside answer generation. It provides a decision-level estimate of evidential support but does not itself revise the answer or restart search in the downstream BrowseComp-Plus experiment.

3. Data sources, supervision, and benchmark construction

SearchDecision-Bench combines Chinese and English resources spanning intent classification, query rewriting, retrieval, multi-hop question answering, agent demonstrations, and claim verification. Listed source datasets include BANKING77, CLINC150, MASSIVE, KUAKE-QIC, LCQMC, AFQMC, ATEC, BQ, CHIP-STS, PAWS-X, QReCC, T2Ranking, DuReader-Retrieval, Multi-CPR, QBQTC, KUAKE-QQR/QTR, MuSiQue, VitaminC, CFEVER, and TrendFact.

The benchmark converts heterogeneous source examples into serialized search states. Depending on the source, a state may contain a query–passage pair, query–query pair, claim–evidence pair, query–source-quality pair, or multi-hop trajectory with solved steps, retrieved evidence, and candidate next sub-queries. Sufficiency and navigation examples are derived partly from MuSiQue decompositions, hard negatives, LLM filtering, and agent demonstrations.

Training supervision has several origins:

Original dataset annotations.
Derived labels.
Agent demonstrations.
LLM teachers or votes.

LLMs participate in labeling 33.7% of training rows. The benchmark uses source-dependent confidence in supervision rather than treating every label as equally certain. Most original labels receive approximately 0.97 confidence, while demonstrations and constructed rewrites receive approximately 0.70. An example MuSiQue sufficiency target assigns 0.85 probability to True.

Source-dependent weights are used during optimization:

$w_f= \begin{cases} 0.5,&\text{LLM-assigned labels, demonstrations, and constructed rewrites},\ 1,&\text{otherwise}. \end{cases}$

The construction pipeline applies deduplication, removes conflicting labels, masks fields that cannot be judged from available information, isolates splits by identity keys and query text, and removes synthetic variants whose source occurs in evaluation data. The reported cleaning removed 56 rows with conflicting labels, 185 training rows, and 407 validation rows whose states appeared in a test set. Inputs exceeding 2,048 tokens are truncated at the state level.

The main six-family table contains 425,398 state–schema pairs:

Decision type	Training	ID test	OOD test
Routing	77,627	26,427	10,000
Rewriting	74,318	32,746	10,000
Relevance	153,990	17,581	5,976
Sufficiency	14,652	1,036	0
Navigation	56,981	10,213	0
Verification	47,830	10,334	20,631

Including auxiliary general-language tasks, the broader corpus contains 487,012 training rows, 159,124 validation rows, 250,899 test rows, and 683,130 expanded training decisions. Approximately 63.7% of training rows are mainly Chinese and 29.7% are mainly in Latin script.

ID tests use held-out samples from datasets represented during training. OOD tests use held-out datasets or splits including FRAMES, RAGTruth, CANDY, Search Arena, AgentRewardBench, and the QReCC test split. OOD results are reported for routing, rewriting, relevance, and verification; the main OOD table does not report sufficiency or navigation.

4. Models and inference mechanism

SearchJev-0.8B and SearchJev-4B are initialized from Qwen3.5-0.8B and Qwen3.5-4B and fine-tuned with LoRA. Both use rank 16, $s_t=(q,H_t,C_t),$ 0, learning rate $s_t=(q,H_t,C_t),$ 1, batch size 132, 5,176 training steps, maximum input length 2,048, AdamW, weight decay 0.01, gradient clipping 1.0, bf16 precision, and cosine scheduling with 5% warm-up.

The principal baselines are Qwen3.5 autoregressive JSON models of 0.8B and 4B parameters, Jev 1.13, and additional open System-1 models. The autoregressive models use constrained JSON decoding and derive confidence from option log-probabilities. Jev is queried through an API.

SearchJev does not generate a JSON response. Each legal option is assigned a single-token label. Given the vocabulary-logit vector $s_t=(q,H_t,C_t),$ 2 at the first assistant output position, the logit for option $s_t=(q,H_t,C_t),$ 3 is

$s_t=(q,H_t,C_t),$ 4

where $s_t=(q,H_t,C_t),$ 5 maps the option to its label token. SearchJev normalizes only over legal options:

$s_t=(q,H_t,C_t),$ 6

The selected decision is

$s_t=(q,H_t,C_t),$ 7

This direct readout avoids autoregressive output generation, JSON parsing, and multi-token answer production. It supports heterogeneous schemas through the field question and legal-option descriptions while using the same output head for Choice, Score, and Noul decisions.

The benchmark’s direct decision formulation is consistent with the broader observation that typed decision models can be substantially faster than generative models while still failing on specialist knowledge, exact computation, uncertainty estimation, or long-horizon composition (Duan et al., 2 Oct 2026). SearchDecision-Bench focuses on the local decision component rather than claiming that direct readout alone solves search planning.

5. Soft-label learning and calibration

SearchJev uses Soft-Label Learning for Calibrated Decisions (SLCD). Each field has a target distribution

$s_t=(q,H_t,C_t),$ 8

The training objective combines soft-label cross-entropy and a Brier-score term:

$s_t=(q,H_t,C_t),$ 9

with $q$ 0.

The cross-entropy term encourages agreement with the target distribution, while the Brier term penalizes squared probability error. The use of soft targets allows supervision to express uncertainty arising from demonstrations, constructed examples, and LLM-derived labels.

After training, SearchJev applies temperature scaling separately to Choice, Score, and Noul outputs. For output type $q$ 1,

$q$ 2

Temperatures are fitted by minimizing validation negative log-likelihood over 5,000 validation decisions. The reported temperatures are:

Model	Choice	Score	Noul
SearchJev-0.8B	1.11	0.96	0.88
SearchJev-4B	1.11	1.01	0.82

In the dual-system agent, a decision is accepted when

$q$ 3

If confidence falls below $q$ 4, the decision is delegated to System 2. The numerical value of $q$ 5 is not recoverable from the aggregate logs and is not treated as a benchmark-defined constant.

SearchDecision-Bench evaluates calibration with expected calibration error:

$q$ 6

The benchmark also evaluates decision quality with accuracy for classification-oriented families and NDCG@10 for relevance. Calibration and accuracy are treated as separate properties: a model may select the correct option while assigning substantially incorrect probabilities to the full option distribution.

The importance of this distinction is reinforced by “General Decision Models: Benchmarking and Insights Beyond Jev,” which reports that decision models can identify the most likely outcome while substantially overstating its probability (Duan et al., 2 Oct 2026). It is also consistent with selective-classification benchmarks, where risk–coverage behavior and coverage-constraint satisfaction cannot be inferred from ordinary accuracy alone (Pugnana et al., 2024).

6. Results, downstream use, and limitations

In-distribution results

The principal ID results are:

Model	Rel. NDCG	Sufficiency accuracy	Routing accuracy	Navigation accuracy	Rewriting accuracy	Verification accuracy	Latency (ms)
Jev 1.13	98.5	79.8	73.5	72.1	79.5	79.8	265.4
Qwen3.5-0.8B AR	85.7	49.9	24.7	42.0	40.6	56.8	144.2
Qwen3.5-4B AR	96.7	57.7	53.3	37.8	66.1	55.1	195.2
SearchJev-0.8B	98.6	89.4	83.2	56.8	84.0	91.7	27.6
SearchJev-4B	99.0	96.0	86.6	57.2	85.9	93.2	37.1

SearchJev outperforms its same-size autoregressive baseline in decision quality across all six families. SearchJev-4B leads five of the six families, while Jev remains stronger on navigation and is often better calibrated on navigation and verification.

The reported latency advantage is approximately 5.2–5.3 times over same-size Qwen3.5 autoregressive models and approximately 7.2–9.6 times over Jev 1.13. Relative to same-size autoregressive models, mean ECE is reduced by 74.4% for SearchJev-0.8B and 40.7% for SearchJev-4B.

Out-of-distribution results

OOD performance is more heterogeneous:

Model	Routing accuracy	Rewriting accuracy	Relevance NDCG	Verification accuracy	Latency (ms)
Jev 1.13	62.3	97.7	98.7	84.5	273.3
Qwen3.5-0.8B AR	29.2	47.3	82.2	53.9	153.2
Qwen3.5-4B AR	43.7	95.8	93.8	78.0	195.8
SearchJev-0.8B	41.1	99.6	95.5	55.7	27.7
SearchJev-4B	55.8	99.6	99.0	65.0	37.1

Both SearchJev models outperform their same-size autoregressive counterparts on routing, rewriting, and relevance quality. SearchJev-4B leads OOD relevance, while Jev remains stronger on routing and verification accuracy. Calibration transfers less consistently than decision preferences: Jev is better calibrated on OOD rewriting, relevance, and verification, whereas SearchJev-4B improves routing calibration.

These results limit the interpretation of SLCD. The evidence supports strong in-distribution improvements and substantial efficiency gains, but not uniformly superior OOD calibration. The paper also reports no conventional ablation isolating the contributions of cross-entropy, the Brier term, soft labels, instruction paraphrasing, option permutation, or temperature scaling.

BrowseComp-Plus

BrowseComp-Plus is a separate end-to-end experiment. It uses three SearchDecision-Bench decision functions—relevance, sufficiency, and verification—inside a search agent with BM25 retrieval and Qwen3.5-27B as System 2.

The results are:

Agent	Accuracy	System-2 calls	Output tokens (k)	Median time (s)
System-2 only	45.0	90.3	73.7	4,080
Dual-system with Jev 1.13	46.0	76.0	18.3	1,018
Dual-system with SearchJev-0.8B	46.0	62.1	15.5	868
Dual-system with SearchJev-4B	54.0	84.6	22.1	1,102

SearchJev-0.8B matches the Jev-based agent’s accuracy, while SearchJev-4B improves accuracy by eight percentage points over that agent. Both SearchJev agents exceed the System-2-only baseline. Relative to System-2-only, active search is reported as 3.7–4.7 times faster, with output tokens reduced by 3.3–4.8 times.

These results concern downstream agent behavior rather than the isolated benchmark. SearchDecision-Bench measures local decision quality, calibration, and latency; BrowseComp-Plus measures final answer quality, model calls, output tokens, and active search time. The distinction is important because a fast local decision can still cause long-horizon failure if its errors trigger unnecessary searches, incorrect stopping, or repeated recovery. Related evidence from $q$ 7-bench shows that replacing local decisions with faster decision-model choices can reduce median episode time while lowering task success and increasing tail latency (Duan et al., 2 Oct 2026).

Limitations and open issues

SearchDecision-Bench has several limitations.

Finite legal-option assumption: the benchmark assumes that the relevant decision space can be enumerated. Open-ended query generation, corpus discovery, and answer composition remain outside its principal task definition.
Candidate retrieval is not directly evaluated: most examples provide a prepared state and candidate space. The benchmark does not systematically measure corpus-wide retrieval, query reformulation, browsing, indexing, retrieval recall before ranking, or search stopping under a budget.
Heterogeneous supervision: 33.7% of training rows involve LLM labeling, and derived labels depend on heuristics, demonstrations, filtering models, or teacher votes.
Calibration uncertainty: soft-label confidence values are source-dependent design choices. ECE depends on binning and does not guarantee that the full probability distribution is correct.
OOD limitations: sufficiency and navigation have no reported OOD results in the principal table, and calibration advantages do not transfer uniformly.
No complete ablation: the individual contributions of soft labels, Brier loss, temperature scaling, paraphrasing, and option permutation are not isolated.
End-to-end sample size: BrowseComp-Plus contains 100 questions with one run per question, and its answer quality is judged by an LLM evaluator.
Threshold disclosure: the confidence threshold $q$ 8 used for delegation is not disclosed or recoverable from aggregate call counts.
Latency comparability: local and hosted measurements involve different serving conditions; hosted Jev latency includes network transport, while local measurements exclude queueing and transport.

For SearchDecision-Bench to function as a broader search benchmark, it would need additional evaluation of query formulation, source selection, candidate coverage, evidence provenance, search cost, stopping decisions, and multi-step trajectories. Candidate-coverage work shows why complete-set classification should be separated from recognizing that the correct candidate is absent (Lu et al., 2 Oct 2026). Evidence-locked selection further suggests that final answer quality should be separated from the admissibility and provenance of the evidence used to select it (Zhang et al., 7 Aug 2026). Execution-replay work adds that stable final decisions can conceal unstable tool paths, arguments, and evidence contacts (Khatchadourian, 10 Jun 2026).

SearchDecision-Bench’s principal contribution is therefore a unified intermediate-decision layer for search agents. It makes routing, rewriting, relevance, sufficiency, navigation, and verification comparable under finite schemas, soft supervision, calibrated probabilities, and direct-logit inference. Its results support the use of fast System-1 decisions for many in-distribution search-control tasks, while leaving open the broader questions of open-world retrieval, candidate discovery, evidence completeness, long-horizon reliability, and calibration under distribution shift.