SearchJev: Decision Model for Search Agents

• SearchJev optimises decision speed and reduces errors in strategic, long-term search operations, achieving up to 5.3-fold reductions in latency while maintaining or improving upon System 2 models, separating query routing, document handling, and retrieval.
• SearchJev improves upon Qwen3.5 AR models across on multiple benchmark families (relevance NDCG, accuracy for Routing, Navigation, Rewriting, Sufficiency, and Verification),
• The calibration of SearchJev, trained with soft-label learning and temperature calibration, allows for dynamic integration of confidence-based delegation by passing decision authority to System 2 unless it achieves a certain confidence level threshold.

SearchJev is a schema-conditioned System-1 decision model for search agents. It performs repeated, finite-choice judgments—such as relevance assessment, evidence sufficiency, query routing, search navigation, rewriting verification, and claim checking—by directly scoring legal options and returning a probability distribution, while delegating planning, query generation, difficult reasoning, browsing strategy, and answer composition to a System-2 generative LLM. Its principal implementation combines direct legal-option logit readout, Soft-Label Learning for Calibrated Decisions (SLCD), temperature calibration, and confidence-based delegation (Cao et al., 4 Oct 2026).

1. Concept and architectural position

SearchJev addresses the computational structure of agentic search. A search agent repeatedly makes short decisions inside a longer trajectory, including whether a retrieved document is relevant, whether the current evidence is sufficient, whether another query should be issued, and whether a claim is supported. Conventional agents commonly use an autoregressive LLM for these decisions as well as for planning and answer generation. This introduces sequential decoding overhead, JSON parsing and formatting failures, and confidence estimates that are difficult to use operationally.

SearchJev separates the workload into two functional systems:

• System 1: SearchJev performs fast, typed decisions with probabilities over a finite legal-option set.
• System 2: a larger generative model performs planning, query generation, difficult judgments, browsing strategy, reasoning, and final answer composition.

At search step $t$, the search state is represented as

$s_t=(q,H_t,C_t),$

where $q$ is the original request, $H_t$ is the accumulated trajectory, and $C_t$ is the current set of candidate documents, links, or sources. A decision schema is

$\sigma=\{f_1,\ldots,f_k\},$

where each field $f_i$ defines a natural-language question, an output type, and a finite set of legal options $\mathcal{V}(f_i)$.

SearchJev is not a conventional vector-space retriever and does not retrieve documents through approximate nearest-neighbor search in the reported experiments. It evaluates alternatives supplied by another component. Retrieval, candidate construction, business constraints, diversity constraints, and open-ended generation remain external responsibilities.

2. Decision schema and supported task families

SearchJev provides a unified interface for six classes of search decisions.

Routing determines the interpretation of a request or the required retrieval strategy. Examples include intent classification, retrieval need—none, one fact, several facts, or multi-source research—and query clarity, such as whether time, location, object, or aspect information is missing.

Rewriting evaluates whether a reformulated query preserves the original intent. It includes query-equivalence and conversational-rewrite-equivalence judgments.

Relevance evaluates retrieved material through binary or graded relevance, document usefulness for a current sub-query, and document quality. The reported source-quality scale consists of untrusted, user-generated, professional, and authoritative.

Sufficiency determines whether the current evidence contains all facts needed to answer the original question.

Navigation selects the next search operation, including search, reading a page, answering, selecting a sub-query, or assessing whether a candidate query addresses the next unsolved step.

Verification evaluates claims and evidence through supported, refuted, or insufficient outcomes, and determines whether a sentence constitutes evidence for a claim.

The interface supports three output types:

1. Choice: categorical selection from arbitrary options.
2. Score: ordinal selection, such as relevance or source quality.
3. Noul: binary judgment, with options such as true and false.

Each option is assigned a distinct single-token vocabulary label, typically A, B, or C. The option description may contain multiple tokens; only the label must be a single token. This allows the model to read a semantically rich schema while extracting probabilities from a small legal vocabulary.

3. Direct decision readout and probability computation

SearchJev is initialized from Qwen3.5-0.8B or Qwen3.5-4B but is not used as a conventional autoregressive JSON generator. Given serialized input $x_f$ for field $f$, the model produces a vocabulary-logit vector at the first output position. If $s_t=(q,H_t,C_t),$0 is the vocabulary index of the label assigned to legal option $s_t=(q,H_t,C_t),$1, its option logit is

$s_t=(q,H_t,C_t),$2

The decision probability is normalized only over legal options:

$s_t=(q,H_t,C_t),$3

Thus, a three-option decision compares only the logits of the three permitted labels. The remaining vocabulary does not participate in the decision softmax.

Inference consists of serializing the search state and schema field, executing one forward pass, reading the logits at the final input position, extracting the logits for legal option labels, normalizing them, and mapping the winning label to its structured option. There is no multi-token answer generation, JSON decoding, or output parsing. Independent fields sharing a search state can be batched using left padding.

This interface differs from a generative baseline even when the same backbone is used. The autoregressive Qwen3.5 baselines generate structured JSON and derive confidence from option log-probabilities. SearchJev directly reads the option-label logits, which accounts for its lower decision latency in the reported comparisons.

4. Soft-Label Learning and calibration

SearchJev is trained with Soft-Label Learning for Calibrated Decisions (SLCD). The training corpus combines human annotations, retrieval judgments, derived labels, agent demonstrations, constructed rewrites, LLM teachers, and votes. Because these sources differ in reliability, supervision is represented as a target distribution rather than necessarily as a one-hot label. A binary example may therefore use a target such as

$s_t=(q,H_t,C_t),$4

Labels without explicit uncertainty information are represented as one-hot targets. LLMs assign labels to approximately 33.7% of training rows.

For each field, the loss combines soft-label cross-entropy and multiclass Brier loss:

$s_t=(q,H_t,C_t),$5

with $s_t=(q,H_t,C_t),$6. The field weight is $s_t=(q,H_t,C_t),$7 for LLM-assigned labels, demonstrations, and constructed rewrites, and $s_t=(q,H_t,C_t),$8 for other labels. The training corpus also includes Chinese and English paraphrases of field instructions, permutations of unordered option lists, and state-schema serialization rather than task-specific output heads.

After training, SearchJev applies temperature scaling separately to Choice, Score, and Noul outputs:

$s_t=(q,H_t,C_t),$9

Temperatures are fitted on 5,000 validation decisions by minimizing validation negative log-likelihood. The reported temperatures are:

Output type	SearchJev-0.8B	SearchJev-4B
Choice	1.11	1.11
Score	0.96	1.01
Noul	0.88	0.82

Temperature scaling preserves option ranking while changing confidence sharpness. The calibrated probability is used by the dual-system agent as a selective-computation signal. A decision is accepted when

$q$0

otherwise the field is delegated to System 2.

The reported calibration metric is expected calibration error (ECE). SearchJev also uses accuracy, NDCG@10 for relevance, and median per-decision latency at batch size one. In BrowseComp-Plus, evaluation includes final answer accuracy, System-2 calls, output tokens, reasoning tokens, and median active search time.

5. SearchDecision-Bench evaluation

SearchDecision-Bench unifies Chinese- and English-language data across the six decision families. Its sources include BANKING77, CLINC150, MASSIVE, KUAKE-QIC, LCQMC, AFQMC, ATEC, BQ, CHIP-STS, PAWS-X, QReCC, T2Ranking, DuReader-Retrieval, Multi-CPR, QBQTC, KUAKE retrieval resources, MuSiQue, VitaminC, CFEVER, and TrendFact.

The reported corpus contains 487,012 training rows, 159,124 validation rows, and 250,899 test rows. The expanded training corpus contains 683,130 decisions. Inputs are truncated at 2,048 tokens, duplicate inputs are deduplicated, contradictory examples are removed, and train–validation–test isolation is enforced by identity keys and query text.

In-distribution results

Model	Relevance NDCG	Sufficiency accuracy	Routing accuracy	Navigation accuracy	Rewrite accuracy	Verification accuracy	Latency
Jev 1.13	98.5	79.8	73.5	72.1	79.5	79.8	265.4 ms
Qwen3.5-0.8B AR	85.7	49.9	24.7	42.0	40.6	56.8	144.2 ms
Qwen3.5-4B AR	96.7	57.7	53.3	37.8	66.1	55.1	195.2 ms
SearchJev-0.8B	98.6	89.4	83.2	56.8	84.0	91.7	27.6 ms
SearchJev-4B	99.0	96.0	86.6	57.2	85.9	93.2	37.1 ms

Relative to the same-size autoregressive Qwen3.5 systems, SearchJev-0.8B and SearchJev-4B improve all six reported quality families except that the 0.8B system is weaker on the difficult navigation comparison relative to Jev. Mean ECE decreases by 74.4% for SearchJev-0.8B and 40.7% for SearchJev-4B. Latency falls from 144.2 ms to 27.6 ms for the 0.8B configuration and from 195.2 ms to 37.1 ms for the 4B configuration, corresponding to approximately 5.2–5.3 times faster decisions.

Out-of-distribution results

Model	Routing accuracy	Rewriting accuracy	Relevance NDCG	Verification accuracy	Latency
Jev 1.13	62.3	97.7	98.7	84.5	273.3 ms
Qwen3.5-0.8B AR	29.2	47.3	82.2	53.9	153.2 ms
Qwen3.5-4B AR	43.7	95.8	93.8	78.0	195.8 ms
SearchJev-0.8B	41.1	99.6	95.5	55.7	27.7 ms
SearchJev-4B	55.8	99.6	99.0	65.0	37.1 ms

Out-of-distribution transfer is mixed. SearchJev retains substantial latency advantages and generally strong accuracy, but calibration transfer is less consistent than decision accuracy. Jev remains superior on several held-out calibration tasks and some navigation or verification settings. The results therefore support task-dependent rather than universal calibration transfer.

6. Dual-system search agent and BrowseComp-Plus

In the dual-system architecture, System 2 maintains the research process and generates queries. A retriever returns candidate documents, after which SearchJev evaluates relevance and ranks candidates using calibrated relevance probabilities. SearchJev also evaluates evidence sufficiency when the agent might answer or when another search is being considered. High-confidence decisions are accepted; uncertain decisions are delegated to System 2.

For BrowseComp-Plus, the reported configuration retrieves the top 50 unseen documents, retains documents with relevance probability at least $q$1, returns at least one and at most five documents per search, checks sufficiency when System 2 proposes an answer or requests a third or later search, and permits insufficient evidence to force further search up to twice per question. Final verification produces a calibrated support score but does not revise the answer or restart search in the reported workflow.

The evaluation sampled 100 questions and used Qwen3.5-27B as System 2 with BM25 retrieval.

Agent	Accuracy	System-2 calls	Output tokens	Median active time
System-2 only	45.0%	90.3	73.7k	4080 s
Dual-system + Jev 1.13	46.0%	76.0	18.3k	1018 s
Dual-system + SearchJev-0.8B	46.0%	62.1	15.5k	868 s
Dual-system + SearchJev-4B	54.0%	84.6	22.1k	1102 s

Relative to the System-2-only agent, SearchJev-0.8B provides approximately a 4.7-fold reduction in active search time, while SearchJev-4B provides approximately a 3.7-fold reduction. Accuracy increases from 45.0% to 46.0% with SearchJev-0.8B and to 54.0% with SearchJev-4B.

The results illustrate that decision acceleration is not equivalent to end-to-end agent reliability. A fast local decision can reduce typical trajectory cost while still causing search errors that affect later retrieval, evidence accumulation, and answer generation. SearchJev is consequently most useful when integrated with delegation, uncertainty handling, and recovery mechanisms rather than used as an unconditional replacement for System 2.

7. Limitations and deployment considerations

SearchJev requires a finite legal-option set. It cannot generate open-ended queries or final answers, which remain System-2 responsibilities. Its performance is also dependent on schema quality, option semantics, retrieval quality, and the evidence supplied in the search state.

Calibration is strongest in-distribution and less consistent out-of-distribution. SearchJev’s confidence should therefore be validated against representative search judgments and the intended human-review capacity. Option probabilities should not automatically be treated as universally calibrated probabilities for expected utility, risk aggregation, or irreversible decisions.

The model is sensitive to context length and evidence placement. SearchDecision-Bench uses inputs truncated at 2,048 tokens, and the reported results do not establish robustness to substantially longer evidence contexts. Search systems should prefer compact candidate-specific contexts, retrieved supporting passages, and explicit evidence localization over concatenating many full documents.

The BrowseComp-Plus evaluation uses 100 sampled questions and one run per question. The supplied results do not provide a conventional ablation suite isolating soft labels, Brier loss, temperature scaling, option permutation, schema paraphrases, or confidence thresholds. The contribution of each component is therefore not independently identified.

The released configurations are SearchJev-0.8B and SearchJev-4B, initialized from Qwen3.5-0.8B and Qwen3.5-4B. Training uses LoRA with rank 16, scale 32, learning rate $q$2, batch size 132, 5,176 steps, maximum input length 2,048, AdamW, weight decay 0.01, gradient clipping 1.0, cosine scheduling, 5% warm-up, and bfloat16 precision. LoRA is applied to attention, Gated DeltaNet, and MLP projections.

SearchJev is therefore best characterized as a fast, schema-conditioned decision layer for high-volume search judgments. Its appropriate role is to filter, rank, verify, score, and route candidate evidence while reserving planning, open-ended generation, difficult reasoning, and recovery for System 2. The reported evidence supports substantial reductions in decision latency and improved in-distribution calibration, but also shows that out-of-distribution transfer, navigation, confidence calibration, and long-horizon reliability remain task-dependent.