DecisionBench: Delegation Benchmark

• DecisionBench is a benchmark substrate for emergent delegation in agentic workflows, distinguishing the orchestration layer from end-task quality.
• It fixes a core architecture with a consistent task suite, heterogeneous peer models, a dedicated delegation interface, and a rule-based skill annotation layer.
• Empirical results reveal that while mean task quality remains constant across conditions, routing fidelity and delegation rates vary significantly, highlighting process-level performance gaps.

Searching arXiv for the DecisionBench paper and closely related benchmark literature. DecisionBench is a benchmark substrate for emergent delegation in long-horizon agentic workflows. It is designed to evaluate not only whether an orchestrating agent completes a task, but whether it delegates at the right time, to the right peer, through the right interface, and with what process-level consequences. The substrate fixes a task suite, a peer-model pool, a delegation interface, a deterministic skill-annotation layer, and a multi-axis metric suite, so that orchestration behavior can be measured separately from ordinary end-task quality. In its reference characterization over $n=23{,}375$ task instances, mean end-task quality is reported as statistically indistinguishable across the four awareness conditions, while routing fidelity-at-1 varies substantially and a counterfactual ceiling remains well above observed performance (Gao et al., 18 May 2026).

1. Problem formulation and benchmark purpose

DecisionBench targets a specific evaluation gap in agent systems. Existing benchmarks typically score the final outcome on a fixed task, but they do not isolate the orchestration layer: whether the agent should delegate, which peer it should select, how peer information should be delivered, and whether those choices affect cost, latency, or bias in delegation. DecisionBench therefore treats delegation itself as a measurable object rather than as an implementation detail (Gao et al., 18 May 2026).

The benchmark is framed around emergent delegation. An orchestrator is given access to a pool of peer models and may either solve work locally or delegate subtasks. The central research question is not merely whether access to peers increases benchmark score, but whether the orchestrator’s routing behavior is appropriate under heterogeneous peer capabilities. This makes DecisionBench especially relevant to settings in which multiple models coexist inside a single agentic workflow.

A common misconception is that orchestration quality can be inferred from final task quality alone. The benchmark’s design explicitly rejects that assumption. Its reference results show that systems can achieve nearly identical mean quality while exhibiting very different routing behavior, which is why DecisionBench includes process metrics such as routing fidelity-at-$k$, delegation rate, vendor self-preference, and a counterfactual-delegation ceiling (Gao et al., 18 May 2026).

2. Fixed substrate architecture

DecisionBench fixes five components so that future delegation methods can be compared against a common substrate. The task suite combines GAIA, tau-bench, and BFCL multi-turn. The peer-model pool contains 11 models across 7 vendor families. The delegation interface consists of call_model and an optional read_profile channel. The evaluation layer adds deterministic skill annotation, and the metric suite measures both outcome and orchestration properties (Gao et al., 18 May 2026).

Component	Fixed design	Function
Task suite	GAIA, tau-bench, BFCL multi-turn	Heterogeneous probes of agentic behavior
Peer-model pool	11 models, 7 vendor families	Makes delegation meaningful
Delegation interface	call_model and optional read_profile	Separates delegation from peer-information access
Annotation layer	Frozen 7-skill taxonomy, rule-based tagger	Supports process-level evaluation
Metric suite	Quality, cost, latency, delegation rate, routing fidelity-at-$k$, vendor self-preference, counterfactual ceiling	Measures orchestration beyond final score

The three task sources play distinct roles. GAIA is scored by exact match and probes retrieval, reasoning, and tool use. tau-bench is a multi-turn tool-agent-user dialogue benchmark scored by pass@$k$, emphasizing multi-turn state tracking and policy compliance. BFCL multi-turn is scored by AST match on per-turn calls and emphasizes structured function-calling correctness. The benchmark uses deterministic Stage-1 / Stage-2 splits, where Stage-1 is used for profiling and peer-description construction and Stage-2 is reserved for held-out evaluation. The split is a deterministic 20/80 stratified sample (Gao et al., 18 May 2026).

The peer pool is intentionally heterogeneous and is pinned to a freeze date and routed through OpenRouter. That heterogeneity is essential to the benchmark’s logic: if all peers were effectively interchangeable, routing quality would be largely unobservable.

3. Delegation interface, skill annotation, and routing measurement

The orchestrator is an LLM agent that can invoke tools and solve tasks. The central delegation primitive is:

$k$3

With this call, the orchestrator sends a subtask string to a named peer in the pool. The peer sees only the subtask text and its own system prompt; it does not see the earlier conversation history. The peer response is returned as a tool result. This design isolates the act of delegation itself rather than conflating it with full conversational state transfer (Gao et al., 18 May 2026).

The optional second channel is:

$k$4

This returns a structured description of a peer model. It is the mechanism through which peer-awareness interventions are delivered. DecisionBench uses this distinction to compare preloaded peer descriptions against on-demand profile retrieval, which allows the benchmark to test whether the delivery channel of peer information matters as much as, or more than, the content of the description.

A deterministic, rule-based annotation layer assigns each trajectory step to one of a frozen seven-skill taxonomy: tool-call schema adherence, multi-turn state tracking, domain-policy compliance, information retrieval, multi-step reasoning, numerical computation, and long-input handling. The tagger is explicitly not an LLM-based component; it uses trace-only signals such as tool names, prompt token count, refusal or confirmation regexes, and whether earlier tool calls occurred. An audit reported that 94.5% of free-form LLM re-tagging maps cleanly onto one of the seven labels (Gao et al., 18 May 2026).

This annotation layer enables the benchmark’s process metrics. Routing fidelity-at-$k$ is computed by inferring the dominant skill of the current trajectory, ranking peers by their Stage-1 pass rate on that skill, and checking whether the chosen peer lies in the top $k$. Vendor self-preference compares same-vendor delegation rates against chance given pool composition. The counterfactual-delegation ceiling imagines, for each blind task, that the orchestrator delegated the whole task to the Stage-1-best peer for the inferred dominant skill and that the peer answered at its empirical Stage-1 pass rate. The paper notes that this ceiling is a lower envelope on a fuller “perfect orchestration” ceiling because real tasks may require multi-step delegation (Gao et al., 18 May 2026).

4. Reference intervention suite and experimental characterization

The paper characterizes the substrate through a five-condition reference sweep. These conditions are not themselves the definition of the benchmark; rather, they are controlled interventions used to expose what the substrate can measure (Gao et al., 18 May 2026).

The conditions are blind, aware-c1, aware-c2, aware-c3, and aware-tool-only. In the blind condition, the orchestrator has only call_model and no peer descriptions. In aware-c1, aware-c2, and aware-c3, the orchestrator has both call_model and read_profile, with peer descriptions built respectively from a curated rubric, deterministic statistics on Stage-1 traces, and dual out-of-pool LLM judges. In aware-tool-only, the same C2 profile is available on demand through read_profile, but no preloaded description is given. The authors present this as a way to separate a content axis from a delivery axis (Gao et al., 18 May 2026).

The reference evaluation spans 11 models, 3 benchmarks, and 5 conditions, yielding 165 cells and $n=23{,}375$ task instances overall. The release contains 220 per-condition run archives, because tau-bench is sharded into airline and retail runs. The scale is therefore not a single leaderboard snapshot, but a structured characterization of how orchestration behavior changes under controlled variations in peer awareness (Gao et al., 18 May 2026).

This design supports an important methodological distinction. A learned router, a richer peer memory, an adaptive profile-construction method, or a multi-step delegation policy can all be evaluated on the same substrate without changing the task suite, peer pool, or metrics. DecisionBench is thus a benchmark substrate rather than a single fixed model comparison.

5. Empirical findings

The benchmark’s first central finding is that mean end-task quality is statistically indistinguishable across awareness conditions. In the mixed-effects regression reported in the paper, all awareness-condition coefficients satisfy $|\beta| \le 0.010$ and all associated p-values satisfy $p \ge 0.21$. The fitted coefficients are aware-c1: $\beta=-0.005, p=0.490$, aware-c2: $k$0, aware-c3: $k$1, and aware-tool-only: $k$2 (Gao et al., 18 May 2026).

The second central finding is that routing fidelity-at-1 changes substantially even when quality remains nearly flat. The reported fidelity@1 values are 14.2% for blind, 7.5% for aware-c1, 20.8% for aware-c2, 15.5% for aware-c3, and 29.5% for aware-tool-only. The range therefore spans 7.5% to 29.5% at near-equal mean quality. The paper interprets this as evidence that delivery channel dominates description content: on-demand access via read_profile performs best, and preloaded peer cards recover less than half of the fidelity gain (Gao et al., 18 May 2026).

The third benchmark-level result is the counterfactual-delegation ceiling, which places perfect single-step delegation 15–31 percentage points above measured performance on every suite. The reported figures are GAIA: blind actual 0.407, ceiling 0.675, gap +0.269; tau-bench: blind actual 0.695, ceiling 0.848, gap +0.153; and BFCL: blind actual 0.536, ceiling 0.849, gap +0.313. This locates large unrealized headroom for future orchestration methods (Gao et al., 18 May 2026).

Several secondary findings refine this picture. Awareness conditions can shift systems into a different cost band without improving quality much. Reported mean latencies include GAIA blind: 105s versus GAIA aware-tool-only: 79s, and BFCL blind: 69s versus BFCL aware-tool-only: 64s. Delegation rate is highly uneven across suites: GAIA invites delegation much more often, BFCL is intermediate, and tau-bench exhibits near-zero delegation overall. The paper also reports vendor self-preference above chance for some models, including Gemini-3-Flash: 1.48×, DeepSeek-V4-Flash: 1.53×, DeepSeek-V4-Pro: 1.85×, and GPT-5.5: 3.65× (Gao et al., 18 May 2026).

Taken together, these results imply that a benchmark focused only on final quality would miss the main orchestration signal. A plausible implication is that future evaluation of agent systems will increasingly need separate axes for task execution and work allocation.

6. Position within the broader decision-benchmark landscape

DecisionBench belongs to a broader movement in benchmark design that measures decision quality under structure, uncertainty, or operational constraints, rather than only answer correctness. Within that landscape, its distinctive focus is the orchestration problem: when multiple peer models are available, can the system manage delegation well (Gao et al., 18 May 2026)?

Other recent benchmarks target different decision layers. ClinDet-Bench formalizes judgment determinability under incomplete clinical information and distinguishes determinable from undeterminable cases rather than treating all incomplete cases as unanswerable (Watanabe et al., 26 Feb 2026). AR-BENCH evaluates appellate review as post-judgment error detection, classification, and correction, framing legal review as anomaly detection rather than prediction (Li et al., 30 Jan 2026). ScenarioBench makes compliance decisions trace-grounded by requiring clause-level evidence from the system’s own retrieval, thereby scoring not only what decision was made but why (Atf et al., 29 Sep 2025). CONDESION-BENCH shifts decision-making from atomic labels to compositional actions under explicit feasibility conditions, with oracle-based scoring of both utility and condition adherence (Hwang et al., 10 Apr 2026). JudgmentBench compares rubric-based scoring and comparative judgment for high-expertise legal evaluation, asking which elicitation protocol better recovers intended quality orderings when no objective gold answer exists (Yang et al., 24 May 2026). Deployment-complete benchmarking goes further by asking whether benchmark evidence actually determines the deployment action, introducing mixed fibers, certifiable fraction, and completion curves as diagnostics of decision readiness (Mansouri et al., 25 May 2026).

Viewed against these resources, DecisionBench is not a general benchmark of decision-making in the abstract. It is specifically a benchmark of delegation policy in agentic systems. Its core contribution is to make routing behavior observable and comparable under fixed tasks, fixed peers, and fixed evaluation machinery. This suggests a division of labor among contemporary benchmarks: some test whether a model can decide correctly under uncertainty, some test whether it can justify or audit a decision, and DecisionBench tests whether it can assign work across peers in a long-horizon workflow.

7. Artifacts, extensibility, and research significance

The release includes the DecisionBench substrate, the annotation layer, the reference intervention suite, the analysis pipeline, and 220 per-condition run archives containing roughly 23,000 task-level traces. The reference intervention suite includes 33 profile cards in three variants and an aware-tool-only ablation harness. The annotation layer includes tagger.py v2.0, the frozen seven-skill taxonomy, and an emergent-audit CSV (Gao et al., 18 May 2026).

This release structure matters because the benchmark is intended to be extensible. The paper explicitly states that the substrate is agnostic to how peer information is generated or delivered, so learned routers, richer peer memories, adaptive profile construction, and multi-step delegation can all be evaluated against it. The benchmark therefore functions as a reusable measurement framework rather than as a one-time ablation.

Its broader significance lies in the claim that future agent systems will be evaluated not only on how well they answer or act, but also on how well they manage work across heterogeneous peers. DecisionBench operationalizes that claim by separating orchestration from outcome. The benchmark’s empirical characterization shows that orchestration effects can be large even when ordinary benchmark scores are nearly unchanged, and that current systems remain far below a counterfactual ceiling. In that sense, DecisionBench establishes delegation as a first-class benchmark target rather than a hidden systems variable (Gao et al., 18 May 2026).