---
title: 'DecisionBench: Delegation Benchmark'
url: https://www.emergentmind.com/topics/decisionbench
type: topic
---

# DecisionBench: Delegation Benchmark

Searching arXiv for the DecisionBench paper and closely related benchmark literature.
DecisionBench is a benchmark substrate for **emergent delegation** in **long-horizon agentic workflows**. It is designed to evaluate not only whether an orchestrating agent completes a task, but whether it delegates at the right time, to the right peer, through the right interface, and with what process-level consequences. The substrate fixes a task suite, a peer-model pool, a delegation interface, a deterministic skill-annotation layer, and a multi-axis metric suite, so that orchestration behavior can be measured separately from ordinary end-task quality. In its reference characterization over **\(n=23{,}375\)** task instances, mean end-task quality is reported as statistically indistinguishable across the four awareness conditions, while routing fidelity-at-1 varies substantially and a counterfactual ceiling remains well above observed performance [2605.19099].

## 1. Problem formulation and benchmark purpose

DecisionBench targets a specific evaluation gap in agent systems. Existing benchmarks typically score the **final outcome** on a fixed task, but they do not isolate the **orchestration layer**: whether the agent should delegate, which peer it should select, how peer information should be delivered, and whether those choices affect cost, latency, or bias in delegation. DecisionBench therefore treats delegation itself as a measurable object rather than as an implementation detail [2605.19099].

The benchmark is framed around **emergent delegation**. An orchestrator is given access to a pool of peer models and may either solve work locally or delegate subtasks. The central research question is not merely whether access to peers increases benchmark score, but whether the orchestrator’s routing behavior is appropriate under heterogeneous peer capabilities. This makes DecisionBench especially relevant to settings in which multiple models coexist inside a single agentic workflow.

A common misconception is that orchestration quality can be inferred from final task quality alone. The benchmark’s design explicitly rejects that assumption. Its reference results show that systems can achieve nearly identical mean quality while exhibiting very different routing behavior, which is why DecisionBench includes process metrics such as routing fidelity-at-\(k\), delegation rate, vendor self-preference, and a counterfactual-delegation ceiling [2605.19099].

## 2. Fixed substrate architecture

DecisionBench fixes five components so that future delegation methods can be compared against a common substrate. The task suite combines **GAIA**, **tau-bench**, and **BFCL multi-turn**. The peer-model pool contains **11 models across 7 vendor families**. The delegation interface consists of `call_model` and an optional `read_profile` channel. The evaluation layer adds deterministic skill annotation, and the metric suite measures both outcome and orchestration properties [2605.19099].

| Component | Fixed design | Function |
|---|---|---|
| Task suite | GAIA, tau-bench, BFCL multi-turn | Heterogeneous probes of agentic behavior |
| Peer-model pool | 11 models, 7 vendor families | Makes delegation meaningful |
| Delegation interface | `call_model` and optional `read_profile` | Separates delegation from peer-information access |
| Annotation layer | Frozen 7-skill taxonomy, rule-based tagger | Supports process-level evaluation |
| Metric suite | Quality, cost, latency, delegation rate, routing fidelity-at-\(k\), vendor self-preference, counterfactual ceiling | Measures orchestration beyond final score |

The three task sources play distinct roles. **GAIA** is scored by exact match and probes retrieval, reasoning, and tool use. **tau-bench** is a multi-turn tool-agent-user dialogue benchmark scored by pass@\(k\), emphasizing multi-turn state tracking and policy compliance. **BFCL multi-turn** is scored by AST match on per-turn calls and emphasizes structured function-calling correctness. The benchmark uses deterministic **Stage-1 / Stage-2** splits, where Stage-1 is used for profiling and peer-description construction and Stage-2 is reserved for held-out evaluation. The split is a deterministic **20/80 stratified sample** [2605.19099].

The peer pool is intentionally heterogeneous and is pinned to a freeze date and routed through OpenRouter. That heterogeneity is essential to the benchmark’s logic: if all peers were effectively interchangeable, routing quality would be largely unobservable.

## 3. Delegation interface, skill annotation, and routing measurement

The orchestrator is an LLM agent that can invoke tools and solve tasks. The central delegation primitive is:

```text
call_model(name, subtask, budget_usd)
```

With this call, the orchestrator sends a **subtask string** to a named peer in the pool. The peer sees only the subtask text and its own system prompt; it does **not** see the earlier conversation history. The peer response is returned as a tool result. This design isolates the act of delegation itself rather than conflating it with full conversational state transfer [2605.19099].

The optional second channel is:

```text
read_profile(model)
```

This returns a structured description of a peer model. It is the mechanism through which peer-awareness interventions are delivered. DecisionBench uses this distinction to compare **preloaded peer descriptions** against **on-demand profile retrieval**, which allows the benchmark to test whether the *delivery channel* of peer information matters as much as, or more than, the content of the description.

A deterministic, rule-based annotation layer assigns each trajectory step to one of a frozen **seven-skill taxonomy**: **tool-call schema adherence**, **multi-turn state tracking**, **domain-policy compliance**, **information retrieval**, **multi-step reasoning**, **numerical computation**, and **long-input handling**. The tagger is explicitly not an LLM-based component; it uses trace-only signals such as tool names, prompt token count, refusal or confirmation regexes, and whether earlier tool calls occurred. An audit reported that **94.5%** of free-form LLM re-tagging maps cleanly onto one of the seven labels [2605.19099].

This annotation layer enables the benchmark’s process metrics. **Routing fidelity-at-\(k\)** is computed by inferring the dominant skill of the current trajectory, ranking peers by their Stage-1 pass rate on that skill, and checking whether the chosen peer lies in the top \(k\). **Vendor self-preference** compares same-vendor delegation rates against chance given pool composition. The **counterfactual-delegation ceiling** imagines, for each blind task, that the orchestrator delegated the whole task to the Stage-1-best peer for the inferred dominant skill and that the peer answered at its empirical Stage-1 pass rate. The paper notes that this ceiling is a **lower envelope** on a fuller “perfect orchestration” ceiling because real tasks may require multi-step delegation [2605.19099].

## 4. Reference intervention suite and experimental characterization

The paper characterizes the substrate through a **five-condition reference sweep**. These conditions are not themselves the definition of the benchmark; rather, they are controlled interventions used to expose what the substrate can measure [2605.19099].

The conditions are **blind**, **aware-c1**, **aware-c2**, **aware-c3**, and **aware-tool-only**. In the **blind** condition, the orchestrator has only `call_model` and no peer descriptions. In **aware-c1**, **aware-c2**, and **aware-c3**, the orchestrator has both `call_model` and `read_profile`, with peer descriptions built respectively from a curated rubric, deterministic statistics on Stage-1 traces, and dual out-of-pool LLM judges. In **aware-tool-only**, the same C2 profile is available **on demand** through `read_profile`, but no preloaded description is given. The authors present this as a way to separate a **content axis** from a **delivery axis** [2605.19099].

The reference evaluation spans **11 models**, **3 benchmarks**, and **5 conditions**, yielding **165 cells** and **\(n=23{,}375\)** task instances overall. The release contains **220 per-condition run archives**, because tau-bench is sharded into airline and retail runs. The scale is therefore not a single leaderboard snapshot, but a structured characterization of how orchestration behavior changes under controlled variations in peer awareness [2605.19099].

This design supports an important methodological distinction. A learned router, a richer peer memory, an adaptive profile-construction method, or a multi-step delegation policy can all be evaluated on the same substrate without changing the task suite, peer pool, or metrics. DecisionBench is thus a **benchmark substrate** rather than a single fixed model comparison.

## 5. Empirical findings

The benchmark’s first central finding is that **mean end-task quality is statistically indistinguishable across awareness conditions**. In the mixed-effects regression reported in the paper, all awareness-condition coefficients satisfy **\(|\beta| \le 0.010\)** and all associated p-values satisfy **\(p \ge 0.21\)**. The fitted coefficients are **aware-c1: \(\beta=-0.005, p=0.490\)**, **aware-c2: \(\beta=-0.010, p=0.220\)**, **aware-c3: \(\beta=-0.008, p=0.296\)**, and **aware-tool-only: \(\beta=+0.001, p=0.889\)** [2605.19099].

The second central finding is that **routing fidelity-at-1 changes substantially even when quality remains nearly flat**. The reported fidelity@1 values are **14.2%** for blind, **7.5%** for aware-c1, **20.8%** for aware-c2, **15.5%** for aware-c3, and **29.5%** for aware-tool-only. The range therefore spans **7.5% to 29.5%** at near-equal mean quality. The paper interprets this as evidence that **delivery channel dominates description content**: on-demand access via `read_profile` performs best, and preloaded peer cards recover less than half of the fidelity gain [2605.19099].

The third benchmark-level result is the **counterfactual-delegation ceiling**, which places perfect single-step delegation **15–31 percentage points above measured performance** on every suite. The reported figures are **GAIA: blind actual 0.407, ceiling 0.675, gap +0.269**; **tau-bench: blind actual 0.695, ceiling 0.848, gap +0.153**; and **BFCL: blind actual 0.536, ceiling 0.849, gap +0.313**. This locates large unrealized headroom for future orchestration methods [2605.19099].

Several secondary findings refine this picture. Awareness conditions can shift systems into a different cost band without improving quality much. Reported mean latencies include **GAIA blind: 105s** versus **GAIA aware-tool-only: 79s**, and **BFCL blind: 69s** versus **BFCL aware-tool-only: 64s**. Delegation rate is highly uneven across suites: **GAIA** invites delegation much more often, **BFCL** is intermediate, and **tau-bench** exhibits near-zero delegation overall. The paper also reports **vendor self-preference** above chance for some models, including **Gemini-3-Flash: 1.48×**, **DeepSeek-V4-Flash: 1.53×**, **DeepSeek-V4-Pro: 1.85×**, and **GPT-5.5: 3.65×** [2605.19099].

Taken together, these results imply that a benchmark focused only on final quality would miss the main orchestration signal. A plausible implication is that future evaluation of agent systems will increasingly need separate axes for **task execution** and **work allocation**.

## 6. Position within the broader decision-benchmark landscape

DecisionBench belongs to a broader movement in benchmark design that measures **decision quality under structure, uncertainty, or operational constraints**, rather than only answer correctness. Within that landscape, its distinctive focus is the orchestration problem: when multiple peer models are available, can the system manage delegation well [2605.19099]?

Other recent benchmarks target different decision layers. **ClinDet-Bench** formalizes **judgment determinability** under incomplete clinical information and distinguishes determinable from undeterminable cases rather than treating all incomplete cases as unanswerable [2602.22771]. **AR-BENCH** evaluates **appellate review** as post-judgment error detection, classification, and correction, framing legal review as anomaly detection rather than prediction [2601.22742]. **ScenarioBench** makes compliance decisions **trace-grounded** by requiring clause-level evidence from the system’s own retrieval, thereby scoring not only what decision was made but why [2509.24212]. **CONDESION-BENCH** shifts decision-making from atomic labels to **compositional actions** under explicit feasibility conditions, with oracle-based scoring of both utility and condition adherence [2604.09029]. **JudgmentBench** compares **rubric-based scoring** and **comparative judgment** for high-expertise legal evaluation, asking which elicitation protocol better recovers intended quality orderings when no objective gold answer exists [2605.25240]. **Deployment-complete benchmarking** goes further by asking whether benchmark evidence actually determines the deployment action, introducing mixed fibers, certifiable fraction, and completion curves as diagnostics of decision readiness [2605.25997].

Viewed against these resources, DecisionBench is not a general benchmark of decision-making in the abstract. It is specifically a benchmark of **delegation policy in agentic systems**. Its core contribution is to make routing behavior observable and comparable under fixed tasks, fixed peers, and fixed evaluation machinery. This suggests a division of labor among contemporary benchmarks: some test whether a model can decide correctly under uncertainty, some test whether it can justify or audit a decision, and DecisionBench tests whether it can **assign work across peers** in a long-horizon workflow.

## 7. Artifacts, extensibility, and research significance

The release includes the **DecisionBench substrate**, the **annotation layer**, the **reference intervention suite**, the **analysis pipeline**, and **220 per-condition run archives** containing roughly **23,000 task-level traces**. The reference intervention suite includes **33 profile cards in three variants** and an **aware-tool-only ablation harness**. The annotation layer includes **tagger.py v2.0**, the frozen seven-skill taxonomy, and an **emergent-audit CSV** [2605.19099].

This release structure matters because the benchmark is intended to be **extensible**. The paper explicitly states that the substrate is agnostic to how peer information is generated or delivered, so **learned routers**, **richer peer memories**, **adaptive profile construction**, and **multi-step delegation** can all be evaluated against it. The benchmark therefore functions as a reusable measurement framework rather than as a one-time ablation.

Its broader significance lies in the claim that future agent systems will be evaluated not only on how well they answer or act, but also on how well they **manage work across heterogeneous peers**. DecisionBench operationalizes that claim by separating orchestration from outcome. The benchmark’s empirical characterization shows that orchestration effects can be large even when ordinary benchmark scores are nearly unchanged, and that current systems remain far below a counterfactual ceiling. In that sense, DecisionBench establishes delegation as a first-class benchmark target rather than a hidden systems variable [2605.19099].

Source: https://www.emergentmind.com/topics/decisionbench