DecisionBench: A Benchmark for Emergent Delegation in Long-Horizon Agentic Workflows

Abstract: We introduce DecisionBench, a benchmark substrate for emergent delegation in long-horizon agentic workflows. The substrate fixes a task suite (GAIA, tau-bench, BFCL multi-turn), a peer-model pool (11 models, 7 vendor families), a delegation interface (call_model plus an optional read_profile channel), a deterministic skill-annotation layer, and a multi-axis metric suite covering quality, cost, latency, delegation rate, routing fidelity-at-k, vendor self-preference, and a counterfactual-delegation ceiling. The substrate is agnostic to how peer information is generated or delivered, so learned routers, richer peer memories, adaptive profile construction, and multi-step delegation can all be evaluated against it. We characterize the substrate with a five-condition reference sweep on the full pool (n=23,375 task instances). Three benchmark-level findings emerge: (i) mean end-task quality is statistically indistinguishable across the four awareness conditions (|beta| <= 0.010, p >= 0.21), so quality-only evaluation would miss the orchestration signal; (ii) routing fidelity-at-1 ranges from 7.5% to 29.5% across conditions at near-equal mean quality, with delivery channel (on-demand tool vs. preloaded description) dominating description content; (iii) a counterfactual ceiling places perfect delegation 15-31 percentage points above measured performance on every suite, locating large unrealized headroom for future orchestration methods. We release the substrate, annotation layer, reference intervention suite, analysis pipeline, and 220 per-condition run archives.

• The paper presents DecisionBench, a benchmark that formalizes emergent delegation in long-horizon workflows through a parameterized substrate combining fixed tasks, a diverse peer-model pool, and explicit delegation interfaces.
• It employs a deterministic annotation layer and comprehensive process-level metrics to assess delegation fidelity, routing precision, and unrealized performance headroom across varied agent capabilities.
• Empirical results show that on-demand tool access significantly improves delegation fidelity over blind or preloaded methods, underscoring the need for refined orchestration approaches.

DecisionBench: A Formal Substrate for Agentic Delegation Evaluation

Benchmark Design and Methodology

DecisionBench introduces a parameterized substrate for evaluating emergent delegation within long-horizon agentic workflows (2605.19099). The benchmark fixes a task suite, a peer-model pool, delegation interfaces, annotation mechanisms, and process-centric metric suites. The task suite comprises representative agentic tasks from GAIA (general tool-use), $\tau$-bench (tool-agent-user dialogue), and BFCL (multi-turn function-calling), stratified via deterministic splits to facilitate profile-building for peer-awareness. The peer-model pool spans 11 LLMs across seven vendor families and three capability tiers, routed via OpenRouter.

Delegation is operationalized as a tool interface: call_model allows direct subtask delegation, while read_profile supplies structured peer descriptions. The substrate is agnostic to peer-information delivery and generation, supporting evaluation of learned routers, profile adaptations, and multi-step delegation—all anchored to process-level metrics rather than solely outcome accuracy.

DecisionBench’s annotation layer freezes a seven-skill taxonomy. A deterministic trace-only tagger assigns skill labels to agent trajectory steps—eliminating the need for LLM judgment and ensuring reproducibility. Metrics include task outcome quality, cost, latency, delegation rate, routing fidelity-at-$k$, vendor self-preference, and a counterfactual-delegation ceiling.

Figure 1: The core DecisionBench substrate, including task suite, peer pool, interface, annotation layer, and the metric suite.

Reference Awareness Interventions

The DecisionBench empirical characterization spans five reference conditions—one blind (no peer skill information) and four aware interventions. The aware variants instantiate profile-card information delivery along content (C1: rubric anchor, C2: deterministic statistics, C3: dual LLM-judge summary) and delivery axes (preloaded vs. tool-on-demand). The aware-tool-only variant isolates delivery channel effects by enabling card retrieval solely via tool without system prompt preloading.

This design enables principled isolation of orchestration, distinguishing outcomes driven by tool accessibility versus priming via preloaded descriptions.

Empirical Results: Contradictory Process-Outcomes Dynamics

Across a sweep of $n=23,375$ task instances (11 agents × 3 benchmarks × 5 conditions), DecisionBench reveals several contradicted intuitions:

• End-task quality is invariant across awareness conditions ($|\beta| \leq 0.010$, $p \geq 0.21$). Outcome-only evaluation would miss the orchestration signal—a claim robust across all benchmarks.
• Delegation fidelity is strongly delivery-channel dependent: on-demand tool access more than doubles routing precision-at-$1$ over blind (14.2% $\to$ 29.5%) at quality parity and lower mean cost; preloaded variants yield lower gains (C2 20.8%, C3 15.5%) with C1 performing worse than blind.
• A large counterfactual-delegation ceiling exists: perfect single-step delegation would improve performance by 15–31 percentage points across suites, quantifying unrealized headroom for orchestration algorithms.

  Figure 2: Quality–cost Pareto frontiers show all awareness conditions cluster tightly on aggregate metrics, but per-agent effects enable Pareto-favorable shifts.

  

  Figure 3: Per-agent Pareto dominance of aware variants over blind, highlighting BFCL’s aware-tool-only contributions.

  

  Figure 4: Per-agent decomposition—tool-availability drives quality lift, system-prompt augmentation is often negative; delegation fidelity is maximized via tool-channel.

Heterogeneity, Delivery Channel, and Skill-Specific Effects

Aggregate measures mask strong heterogeneity:

• Mid-capability agents benefit most from peer-awareness; frontier agents saturate, and weak agents lack discrimination for effective peer utilization.
• On-demand tool access is the dominant driver for delegation fidelity and any quality improvement; preloaded descriptions are consistently less effective and sometimes regressive.

Per-agent and per-skill breakdowns further clarify:

Figure 5: Parabolic fit illustrates mid-capability agents extract maximal gain from peer-aware orchestration.

Figure 6: Per-agent GAIA quality lift—aware-tool-only is the most effective aware variant, but right-tail regression occurs in some frontier agents.

Figure 7: Skill-level analysis—information retrieval and multi-step reasoning tasks derive maximal benefit from agentic peer-awareness.

Vendor Self-Preference Bias and Delegation Dynamics

The benchmark reveals explicit vendor self-preference: orchestrators from several families (Gemini-3-Flash, DeepSeek-V4-Flash, GPT-5.5) over-delegate to same-vendor models at rates up to 3.65× above chance. Anthropic models are neutral in self-preference; vendor columns from Anthropic and Google absorb cross-vendor delegations disproportionally, implying perceived peer strength independent of actual performance.

Figure 8: Vendor-by-vendor heatmap—cross-vendor delegation flows and self-preference ratios surface orchestration biases.

Upper Bound: Unrealized Delegation Headroom

By constructing a counterfactual—delegating each task to the Stage-1-best peer for its dominant skill—DecisionBench establishes substantial unrealized potential:

• GAIA: blind 0.407, ceiling 0.675, gap +0.269
• $\tau$-bench: blind 0.695, ceiling 0.848, gap +0.153
• BFCL: blind 0.536, ceiling 0.849, gap +0.313

This gap is largest for weak and mid-tier agents, with MiniMax-M2.5 on GAIA ($0.17 \to 0.80$, +63pp possible); frontier cells are near-saturated.

Figure 9: Counterfactual-delegation ceiling quantifies maximal achievable quality, largest for mid/weak agents.

Implications and Future Directions

DecisionBench formalizes evaluation of emergent delegation mechanics with rigor unattainable via outcome-only agentic benchmarks. The dissociation between process metrics (delegation rate/fidelity) and outcomes (task quality/cost) exposes orchestration levers invisible to traditional agent evaluations.

Bold claim: Delivery channel dominates description content—on-demand peer skill information is vastly more effective for routing precision than system-prompt preloading. This insight directly informs practical orchestrator implementations: design should prioritize explicit tool-driven skill discovery, not priming via context window descriptions.

The counterfactual ceiling quantifies a measurable orchestration channel, substantiating the methodological necessity for process-level instrumentation. The DecisionBench substrate—frozen tasks, peer pool, interface, annotation, and metrics—admits head-to-head evaluation of learned routers, adaptive profile construction, and richer peer pool composition.

On the theoretical axis, DecisionBench reveals the limits of current agentic routing and the suite-specificity of orchestration ability. The prevalence of vendor self-preference bias warrants deeper exploration regarding both agent design and benchmarking fidelity.

Conclusion

DecisionBench defines a stable, reproducible substrate for benchmarking emergent delegation in agentic workflows. It decouples dependency on description content from delivery channel, quantifying orchestration mechanisms missed by outcome-only assessment. The dominant empirical finding—on-demand tool access vastly improves delegation fidelity with quality parity—substantiates practical orchestration recommendations. Benchmarking orchestration algorithms, profile adaptors, and heterogeneous agent pools is now systematically feasible, with substantial unrealized headroom quantified for future algorithmic advances. DecisionBench will be central to next-phase orchestration research, providing robust process-level metrics and datasets for reproducible and interpretable agentic delegation evaluation.