Multi-Agent Routing as Set-Valued Prediction: A WildChat Benchmark and Cost-Aware Evaluation

Abstract: Tool and agent routing from natural-language prompts is naturally a set-valued prediction problem: a single query may require multiple agents, while over-selection increases execution cost. The benchmark introduced here is derived from WildChat and contains 3,000 prompts over a fixed 12-agent catalog, with AI-assisted heuristic labels under a fixed schema and controlled rebalancing for multi-label evaluation. The evaluation protocol combines set-level metrics (Precision, Recall, F1, Jaccard, and Exact Match), latency, an execution-oriented capability-coverage simulation, and a constrained weighted-routing setting based on ordinal agent-cost tiers. Compared methods include nearest-neighbor matching, linear multilabel classification, dependency-aware baselines, a fine-tuned encoder, deterministic weighted post-scoring via Weighted Agent Routing (WAR), and a zero-shot LLM baseline. Results show that supervised routers substantially outperform nearest-neighbor and zero-shot LLM routing. The fine-tuned encoder achieves the strongest unconstrained set accuracy, while the linear multilabel model provides the strongest practical baseline. In the constrained setting, the weighted routing layer improves utility when applied on top of strong supervised scorers, with the largest gain observed for Encoder+WAR. Overall, the benchmark and evaluation protocol support reproducible study of accuracy-cost trade-offs in fixed-catalog multi-agent routing.

• The paper introduces set-valued agent routing to map natural language queries to multi-agent selections, demonstrating its necessity for effective tool orchestration.
• It employs a benchmark-driven evaluation with a balanced label distribution and compares multiple algorithms, with a fine-tuned encoder achieving top performance.
• The study integrates a deterministic cost-aware layer (WAR) to optimize the trade-offs between operational costs and agent utility in realistic multi-agent systems.

Multi-Agent Routing as Set-Valued Prediction: Methods, Benchmarking, and Cost-Aware Evaluation

Introduction and Motivation

The deployment of multi-agent and tool-augmented LLM systems necessitates effective mapping from free-form natural language queries to fixed catalogs of specialized agents. Unlike typical top-1 intent classification or retrieval, many real-user prompts demand coordinated capability coverage across multiple agents. This paper systematically formulates agent routing as a set-valued prediction problem, addressing both the semantic adequacy of chosen agent sets and the practical constraints imposed by cost and execution.

To support empirical study of set-valued agent routing, the paper introduces a WildChat-derived benchmark, enforces set-size and agent-coverage balancing, and constructs AI-assisted, heuristic reference label sets. This enables meaningful comparative evaluation of routing algorithms, including both unconstrained set accuracy and cost-aware trade-offs.

Benchmark Construction and Dataset Properties

The benchmark is derived from 3,000 linguistic-diverse prompts over a fixed catalog of 12 agents, selected and labeled to capture realistic multi-agent workflows such as retrieval plus analysis or multi-stage reporting. Labeling is performed under a protocol-driven schema, with explicit set sizes (60% single-agent, 30% dual-agent, 10% triple-agent), ensuring stable evaluation and mitigating label skew.

The dataset enforces nearly uniform per-agent coverage and demonstrates high prompt-label consistency: pairwise text similarities are strongly correlated with label-set overlaps (mean rank-1 Jaccard 0.601 vs. 0.091 for random pairs), establishing the learnability and internal coherence of the routing protocol.

Figure 1: Gold set-size distribution in the benchmark dataset, showing the enforced 1-/2-/3-agent split.

Figure 2: Per-agent appearance rate indicating balanced label coverage across all 12 agents.

Routing Methods and Evaluation Protocol

The routing architecture is modular (Figure 3) and decouples feature construction, scoring, and selection. All methods share normalized sentence-transformer (MPNet) prompt and agent profile embeddings. Evaluated algorithms include:

• KNN retrieval: Non-parametric semantic matching using cosine similarity on embeddings.
• Linear one-vs-rest SVM (ML): Task-specific multilabel classifier.
• Classifier Chains / ML-kNN: Dependency-aware approaches modeling label correlations.
• Fine-tuned encoder: Supervised MPNet backbone further trained for prompt-agent multi-labeling.
• Zero-shot LLM: GPT-4o with catalog-constrained, JSON-format output.
• WAR post-scoring: Deterministic layer manually adjusting base scores for agent cost tiers, enabling accuracy--cost trade-off under operational constraints.

The pipeline outputs a prompt-specific set of agents, either unconstrained (by fixed threshold) or with costs reweighted by WAR (Figure 4).

Figure 4: Overview of routing system components and decision flow for set prediction and evaluation metrics.

Main Results: Set Accuracy and Practical Utility

Under the unconstrained protocol (threshold $t=0.60$), the fine-tuned encoder is dominant by all set-based metrics: F1 89.59, Jaccard 85.52, Exact Match 73.11. The linear ML classifier is optimal among practical, lightweight baselines (F1 71.79), outperforming both simple KNN matching (F1 42.56) and dependency-aware methods. Zero-shot LLM routing lags substantially (F1 41.51), demonstrating that generative capabilities are not sufficient for precise, cost-regularized, catalog routing.

Threshold sweeps (Figure 5) reveal classical precision-recall tradeoffs; both ML and encoder peak at $t=0.60$, with precision dropping at lower thresholds (over-selection) and recall falling at higher thresholds (under-selection).

Figure 5: Threshold sweep on dev showing the F1 surface and optimal operating points for ML and Encoder.

Precision-recall interplay across methods is visualized in Figure 6, further indicating the clear separation between the encoder and all alternatives.

Figure 6: Precision-recall scatter under the unconstrained set evaluation protocol for all routing baselines.

The distribution of predicted set size is controlled (Figure 7), ensuring that multi-agent dispatches match realistic workflow requirements rather than simply maximizing set overlap.

Figure 7: Average predicted set size across evaluated routing methods at the shared operating point.

Execution-Oriented Simulation and Cost-Aware Selection

The practical relevance of routing is assessed via execution-oriented simulation, modeling capability coverage, median cost (via ordinal agent tiers), and utility (coverage penalized by cost and extra-agent selection). The fine-tuned encoder achieves success 74.33%, coverage 86.07%, and utility 0.654, with minimal cost overhead and highly selective dispatch. ML provides robust utility at much lower computational expense but with decreased precision.

The introduction of Weighted Agent Routing (WAR) as a deterministic constrained-selection layer enables explicit calibration of utility under cost constraints (Figure 8). WAR confers the largest gain when applied on top of the already performant encoder, boosting success from 74.33% to 92.11%, coverage from 86.07% to 96.39%, and utility from 0.654 to 0.670, albeit with higher cost and a small uptick in extra agents.

Figure 8: Trade-off between utility and cost penalty for WAR-modified ML and Encoder over dev, with operating point markers.

The empirical distinction is clear: for strong scorers, cost-aware adjustment mostly improves capability coverage without over-selecting, whereas for weaker scorers, WAR primarily trims unnecessary cost with only modest gain in coverage.

Implications and Future Directions

This study demonstrates that set-valued agent routing—rather than top-1 selection or unconstrained ranking—is essential for effective multi-agent orchestration. Benchmarks built around catalog balancing and protocol-driven reference sets enable reproducible, statistically stable evaluation of routing strategies. The findings indicate:

• Set-valued supervised models with capacity for semantic fine-tuning consistently outperform both non-parametric and dependency-only baselines.
• Zero-shot LLM routing yields inferior results for fixed-catalog, cost-sensitive deployments, despite advances in LLM tool use and generalization.
• Simple, deterministic, post-hoc cost-aware weighting is sufficient for identifying robust operating points, provided base scores are well calibrated.
• The most significant routing bottleneck is semantic representation fidelity, not label-dependency modeling.

Theoretically, this reframing encourages research into learnable, utility-aware selection layers and dynamic cost modeling instead of static protocol engineering. Practically, the separation of set scoring and constrained selection aligns well with deployment needs, supporting retrainability and operational adaptation as agent catalogs evolve. Future work should address large-scale catalogs, real-time cost latency, dynamic execution feedback, and user-grounded task success.

Conclusion

The results strongly support the formulation of agent routing as a set-valued, cost-aware prediction problem. Controlled benchmarks informed by protocol-consistent labeling enable rigorous empirical comparison. High-capacity, fine-tuned semantic encoders, optionally augmented with deterministic cost-aware layers, define the practical accuracy and utility envelope for multi-agent prompt routing. Broader progress will be driven by both improved semantic modeling and richer, context-dependent cost tradeoff mechanisms.

(2606.28925)