Reward Modeling for Multi-Agent Orchestration

Abstract: Multi-Agent Systems (MAS) built on LLMs require effective orchestration to coordinate specialized agents, yet training such orchestrators is hindered by limited supervision and high computational cost. We propose Orchestration Reward Modeling (OrchRM), a self-supervised framework for evaluating orchestration quality without human annotations. OrchRM leverages intermediate artifacts from multi-agent executions to construct win-lose pairs for Bradley-Terry reward model training. Unlike existing MAS test-time scaling and orchestrator training frameworks that rely on costly sub-agent rollouts, OrchRM operates directly at the orchestration level, enabling efficient and high-performing reward-guided orchestrator training and MAS test-time scaling. OrchRM improves training efficiency by up to 10x in token usage while improving MAS test-time scaling performance by up to 8% in accuracy. These gains consistently transfer across multiple domains, including mathematical reasoning, web-based question answering, and multi-hop reasoning, demonstrating orchestration-level reward modeling as a scalable direction for robust multi-agent orchestration. Code will be available at https://github.com/Wang-ML-Lab/OrchRM.

• Orch-RM trains a reward model on self-supervised comparisons from an MAS orchestrator's logs, thereby allowing judging the orchestration quality long before any sub-agent executes and yielding significant computational savings.
• Orch-RM achieves an approximately 17x reduction in verification token cost and demonstrates a stronger accuracy-efficiency trade-off on most benchmarks compared to traditional methods, for example, 68.33% accuracy on AIME 24&25 with 2.38M verification tokens versus 142.80M at 66.33% using a baseline model.
• While Orch-RM is effective in test-time scaling and orchestrator training for multi-agent orchestration, its performance is impacted in scenarios with low orchestration diversity. Its domain-specific training also leaves multi-domain generalization an unresolved area for future research.

Orch-RM is a self-supervised reward modeling framework for LLM-based Multi-Agent Systems (MAS) that learns to score orchestration quality directly, without human annotations or full sub-agent rollouts. The framework addresses two persistent bottlenecks in automated multi-agent orchestration: the scarcity of supervision for judging whether a generated multi-agent workflow is good, and the prohibitive computational cost of obtaining feedback through complete trajectory execution. By training a Bradley-Terry reward model on preference pairs mined from intermediate artifacts of an existing orchestrator's training logs, the authors show that orchestration-level reward signals can improve both test-time scaling and orchestrator training at a fraction of the token cost of trajectory-level alternatives.

Motivation and problem setting

The work builds on MAS-Orchestra [ke2026masorchestraunderstandingimprovingmultiagent], which formulates agent coordination as holistic function-calling reinforcement learning over modular sub-agents. While effective, its training relies on sparse final-answer rewards, requiring full multi-agent rollouts per update; the original GRPO training consumed over 1B tokens for only 100 orchestrator update steps. Prior analysis (MAS-ProVe) further established that scalar and process reward models trained on single-agent paths fail to reliably evaluate out-of-distribution conversational dynamics in MAS.

The key observation is that a MAS trajectory decomposes into an orchestration plan and its execution. Orchestration quality determines performance, but supervising it conventionally requires executing the full workflow. Orch-RM instead trains a reward model $r_\phi(q, o)$ that scores an orchestration $o$ given query $q$ before any sub-agent inference occurs, enabling early intervention at both inference and training time.

Method

Preference data is constructed from two sources, both derived from accessible artifacts of MAS-Orchestra training:

• Specialized-over-base: trajectories from the trained orchestrator policy are preferred over those from the backbone model, exploiting task-specific optimization.
• Correct-over-incorrect: sampled trajectories are labeled by final-answer correctness, requiring no additional annotation.

These are mixed at a fixed ratio (yielding 12 specialized-over-base and 4 correct-over-incorrect pairs per query) and used to fine-tune Skywork-Reward-LLaMA-3.1-8B with the standard Bradley-Terry logistic loss:

$$\mathcal{L}_{\text{Orch-RM}}(\phi) = -\mathbb{E}\left[\log \sigma\big(r_\phi(q, o_w) - r_\phi(q, o_l)\big)\right].$$

At inference, $N$ orchestrations are sampled in parallel and the highest-scoring one is selected (best-of-$N$) before any sub-agent execution, avoiding redundant rollouts that dominate typical MAS test-time scaling. For training, the reward model replaces task-level correctness as the GRPO advantage signal, computed via group-normalized advantages over sampled orchestrations—again without executing sub-agents.

Test-time scaling results

Using Qwen2.5-7B-Instruct as the orchestration policy and GPT-OSS-120B as the sub-agent backbone, with best-of-$N$ ($N{=}8$), Orch-RM achieves the strongest accuracy-efficiency trade-off among orchestration-level verifiers:

Benchmark	Majority vote	Orch-RM	Verification tokens
AIME 24&25	63.33%	68.33%	2.38M
BrowseComp+	9.50%	14.00%	8.26M
HotpotQA	42.50%	42.50%	7.32M
GPQA (OOD)	63.63%	66.16%	7.49M

Notably, on BrowseComp+ Orch-RM surpasses the strongest trajectory-level LLM judge (GPT-5-mini at 12.50%) while reducing verification cost from 142.80M to 8.26M tokens—a roughly 17× reduction. Ablations show the mixed data configuration (specialized-over-other-domains plus correct-over-incorrect at 1:3) is essential; single-source variants underperform, confirming that combining domain-level and correctness-based comparisons yields the most discriminative supervision.

Two caveats temper these results. Gains are small on HotpotQA, where all methods cluster between 40.5–46% and normalized advantages turn slightly negative—the authors attribute this to limited orchestration diversity, meaning many trajectories share similar plans and cannot be distinguished at the orchestration level. On GPQA, off-the-shelf log-probability confidence actually outperforms Orch-RM (71.21% vs. 66.16%), indicating the learned signal does not uniformly dominate simpler baselines out of distribution.

Orchestrator training results

For continued training of the pretrained MAS-Orchestra checkpoint, Orch-RM reaches 68.33% majority-vote accuracy on AIME 24&25 and 11.00% on BrowseComp+, and is reported as the only method that consistently improves performance across benchmarks in this setting. On efficiency, it attains comparable or better accuracy than trajectory-level RFT and GRPO using up to ~10× fewer tokens on AIME and ~46× fewer on BrowseComp+. For training from scratch, Orch-RM raises majority-vote accuracy from 23.33% to 61.67% on AIME and from 0.50% to 4.50% on BrowseComp+, approaching—but not matching—the trajectory-level RL-trained teacher (63.33% and 9.50%). This gap indicates that orchestration-level supervision alone recovers most, though not all, of the benefit of expensive rollout-based RL; the residual difference is a concrete open question about what information final-trajectory rewards carry that orchestration-level scores do not.

Limitations

The authors concede two principal limitations. First, although human annotation is avoided, performance remains bounded by the diversity and quality of available orchestrator checkpoints and sampled trajectories; scaling orchestration datasets is left unresolved. Second, reward models are trained domain-specifically, so cross-domain generalization is untested—the GPQA result above hints that transfer may be uneven. Additionally, the method presupposes access to a strong pre-trained orchestrator's training artifacts, which constrains applicability when such checkpoints are unavailable.

Conclusion

Orch-RM demonstrates that a standalone reward model trained on self-supervised orchestration preferences can serve as an effective, cheap evaluative signal for both MAS test-time selection and orchestrator optimization, delivering up to 8% accuracy gains while cutting verification and training token costs by an order of magnitude or more. Its effectiveness degrades where orchestration diversity is low, and its domain-specific training leaves multi-domain generalization as the central open question raised by the work.