To view this video please enable JavaScript, and consider upgrading to a web browser that supports HTML5 video.

Create a Video View Paper

Unlocking Multiagent Scaling with Process Rewards

This presentation explores MAPPA, a novel reinforcement learning framework for multiagent LLM systems that replaces sparse rewards with dense, per-action process rewards. By utilizing an AI coach to provide feedback on every step of a long-horizon task, the authors solve critical credit assignment problems and enable agent specialization in complex tool-augmented environments.
Script
How do we teach a choir of independent AI agents to harmonize when they only receive a single round of applause at the very end of a performance? This paper introduces MAPPA, a method for scaling multiagent systems using fine-grained process rewards.
Building on the challenge of long-horizon tasks, the authors identify that sparse outcome rewards make it nearly impossible to pinpoint which specific agent failed in a complex pipeline. This leads to massive sample inefficiency and high costs during training.
To solve these issues, the researchers propose a framework called Multiagent finetuning with Per-action Process rewards from AI feedback.
Instead of waiting for the final result, MAPPA inserts a strong AI coach into the loop to score every single action on a scale of 0 to 10. This allows the system to extract learning signals even from failed runs by evaluating the intermediate reasoning and tool usage.
Moving to the optimization strategy, the authors chose REINFORCE++ over GRPO because multiagent pipelines create highly diverse states that break standard group-relative assumptions. This approach ensures stable learning by normalizing advantages across the entire global batch of agent experiences.
In their DSBench experiment, the researchers tested a three-stage pipeline where a Data Engineer, Modeler, and Analyst collaborated in a shared sandbox to solve real-world Kaggle tasks. Each agent must manage file dependencies, and the coach uses ground-truth metrics to anchor its process rewards.
The empirical results are striking; they saw double-digit improvements in math accuracy and a significant reduction in error rates for data science tasks. These gains demonstrate that fine-grained supervision effectively drives agent specialization.
Despite the success, the authors warn about coach bias, noting that their models prioritized regression tasks over classification due to higher relative scores from the AI judge. Future work must address this potential for reward hacking and systematic evaluation bias.
The implications of this work point toward a future where we move beyond scalar rewards to richer, language-based corrections for massive agent networks. This shift could finally allow us to scale multiagent systems to tackle the most complex, long-duration human endeavors.
MAPPA proves that granular feedback is the key to mastering the complexity of multiagent systems in long-horizon tasks. To dive deeper into this research, visit EmergentMind.com.