Reinforced Collaboration in Multi-Agent Flow Networks

Abstract: Multi-agent systems provide a powerful way to extend LLMs by decomposing a complex task into specialized subtasks handled by different agents. However, their performance is often hindered by error propagation, arising from suboptimal workflow design or inaccurate agent outputs, which can propagate through the agent collaboration process and degrade final results. To address the challenges, we present MANGO (Multi-Agent Network Gradient Optimization), a data-driven framework that organizes and refines agent collaboration via a flow network constructed from past successful workflows. MANGO integrates reinforcement learning and textual gradients to jointly optimize workflow paths and agent behaviors, while a skipping mechanism prevents redundant updates to well-optimized agents for improving efficiency. Extensive experiments on seven benchmarks show that MANGO achieves up to 12.8% performance improvement over state-of-the-art baselines, enhances efficiency by 47.4%, and generalizes effectively to unseen domains. Our code and datasets are publicly available at https://github.com/openJiuwen-ai/agent-store/tree/main/community/mango.

• The paper demonstrates that reinforced collaboration using a flow network reduces error propagation in multi-agent LLM systems.
• It introduces the MANGO framework, which jointly optimizes workflow path selection with RL and prompt refinement with textual gradients.
• Experimental results show up to 12.8% accuracy gains and significant reductions in training and inference times over baseline methods.

Reinforced Collaboration in Multi-Agent Flow Networks: An Expert Analysis

Motivation and Problem Setting

Multi-agent systems (MAS) promise the modularization of complex tasks by delegating subtasks to specialized agents, often leveraging LLMs as the underlying computation and reasoning engines. While such systems enhance flexibility and promote division of labor, they suffer acutely from error propagation—stemming from both workflow generation errors (suboptimal decomposition or misassigned subtasks) and single-agent execution errors (hallucination, miscomputation, or factual error within an agent). These errors can propagate throughout the agent collaboration chain, ultimately corrupting the final output.

Figure 1: An illustration of error propagation in multi-agent collaboration; a workflow misplanning triggers downstream execution errors.

Earlier work has explored both rule- and heuristic-driven agent orchestration, employing approaches such as cyclic dialogue, tree search, evolutionary recombination, or meta-agent search. However, these are empirically brittle and require manual heuristics for adaptation across domains. Moreover, methods relying solely on gradient signals at the final output suffer from weak credit assignment, analogous to the classic vanishing gradient problem in deep learning, which impedes efficient agent prompt optimization in long reasoning chains.

The MANGO Framework: Flow Network-Based Optimization

Flow Network Construction

The proposed MANGO (Multi-Agent Network Gradient Optimization) framework constructs a flow network from clusters of agent operations found in successful historical workflows. Each node represents a cluster of similar operations associated with a particular LLM and prompt configuration, and each edge describes a valid transition between operations (i.e., subtask dependencies or execution orderings). Learning is data-driven: the flow network is built incrementally, clustering operations through embedding similarity with a configurable threshold, thus continuously extending its capacity as new workflows are discovered or refined.

Formulating Collaboration as Joint RL and Prompt Optimization

MANGO centralizes agent collaboration as a joint optimization over workflow path selection and prompt parameterization:

• Policy Network for Path Selection: Workflow generation is cast as a discrete path-finding problem within the flow network, optimized via a Markov Decision Process (MDP) using policy gradients (REINFORCE). States comprise role/experience embeddings and their similarity to plan steps and task description, while actions correspond to choosing the next agent node. A process-oriented reward term (agreement with ground-truth workflow) is linearly combined with a result-oriented term (task success).
• Textual Gradient Optimization of Prompts: Each node/agent's prompt is refined via textual gradients, which leverage both global (output-based) and local (intermediate step-based) feedback as optimization signals. This addresses the gradient vanishing problem by distributing supervision more evenly along the workflow path.

Skipping Mechanism for Efficiency

Significant computational cost arises from repeated prompt updates for already well-optimized agents. MANGO introduces a skipping mechanism: at each path selection point, the policy can choose to skip up to $K$ nodes, provided the resultant path remains valid. Skipped nodes are supervised with ground-truth traces where available, dramatically reducing redundant LLM invocations during both training and inference while maintaining performance.

Figure 2: MANGO framework overview—joint RL and prompt/textual gradient training over a flow network of historical workflows, with an explicit skip mechanism for computational efficiency.

Experimental Results and Analysis

Benchmark Performance and Claims

On seven standard reasoning and code benchmarks (HumanEval, MBPP, MATH, GSM8K, DROP, MMLU, GPQA), MANGO achieves strong numerical improvements:

• Up to 12.8% absolute accuracy improvement over the strongest multi-agent baseline (MaAS) on MATH.
• On average, MANGO demonstrates superior or state-of-the-art results across all evaluated benchmarks.
• Training and inference times are reduced by 41.5% and 47.4%, respectively, compared to the best prior systems (with Skip-3).
• Robust generalization: The framework proves robust under cross-domain and cross-backbone transfer, outperforming all baselines even when the flow network is constructed from noisy or weak initial workflows.

Robustness and Ablations

• MANGO retains performance even with high noise rates (up to 50% failed workflows) in network construction, exhibiting less than a 10% drop in accuracy.
• Ablation studies isolate the contributions of: (i) reinforcement learning (outperforming heuristic and LLM-based selection by 3-7%); (ii) textual gradients (adding up to 19.3% gain, with local signals conferring another 6.8% on top of global only supervision).
• Parameter sweeps over clustering threshold $\delta$, reward balance $\alpha$, and skip distance $K$ reveal robust and monotonic trade-offs—higher $K$ reduces computational cost but slightly degrades accuracy.

Figure 3: Flow network robustness—performance degrades gracefully with increased workflow noise or smaller planner models.

Error Case Studies and Practical Interpretability

Case studies demonstrate MANGO's ability to (i) correct misplanned workflows by leveraging recombination and RL over the flow network and (ii) repair agent execution errors via local prompt optimization. The learned agent network is interpretable, and nodes/edges can be inspected for diagnosis or manual augmentation.

Theoretical and Practical Implications

The explicit modeling of past successful workflows as a flow network enables both sample-efficient learning and modular generalization. Policy-based routing ensures combinatorial exploration and reuse of solution fragments, facilitating transfer to new domains and backbones. The use of textual gradients at both global and local levels simultaneously strengthens credit assignment and sample efficiency, and the skip mechanism points toward scalable, cost-sensitive deployment in applied agentic scenarios.

Practically, MANGO can underpin systems requiring robust, adaptive, and self-improving collaboration among LLM agents in environments such as code generation, mathematical reasoning, reading comprehension, and interactive web tasks. The framework's data-driven construction and learning paradigm reduce the need for expert heuristic engineering, fostering rapid adaptability.

Limitations and Future Directions

• The reliance on high-quality historical workflows for initialization may introduce bias if those samples are suboptimal; expanding the discovery and aggregation of successful workflows will be critical.
• Hyperparameters (clustering threshold, skip distance, reward balancing) may require dataset-specific tuning, though defaults generalize reasonably well in current experiments.
• Integrating continual learning and online workflow extension, especially in open-ended interactive environments, is a natural next step.
• Cross-modal and tool-using agent extensions, as well as the adoption of even more granular and structured feedback signals for prompt optimization, remain promising avenues.
• Scaling the flow network and policy training to thousands or millions of agents/operations will demand further engineering innovation.

Conclusion

MANGO presents a unified, data-driven framework for optimizing multi-agent LLM collaboration using a flow network built from historical workflows, jointly trained with reinforcement learning and distributed textual gradient supervision. It attains strong empirical performance, high efficiency, interpretability, and robust generalization, standing as a compelling template for future multi-agent AI architectures in both research and applied settings (2605.12943).