Reinforcement Learning for LLM-based Multi-Agent Systems through Orchestration Traces

Abstract: As LLM agents evolve from isolated tool users into coordinated teams, reinforcement learning (RL) must optimize not only individual actions but also how work is spawned, delegated, communicated, aggregated, and stopped. This paper studies RL for LLM-based multi-agent systems through orchestration traces: temporal interaction graphs whose events include sub-agent spawning, delegation, communication, tool use, return, aggregation, and stopping decisions. Using this lens, we identify three technical axes. First, reward design spans eight families, including orchestration rewards for parallelism speedup, split correctness, and aggregation quality. Second, reward and credit signals attach to eight credit- or signal-bearing units from token to team; explicit counterfactual message-level credit remains especially sparse in our curated pool. Third, orchestration learning decomposes into five sub-decisions: when to spawn, whom to delegate to, how to communicate, how to aggregate, and when to stop. In our curated pool as of May 4, 2026, we found no explicit RL training method for the stopping decision. We connect academic methods to public industrial evidence from Kimi Agent Swarm, OpenAI Codex, and Anthropic Claude Code. The resulting scale gap is a gap between publicly reported deployment envelopes and open academic evaluation regimes, not independent verification of industrial training traces. We release the artifact at https://github.com/xxzcc/awesome-LLM-mas-rl, including an 84-entry tagged paper pool, a 32-record exclusion log, scripted corpus statistics, and a minimal JSON schema for replayable orchestration traces.

• The paper introduces orchestration traces as a novel abstraction to design rewards, assign multi-level credit, and learn orchestration policies for LLM-based multi-agent systems.
• It formalizes these traces as temporal interaction graphs that extend Dec-POMDP with spawn/despawn actions, enabling dynamic multi-agent coordination and communication strategies.
• The findings reveal key open challenges including long-horizon credit assignment, lack of MAS-native benchmarks, and safety vulnerabilities in large-scale industrial deployments.

Reinforcement Learning for LLM-Based Multi-Agent Systems via Orchestration Traces

Introduction and Motivation

As LLM agents transition from isolated tool users to complex, coordinated teams, the optimization landscape for reinforcement learning (RL) fundamentally shifts. RL must now address not only individual-agent trajectories, but also the higher-order orchestration of work—when to spawn agents, how to delegate and coordinate, how to aggregate results, and when to halt collaborative processes. This paper introduces the concept of orchestration traces as temporal interaction graphs that encapsulate these multi-agent decisions. The orchestration trace serves as an organizing abstraction for reward design, credit assignment, and orchestration policy learning in LLM-based multi-agent systems (MAS).

The motivation for this reframing comes from recent industrial and academic advances. Industrial systems such as Moonshot’s Kimi Agent Swarm have reported orchestration over hundreds of agents and thousands of coordinated steps, moving far beyond the assumptions of laboratory-scale MARL. Academia is converging on primitives (e.g., multi-agent fine-tuning, hierarchical decompositions, message-level counterfactual credit) that challenge and extend classical MARL constructs. The scale and dynamics of contemporary LLM-MAS call for a survey taxonomy that captures event-level credit and orchestration, rather than agent- or token-centric abstractions.

Orchestration Trace: Formalism and Taxonomy

The orchestration trace is formalized as a temporal interaction graph, extending the Dec-POMDP to a dynamic-Dec-POMDP ($\mathcal{M}^+$) that supports spawn and despawn actions. Each trace is a graph whose nodes are events (orchestrator decisions, agent spawn/return/aggregation, tool calls, messages) and edges denote temporal or causal dependencies.

Two informal observations anchor the rest of the technical discussion:

• Credit Diffusion under Uniform Credit: As trace length and granularity increase, naïve uniform credit allocation results in degraded signal-to-noise ratio per unit. This poses significant learning instability as empirically observed in long-trace RL for LLM-MAS [dr-mas2026].
• Non-Identifiability of Counterfactual Orchestrator Credit: For policy branches involving spawn/no-spawn choices, the causal effect of a decision at a trace prefix becomes unidentifiable from on-policy data alone, absent explicit off-policy evaluation. This is a fundamental barrier for credit assignment at the orchestration level.

The event-graph perspective enables the partitioning of RL for LLM-MAS along three orthogonal axes:

1. Reward Design (eight families): From shared outcome to orchestration-specific signals, with hybrid and system-level compositions dominating recent practice.
2. Credit and Signal Assignment (eight granularities): Team, orchestrator, role, agent, turn, message, tool, and token—each furnishing distinct loci for reward and learning signal.
3. Orchestration Learning Sub-decisions: When to spawn, whom to delegate, how to communicate, how to aggregate, when to stop.

Taxonomy of Methods and Coverage Gaps

Reward Design

Reward shaping in LLM-MAS is characterized by compositions across outcome, process, verifier, tool-use, and orchestration-specific signals. No single family suffices; most large-scale deployments utilize hybrid (R8) composites that involve time-varying reward weights. Notably, orchestration rewards (R7), which target system-level desiderata such as parallel execution speedup, split correctness, and aggregation quality, occupy a central position distinct from both single-agent RL and classical MARL.

Kimi PARL exemplifies this: the orchestrator’s reward combines task success, parallelism bonuses, and sub-agent finish-rate, with auxiliary weights annealed to zero over training. This staged annealing prevents pseudo-parallelism and ensures the policy ultimately optimizes the primary objective.

Credit Assignment

Credit assignment departs sharply from prior work. The hierarchy of credit-bearing units in LLM-MAS is multi-level and includes newly central units—especially orchestrator decisions and inter-agent messages. Most methods only target one or two levels (e.g., agent-wise normalization, role-specific PRM, message-level counterfactuals). Explicit counterfactual message-level credit is rare (C3 [c3-2026] constitutes a principal example), and orchestrator-level counterfactual credit is essentially unaddressed due to intrinsic non-identifiability.

SHARP [sharp2026] applies a Shapley-complete credit allocation, but computational intractability at large agent counts remains. Role-based and message-level interventions are beginning to populate the taxonomy, but multi-level composition is non-trivial due to possible credit double-counting.

Orchestration Learning

Five sub-decisions structure orchestration:

• O1 (Spawn timing): Dynamic extension of the agent set; only reward-shaping proxies for the true counterfactual are present.
• O2 (Delegation target): Learned central critics (e.g., Puppeteer [puppeteer2025]) and unified agent-tool action spaces (ParaManager [paramanager2026]).
• O3 (Communication strategy): Debate-as-Reward [debate-as-reward2026], Agent Q-Mix [agent-qmix2026], and latent-channel approaches have begun to explore direct learning over communication topology and content.
• O4 (Aggregation): Summarization and context-folding are utilized, though explicit uncertainty modeling remains unexplored.
• O5 (Stopping/Halting): To date, no method has proposed or trained an explicit RL-based stopping policy.

Practically, orchestrator and sub-agents are commonly trained under a regime where only the orchestrator (or central critic) is updated, due to cost and stability. Joint and fully decoupled training remain high-overhead and are only sparsely demonstrated.

Engineering, Evaluation, and Safety Implications

Industrial systems highlight a scale gap: public deployments (e.g., Kimi K2.5/K2.6) operate with hundreds of agents and traces spanning thousands of steps—regimes in which signal diffusion and rollout cost dominate, and RL signal sparsity is most acute. Harness boundaries in deployment, which fix the interface between model, tools, and orchestration logic, further constrain what parts of the system are RL-optimiz­able and what must be treated as fixed context.

The absence of open, MAS-native benchmarks reporting both traditional and multi-agent coordination metrics (task success, parallelism efficiency, collaboration quality, and protocol overhead) undermines fair evaluation. No open-source evaluation suite captures the trace length, scale, or credit-sparseness of real industrial systems, making cross-method comparison and reproducibility problematic.

Safety is increasingly shaped by multi-level, cross-agent vulnerabilities. Message pollution, shared-memory poisoning, orchestrator misdirection, and sandbox escape are all emergent risks that appear at distinct levels of the orchestration trace. Current defenses are predominantly heuristic or localized, and neither systematic trace-level constrained optimization nor steerability have yet been formalized in RL objectives.

Open Problems and Future Directions

Fifteen concrete open problems are articulated, with particular focus on:

• Long-horizon credit assignment under reward diffusion (P1): Theoretical and algorithmic developments for high-SNR credit assignment on Kimi-scale traces are urgently needed.
• Counterfactual credit for unrealized orchestration branches (P4): Advances in off-policy evaluation, perhaps via learned counterfactual estimators or importance sampling, are prerequisite for principled orchestrator optimization.
• MAS-native long-horizon benchmarks (P15): Evaluation infrastructure must be constructed to reflect realistic deployment envelopes and support reproducible, coordination-sensitive measurement.
• Systematic composition across credit layers (P1–P4): Principled frameworks for multi-level credit assignment and avoidance of double-counting are required.
• Safety and steerability as RL objectives (P12–P14): Cross-agent safety and mid-trace human intervention mechanisms must move from UX paradigms to RL-optimizable properties.

Conclusion

This paper redefines RL for LLM-MAS through the unifying abstraction of orchestration traces, organizing the field around event-centric taxonomies of reward, credit, and orchestration learning. The analysis rigorously demonstrates that classical MARL and single-agent RL are insufficient in this regime—dynamic agent sets, graph-structured traces, and multi-level credit pose novel theoretical and practical challenges. The gap between academic methods and public industrial deployments is primarily infrastructural, not solely algorithmic, with scale, evaluation, and safety constraints dominating.

Addressing the methodological and infrastructural gaps identified in this paper—especially MAS-native benchmarks and multi-level counterfactual credit assignment—will be central to the scientific maturation of LLM-MAS RL. The orchestration trace formalism provides a robust substrate for future theoretical, empirical, and systems work in this domain.