Round-Robin Behavior Cloning (R2BC)

• R2BC is a multi-agent imitation learning paradigm that replaces synchronized joint demonstrations with sequential, single-agent teleoperation.
• It leverages per-agent behavior cloning with masked loss functions, enabling realistic online adaptation to partner-induced state fluctuations.
• Empirical evaluations show that R2BC outperforms joint behavior cloning by reducing covariate shift and enhancing coordination in multi-robot tasks.

Round-Robin Behavior Cloning (R2BC) is a multi-agent imitation learning paradigm for settings in which a single human can teleoperate only one agent at a time. It was introduced as an online, decentralized behavior cloning procedure that replaces synchronized joint-action demonstrations with sequential, single-agent demonstrations collected in a round-robin schedule. During data collection, exactly one agent receives expert action labels, while non-teleoperated agents execute their current learned policies; training then proceeds through per-agent behavior cloning on agent-specific buffers. The method is motivated by the observation that a lone human cannot reliably provide high-quality, synchronized joint actions for all agents in a multi-robot team, and it is presented as a practical alternative to joint behavior cloning (JBC) in cooperative multi-agent systems (Mattson et al., 20 Oct 2025).

1. Definition and motivating setting

R2BC is designed for realistic multi-robot teaching scenarios in which the demonstration interface is underactuated relative to the full joint action space. Instead of assuming that a demonstrator can specify a joint action $a_t \in A_1 \times \dots \times A_N$ for all $N$ agents at every timestep, R2BC assumes that the demonstrator teleoperates one agent at a time. The remaining agents act autonomously using their current policies, so the demonstrated agent is trained in the context of imperfect teammates rather than alongside privileged oracle labels (Mattson et al., 20 Oct 2025).

This formulation differs from standard multi-agent behavior cloning in two central respects. First, supervision is partial: only one agent’s action is labeled at a timestep. Second, training is online rather than purely offline: policies are updated periodically as new demonstrations are collected, and the behavior of non-demonstrated agents affects the state distribution encountered by the demonstrator. The resulting dataset contains interaction-induced variability and corrective examples, which the method identifies as a mechanism for reducing covariate shift relative to offline JBC (Mattson et al., 20 Oct 2025).

The motivating claim is therefore not merely that synchronized demonstrations are inconvenient, but that they may be systematically misaligned with actual deployment conditions. R2BC exploits the fact that teammates executing learned policies generate realistic multi-agent contexts, including coordination variability and occasional misalignment, and these conditions allow the demonstrator to provide corrections directly in the states that arise during learning. A plausible implication is that the method treats demonstration not as static labeling of optimal trajectories, but as iterative supervision under a changing team policy.

2. Formal problem formulation

The method is posed in a Markov (stochastic) game with tuple $\langle I,S,O,A,R,T\rangle$, where agents are indexed by $i \in \{1,\dots,N\}$. The global state space is $S$, each agent receives a local observation $o_t^i \in O_i$, and the joint observation space is $O = O_1 \times \dots \times O_N$. The joint action space is $A = A_1 \times \dots \times A_N$, with joint action $a_t = (a_t^1,\dots,a_t^N)$ and transition kernel $T(s_{t+1}\mid s_t,a_t)$. The tasks considered use a shared/common reward, although R2BC itself is trained from demonstrations only (Mattson et al., 20 Oct 2025).

The main instantiation learns decentralized per-agent policies

$N$0

and executes them in a decentralized manner. No communication or shared latent state is required. The paper also notes that a centralized server may aggregate sensing to construct per-agent observations, such as mock LiDAR or vision-derived features, but the learned policies themselves remain decentralized. CTDE is explicitly not required, and agents can be trained independently with their own buffers (Mattson et al., 20 Oct 2025).

Data collection produces a partially labeled multi-agent dataset. At each timestep $N$1, the logged tuple includes the global or environment state $N$2 if available, all agents’ observations $N$3, the demonstrated agent’s action label $N$4, and the identity of the teleoperated agent $N$5. The dataset is written as

$N$6

with an equivalent per-agent view

$N$7

This representation makes the supervision structure explicit: the full observation context is available, but action labels are sparse and agent-indexed (Mattson et al., 20 Oct 2025).

3. Demonstration protocol and learning objective

The defining operational mechanism is round-robin teleoperation. A single human or simulated expert teleoperates one agent $N$8 for an episode; after that episode, control switches to agent $N$9, and the cycle repeats. This schedule ensures that each agent receives demonstrations at regular intervals. During an episode in which agent $\langle I,S,O,A,R,T\rangle$0 is controlled, every agent $\langle I,S,O,A,R,T\rangle$1 executes its current learned policy $\langle I,S,O,A,R,T\rangle$2 (Mattson et al., 20 Oct 2025).

The learning objective can be written through a masked multi-agent behavior cloning loss. Let $\langle I,S,O,A,R,T\rangle$3 indicate whether agent $\langle I,S,O,A,R,T\rangle$4 was teleoperated at timestep $\langle I,S,O,A,R,T\rangle$5, and let $\langle I,S,O,A,R,T\rangle$6 denote a discrepancy loss, specified as cross-entropy for discrete actions or mean squared error for continuous actions. Then the general masked loss is

$\langle I,S,O,A,R,T\rangle$7

where $\langle I,S,O,A,R,T\rangle$8 collects parameters across agents. In the decentralized case, this reduces to independent per-agent objectives,

$\langle I,S,O,A,R,T\rangle$9

The masking is not an auxiliary implementation detail; it is the formal device that permits training from single-agent labels within a multi-agent trajectory (Mattson et al., 20 Oct 2025).

An optional centralized variant can use a shared encoder $i \in \{1,\dots,N\}$0 with agent-specific heads $i \in \{1,\dots,N\}$1, so that

$i \in \{1,\dots,N\}$2

In that setting, masked updates backpropagate only through the demonstrated agent’s head, and optionally through the shared parameters $i \in \{1,\dots,N\}$3. Agreement or consistency losses across agents can also be added, but the reported main results are achieved without such regularizers. The paper further states that the centralized masked variant performed slightly worse than decentralized R2BC in most tasks, while remaining useful when parameter sharing is desired (Mattson et al., 20 Oct 2025).

The conceptual claim attached to this objective is that coordination can emerge without joint labels. Because the demonstrator repeatedly acts in environments shaped by the current teammates’ policies, the collected supervision is conditioned on realistic partner behavior rather than on a privileged centralized controller. This suggests that R2BC substitutes synchronized labeling with iterative conditioning on evolving team dynamics.

4. Algorithmic procedure and online adaptation

The end-to-end procedure begins by instantiating $i \in \{1,\dots,N\}$4 policies $i \in \{1,\dots,N\}$5, initializing empty per-agent buffers $i \in \{1,\dots,N\}$6, and selecting an update frequency $i \in \{1,\dots,N\}$7, such as updating after every $i \in \{1,\dots,N\}$8 episodes per agent. Data collection is then performed online in round-robin order. For each teleoperated agent $i \in \{1,\dots,N\}$9, the environment is reset to $S$0, the human or expert supplies $S$1 for $S$2, every non-teleoperated agent applies its current policy, the environment transitions are executed, and $S$3 is appended to $S$4. Every $S$5 episodes, each agent is updated by behavior cloning on its own buffer using the appropriate discrepancy loss (Mattson et al., 20 Oct 2025).

The paper’s decentralized pseudocode emphasizes that this is a replay-based iterative procedure rather than one-shot supervised fitting. Per-agent buffers accumulate demonstrations over time, which is described as helping mitigate catastrophic forgetting by continuing to train on the full dataset. The main procedure co-trains all agents across rounds, but if instability is observed, non-target agents can be frozen during certain updates or $S$6 can be varied to stabilize learning. The round-robin schedule is intended to encourage balanced data collection; if imbalance emerges because some agents are harder to train, weighting or targeted extra demonstrations can be added (Mattson et al., 20 Oct 2025).

The justification for online adaptation is tied directly to non-stationarity. Since non-demonstrating agents act autonomously, the demonstrator encounters diverse states, including off-nominal situations. These states generate corrective examples that the authors relate to the effects of DAgger and DART: non-demonstrating agents induce observation noise akin to DART’s injected perturbations, and the demonstrator supplies corrections in off-nominal states akin to DAgger. The paper reports that R2BC achieves similar reductions in train-test loss gaps without requiring centralized oracle corrections (Mattson et al., 20 Oct 2025).

This framing also clarifies why R2BC is not merely a data-efficiency trick. Its online regime changes the state distribution under which demonstrations are gathered. The stated theoretical intuition is that, in cooperative tasks with shared reward and sufficient local observations, if each agent learns a policy minimizing $S$7 under the distribution induced by teammates executing their current policies, then the joint policy approximates coordinated behavior. The paper also notes a conditional-independence intuition: if each optimal action $S$8 is conditionally independent given $S$9 and the environment state $o_t^i \in O_i$0, then learning from round-robin single-agent labels can approach the performance of joint BC as the dataset covers relevant interaction states (Mattson et al., 20 Oct 2025).

5. Empirical evaluation in simulation and on hardware

The principal comparison is against Joint Behavior Cloning (JBC) trained from privileged synchronized demonstrations provided by an oracle MAPPO policy, as well as online single-agent imitation learning baselines adapted to multi-agent settings, specifically DART and DAgger with centralized expert corrections. Across four VMAS tasks—Navigation, Balance, Buzz Wire, and Transport—the paper states that R2BC consistently outperforms or matches JBC despite having no access to joint-action labels and only single-agent demonstrations. It also reports that R2BC achieves similar benefits to DART and DAgger as the number of online demonstrations increases, while reducing the train-test loss gap relative to JBC and showing comparable generalization trends to DART and DAgger (Mattson et al., 20 Oct 2025).

The four simulated tasks span different coordination regimes. In Navigation ($o_t^i \in O_i$1), agents must reach individual goals while avoiding collisions using LiDAR-style sensing. In Balance ($o_t^i \in O_i$2), agents stabilize a freely rotating line carrying a spherical package while moving it to a goal without letting it fall. In Buzz Wire ($o_t^i \in O_i$3), two agents connected by rigid linkages push a mass through an enclosure, with coupled dynamics penalizing contact with boundaries. In Transport ($o_t^i \in O_i$4), agents cooperatively push a heavy package into a goal region, requiring joint effort beyond the capacity of any single agent (Mattson et al., 20 Oct 2025).

Physical deployment uses three HeRo+ differential-drive robots whose action space consists of forward/backward velocity and angular velocity. Tracking is performed with AruCo markers, vision uses an Intel RealSense D435i, and a centralized ROS server computes per-agent observations and dispatches actions. Two tasks are reported: a real-world Navigation task in which agents navigate to colored goals while avoiding collisions using LiDAR-like features derived from the camera feed, and a Block Pusher task in which three robots push an elongated block of approximately $o_t^i \in O_i$5 kg to a goal, requiring balancing forces and coordinated contact (Mattson et al., 20 Oct 2025).

For data collection, two sets of 240 demonstrations per task were collected in simulation using Xbox controllers. For JBC, two controllers simultaneously controlled multiple agents, which the paper characterizes as difficult in practice. For R2BC, the operator controlled one agent per episode in round-robin order. The simulation environments matched the real robots’ kinematics and velocity limits, and policies trained in simulation were deployed directly to hardware (Mattson et al., 20 Oct 2025).

The reported hardware results use five randomized initial configurations and Welch t-tests. In Navigation, R2BC achieves a $o_t^i \in O_i$6 average improvement over JBC with zero interventions ($o_t^i \in O_i$7), $o_t^i \in O_i$8 with one intervention ($o_t^i \in O_i$9), and $O = O_1 \times \dots \times O_N$0 with two interventions ($O = O_1 \times \dots \times O_N$1). In Block Pusher, R2BC achieves $O = O_1 \times \dots \times O_N$2 over JBC with zero interventions ($O = O_1 \times \dots \times O_N$3), $O = O_1 \times \dots \times O_N$4 with one intervention ($O = O_1 \times \dots \times O_N$5), and $O = O_1 \times \dots \times O_N$6 with two interventions ($O = O_1 \times \dots \times O_N$7). The interventions are brief supervisor corrections lasting three seconds, and they are reported to improve both methods while benefiting R2BC substantially more (Mattson et al., 20 Oct 2025).

6. Relation to other imitation-learning paradigms, limitations, and practical use

Within multi-agent imitation learning, R2BC is distinguished by its explicit rejection of synchronized joint labels as a training requirement. The paper situates prior multi-agent IL methods as often assuming privileged joint demonstrations or centralized experts, including coordinated multi-agent IL, GAIL-style extensions, and inverse factorized soft Q-learning. By contrast, R2BC keeps the learning problem in each agent’s own action space and targets the practical constraint of a single human demonstrator (Mattson et al., 20 Oct 2025).

Its nearest methodological analogues are DAgger and DART, but the relationship is qualified. R2BC shares their emphasis on online correction under non-ideal state distributions, yet adapts that logic to a multi-agent setting where teammate policies themselves generate the perturbations and distribution shift. This suggests that the method can be understood as a multi-agent analogue of dataset aggregation under decentralized supervision, although the paper does not label it in those terms.

Several limitations are identified explicitly. Tasks with strong instantaneous coupling may still benefit from joint labels; Buzz Wire is cited as a case in which R2BC can succeed but may require more demonstrations or better observation features. Severe partial observability or the absence of communication can make coordination difficult if an agent’s local observation is insufficient to infer teammate intent or state. Teleoperation latency and long, rapid-dynamics episodes can increase human burden. The round-robin schedule can also introduce order-dependent non-stationarity that affects early agents, though repeated cycling and periodic retraining are described as mitigating factors. Finally, scaling to larger $O = O_1 \times \dots \times O_N$8 increases demonstration time, and dataset imbalance may require targeted extra rounds or loss weighting (Mattson et al., 20 Oct 2025).

The practical guidance given in the paper follows directly from these constraints. Round-robin scheduling is recommended to maintain balanced demonstrations, with additional focused rounds for persistently underperforming agents. In decentralized training, per-agent buffers are sufficient; with shared backbones, losses should be masked so only the demonstrated agent’s head is updated at a given timestep. Observation normalization, action clamping to hardware capabilities, and consistent sim-real kinematics are recommended preprocessing measures. Online training with frequent updates, smaller $O = O_1 \times \dots \times O_N$9, and full logging of all agents’ observations are recommended to reduce covariate shift. Evaluation should track train-test loss gaps and success metrics, and limited supervisor interventions can be used in real-world evaluations to probe robustness and escape local minima. Reproducibility recommendations include logging seeds, random initializations, environment parameters, camera calibration, marker setups, and robot velocity limits (Mattson et al., 20 Oct 2025).