Learning to Coordinate (L2C)

• Learning to Coordinate (L2C) is a research umbrella addressing how multiple decision-makers learn mutually compatible behaviors amid coupling, uncertainty, and adaptation pressures.
• It encompasses methods from cooperative multi-agent reinforcement learning, decentralized actor-critic systems, and role-based coordination in robotics and distributed control.
• Key algorithmic approaches include deterministic model-based reductions, shared actor parameter policies, hierarchical advising, and optimization of distributed hyperparameters to achieve synchronized behavior.

Learning to Coordinate (L2C) denotes a family of research problems concerned with how multiple decision-making entities learn mutually compatible behavior under coupling, uncertainty, and adaptation pressure. Across the literature, the term is used most prominently in cooperative multi-agent reinforcement learning, multi-robot control, and coordination under partial observability, but its operational meaning varies by domain: coordination may mean convergence to an optimal joint strategy in common-interest stochastic games, learning partially shared policies in decentralized actor-critic systems, inferring partner intent without explicit communication, assigning complementary roles in bimanual manipulation, coordinating with previously unseen teammates, or tuning the hyperparameters of distributed trajectory optimizers (Brafman et al., 2011, Zeng et al., 2021, Jin et al., 2022, Grannen et al., 2023, Yuan et al., 2023, Chen et al., 2024, Wang et al., 1 Sep 2025). A recent extension also treats expert assistance as a coordination problem, introducing Learning to Yield and Request Control (YRC), where an agent must decide when to act autonomously and when to request help from an expert (Danesh et al., 13 Feb 2025).

1. Historical emergence and problem scope

An early explicit formulation of L2C appears in the study of common-interest stochastic games (CISGs), where all agents receive an identical payoff and must learn to coordinate to maximize the common long-run reward (Brafman et al., 2011). In that setting, the central difficulty is not merely environmental uncertainty, but synchronized exploration and agreement on high-value joint actions under imperfect monitoring. The 2011 model-based line showed that coordination could be reduced to repeated execution of a deterministic single-agent learner such as R-MAX, yielding polynomial-time convergence guarantees rather than convergence only “in the limit” (Brafman et al., 2011).

Subsequent work broadened L2C from common-interest games to more heterogeneous cooperative systems. In coordinated actor-critic (CAC), coordination is induced by a policy architecture with a shared part and a personalized part, allowing agents to learn common structure while preserving specialization (Zeng et al., 2021). LALA reframed coordination as a multilevel emergence problem, distinguishing micro, meso, and macro levels and placing a centralized advisor at the meso level to capture spatiotemporal decision structure (Jin et al., 2022). This marked a shift from viewing coordination as a byproduct of shared reward or communication toward treating it as an intermediate structure to be modeled directly.

The scope later expanded along at least four additional axes. First, coordination with previously unseen teammates became a central problem in open multi-agent environments, motivating continual teammate generation and compatibility learning in Macop (Yuan et al., 2023). Second, coordination without direct communication was formalized under incomplete information, where one agent must infer another’s intent from action sequences rather than messages (Chen et al., 2024). Third, coordination entered embodied robotics through bimanual manipulation, where role assignment between a stabilizing arm and an acting arm became the main organizational principle of BUDS (Grannen et al., 2023). Fourth, trajectory optimization work used L2C to denote distributed meta-learning of hyperparameters for ADMM-DDP, where the objective is not to learn control trajectories directly but to learn how local task performance and global coordination should be weighted across agents (Wang et al., 1 Sep 2025).

This breadth implies that L2C is not a single algorithmic family with a canonical objective. It is better understood as a research umbrella organized around the acquisition of joint structure under coupled decision-making.

2. Formal problem formulations

In CISGs, the environment is a stochastic game with finite state set $S=\{1,\dots,N\}$, common action set $A=\{a_1,\dots,a_k\}$ for each player, and transition function

$\mathrm{tr}(s,t,a,a').$

With the average reward criterion, $U(s,T,\pi,p)$ denotes the expected $T$-step undiscounted average reward, and

$$U(s,T)=\max_{\pi,p}U(s,T,\pi,p),\qquad U(s)=\liminf_{T\to\infty} U(s,T).$$

Under ergodicity, the optimal value is written as $v(M)$ and becomes independent of the initial state (Brafman et al., 2011). In this formulation, L2C means learning a joint policy that achieves the optimal common value efficiently.

In coordinated actor-critic, each agent’s policy parameters are decomposed as

$$\theta_{i} := \{ \theta_{i}^{s}, \theta_{i}^{p} \},$$

where $\theta_i^s$ is the shared part and $\theta_i^p$ is the personalized part (Zeng et al., 2021). Coordination is encoded directly in the actor rather than only in the critic. The stationarity notion used in the analysis jointly constrains consensus on shared parameters and optimality of both shared and personalized gradients:

$A=\{a_1,\dots,a_k\}$0

This formulation makes coordination partly an optimization-consensus problem (Zeng et al., 2021).

LALA defines coordination through multilevel emergence dynamics. At the meso level, an advisor operates on a spatiotemporal graph of agents’ decisions, with spatial neighbors $A=\{a_1,\dots,a_k\}$1 capturing conflicts and temporal neighbors $A=\{a_1,\dots,a_k\}$2 capturing continuity:

$A=\{a_1,\dots,a_k\}$3

Its advisor objective explicitly combines temporal continuity, spatial conflict reduction, and max-confidence consistency with confident agents (Jin et al., 2022).

In communication-free coordination under incomplete information, the helper policy is defined over the current state and the seeker’s last action sequence,

$A=\{a_1,\dots,a_k\}$4

with objective

$A=\{a_1,\dots,a_k\}$5

The seeker policy is

$A=\{a_1,\dots,a_k\}$6

Here the coordination problem is explicitly non-Markovian, because the helper must condition on a full action sequence emitted by the seeker in the previous turn (Chen et al., 2024).

In bimanual manipulation, coordination is formalized as a decomposition of the joint bimanual policy

$A=\{a_1,\dots,a_k\}$7

into mutually dependent stabilizing and acting policies

$A=\{a_1,\dots,a_k\}$8

The stabilizing objective is to minimize change in a task-relevant representation $A=\{a_1,\dots,a_k\}$9,

$\mathrm{tr}(s,t,a,a').$0

A stabilization remains valid until

$\mathrm{tr}(s,t,a,a').$1

(Grannen et al., 2023). Coordination is therefore expressed as role assignment plus event-triggered recoordination.

A further formulation appears in YRC. The abstract defines a strategy that determines when to act autonomously and when to seek expert assistance, in a setting where the agent does not interact with experts during training but must adapt to novel environmental changes and expert interventions at test time (Danesh et al., 13 Feb 2025). The available figure suggests a novice policy $\mathrm{tr}(s,t,a,a').$2, state $\mathrm{tr}(s,t,a,a').$3, validation signal $\mathrm{tr}(s,t,a,a').$4, novice action $\mathrm{tr}(s,t,a,a').$5, expert action $\mathrm{tr}(s,t,a,a').$6, and a coordination decision $\mathrm{tr}(s,t,a,a').$7 indicating whether control remains with the novice or is yielded to the expert. This suggests a router-like controller, although the formal objective is not available in the provided material (Danesh et al., 13 Feb 2025).

3. Algorithmic mechanisms

A first major algorithmic line is deterministic model-based reduction. In the CISG setting, R-MAX is used as a deterministic single-agent engine on the induced MDP. Unknown state-action pairs are assigned reward $\mathrm{tr}(s,t,a,a').$8, the learner repeatedly computes the optimal $\mathrm{tr}(s,t,a,a').$9-step policy in the current optimistic model, and tie-breaking is fixed so that all agents execute the same joint action whenever their internal models coincide (Brafman et al., 2011). This determinism is not incidental; it is the mechanism that synchronizes exploration and exploitation without requiring stronger monitoring assumptions.

A second line is partial policy sharing in actor-critic. CAC preserves fully decentralized and federated deployment while allowing agents to coordinate through shared actor parameters. The framework uses linear function approximation,

$U(s,T,\pi,p)$0

and a TD-like signal

$U(s,T,\pi,p)$1

Shared actor parameters are mixed via neighbor communication, for example

$U(s,T,\pi,p)$2

while personalized parameters are updated locally (Zeng et al., 2021). The paper proves that the algorithm requires $U(s,T,\pi,p)$3 samples to achieve an $U(s,T,\pi,p)$4-stationary solution and gives a double-sampling refinement removing the sampling mismatch term $U(s,T,\pi,p)$5 (Zeng et al., 2021).

A third line is hierarchical coordination via advising. LALA introduces a centralized advisor based on a spatiotemporal DualGCN and couples it to agents through policy generative adversarial learning. The discriminator loss is

$U(s,T,\pi,p)$6

and the agent loss is

$U(s,T,\pi,p)$7

The discriminator also boosts the advisor through

$U(s,T,\pi,p)$8

This architecture treats coordination as advice generation at the meso level and advice assimilation at the micro level (Jin et al., 2022).

A fourth line learns coordination conventions from observed behavior alone. In the no-communication setting, one deterministic finite automaton (DFA) is learned per helper action,

$U(s,T,\pi,p)$9

using Angluin’s $T$0, a Capping procedure that removes shortest-path behavior from the seeker’s trace, and a finite-state transducer with transition relation $T$1 and output function $T$2 (Chen et al., 2024). The resulting NCC algorithm counts how often trajectory segments are accepted by each DFA and selects the helper action with maximal frequency. The strategy is explicitly non-Markovian because it depends on the entire observed seeker sequence rather than only the current state (Chen et al., 2024).

A fifth line uses role assignment and event detection rather than symmetric control. BUDS learns three models: a stabilizing keypoint model $T$3, an acting policy $T$4, and a restabilizing classifier $T$5 (Grannen et al., 2023). The stabilizer predicts a workspace keypoint via

$T$6

and the acting arm follows

$T$7

The coordination loop alternates between keeping the current stabilizing point and recomputing it when $T$8 signals that the old hold is no longer effective (Grannen et al., 2023).

A sixth line treats coordination as differentiable distributed optimization. In distributed ADMM-DDP, each agent solves a local DDP subproblem inside an ADMM loop, while copy variables enforce consensus:

$T$9

The augmented Lagrangian includes agent-specific penalties $$U(s,T)=\max_{\pi,p}U(s,T,\pi,p),\qquad U(s)=\liminf_{T\to\infty} U(s,T).$$0, and DDP yields feedback gains

$$U(s,T)=\max_{\pi,p}U(s,T,\pi,p),\qquad U(s)=\liminf_{T\to\infty} U(s,T).$$1

L2C meta-learns per-agent hyperparameters

$$U(s,T)=\max_{\pi,p}U(s,T,\pi,p),\qquad U(s)=\liminf_{T\to\infty} U(s,T).$$2

via lightweight agent-wise neural networks, differentiating end-to-end through the ADMM-DDP pipeline and reusing Riccati terms and feedback gains in the backward pass (Wang et al., 1 Sep 2025).

4. Coordination under adaptation, novelty, and partial observability

A central contemporary theme is that L2C must generalize beyond training-time partners and conditions. Macop addresses open multi-agent environments by defining a Continual Teammate Dec-POMDP in which controllable agents face a sequence of previously unseen teammate groups $$U(s,T)=\max_{\pi,p}U(s,T,\pi,p),\qquad U(s)=\liminf_{T\to\infty} U(s,T).$$3 and must retain coordination ability across stages (Yuan et al., 2023). Rather than sampling a fixed diversity set, Macop continually generates teammates that are deliberately incompatible with the current ego policy. The teammate objective

$$U(s,T)=\max_{\pi,p}U(s,T,\pi,p),\qquad U(s)=\liminf_{T\to\infty} U(s,T).$$4

combines self-play quality, diversity, and incompatibility. A stopping criterion

$$U(s,T)=\max_{\pi,p}U(s,T,\pi,p),\qquad U(s)=\liminf_{T\to\infty} U(s,T).$$5

terminates continual training when no meaningfully more challenging teammate population can be found (Yuan et al., 2023). This makes compatibility coverage, rather than static diversity, the organizing principle.

Partial observability motivates a different form of adaptation. In the no-communication game setting, the helper never observes the seeker’s private transition dynamics or goal; only the seeker knows the true goal location (Chen et al., 2024). Coordination therefore depends on interpreting deviations from shortest-path motion as intent signals. This shifts L2C from explicit message exchange to learned conventions grounded in action traces. A plausible implication is that some coordination problems are better characterized as convention learning than as communication learning.

Embodied coordination introduces adaptation to task phase and scene change. In BUDS, a single stabilizing point is insufficient for tasks such as Jacket Zip or Cut Vegetable, so the system must repeatedly decide when the environment has changed enough to trigger restabilization (Grannen et al., 2023). The acting arm is trained from single-arm demonstrations, but its success depends on the stabilizer maintaining a task-relevant fixture. Coordination is therefore temporally segmented rather than simultaneous and symmetric.

Expert-assisted coordination adds a further adaptation dimension. YRC assumes that the agent does not interact with experts during training, yet must adapt to novel environmental changes and expert interventions at test time (Danesh et al., 13 Feb 2025). This formulation is notable because expert querying is explicitly described as costly. The coordination problem is not only behavioral compatibility but also selective escalation: when autonomous competence should yield to external control.

In distributed trajectory optimization, adaptation occurs through hyperparameter meta-learning rather than direct policy adaptation. L2C learns cost weights, constraint parameters, and ADMM penalties that vary with task conditions, such as load center-of-mass offset or team size (Wang et al., 1 Sep 2025). This suggests a broader interpretation of L2C in control: coordination can be learned by shaping the optimizer that produces local decisions, not only by learning the decisions themselves.

5. Empirical regimes, benchmarks, and reported outcomes

The empirical literature on L2C is correspondingly heterogeneous. YRC-Bench is introduced as an open-source benchmark with diverse domains, a standardized Gym-like API, simulated experts, an evaluation pipeline, and competitive baselines (Danesh et al., 13 Feb 2025). Although the provided material does not include benchmark statistics, the benchmark’s role is to make expert-coordination research empirically comparable.

LALA reports improvements in learning efficiency and coordination capability, with evaluation in terms of average episode reward, cooperation success rate, and normalized navigation time (Jin et al., 2022). It also analyzes coordination through mutual information, studying $$U(s,T)=\max_{\pi,p}U(s,T,\pi,p),\qquad U(s)=\liminf_{T\to\infty} U(s,T).$$6 between an agent’s latent representation and other agents’ actions. The paper states that LALA increases this mutual information, linking meso-level coordination advice to improved micro-level teammate modeling (Jin et al., 2022).

CAC contributes a different empirical-theoretical profile. Its distinguishing result is a finite-time sample complexity bound rather than a benchmark headline: $$U(s,T)=\max_{\pi,p}U(s,T,\pi,p),\qquad U(s)=\liminf_{T\to\infty} U(s,T).$$7 samples to reach an $$U(s,T)=\max_{\pi,p}U(s,T,\pi,p),\qquad U(s)=\liminf_{T\to\infty} U(s,T).$$8-stationary solution, with actor and critic stationarity measures decaying at rate $$U(s,T)=\max_{\pi,p}U(s,T,\pi,p),\qquad U(s)=\liminf_{T\to\infty} U(s,T).$$9 up to approximation and sampling mismatch terms (Zeng et al., 2021). This makes it one of the few L2C formulations centered on non-asymptotic analysis.

Communication-free coordination is evaluated in a modified Gnomes at Night testbed with $v(M)$0 and $v(M)$1 mazes, 10 maze layouts per size, 5 treasure positions per layout, and 100 trials per configuration (Chen et al., 2024). Compared with No Coordination (NC), the proposed No-Communication Coordination (NCC) substantially improves success rate: NC achieves 28.62% and 17.70%, NCC achieves 90.16% and 90.54%, and Direct-Communication Coordination (DCC) achieves 94.08% and 98.02% on the two maze sizes (Chen et al., 2024). NCC also reduces steps taken and lowers both the number of memorized walls and the wall error rate. The learned DFAs have Jaccard similarities of 0.8 for right, 0.58 for up, 0.67 for left, and 0.58 for down (Chen et al., 2024).

BUDS is evaluated on four real-world dual-arm tasks—Pepper Grinder, Jacket Zip, Marker Cap, and Cut Vegetable—using a dual-UR16e platform with Robotiq 2F-85 grippers, three Intel RealSense cameras, and RTDE-based impedance control at 10 Hz (Grannen et al., 2023). Given only 20 demonstrations, BUDS achieves 76.9% average success across the task suite and 52.7% success on out-of-distribution objects within a class (Grannen et al., 2023). It is reported as 56.0% more successful than a BC-Stabilizer baseline and far stronger than a monolithic 14-DoF policy, which achieves zero success (Grannen et al., 2023).

Macop is evaluated on 8 scenarios from 4 environments: Level-based Foraging, Predator-Prey, Cooperative Navigation, and SMAC (Yuan et al., 2023). The reported result is that Macop achieves the best average coordination performance in all 8 scenarios and a 60.44% average improvement over the Finetune anchor (Yuan et al., 2023). It also records the best BWT in all evaluated environments, indicating minimal forgetting, while cross-play heatmaps and t-SNE of self-play trajectory embeddings are used to show that generated teammate populations are both well dispersed and low in compatibility (Yuan et al., 2023).

Distributed meta-trajectory L2C is validated on cooperative aerial transport with cable-suspended rigid loads in high-fidelity simulation and IsaacSIM-based software-in-the-loop evaluation (Wang et al., 1 Sep 2025). The method is reported to achieve up to 88% faster gradient computation than SPDP and ADDP while maintaining relative gradient errors below 10% (Wang et al., 1 Sep 2025). It is also reported to generalize without additional tuning to unseen 3-quadrotor and 7-quadrotor teams, unseen load offsets, new obstacle placements, and new reference trajectories (Wang et al., 1 Sep 2025).

6. Interpretive issues, misconceptions, and acronym ambiguity

A recurring misconception is that L2C is simply another name for communication learning. The surveyed work does not support that reduction. Some formulations rely on no explicit communication at all and instead learn coordination through deterministic policies, shared actor structure, behavioral conventions, role timing, or optimizer-level hyperparameter adaptation (Brafman et al., 2011, Zeng et al., 2021, Grannen et al., 2023, Chen et al., 2024, Wang et al., 1 Sep 2025). Communication is therefore only one possible substrate for coordination, not its defining property.

Another misconception is that shared reward suffices to produce coordination automatically. The empirical and algorithmic designs argue otherwise. The CISG work emphasizes coordinated exploration under imperfect monitoring (Brafman et al., 2011). CAC adds shared policy parameters rather than only shared critics (Zeng et al., 2021). LALA introduces an explicit advisor to model spatial conflicts and temporal smoothness (Jin et al., 2022). Macop shows that even broad teammate diversity is insufficient unless new teammates are actively evolved to be incompatible with the current ego policy (Yuan et al., 2023). These results suggest that coordination is usually an architectural or algorithmic object in its own right.

The literature also differs on what is being coordinated. In MARL, the objects are usually policies or action distributions; in bimanual robotics, they are roles and timing; in no-communication games, they are conventions between trajectories and latent intent; in trajectory optimization, they are hyperparameters that shape local and global trade-offs (Jin et al., 2022, Grannen et al., 2023, Chen et al., 2024, Wang et al., 1 Sep 2025). A plausible implication is that “coordination” is best treated as a relational property of coupled decision variables rather than as a fixed task class.

Finally, the acronym itself is overloaded. “L2C” is also used for “Learning-to-Compare” in visual difference description, where the goal is to compare semantic representations of two images rather than to coordinate agents (Yan et al., 2021). It is likewise used for “Learning to Condition” in scalable MPE inference, where a neural heuristic scores variable-value assignments for conditioning in probabilistic graphical models (Malhotra et al., 22 Sep 2025). In the cooperative systems literature, however, L2C refers specifically to learning structured compatibility under joint decision-making.

Within that cooperative meaning, the field now spans optimal-value guarantees in ergodic stochastic games, decentralized finite-time actor-critic analysis, hierarchical advising, convention learning without direct communication, role-based robot coordination, continual compatibility learning with arbitrary teammates, and differentiable distributed optimization with learned coordination hyperparameters (Brafman et al., 2011, Zeng et al., 2021, Jin et al., 2022, Grannen et al., 2023, Yuan et al., 2023, Chen et al., 2024, Danesh et al., 13 Feb 2025, Wang et al., 1 Sep 2025). The common thread is not a shared implementation recipe, but the explicit treatment of joint structure as something to be learned rather than assumed.