Centralized Imitation Learning

• Centralized imitation learning is a framework that leverages complete global information during training to mimic expert behaviors in multi-agent settings.
• Techniques such as mixing networks, diffusion models, and graph-based methods enable coordinated decentralized execution while addressing partial observability.
• Empirical evaluations show enhanced sample efficiency, stable convergence, and improved win-rates in complex environments compared to decentralized baselines.

Centralized imitation learning is a paradigm within multi-agent imitation learning (IL) in which agents are trained using global state, action, or observation information, often via a single joint learner or centralized coordination mechanism. This approach addresses the difficulties of multi-agent decision making, such as non-stationarity and partial observability, by leveraging centralized information during training while typically devolving into decentralized execution or per-agent policies post-training. Key motivations include circumventing the challenges of direct multi-agent reinforcement learning (RL) in partially observable Markov games, and closing the gap to the performance of an expert, with theoretical guarantees on sample efficiency, stability, and coordination robustness.

1. Problem Formulation and Motivation

Centralized imitation learning typically operates in decentralized partially observable Markov decision processes (Dec-POMDPs) or Markov games characterized by a tuple $$(S, \{O_i\}_{i=1}^M, \{A_i\}_{i=1}^M, P, R, \gamma)$$, where $M$ agents observe only local views $o_i$ and select actions $a_i$ to maximize a shared reward. The critical challenge is to learn decentralized (per-agent) policies that, at execution, act using only local information but can coordinate as effectively as a fully centralized policy exploiting the global state or joint observations.

Centralized training enables the use of global state and expert joint-action datasets, affording the learner access to information unavailable to any individual agent at execution time. This framework is especially advantageous in scenarios where coordination—often reflected in subtle dependencies between agent strategies—is critical for task success, as in high-dimensional or competitive/cooperative domains (Lin et al., 2019, Bui et al., 2023, Dharmavaram et al., 2021, Dong et al., 17 Sep 2025).

2. Methodological Frameworks

There are several methodological instantiations of centralized imitation learning, distinguished by their treatment of the expert, learning objectives, and subsequent decentralization:

• Centralized Expert, Decentralized Imitation: A centralized expert policy $\pi^*(o)$ is trained with global observability, then individual agent policies $\{\pi_i\}$ are trained to imitate the expert's decisions based on local observations via DAgger-style dataset aggregation or supervised learning. Performance bounds can be derived based on the imitation error and expert regret (Lin et al., 2019).
• Centralized Training, Decentralized Execution (CTDE) with Structured Value Decomposition: Local Q-functions $Q_i(o_i, a_i)$ are learned, and their contributions to the global objective are aggregated via mixing networks parameterized by the global state. These mixing networks are constructed to preserve convexity and functional consistency, allowing stable and expressive sharing of information between agents at training, while extracting local policies for deployment (Bui et al., 2023).
• Centralized Diffusion Policy Training: Each agent's policy is parameterized as a conditional diffusion model over trajectories. All agent policies are trained jointly (centralized) by minimizing the sum of per-agent denoising losses, with full access to the joint demonstration dataset. At deployment, agents sample actions using their own local policy and observation without inter-agent communication (Dong et al., 17 Sep 2025).
• Graph-based Centralized Actor–Critic: Centralized policies are built over latent interaction graphs that encode agent dependencies, using graph convolution and soft actor–critic methods. These models support fully centralized training, with per-agent factorization and decentralized sampling during execution (Dharmavaram et al., 2021).

Method	Centralized Component	Decentralized Component
CESMA (Lin et al., 2019)	Centralized expert policy $\pi^*(o)$	DAgger/supervised learning for $\pi_i$
MIFQ (Bui et al., 2023)	Mixing networks $f_R$, $M$0 (state-based)	Policies $M$1 from $M$2
MIMIC-D (Dong et al., 17 Sep 2025)	Joint diffusion policy training	Per-agent diffusion sampling
SS-MAIL (Dharmavaram et al., 2021)	Graph-structured SAC/AIL	Per-agent selection from $M$3

3. Architectures and Algorithms

A variety of architectures have been proposed to realize centralized imitation learning in practice:

• Graph-Based Architectures: MAIL/SS-MAIL uses a graph encoder (LSTM and MLP) to infer a latent interaction graph, which is then used to propagate information through graph convolutions in both the actor and critic networks, enabling centralized conditioning on inferred interaction structure (Dharmavaram et al., 2021).
• Mixing Networks and Convex Factorization: By aggregating per-agent Q-values using parameterized mixing networks (with non-negativity and convexity constraints on weights), MIFQ can exploit global state for coordination, preserve convexity (ensuring a unique global optimum), and allow stable multi-agent learning (Bui et al., 2023).
• Diffusion Models: In MIMIC-D, per-agent policies are transformer-based denoising networks trained to invert noise-added trajectories. Centralized training conditions the denoiser on all agents’ observations, implicitly coupling their behavior and coordination (Dong et al., 17 Sep 2025).
• Self-Supervised Adversarial Discriminators: SS-MAIL introduces a mean-squared-error regression loss for the discriminator, matching generated, expert, and interpolated pairs to 0, 1, and interpolation weights, respectively, yielding dense reward gradients that stabilize training and promote multimodal policy learning (Dharmavaram et al., 2021).

4. Sample Efficiency, Stability, and Curriculum Strategies

Centralized imitation learning frameworks often introduce mechanisms to stabilize optimization and improve sample efficiency:

• Curriculum via Trajectory Forcing: SS-MAIL incorporates a teacher-forcing curriculum where the probability of following expert trajectories decays exponentially, enabling smooth transition from imitation to autonomy and accelerating early learning. The schedule parameter $M$4 can be tuned to optimally trade off bias and variance, with intermediate values yielding lowest imitation errors (Dharmavaram et al., 2021).
• Convex Losses and Mixing: MIFQ’s convexity guarantees for its objective function ensure no spurious minima, lower variance in learning curves, and stable convergence even in high-dimensional state-action spaces (Bui et al., 2023).
• Diffusion-Based Exploration: The stochasticity provided by diffusion process training naturally mitigates mode collapse and enables recovery of rich, multi-modal strategies essential for robust multi-agent coordination (Dong et al., 17 Sep 2025).

5. Empirical Evaluation and Results

Centralized imitation learning methods consistently outperform decentralized or per-agent baselines across a range of complex cooperative and competitive benchmarks:

• Policy Fidelity and Coordination: In StarCraft II micromanagement (SMACv2), Gold Miner, and cooperative particle-world environments, MIFQ achieves $M$5 win-rates compared to $M$6 for per-agent BC, IQ-Learn, IQVDN, and multi-agent GAIL. MIFQ closes $M$7 of the gap to expert performance, with substantial improvements in convergence rate and stability (Bui et al., 2023).
• Mode Coverage and Multi-Modality: On synthetic Y-Junction and motion-capture prediction tasks, SS-MAIL cleanly captures multi-modal expert behaviors and demonstrates robust noise resilience, outperforming GAIL/WAIL (unstable losses) and BC (mode averaging). Trajectory Forcing yields minimal error and variance at intermediate decay (Dharmavaram et al., 2021).
• Collision Avoidance and Implicit Synchronization: MIMIC-D achieves substantial reductions in collision rates (e.g., $M$8 vs $M$9 collisions for BC in two-agent swap), recovers implicit “mode locking” in decentralized execution, and approaches $o_i$0 real-world hardware success rates in dual-arm robot lifts (Dong et al., 17 Sep 2025).

Paper	Testbed	Key Metric / Result
MIFQ	SMACv2, Gold Miner	$o_i$1 win ($o_i$22-3$o_i$3 BC)
SS-MAIL	Y-Junction, Mocap	Smooth loss decay, robust to noise
MIMIC-D	Swap, Road Crossing	15/100 collisions ($o_i$4BC,GAIL)

6. Theoretical Guarantees and Analysis

Centralized imitation learning approaches provide rigorous performance bounds:

• No-Regret and Performance Guarantee: In CESMA, the regret bound shows that the decentralized team policy reward is within $o_i$5 of the centralized expert, with $o_i$6 the average imitation loss. Key assumptions include bounded per-step rewards and no-regret supervised learning (Lin et al., 2019).
• Convexity of Loss Functions: MIFQ establishes that with convex, non-decreasing mixing nets, the global imitation objective is convex in the local Q-functions, precluding suboptimal local minima (Bui et al., 2023).
• Cost-Regularized Apprenticeship Learning Connections: SS-MAIL demonstrates that its self-supervised AIL loss can be interpreted as a cost-regularized apprenticeship learning objective with a quadratic regularizer that interpolates between expert and generated trajectories (Dharmavaram et al., 2021).

7. Limitations and Practical Considerations

While centralized imitation learning offers robust coordination and sample efficiency benefits, it is contingent upon the quality and coverage of expert demonstrations. Partial observability can introduce imitation errors that are irreducible without expanding per-agent information (e.g., through communication or increased history). The efficacy of decentralized deployment is bounded by the decentralized agents’ capacity to approximate the expert from local observations alone. Sample complexity during centralized RL training and subsequent data aggregation can be high for large agent teams or long-horizon tasks (Lin et al., 2019, Dong et al., 17 Sep 2025).

Centralized imitation learning remains a highly effective paradigm for multi-agent control, unifying the strengths of global coordination, stability, and expressiveness, while enabling efficient and robust decentralized execution in challenging environments.