GCT-MARL: Graph-Based Contrastive Transfer for Sample-Efficient Cooperative Multi-Agent Reinforcement Learning

Abstract: In cooperative multi-agent reinforcement learning (MARL), from a deployment perspective, it is challenging and expensive to train agents from scratch for each new environment or task. In this work, we propose GCT-MARL, a transfer learning framework that builds on the multi-view graph contrastive backbone of MAIL and augments it with a per-view, adaptively weighted alignment loss and a two-phase training protocol specifically designed for transfer across populations of varying sizes and compositions. We empirically demonstrate that the proposed framework markedly accelerates convergence on the target task relative to from-scratch training, in both homogeneous (within-faction, varying N) and heterogeneous (cross-faction and mixed unit-type) transfer scenarios. Furthermore, we show that the framework naturally supports continual learning by sequentially chaining the two-phase transfer protocol across a series of related tasks. Overall, this work provides a unified approach to mitigating key limitations in current MARL transfer methods with new insights at both methodological and empirical levels.

• The paper introduces a population-invariant graph-based encoder and adaptive cross-task alignment loss to enable effective transfer across heterogeneous MARL settings.
• Empirical evaluation on the SMAC benchmark shows 2–3× faster convergence and near-perfect win rates in variable agent populations.
• The two-phase training protocol supports continual learning with strong retention, underscoring the benefits of representation-level alignment for cooperative MARL.

GCT-MARL: A Graph-Contrastive Transfer Framework for Cooperative Multi-Agent Reinforcement Learning

Motivation and Background

Sample inefficiency remains a principal challenge in cooperative multi-agent reinforcement learning (MARL), especially when deploying agents across diverse tasks with variable population sizes and unit types. Conventional architectures for MARL rely on fixed input dimensions, precluding straightforward parameter transfer between environments with observation and action space mismatch. Prior transfer methods have attempted lateral adapters, curriculum learning, or permutation-invariant pooling via transformers, but typically lack explicit representation-level alignment mechanisms, leading to negative transfer and vulnerability when coordination dynamics diverge.

Graph-based representations, particularly GNNs, offer parameter shape invariance due to input permutation symmetry, making them suitable for environments with variable agent populations. The MAIL framework utilizes a multi-view graph contrastive backbone to enhance intra-task communication, but its original design does not facilitate transfer across heterogeneous tasks. "GCT-MARL: Graph-Based Contrastive Transfer for Sample-Efficient Cooperative Multi-Agent Reinforcement Learning" (2606.25073) addresses these gaps by recasting graph-contrastive learning as a transferable prior that is population and scale invariant, enabling robust transfer and continual learning.

Methodological Advancements

GCT-MARL builds upon the MAIL backbone, introducing three principal developments:

1. Population-Invariant Entity Encoder: The entity encoder decouples the input encoding from observation dimensionality by aggregating per-entity type features using masked mean pooling and shared MLP projections for each entity type. This enables structural compatibility across tasks with differing team compositions, allowing for seamless reuse of the encoder across environments sharing entity-type schemas.
2. Per-View, Adaptively-Weighted Cross-Task Alignment Loss ($\mathcal{L}_{xfer}$): Transfer is operationalized via a multi-view InfoNCE alignment loss that explicitly anchors target representations to a frozen source backbone. Alignment is performed separately for the original adjacency, feature-similarity, and higher-order topology views. Empirically, learnable weighting of these views (via softmaxed $\alpha$ parameters) allows the model to prioritize the most transferable signal, typically favoring the topological view.
3. Two-Phase Training Protocol and Continual Learning: Transfer proceeds in two stages: single-task source training followed by target task adaptation with cross-task alignment. Natural chaining of the protocol permits continual learning over task sequences, yielding empirical retention and forward transfer on sequential population-varying tasks.

Empirical Evaluation

The evaluation employs the StarCraft Multi-Agent Challenge (SMAC) benchmark across up-scaling ($3m \to 8m$), down-scaling ($8m \to 3m$), heterogeneous transfer ($3m \to 1c3s5z$), and sequential continual learning scenarios. GCT-MARL demonstrates substantial acceleration in convergence—reaching $80\%$ of final performance $2$–$3\times$ faster than both domain adaptation and MARL transfer baselines, including LA-QT and MAIL. Notably, GCT-MARL achieves perfect final win rates ($1.00$) across most maps and strong continual learning retention (final ACC $= 0.898$, avg backward transfer $\alpha$0).

Figure 1: Transfer learning performance from $\alpha$1 to $\alpha$2, with GCT-MARL exhibiting fast convergence under population up-scaling.

Figure 2: Continual learning win rates during sequential SMAC tasks ($\alpha$3), demonstrating strong retention and forward transfer.

Ablation studies reveal adaptive view weighting as essential: the learned $\alpha$4 parameters consistently converge to exclusively favor the topological view ($\alpha$5), indicating that global coordination structure is the most robust and transferable representation under distributional shift.

Figure 3: Ablation of alignment views; learnable weights outperform fixed strategies and converge to topological view dominance.

Theoretical and Practical Implications

The architectural innovations in GCT-MARL demonstrate that explicit representation-level alignment via graph contrastive learning circumvents the limitations of parameter shape mismatch in population-varying MARL. The adaptive cross-task loss allows task-specific selection of the most informative view, mitigating negative transfer. Continual learning emerges as a byproduct of the transfer protocol, without requiring anti-forgetting regularization, suggesting implicit knowledge distillation at the representation level.

The principal constraint is the requirement of a shared entity-type schema between source and target, which restricts transferability across radically heterogeneous tasks. Future research may investigate schema-agnostic encoders and generalization across diverse environments with broader entity modalities. The empirical results underscore the potential of population-invariant graph transfer mechanisms for rapid MARL deployment in complex, multi-agent domains such as traffic control, auction mechanisms, and fleet management.

Conclusion

GCT-MARL establishes a robust framework for sample-efficient cooperative MARL under population and observation mismatch, leveraging graph-contrastive learning and adaptive cross-task alignment. The approach accelerates transfer, achieves strong final performance, and exhibits stable continual learning retention. Extension to schema-agnostic entity encoders and validation on broader multi-agent environments constitute promising future directions.