---
title: 'HeMARL: Heterogeneous Multi-Agent Reinforcement Learning'
url: https://www.emergentmind.com/topics/heterogeneous-multi-agent-reinforcement-learning-hemarl
type: topic
---

# HeMARL: Heterogeneous Multi-Agent Reinforcement Learning

to=arxiv_search.search ,最新高清无码专区  北京赛车如何json code
{"query":"all:(heterogeneous multi-agent reinforcement learning) OR ti:(Heterogeneous-Agent Reinforcement Learning) OR ti:(Learning Communication Between Heterogeneous Agents in Multi-Agent Reinforcement Learning for Autonomous Cyber Defence)","max_results":10,"sort_by":"submittedDate","sort_order":"descending"}Japgollyassistant to=arxiv_search.search
to=arxiv_search.search  ฝ่ายขายละครണം  天天中彩票足球json code
{"query":"id:2603.20279 OR id:2304.09870 OR id:2605.17393 OR id:2410.08540 OR id:2211.11616 OR id:2208.01682 OR id:2303.01070 OR id:2012.07617 OR id:2009.13161 OR id:2210.03022","max_results":20,"sort_by":"relevance","sort_order":"descending"}
Heterogeneous Multi-Agent Reinforcement Learning (HeMARL) studies cooperative or mixed multi-agent decision-making when agents are not interchangeable: they may differ in observation spaces, action spaces, transition dynamics, reward functions, roles, resources, sensing modalities, or communication capacities. In the recent literature, HeMARL has been treated not merely as “MARL without full parameter sharing,” but as a broader problem class encompassing agent-level heterogeneity, relation-level heterogeneity, and even environment-level heterogeneity, with corresponding consequences for representation learning, coordination, optimization, and benchmarking [2304.09870] [2512.22941].

## 1. Formal scope and problem formulations

A rigorous agent-level account models MARL as a Partially Observable Markov Game,
$$
(N, \{S_i\}_{i \in N}, \{O_i\}_{i \in N}, \{A_i\}_{i \in N}, \{Q_i\}_{i \in N}, \{T_i\}_{i \in N}, \{r_i\}_{i \in N}, \gamma),
$$
and classifies heterogeneity into five types: observation heterogeneity, response transition heterogeneity, effect transition heterogeneity, objective heterogeneity, and policy heterogeneity [2512.22941]. In that framework, heterogeneity is not restricted to visible capability differences; it also includes differences in how agents affect the environment and how their policies diverge under a common task.

The same literature introduces a quantitative notion of heterogeneity distance,
$$
d_{ij} = \int_{x \in X} D\big[ F_i(\cdot|x), F_j(\cdot|x) \big]\, p(x)\, dx,
$$
with specialized forms for each heterogeneity type and a representation-learning-based estimation procedure using conditional variational autoencoders [2512.22941]. This provides an explicit bridge from descriptive heterogeneity to algorithmic mechanisms such as dynamic parameter sharing.

A complementary formalization casts HeMARL as a generalized Dec-POMDP,
$$
(n, S, \{A_i\}_{i=1}^n, T, \{O_i\}_{i=1}^n, O, R, \gamma),
$$
where each agent can have its own action space $A_i$ and observation space $O_i$ [2509.19512]. In communication-centric settings, the underlying game is often written as the Markov game tuple
$$
\langle \mathcal{N}, \mathcal{O}, \mathcal{A}, \mathcal{R}, \mathcal{P}, \gamma \rangle,
$$
with heterogeneous observation and action spaces embedded directly into $\mathcal{O}$ and $\mathcal{A}$ [2603.20279]. Taken together, these formulations indicate that HeMARL is defined as much by asymmetric information structure and interaction structure as by distinct embodiments.

## 2. Sources of difficulty: agents, relations, and environments

The main technical difficulties in HeMARL are sharper than in homogeneous MARL because heterogeneity amplifies non-stationarity, complicates credit assignment, and weakens the assumptions behind parameter sharing. One strand of work identifies the non-stationary problem and the policy version iteration issue as especially acute in heterogeneous systems, where agents may have imbalanced numbers, distinct capabilities, and asynchronous update requirements [2211.11616]. Another emphasizes partial observability and task specialization: one agent may observe only a subnet or host, while another can only execute a restricted defensive action such as “Block subnet,” making coordination structurally necessary rather than merely beneficial [2603.20279].

A second axis concerns environmental heterogeneity. The HECOGrid work formalizes the coordination level $c$ of an environment as the minimal number of agents required to obtain certain positive rewards at any time, and defines heterogeneity level $h$ as the number of qualitatively distinct transition regimes within the same environment [2210.03022]. In that benchmark, heterogeneity is not a property of the agents alone; it is implemented by spatially dividing the gridworld into zones with different transition functions. This result matters because it decouples agent heterogeneity from environment heterogeneity and shows that the latter can independently degrade performance.

The distinction is important for interpretation. Some tasks are hard because agents differ; others are hard because the environment demands coordination or because local dynamics vary across space. A plausible implication is that “heterogeneous MARL” is not a single failure mode for standard algorithms, but a family of related failure modes arising from mismatched assumptions about symmetry, exchangeability, and stationarity.

## 3. Representation learning and the parameter-sharing spectrum

Early graph-based HeMARL architectures represented states as directed labeled graphs whose nodes denote agents or entities and whose edge labels encode class-pair relations [2009.13161]. HMAGQ-Net uses class-specific encoding functions for variable-sized feature vectors, relational graph convolutional layers to model different communication channels between entity types, and class-specific Q-networks while sharing parameters wherever possible [2009.13161]. The key claim is that specializing communication channels between entity classes is a promising step toward higher performance in heterogeneous settings.

A more explicit class-aware communication model represents inter-agent communication as a directed labeled heterogeneous agent graph $G=(V,E,C,R)$, where node labels denote agent classes and relation labels encode communication types between classes [2012.07617]. Its R-GCN communication module learns individual transformations for each class-to-class message type, while zero padding and action masking permit full parameter sharing in the encoding and action-selection modules despite heterogeneous observation and action spaces [2012.07617]. On heterogeneous, class-rich SMAC scenarios, this specialized communication achieves equal or superior performance relative to attention-based or no-communication baselines.

Embedding-based approaches pursue a different form of invariance. EMAC transforms variable-length heterogeneous observations into fixed-length latent vectors and couples centralized training with decentralized execution through a shared critic and decentralized actors [2010.02663]. Its triplet loss encourages same-timestep multi-view embeddings to be similar and temporally distant embeddings to be dissimilar, supporting shared policy parameters even when agents have different sensor modalities [2010.02663].

Recent work reframes parameter sharing itself as a continuum rather than a binary choice. Kaleidoscope maintains one set of common parameters together with learnable masks for different agents, and adds diversity regularization so that masks diverge structurally while retaining the sample-efficiency benefits of sharing [2410.08540]. It also extends the same mechanism to critic ensembles. This suggests that HeMARL increasingly treats “how much to share” as a learned object, not a fixed design decision.

## 4. Communication, coordination graphs, and structured cooperation

A major HeMARL theme is that communication should itself be learned under heterogeneity constraints. In autonomous cyber defence, CommFormer models communication as a learnable directed graph $\mathcal{G}=\langle \mathcal{V}, \mathcal{E} \rangle$, uses a sparsity parameter $\mathcal{S}$ to control the number of active links, masks messages according to the learned graph, and outputs actions auto-regressively through an encoder-decoder transformer with self-attention [2603.20279]. In the CybORG/CyMARL environment, CommFormer with heterogeneous capabilities converged to an optimal policy up to four times faster and improved standard error by up to $38\%$, while also exhibiting role-based behaviors such as relaying scan detections to specialized peers [2603.20279].

HIBCG pushes this line toward a theoretically grounded coordination-graph formalism. It introduces a group-aware sparse graph in which both edge existence and message capacity are derived using the graph information bottleneck, with a group-aligned block-diagonal prior that yields a closed-form criterion for edge retention and decomposes the objective per group block [2605.17393]. The same work proves that capacity allocation follows a water-filling principle and reports a variational bound that is $2.9\times$–$6.5\times$ tighter in experiments, with the largest relative improvements on heterogeneous, role-rich tasks and only approximately $4\%$ wall-clock overhead [2605.17393].

Grouped factorization is an alternative to explicit message passing. GHQ partitions agents into groups according to transition-function properties, introduces Grouped Individual-Global-Max Consistency, and maximizes Inter-group Mutual Information between group trajectories to enhance coordination across groups [2303.01070]. Its empirical case is that grouped parameterization and inter-group mutual information are both necessary on heterogeneous SMAC maps, where baseline algorithms degrade as local transition heterogeneity increases.

SAF addresses coordination and heterogeneity through training-time bottlenecks rather than explicit communication at execution. It combines a differentiable shared knowledge source with a shared pool of $U$ policies, from which each agent selects dynamically at each timestep using attention and the Gumbel-Softmax trick [2210.03022]. In HECOGrid, the knowledge source matters most as coordination level rises, while the shared policy pool matters most as heterogeneity rises, supporting a functional separation between coordination mechanisms and heterogeneity mechanisms [2210.03022].

| Mechanism | Representative method | Core idea |
|---|---|---|
| Learnable sparse communication | CommFormer | Directed graph with masked transformer messages |
| Information-bottleneck topology learning | HIBCG | Group-aware sparse graph with per-block capacity control |
| Grouped value factorization | GHQ | Group-specific Q-functions with inter-group mutual information |
| Training-time shared bottleneck | SAF | Knowledge source plus dynamic policy-pool selection |

## 5. Optimization frameworks and theoretical guarantees

A substantial part of modern HeMARL is theoretical: how to update heterogeneous policies without sacrificing monotonicity or convergence guarantees. HARL introduces the multi-agent advantage decomposition lemma and a sequential update scheme in which agents are updated one after another in a random order, producing HATRL as the theoretical foundation and HATRPO and HAPPO as tractable approximations [2304.09870]. The same line develops HAML, a mirror-learning template under which induced algorithms enjoy monotonic joint-return improvement and convergence to Nash equilibrium [2304.09870].

The HAML update takes the mirror-descent form
$$
\pi_i^{t+1} = \arg\max_{\pi_i} \left[ \mathbb{E}_{\pi_i, \pi_{-i}^t}\left[ A^{i}_{\pi^t}(s, a) \right] - \frac{1}{\alpha_i} D(\pi_i \| \pi_i^t) \right],
$$
allowing each agent its own policy parameterization, update scheme, and step size [2208.01682]. This heterogeneity-aware optimization view unifies HATRPO and HAPPO and also yields HAA2C and HADDPG as natural instances [2208.01682].

Maximum-entropy HeMARL modifies the target equilibrium. HASAC embeds cooperative MARL into a probabilistic graphical model and optimizes
$$
J(\boldsymbol{\pi}) = \mathbb{E}\left[ \sum_{t=1}^T \Big( r(s_t, \mathbf{a}_t) + \alpha \sum_{i=1}^n \mathcal{H}(\pi^i(\cdot | s_t)) \Big) \right],
$$
leading to convergence to quantal response equilibrium rather than classical Nash equilibrium [2306.10715]. The broader MEHAML template extends the same monotonic-improvement and convergence guarantees to other maximum-entropy heterogeneous-agent designs [2306.10715].

League-based training targets a different pathology: compatibility across policy versions. HLT maintains a league of historical policy groups, mixes current and past policies during training, and conditions a hyper-network on agent type and mixed team composition [2211.11616]. On UHMP, HLT reaches a best win rate of $98.09\%$ and best test reward of $1.782$, while also providing a practical way to assess the difficulty of learning each role in a heterogeneous team [2211.11616].

Scalability with large heterogeneous populations motivates graphon methods. GMFS approximates graphon-weighted mean-field interactions by subsampling $\kappa$ agents according to interaction strength, obtaining sample complexity $\mathrm{poly}(\kappa)$ and an optimality gap $O(1/\sqrt{\kappa})$ [2602.16196]. This result is explicitly posed as a remedy for the computational expense of graphon-based heterogeneous interaction models as the number of agents grows.

## 6. Benchmarks, applications, and recurring controversies

Benchmark construction has become a central part of HeMARL because weak heterogeneity can make disparate algorithms look deceptively similar. HECOGrid offers explicit quantitative control over coordination level $c$ and heterogeneity level $h$ across TeamTogether, TeamSupport, and KeyForTreasure, and shows that performance drops as either coordination or heterogeneity increases, with three-way coordination becoming extremely difficult for all methods [2210.03022]. CH-MARL introduces a multimodal benchmark with a humanoid and a drone in multi-room home environments, where message passing via binary room vectors significantly improves decentralized performance, visual input is much harder than scene-graph input, and language feedback yields approximately $10\%$ relative improvement in success rates [2208.13626]. HeMAC is proposed to fill the lack of a standardized cooperative HeMARL testbed by providing fully heterogeneous observation, action, and capability structure through Quadcopter, Observer, and Provisioner agents [2509.19512].

Application domains are correspondingly diverse. In autonomous cyber defence, heterogeneous CommFormer agents in CybORG/CyMARL converge faster and more stably than prior baselines under partial observability and task specialization [2603.20279]. In physical layer security for heterogeneous UAV networks, LLM-HeMARL combines an inner S2DC precoding optimizer with offline LLM policy generation, conservative Q-learning distillation, and online ISAC adaptation, reporting approximately $25\%$ improvement over a non-LLM MARL baseline together with better secrecy–energy trade-offs [2507.17188]. In autonomous Earth observation, MAPPO, HAPPO, and HATRPO coordinate two optical satellites and one SAR satellite under energy and memory constraints, with heterogeneous-aware methods maintaining specialization and stability under resource scarcity [2511.12792]. In a single humanoid robot, MASH treats limbs as heterogeneous agents with a shared global critic and reports roughly $40\%$ fewer iterations to convergence than single-agent PPO, alongside improved whole-body cooperation [2508.10423].

The literature also contains genuine counterpoints to simple narratives about heterogeneity. One frequent claim is that parameter sharing and self-play necessarily overfit to homogeneous teammate behavior. Yet, in HeMAC, Rotating Policy Training does not outperform a standard IPPO baseline when paired with a withheld DDQN teammate; both achieve similar performance, which the study interprets as evidence that IPPO learns the game, not the team [2512.08877]. Conversely, several other works report that full parameter sharing degrades under stronger heterogeneity, and that padding-based compatibility can be brittle or suboptimal [2410.08540] [2509.19512]. The most defensible synthesis is therefore conditional rather than absolute: heterogeneity-aware structure is often necessary, but the degree and kind of structure depend on whether the dominant obstacle is agent asymmetry, environmental heterogeneity, teammate variation, or communication topology.

Source: https://www.emergentmind.com/topics/heterogeneous-multi-agent-reinforcement-learning-hemarl