HeMARL: Heterogeneous Multi-Agent Reinforcement Learning

• HeMARL is the study of multi-agent systems where agents differ in observation, action spaces, dynamics, and rewards, leading to unique coordination challenges.
• It employs specialized communication protocols, graph-based models, and embedding techniques to manage heterogeneity and facilitate effective parameter sharing.
• Optimization frameworks and benchmarks validate theoretical guarantees and practical performance improvements in heterogeneous multi-agent environments.

to=arxiv_search.search ,最新高清无码专区 北京赛车如何json code {"query":"all:(heterogeneous multi-agent reinforcement learning) OR ti:(Heterogeneous-Agent Reinforcement Learning) OR ti:(Learning Communication Between Heterogeneous Agents in Multi-Agent Reinforcement Learning for Autonomous Cyber Defence)","max_results":10,"sort_by":"submittedDate","sort_order":"descending"}Japgollyassistant to=arxiv_search.search to=arxiv_search.search ฝ่ายขายละครണം 天天中彩票足球json code {"query":"id:(Popa et al., 17 Mar 2026) OR id:(Zhong et al., 2023) OR id:(Duan et al., 17 May 2026) OR id:(Li et al., 2024) OR id:(Fu et al., 2022) OR id:(Kuba et al., 2022) OR id:(Yu et al., 2023) OR id:(Meneghetti et al., 2020) OR id:(Meneghetti et al., 2020) OR id:(Liu et al., 2022)","max_results":20,"sort_by":"relevance","sort_order":"descending"} Heterogeneous Multi-Agent Reinforcement Learning (HeMARL) studies cooperative or mixed multi-agent decision-making when agents are not interchangeable: they may differ in observation spaces, action spaces, transition dynamics, reward functions, roles, resources, sensing modalities, or communication capacities. In the recent literature, HeMARL has been treated not merely as “MARL without full parameter sharing,” but as a broader problem class encompassing agent-level heterogeneity, relation-level heterogeneity, and even environment-level heterogeneity, with corresponding consequences for representation learning, coordination, optimization, and benchmarking (Zhong et al., 2023, Hu et al., 28 Dec 2025).

1. Formal scope and problem formulations

A rigorous agent-level account models MARL as a Partially Observable Markov Game,

$$(N, \{S_i\}_{i \in N}, \{O_i\}_{i \in N}, \{A_i\}_{i \in N}, \{Q_i\}_{i \in N}, \{T_i\}_{i \in N}, \{r_i\}_{i \in N}, \gamma),$$

and classifies heterogeneity into five types: observation heterogeneity, response transition heterogeneity, effect transition heterogeneity, objective heterogeneity, and policy heterogeneity (Hu et al., 28 Dec 2025). In that framework, heterogeneity is not restricted to visible capability differences; it also includes differences in how agents affect the environment and how their policies diverge under a common task.

The same literature introduces a quantitative notion of heterogeneity distance,

$$d_{ij} = \int_{x \in X} D\big[ F_i(\cdot|x), F_j(\cdot|x) \big]\, p(x)\, dx,$$

with specialized forms for each heterogeneity type and a representation-learning-based estimation procedure using conditional variational autoencoders (Hu et al., 28 Dec 2025). This provides an explicit bridge from descriptive heterogeneity to algorithmic mechanisms such as dynamic parameter sharing.

A complementary formalization casts HeMARL as a generalized Dec-POMDP,

$$(n, S, \{A_i\}_{i=1}^n, T, \{O_i\}_{i=1}^n, O, R, \gamma),$$

where each agent can have its own action space $A_i$ and observation space $O_i$ (Dansereau et al., 23 Sep 2025). In communication-centric settings, the underlying game is often written as the Markov game tuple

$$\langle \mathcal{N}, \mathcal{O}, \mathcal{A}, \mathcal{R}, \mathcal{P}, \gamma \rangle,$$

with heterogeneous observation and action spaces embedded directly into $\mathcal{O}$ and $\mathcal{A}$ (Popa et al., 17 Mar 2026). Taken together, these formulations indicate that HeMARL is defined as much by asymmetric information structure and interaction structure as by distinct embodiments.

2. Sources of difficulty: agents, relations, and environments

The main technical difficulties in HeMARL are sharper than in homogeneous MARL because heterogeneity amplifies non-stationarity, complicates credit assignment, and weakens the assumptions behind parameter sharing. One strand of work identifies the non-stationary problem and the policy version iteration issue as especially acute in heterogeneous systems, where agents may have imbalanced numbers, distinct capabilities, and asynchronous update requirements (Fu et al., 2022). Another emphasizes partial observability and task specialization: one agent may observe only a subnet or host, while another can only execute a restricted defensive action such as “Block subnet,” making coordination structurally necessary rather than merely beneficial (Popa et al., 17 Mar 2026).

A second axis concerns environmental heterogeneity. The HECOGrid work formalizes the coordination level $c$ of an environment as the minimal number of agents required to obtain certain positive rewards at any time, and defines heterogeneity level $h$ as the number of qualitatively distinct transition regimes within the same environment (Liu et al., 2022). In that benchmark, heterogeneity is not a property of the agents alone; it is implemented by spatially dividing the gridworld into zones with different transition functions. This result matters because it decouples agent heterogeneity from environment heterogeneity and shows that the latter can independently degrade performance.

The distinction is important for interpretation. Some tasks are hard because agents differ; others are hard because the environment demands coordination or because local dynamics vary across space. A plausible implication is that “heterogeneous MARL” is not a single failure mode for standard algorithms, but a family of related failure modes arising from mismatched assumptions about symmetry, exchangeability, and stationarity.

3. Representation learning and the parameter-sharing spectrum

Early graph-based HeMARL architectures represented states as directed labeled graphs whose nodes denote agents or entities and whose edge labels encode class-pair relations (Meneghetti et al., 2020). HMAGQ-Net uses class-specific encoding functions for variable-sized feature vectors, relational graph convolutional layers to model different communication channels between entity types, and class-specific Q-networks while sharing parameters wherever possible (Meneghetti et al., 2020). The key claim is that specializing communication channels between entity classes is a promising step toward higher performance in heterogeneous settings.

A more explicit class-aware communication model represents inter-agent communication as a directed labeled heterogeneous agent graph $$d_{ij} = \int_{x \in X} D\big[ F_i(\cdot|x), F_j(\cdot|x) \big]\, p(x)\, dx,$$0, where node labels denote agent classes and relation labels encode communication types between classes (Meneghetti et al., 2020). Its R-GCN communication module learns individual transformations for each class-to-class message type, while zero padding and action masking permit full parameter sharing in the encoding and action-selection modules despite heterogeneous observation and action spaces (Meneghetti et al., 2020). On heterogeneous, class-rich SMAC scenarios, this specialized communication achieves equal or superior performance relative to attention-based or no-communication baselines.

Embedding-based approaches pursue a different form of invariance. EMAC transforms variable-length heterogeneous observations into fixed-length latent vectors and couples centralized training with decentralized execution through a shared critic and decentralized actors (Wakilpoor et al., 2020). Its triplet loss encourages same-timestep multi-view embeddings to be similar and temporally distant embeddings to be dissimilar, supporting shared policy parameters even when agents have different sensor modalities (Wakilpoor et al., 2020).

Recent work reframes parameter sharing itself as a continuum rather than a binary choice. Kaleidoscope maintains one set of common parameters together with learnable masks for different agents, and adds diversity regularization so that masks diverge structurally while retaining the sample-efficiency benefits of sharing (Li et al., 2024). It also extends the same mechanism to critic ensembles. This suggests that HeMARL increasingly treats “how much to share” as a learned object, not a fixed design decision.

4. Communication, coordination graphs, and structured cooperation

A major HeMARL theme is that communication should itself be learned under heterogeneity constraints. In autonomous cyber defence, CommFormer models communication as a learnable directed graph $$d_{ij} = \int_{x \in X} D\big[ F_i(\cdot|x), F_j(\cdot|x) \big]\, p(x)\, dx,$$1, uses a sparsity parameter $$d_{ij} = \int_{x \in X} D\big[ F_i(\cdot|x), F_j(\cdot|x) \big]\, p(x)\, dx,$$2 to control the number of active links, masks messages according to the learned graph, and outputs actions auto-regressively through an encoder-decoder transformer with self-attention (Popa et al., 17 Mar 2026). In the CybORG/CyMARL environment, CommFormer with heterogeneous capabilities converged to an optimal policy up to four times faster and improved standard error by up to $$d_{ij} = \int_{x \in X} D\big[ F_i(\cdot|x), F_j(\cdot|x) \big]\, p(x)\, dx,$$3, while also exhibiting role-based behaviors such as relaying scan detections to specialized peers (Popa et al., 17 Mar 2026).

HIBCG pushes this line toward a theoretically grounded coordination-graph formalism. It introduces a group-aware sparse graph in which both edge existence and message capacity are derived using the graph information bottleneck, with a group-aligned block-diagonal prior that yields a closed-form criterion for edge retention and decomposes the objective per group block (Duan et al., 17 May 2026). The same work proves that capacity allocation follows a water-filling principle and reports a variational bound that is $$d_{ij} = \int_{x \in X} D\big[ F_i(\cdot|x), F_j(\cdot|x) \big]\, p(x)\, dx,$$4–$$d_{ij} = \int_{x \in X} D\big[ F_i(\cdot|x), F_j(\cdot|x) \big]\, p(x)\, dx,$$5 tighter in experiments, with the largest relative improvements on heterogeneous, role-rich tasks and only approximately $$d_{ij} = \int_{x \in X} D\big[ F_i(\cdot|x), F_j(\cdot|x) \big]\, p(x)\, dx,$$6 wall-clock overhead (Duan et al., 17 May 2026).

Grouped factorization is an alternative to explicit message passing. GHQ partitions agents into groups according to transition-function properties, introduces Grouped Individual-Global-Max Consistency, and maximizes Inter-group Mutual Information between group trajectories to enhance coordination across groups (Yu et al., 2023). Its empirical case is that grouped parameterization and inter-group mutual information are both necessary on heterogeneous SMAC maps, where baseline algorithms degrade as local transition heterogeneity increases.

SAF addresses coordination and heterogeneity through training-time bottlenecks rather than explicit communication at execution. It combines a differentiable shared knowledge source with a shared pool of $$d_{ij} = \int_{x \in X} D\big[ F_i(\cdot|x), F_j(\cdot|x) \big]\, p(x)\, dx,$$7 policies, from which each agent selects dynamically at each timestep using attention and the Gumbel-Softmax trick (Liu et al., 2022). In HECOGrid, the knowledge source matters most as coordination level rises, while the shared policy pool matters most as heterogeneity rises, supporting a functional separation between coordination mechanisms and heterogeneity mechanisms (Liu et al., 2022).

Mechanism	Representative method	Core idea
Learnable sparse communication	CommFormer	Directed graph with masked transformer messages
Information-bottleneck topology learning	HIBCG	Group-aware sparse graph with per-block capacity control
Grouped value factorization	GHQ	Group-specific Q-functions with inter-group mutual information
Training-time shared bottleneck	SAF	Knowledge source plus dynamic policy-pool selection

5. Optimization frameworks and theoretical guarantees

A substantial part of modern HeMARL is theoretical: how to update heterogeneous policies without sacrificing monotonicity or convergence guarantees. HARL introduces the multi-agent advantage decomposition lemma and a sequential update scheme in which agents are updated one after another in a random order, producing HATRL as the theoretical foundation and HATRPO and HAPPO as tractable approximations (Zhong et al., 2023). The same line develops HAML, a mirror-learning template under which induced algorithms enjoy monotonic joint-return improvement and convergence to Nash equilibrium (Zhong et al., 2023).

The HAML update takes the mirror-descent form

$$d_{ij} = \int_{x \in X} D\big[ F_i(\cdot|x), F_j(\cdot|x) \big]\, p(x)\, dx,$$8

allowing each agent its own policy parameterization, update scheme, and step size (Kuba et al., 2022). This heterogeneity-aware optimization view unifies HATRPO and HAPPO and also yields HAA2C and HADDPG as natural instances (Kuba et al., 2022).

Maximum-entropy HeMARL modifies the target equilibrium. HASAC embeds cooperative MARL into a probabilistic graphical model and optimizes

$$d_{ij} = \int_{x \in X} D\big[ F_i(\cdot|x), F_j(\cdot|x) \big]\, p(x)\, dx,$$9

leading to convergence to quantal response equilibrium rather than classical Nash equilibrium (Liu et al., 2023). The broader MEHAML template extends the same monotonic-improvement and convergence guarantees to other maximum-entropy heterogeneous-agent designs (Liu et al., 2023).

League-based training targets a different pathology: compatibility across policy versions. HLT maintains a league of historical policy groups, mixes current and past policies during training, and conditions a hyper-network on agent type and mixed team composition (Fu et al., 2022). On UHMP, HLT reaches a best win rate of $$(n, S, \{A_i\}_{i=1}^n, T, \{O_i\}_{i=1}^n, O, R, \gamma),$$0 and best test reward of $$(n, S, \{A_i\}_{i=1}^n, T, \{O_i\}_{i=1}^n, O, R, \gamma),$$1, while also providing a practical way to assess the difficulty of learning each role in a heterogeneous team (Fu et al., 2022).

Scalability with large heterogeneous populations motivates graphon methods. GMFS approximates graphon-weighted mean-field interactions by subsampling $$(n, S, \{A_i\}_{i=1}^n, T, \{O_i\}_{i=1}^n, O, R, \gamma),$$2 agents according to interaction strength, obtaining sample complexity $$(n, S, \{A_i\}_{i=1}^n, T, \{O_i\}_{i=1}^n, O, R, \gamma),$$3 and an optimality gap $$(n, S, \{A_i\}_{i=1}^n, T, \{O_i\}_{i=1}^n, O, R, \gamma),$$4 (Anand et al., 18 Feb 2026). This result is explicitly posed as a remedy for the computational expense of graphon-based heterogeneous interaction models as the number of agents grows.

6. Benchmarks, applications, and recurring controversies

Benchmark construction has become a central part of HeMARL because weak heterogeneity can make disparate algorithms look deceptively similar. HECOGrid offers explicit quantitative control over coordination level $$(n, S, \{A_i\}_{i=1}^n, T, \{O_i\}_{i=1}^n, O, R, \gamma),$$5 and heterogeneity level $$(n, S, \{A_i\}_{i=1}^n, T, \{O_i\}_{i=1}^n, O, R, \gamma),$$6 across TeamTogether, TeamSupport, and KeyForTreasure, and shows that performance drops as either coordination or heterogeneity increases, with three-way coordination becoming extremely difficult for all methods (Liu et al., 2022). CH-MARL introduces a multimodal benchmark with a humanoid and a drone in multi-room home environments, where message passing via binary room vectors significantly improves decentralized performance, visual input is much harder than scene-graph input, and language feedback yields approximately $$(n, S, \{A_i\}_{i=1}^n, T, \{O_i\}_{i=1}^n, O, R, \gamma),$$7 relative improvement in success rates (Sharma et al., 2022). HeMAC is proposed to fill the lack of a standardized cooperative HeMARL testbed by providing fully heterogeneous observation, action, and capability structure through Quadcopter, Observer, and Provisioner agents (Dansereau et al., 23 Sep 2025).

Application domains are correspondingly diverse. In autonomous cyber defence, heterogeneous CommFormer agents in CybORG/CyMARL converge faster and more stably than prior baselines under partial observability and task specialization (Popa et al., 17 Mar 2026). In physical layer security for heterogeneous UAV networks, LLM-HeMARL combines an inner S2DC precoding optimizer with offline LLM policy generation, conservative Q-learning distillation, and online ISAC adaptation, reporting approximately $$(n, S, \{A_i\}_{i=1}^n, T, \{O_i\}_{i=1}^n, O, R, \gamma),$$8 improvement over a non-LLM MARL baseline together with better secrecy–energy trade-offs (Zheng et al., 23 Jul 2025). In autonomous Earth observation, MAPPO, HAPPO, and HATRPO coordinate two optical satellites and one SAR satellite under energy and memory constraints, with heterogeneous-aware methods maintaining specialization and stability under resource scarcity (Hady et al., 16 Nov 2025). In a single humanoid robot, MASH treats limbs as heterogeneous agents with a shared global critic and reports roughly $$(n, S, \{A_i\}_{i=1}^n, T, \{O_i\}_{i=1}^n, O, R, \gamma),$$9 fewer iterations to convergence than single-agent PPO, alongside improved whole-body cooperation (Liu et al., 14 Aug 2025).

The literature also contains genuine counterpoints to simple narratives about heterogeneity. One frequent claim is that parameter sharing and self-play necessarily overfit to homogeneous teammate behavior. Yet, in HeMAC, Rotating Policy Training does not outperform a standard IPPO baseline when paired with a withheld DDQN teammate; both achieve similar performance, which the study interprets as evidence that IPPO learns the game, not the team (LeRoy et al., 9 Dec 2025). Conversely, several other works report that full parameter sharing degrades under stronger heterogeneity, and that padding-based compatibility can be brittle or suboptimal (Li et al., 2024, Dansereau et al., 23 Sep 2025). The most defensible synthesis is therefore conditional rather than absolute: heterogeneity-aware structure is often necessary, but the degree and kind of structure depend on whether the dominant obstacle is agent asymmetry, environmental heterogeneity, teammate variation, or communication topology.