GNN-Based Actor-Critic RL

• Graph Neural Network-based Actor-Critic RL is an approach that integrates graph-structured representations into actor-critic models, enabling flexible handling of variable entity interactions.
• It employs hypergraph and message-passing architectures to facilitate local and global relational reasoning in applications like multi-agent coordination and autonomous driving.
• The method achieves permutation invariance and decentralized execution for robust performance, though it demands careful tuning of hyperparameters and management of computational costs.

Graph neural network (GNN)-based actor-critic reinforcement learning (RL) integrates graph-based relational inductive biases directly into the architecture of actor-critic RL algorithms. This paradigm has been applied in both single-agent and multi-agent settings, encompassing scenarios with combinatorial underlying structure, semantic object-centric environments, and decentralized cooperation domains. By embedding flexible, permutation-invariant, and message-passing representations in the policy and/or value function approximators, GNN-enhanced actor-critic methods are able to exploit relational information in environments where entities naturally interact via graphs or hypergraphs.

1. Problem Formulation and Rationale

Traditional RL architectures rely on fixed-size, vectorial state representations, which are suboptimal for environments with a variable or set-based composition (e.g., varying numbers of agents, objects, or vehicles) or those requiring explicit modeling of relational dependencies. In contrast, GNN-based actor-critic RL addresses these challenges by:

• Using graph-structured inputs where nodes are agents or entities and edges (or hyperedges) encode interactions, spatial proximity, or higher-order coordination.
• Achieving invariance to the number and order of entities, which is crucial for multi-agent systems or object-centric perception.
• Propagating and aggregating neighborhood information to enable both local and global relational reasoning.

These advances are motivated by a variety of application domains, ranging from multi-agent coordination under partial observability (Zhang et al., 2022, Ryu et al., 2019) to autonomous driving in semantic environments (Hart et al., 2020) and combinatorial optimization on graphs (Khakhulin et al., 2019).

2. GNN Architectures for Actor-Critic RL

Hypergraph Neural Critic and Convolutional Architectures

The most general class extends traditional GNNs to hypergraph neural networks for the critic module. For instance, in multi-agent coordination tasks, a hypergraph $\mathcal{G}=(\mathcal{V},\mathcal{E})$ is constructed where vertices are agents and hyperedges represent indecomposable neighborhoods for coordination. The hypergraph is algebraically represented by an incidence matrix $H\in\mathbb{R}^{N\times M}$, with $H_{ie}$ denoting “membership strength” of agent $i$ in hyperedge $\epsilon_e$. The hypergraph convolution is implemented as:

$X' = D_v^{-1/2} H W D_e^{-1} H^T D_v^{-1/2} X P$

where $X$ is the input feature matrix, $W$ is a learnable diagonal matrix of hyperedge weights, $D_v$ and $D_e$ are degree matrices, and $H\in\mathbb{R}^{N\times M}$0 is a linear projection (Zhang et al., 2022).

Graph Convolutional and Attention-based Layers

Alternatively, message-passing neural networks—such as GCNs or graph attention networks—are used in both actor and critic. The node and edge updates can be formalized as:

• Edge update: $H\in\mathbb{R}^{N\times M}$1.
• Aggregation: $H\in\mathbb{R}^{N\times M}$2.
• Node update: $H\in\mathbb{R}^{N\times M}$3,

with $H\in\mathbb{R}^{N\times M}$4, $H\in\mathbb{R}^{N\times M}$5 learnable multilayer perceptrons. Stack $H\in\mathbb{R}^{N\times M}$6 message-passing iterations to allow information diffusion across $H\in\mathbb{R}^{N\times M}$7 hops (Hart et al., 2020). Graph attention variants further weight neighbors or groups by learned attention coefficients (Ryu et al., 2019).

3. Actor-Critic Integration and Optimization

Critic Networks

Critic networks ($H\in\mathbb{R}^{N\times M}$8-functions or value networks) ingest GNN-processed joint observations (or local plus aggregated information), enabling relational value estimation. For the hypergraph critic (HGAC, ATT-HGAC), the Q-function per agent is:

$H\in\mathbb{R}^{N\times M}$9

with $H_{ie}$0 the input feature and $H_{ie}$1 the hypergraph-processed embedding from each head. The critic loss aggregates temporal-difference errors across agents (Zhang et al., 2022).

Value heads in single-agent RL are constructed by pooling the ego-node GNN embedding and regressing to the scalar value estimate $H_{ie}$2 (Hart et al., 2020).

Actor Networks

Policy networks leverage GNN-encoded agent-centric embeddings for action selection. For continuous actions, policy heads produce mean and variance for parameterized distributions; for discrete spaces, a Gumbel-softmax or categorical distribution is used. The actor loss is computed via policy gradient:

$H_{ie}$3

where $H_{ie}$4 is a counterfactual advantage function, quantifying the marginal value of an agent’s action compared to its policy mean (Zhang et al., 2022).

In the hierarchical multi-agent setups, actor and critic share or duplicate the graph-embedding pipeline but differ in the heads, promoting separation of local and global value estimation (Ryu et al., 2019).

Training Protocols

Nearly all variants adopt a centralized-training, decentralized-execution (CTDE) framework: shared critics are trained on global (joint) trajectories, but actors execute policies from local observations. Training involves experience replay (off-policy) or on-policy (e.g., PPO) updates, soft target-network synchronization, and, where relevant, additional mechanisms such as entropy regularization or generalized advantage estimation (Khakhulin et al., 2019).

A representative training step for HGAC/ATT-HGAC consists of:

1. Collecting transitions $H_{ie}$5 via interaction.
2. Building dynamic hypergraph or attention graphs per sample.
3. Minibatch gradient descent for critic and actor parameters.
4. Soft update of target networks (Zhang et al., 2022).

4. Specialized Architectures: Hierarchies, Attention, and Inductive Transfer

Hierarchical Graph Attention Networks (HGAT)

Hierarchical architectures extend GNN-based RL by introducing multi-level attention: agents are grouped a priori, with intra-group attention learning fine coordination and inter-group attention enabling strategic focus allocation. Two-stage attention produces, for each agent $H_{ie}$6, hierarchical embeddings for use in actor and critic. This facilitates transfer to environments with different numbers or groupings of agents (Ryu et al., 2019).

Invariant and Adaptive Representations

Across all systems, GNN layers deliver permutation invariance to agent ordering and cardinality—key for transferability and scalability. GNNs encode relational information explicitly, as opposed to inferring them from concatenated state vectors. Experiments confirm robust performance when the number and composition of entities changes between train and test (Hart et al., 2020, Ryu et al., 2019).

Combinatorial and Object-centric Domains

GCN-based actor-critic RL has been applied to purely combinatorial problems, such as tree decomposition, where the environment state is a graph and actions are structured (node eliminations) (Khakhulin et al., 2019). Here, the policy and value functions operate over node-level GCN embeddings, with mean-pooling and independent heads for stochastic policy and value estimation.

5. Empirical Results and Comparative Performance

A cross-section of experiments demonstrates the practical benefits of GNN-based actor-critic RL:

• Cooperation in multi-agent settings: Hypergraph convolutional critics outperform conventional (MADDPG, MAAC) and basic attention-based baselines in particle environments by achieving faster convergence and higher asymptotic returns, particularly in scenarios with higher-order or dynamic neighborhoods (Zhang et al., 2022).
• Semantic driving environments: GNN-based policies for autonomous lane change maintain superiority under variations in the number and arrangement of vehicles, proving robust to randomized observation ordering and noisy input (Hart et al., 2020).
• Transfer and scalability: Hierarchical attention architectures retain performance when deployed in scenarios with significantly more agents than seen during training, due to the inductive bias of their attention-pooling mechanisms (Ryu et al., 2019).
• Combinatorial optimization: A GCN-actor-critic agent trained on small graphs generalizes to graphs with an order of magnitude more nodes, achieving lower approximation ratios compared to classical greedy heuristics (Khakhulin et al., 2019).

Summary results from key domains:

Domain	Architecture	Key Comparative Result
Multi-agent cooperation	HGAC / ATT-HGAC	Higher/faster return than MADDPG/MASAC/MAAC in CN, CTC, RT
Lane-change scenario	GNN+Actor-Critic	Lower collision rate under noise/randomization than conv. NN
Mix cooperative/competitive	HAMA (HGAT + AC)	Superior final performance, scalable transfer to large $H_{ie}$7
Tree decomposition	GCN-Actor-Critic	Improves on greedy/heuristic AR; scales to 1000-node graphs

6. Limitations and Open Challenges

GNN-based actor-critic RL offers substantial gains but is subject to practical trade-offs:

• Graph construction hyperparameters: Performance depends on neighborhood-radius, number of neighbors, hyperedge design, or group assignment. Poor choices may reduce effectiveness (Hart et al., 2020, Zhang et al., 2022).
• Computational cost: Message-passing and hypergraph convolutions induce nontrivial per-step computation compared to pure MLP baselines, which may limit throughput in high-frequency control or large-scale deployments (Hart et al., 2020).
• Design of relational priors: The selection between pairwise, hypergraph, or hierarchical relation structure is domain-specific and can impact both sample complexity and transfer (Zhang et al., 2022, Ryu et al., 2019).
• Scalability to ultra-large graphs: While GNNs help scale policies to variable-size problems, very large entity sets may require sparse updates or sampling for tractability.

7. Conclusions and Prospective Directions

The integration of graph (and hypergraph) neural networks into actor-critic RL frameworks provides a principled approach for leveraging relational structure inherent to multi-agent, semantic, and combinatorial environments. This approach enables:

• Permutation-invariant and scalable policy/value estimation.
• Robustness to noise and changes in environment composition.
• Expressive coordination and information aggregation strategies tractable for both local and distributed learning.

A plausible implication is that future research will further investigate adaptive graph generation (including object-centric unstructured perception), improved scalability (through sparse or sampled GNN updates), and meta-learning strategies to further enhance generalization across a broader spectrum of relational RL domains (Zhang et al., 2022, Hart et al., 2020, Ryu et al., 2019, Khakhulin et al., 2019).