Action Graph Policies in Multi-Agent RL

• Action Graph Policies (AGP) are graph-structured mechanisms that encode action dependencies, enabling coordinated decision-making in decentralized multi-agent RL.
• They construct a global action graph over agent–action pairs and perform attention-based message passing to capture higher-order interactions and enforce global constraints.
• AGP expands the decentralized policy class beyond independent actions, yielding theoretical and empirical improvements in tasks with constraints like cardinality and synchronization.

Action Graph Policies (AGP) denote graph-structured policy mechanisms in which action selection is conditioned on an explicit graph encoding dependencies relevant to control. In the cooperative multi-agent reinforcement learning formulation that most directly defines the term, AGP builds a global action graph over agent–action pairs, performs attention-based message passing on that graph, and conditions each agent’s local policy on a permutation-invariant coordination context, thereby enlarging the decentralized executable policy class beyond fully independent policies (Gupta et al., 19 Feb 2026). Closely related arXiv usage extends the label, or an equivalent graph-policy interpretation, to directed acyclic action-dependency graphs in coordinated MARL (Ding et al., 1 Jun 2025), global state-transition graphs for long-horizon agentic RL (Wang et al., 22 Jun 2026), graph policies over variable robot morphologies and variable action spaces (Gallien, 16 Aug 2025), and action-centric ranking policies in relational planning (Mangannavar et al., 2024).

1. Terminological scope and core idea

In recent literature, AGP does not denote a single invariant architecture. Rather, it names a class of policy constructions in which the policy is mediated by a graph whose nodes and edges encode the structure of action selection. The graph may be built over concrete agent–action choices, parent–child action dependencies among agents, aggregated state groups connected by action-labeled transitions, kinematic components of a robot, or action schemas and their argument bindings. This suggests that the unifying principle is not a specific neural block, but the relocation of policy structure from a flat per-agent or per-dimension mapping into an explicit graph over action-relevant entities.

Formulation	Graph object	Policy mechanism
AGP in Dec-POMDP MARL	Agent–action pairs	Local policy conditioned on coordination context
ADG as AGP	DAG over agents	$\prod_i \pi_i(a_i \mid s, a_{Pa(i)})$
AGP in G2PO	Global state-transition graph	Edge-selection policy $\pi_\theta(e \mid v)$
Graph policies for variable morphologies	Kinematic graph	Per-node actions concatenated into full action vector
GABAR-style AGP	Action-centric planning graph	Action ranking over grounded actions

The most explicit AGP definition appears in cooperative Dec-POMDPs where decentralized execution is preserved, but global action dependencies are injected into each local policy through a shared graph computation. The motivation is that decentralized systems often fail not because they cannot learn good individual actions, but because independently executed policies cannot enforce global coordination constraints on the joint action, including cardinality constraints, exclusivity, and synchronization patterns (Gupta et al., 19 Feb 2026).

A distinct but mathematically adjacent line formulates the policy as a directed acyclic action dependency graph (ADG), where agent $i$ conditions on the actions of parent agents $Pa(i)$. There, the joint policy factorizes as

$$\pi(a \mid s) = \prod_{i=1}^n \pi_i(a_i \mid s, a_{Pa(i)}),$$

and sparsity of $Pa(i)$ is central to scalability (Ding et al., 1 Jun 2025).

2. Canonical AGP in decentralized multi-agent reinforcement learning

The canonical AGP setting is a cooperative Dec-POMDP

$$\mathcal{M} = \langle \mathcal{I}, \mathcal{S}, \{\mathcal{A}_i\}_{i\in\mathcal{I}}, \mathcal{T}, \mathcal{R}, \{\mathcal{O}_i\}_{i\in\mathcal{I}}, \Omega, \gamma \rangle,$$

with agent set $\mathcal{I}=\{1,\dots,N\}$, discrete per-agent action spaces $\mathcal{A}_i$, joint action space $\mathcal{A}=\prod_{i=1}^N \mathcal{A}_i$, shared reward, partial observability, and decentralized execution under CTDE. The objective is

$\pi_\theta(e \mid v)$0

In most decentralized MARL, the executable joint policy is restricted to the independent factorization

$\pi_\theta(e \mid v)$1

so the policy class is

$\pi_\theta(e \mid v)$2

AGP targets this policy-side bottleneck directly (Gupta et al., 19 Feb 2026).

The architecture constructs a global action-dependency graph

$\pi_\theta(e \mid v)$3

whose nodes represent concrete agent–action choices $\pi_\theta(e \mid v)$4. Each node uses an action-centric embedding

$\pi_\theta(e \mid v)$5

where $\pi_\theta(e \mid v)$6 is a shared encoder and may optionally include recurrent history $\pi_\theta(e \mid v)$7. AGP then performs attention-based propagation across all action nodes:

$\pi_\theta(e \mid v)$8

with learned attention weights normalized over $\pi_\theta(e \mid v)$9. Stacking $i$0 layers captures higher-order dependencies.

For agent $i$1, the action-node representations are aggregated into a permutation-invariant coordination context

$i$2

for example masked mean pooling over available actions. The local decision rule then conditions on both the local observation and the coordination context. In the value-based instantiation,

$i$3

and decentralized execution uses $i$4-greedy action selection. The induced joint policy factorizes conditionally as

$i$5

where the dependencies arise through the shared construction of the $i$6 before any agent commits to an action.

The key claim is architectural rather than merely algorithmic. Value-decomposition methods such as VDN and QMIX improve value-side expressiveness but typically preserve independent greedy execution. AGP instead enlarges the decentralized executable policy class itself, so global constraints are enforced via policy structure rather than explicit joint action inference at execution time.

3. Expressivity, value–policy mismatch, and optimality

A central theoretical claim is that independent decentralized execution is not universally optimal. In the canonical witness problem with binary actions and an “exactly $i$7 agents act” constraint, if each agent acts independently with $i$8, the success probability is

$i$9

maximized at $Pa(i)$0 yet strictly less than $Pa(i)$1. The stated example is $Pa(i)$2, $Pa(i)$3, where the maximum is $Pa(i)$4. Independent execution therefore cannot concentrate probability only on valid joint configurations (Gupta et al., 19 Feb 2026).

The formal results in the Dec-POMDP AGP paper proceed in four steps. First, there exist cooperative Dec-POMDPs with deterministic optimal joint policies $Pa(i)$5 such that $Pa(i)$6. Second, value–policy mismatch persists even when a centralized joint action-value function $Pa(i)$7 admits an exact value-decomposition representation: the optimal joint action $Pa(i)$8 still may not be realizable by decentralized greedy execution under independent policies. Third, if $Pa(i)$9 denotes the class induced by AGP, then

$$\pi(a \mid s) = \prod_{i=1}^n \pi_i(a_i \mid s, a_{Pa(i)}),$$0

Fourth, for any centralized policy $$\pi(a \mid s) = \prod_{i=1}^n \pi_i(a_i \mid s, a_{Pa(i)}),$$1 with full support, the best achievable forward-KL approximation error of AGP is no worse than that of independent policies, and can be strictly better:

$$\pi(a \mid s) = \prod_{i=1}^n \pi_i(a_i \mid s, a_{Pa(i)}),$$2

with instances satisfying $$\pi(a \mid s) = \prod_{i=1}^n \pi_i(a_i \mid s, a_{Pa(i)}),$$3 (Gupta et al., 19 Feb 2026).

A separate theory line establishes global optimality guarantees for graph-structured action-dependent policies in Markov games structured by coordination graphs. There, the action dependency graph $$\pi(a \mid s) = \prod_{i=1}^n \pi_i(a_i \mid s, a_{Pa(i)}),$$4 is a DAG, and the policy again factorizes as

$$\pi(a \mid s) = \prod_{i=1}^n \pi_i(a_i \mid s, a_{Pa(i)}),$$5

If $$\pi(a \mid s) = \prod_{i=1}^n \pi_i(a_i \mid s, a_{Pa(i)}),$$6 is a coordination graph for $$\pi(a \mid s) = \prod_{i=1}^n \pi_i(a_i \mid s, a_{Pa(i)}),$$7 and the compatibility condition

$$\pi(a \mid s) = \prod_{i=1}^n \pi_i(a_i \mid s, a_{Pa(i)}),$$8

holds for a topological order, then any $$\pi(a \mid s) = \prod_{i=1}^n \pi_i(a_i \mid s, a_{Pa(i)}),$$9-locally optimal policy is globally optimal, that is, $Pa(i)$0 (Ding et al., 1 Jun 2025). This result is different in scope from the Dec-POMDP expressivity results. The former is a sufficient condition for global optimality under coordination-graph structure; the latter shows strict enlargement of the decentralized executable policy class and resolution of the value–policy mismatch.

These results support a common misconception correction: stronger centralized critics or richer value decompositions do not, by themselves, remove policy-side coordination limits. Theoretical gains arise when the executable policy itself is allowed to encode action dependencies.

4. Training, execution, and empirical behavior in MARL

Value-based AGP uses a centralized TD loss with decentralized execution. For a batch of transitions $Pa(i)$1, the action graph is built from current observations, the coordination contexts $Pa(i)$2 are computed, and the objective minimizes

$Pa(i)$3

with targets

$Pa(i)$4

CTDE uses a centralized replay buffer, shared optimization with Adam at learning rate $Pa(i)$5, and periodic target-network updates, while execution remains decentralized and parallel. A policy-gradient variant, AGP-PG, directly parameterizes $Pa(i)$6 and uses centralized advantage estimation in a PPO-style scheme with the same action-graph encoder (Gupta et al., 19 Feb 2026).

The stated decentralized execution procedure is: construct the global action-node set $Pa(i)$7; compute node embeddings $Pa(i)$8; run $Pa(i)$9 layers of multi-head graph attention; pool per-agent action-node representations into $$\mathcal{M} = \langle \mathcal{I}, \mathcal{S}, \{\mathcal{A}_i\}_{i\in\mathcal{I}}, \mathcal{T}, \mathcal{R}, \{\mathcal{O}_i\}_{i\in\mathcal{I}}, \Omega, \gamma \rangle,$$0; and select $$\mathcal{M} = \langle \mathcal{I}, \mathcal{S}, \{\mathcal{A}_i\}_{i\in\mathcal{I}}, \mathcal{T}, \mathcal{R}, \{\mathcal{O}_i\}_{i\in\mathcal{I}}, \Omega, \gamma \rangle,$$1 via masked $$\mathcal{M} = \langle \mathcal{I}, \mathcal{S}, \{\mathcal{A}_i\}_{i\in\mathcal{I}}, \mathcal{T}, \mathcal{R}, \{\mathcal{O}_i\}_{i\in\mathcal{I}}, \Omega, \gamma \rangle,$$2-greedy maximization of $$\mathcal{M} = \langle \mathcal{I}, \mathcal{S}, \{\mathcal{A}_i\}_{i\in\mathcal{I}}, \mathcal{T}, \mathcal{R}, \{\mathcal{O}_i\}_{i\in\mathcal{I}}, \Omega, \gamma \rangle,$$3. The worst-case per-step attention cost is

$$\mathcal{M} = \langle \mathcal{I}, \mathcal{S}, \{\mathcal{A}_i\}_{i\in\mathcal{I}}, \mathcal{T}, \mathcal{R}, \{\mathcal{O}_i\}_{i\in\mathcal{I}}, \Omega, \gamma \rangle,$$4

while pooling and local heads are $$\mathcal{M} = \langle \mathcal{I}, \mathcal{S}, \{\mathcal{A}_i\}_{i\in\mathcal{I}}, \mathcal{T}, \mathcal{R}, \{\mathcal{O}_i\}_{i\in\mathcal{I}}, \Omega, \gamma \rangle,$$5. The method is fully parallelizable and avoids combinatorial joint maximization at test time.

The implementation details reported for the main MARL architecture are a shared encoder $$\mathcal{M} = \langle \mathcal{I}, \mathcal{S}, \{\mathcal{A}_i\}_{i\in\mathcal{I}}, \mathcal{T}, \mathcal{R}, \{\mathcal{O}_i\}_{i\in\mathcal{I}}, \Omega, \gamma \rangle,$$6 as a two-layer MLP with ReLU and hidden dimension approximately $$\mathcal{M} = \langle \mathcal{I}, \mathcal{S}, \{\mathcal{A}_i\}_{i\in\mathcal{I}}, \mathcal{T}, \mathcal{R}, \{\mathcal{O}_i\}_{i\in\mathcal{I}}, \Omega, \gamma \rangle,$$7, graph attention with $$\mathcal{M} = \langle \mathcal{I}, \mathcal{S}, \{\mathcal{A}_i\}_{i\in\mathcal{I}}, \mathcal{T}, \mathcal{R}, \{\mathcal{O}_i\}_{i\in\mathcal{I}}, \Omega, \gamma \rangle,$$8 layers and $$\mathcal{M} = \langle \mathcal{I}, \mathcal{S}, \{\mathcal{A}_i\}_{i\in\mathcal{I}}, \mathcal{T}, \mathcal{R}, \{\mathcal{O}_i\}_{i\in\mathcal{I}}, \Omega, \gamma \rangle,$$9 heads, embedding dimension approximately $\mathcal{I}=\{1,\dots,N\}$0, masked mean pooling, two-layer MLP local heads shared across agents, discount $\mathcal{I}=\{1,\dots,N\}$1, batch size $\mathcal{I}=\{1,\dots,N\}$2, replay size $\mathcal{I}=\{1,\dots,N\}$3, and target updates every $\mathcal{I}=\{1,\dots,N\}$4 episodes.

Empirically, the strongest policy-side evidence comes from coordination matrix games and Multi-Agent Particle Environments. In the top-$\mathcal{I}=\{1,\dots,N\}$5 selection matrix game with $\mathcal{I}=\{1,\dots,N\}$6 and $\mathcal{I}=\{1,\dots,N\}$7, the independent-execution bound is approximately $\mathcal{I}=\{1,\dots,N\}$8 success, baselines cluster from $\mathcal{I}=\{1,\dots,N\}$9 for QMIX to $\mathcal{A}_i$0 for DCG, and AGP achieves $\mathcal{A}_i$1 success. In the top-$\mathcal{A}_i$2 task with anti-coordination penalty, AGP is reported at approximately $\mathcal{A}_i$3 success while baselines drop to low success because of relative overgeneralization. Across canonical coordination tasks with partial observability and anti-coordination penalties, AGP reaches $\mathcal{A}_i$4–$\mathcal{A}_i$5 success while other methods often reach $\mathcal{A}_i$6–$\mathcal{A}_i$7. In MPE tasks—Reference, Push, World-Comm, Speaker-Listener, Crypto, and Tag—AGP achieves the highest final test return across all six tasks, outperforming IQL, VDN, QMIX, DCG/DICG, MACPF, and FOP (Gupta et al., 19 Feb 2026).

The ablations are especially diagnostic. Removing cross-agent edges collapses performance to the independent-baseline regime, removing the action graph entirely yields identical failure modes, and AGP-PG attains similar high success. The reported interpretation is that the gain is architectural—action-level coordination—rather than specific to a particular learning algorithm.

5. Variants and adjacent formulations across domains

The graph-policy idea appears in several adjacent literatures with different graph semantics. In long-horizon agentic RL, G2PO adopts an Action Graph Policy perspective by transforming multiple trajectories into a global state-transition graph $\mathcal{A}_i$8, aggregating identical observations into state groups, and redefining actions as directed edges between state nodes. The edge-level policy is

$\mathcal{A}_i$9

restricted to admissible outgoing edges. Group-aggregation node values,

$\mathcal{A}=\prod_{i=1}^N \mathcal{A}_i$0

and edge-centric TD errors,

$\mathcal{A}=\prod_{i=1}^N \mathcal{A}_i$1

are combined with global standardization to form edge-centric advantages. On WebShop, ALFWorld, and AppWorld, G2PO is reported to outperform prompt-based and RL baselines, with improvements up to $\mathcal{A}=\prod_{i=1}^N \mathcal{A}_i$2 over GRPO (Wang et al., 22 Jun 2026).

In coordinated MARL with coordination graphs, the ADG formulation treats AGP as a DAG over agents rather than over agent–action pairs. This produces globally optimal coordinated policies when the parent sets satisfy the stated compatibility condition with the coordination graph, and it yields a tabular policy-iteration algorithm with finite-step convergence under the paper’s assumptions. The same framework is integrated into MAPPO-, QMIX-, and VDN-style systems by replacing independent actors or local value heads with graph-conditioned ones (Ding et al., 1 Jun 2025).

In robotics with variable morphologies, AGP refers to graph-based control policies defined on the robot’s kinematic graph. Each node represents a body part or joint, edges represent kinematic connections, message passing produces node embeddings, and per-node actor heads output actions that are concatenated into the full action vector. The policy is parameterized as a factorized Gaussian,

$\mathcal{A}=\prod_{i=1}^N \mathcal{A}_i$3

The distinctive issue is that PPO and TRPO behave differently as action dimension $\mathcal{A}=\prod_{i=1}^N \mathcal{A}_i$4 varies. The paper derives that PPO clipping becomes effectively tighter as $\mathcal{A}=\prod_{i=1}^N \mathcal{A}_i$5 grows and proposes trust-region compensation

$\mathcal{A}=\prod_{i=1}^N \mathcal{A}_i$6

to equalize clipping likelihood across morphologies. In variable-joint Swimmer with $\mathcal{A}=\prod_{i=1}^N \mathcal{A}_i$7, compensation is reported to reach plateau approximately $\mathcal{A}=\prod_{i=1}^N \mathcal{A}_i$8 timesteps earlier while maintaining performance (Gallien, 16 Aug 2025).

In relational planning, GABAR operationalizes an action-centric AGP that ranks actions rather than outputs control signals. The graph includes object nodes, predicate nodes, action schema nodes, and a global node; edges encode predicate arguments and action-argument bindings. After $\mathcal{A}=\prod_{i=1}^N \mathcal{A}_i$9 rounds of attention-based message passing and global-node updates, a GRU decoder scores action schemas and then sequentially scores argument objects. The training loss is pointwise cross-entropy over schemas and per-parameter object choices. Trained on small instances, the method generalizes to larger ones, with reported coverage of $\pi_\theta(e \mid v)$00 in Blocks World, Gripper, and Miconic, $\pi_\theta(e \mid v)$01 in Logistics, $\pi_\theta(e \mid v)$02 in Visitall, and $\pi_\theta(e \mid v)$03 in Grid (Mangannavar et al., 2024).

These variants share a structural commitment: policy computation is performed over a graph that directly encodes action-relevant relations. They differ in what constitutes a node, what constitutes an edge, and whether the output is a decentralized action, a sequentially generated joint action, an action ranking, or an optimizer-adjusted continuous control distribution.

6. Limitations, misconceptions, and open research directions

A persistent misconception is that AGP simply means “using a graph neural network in RL.” The cited papers support a narrower reading: the graph must be policy-relevant at the level of action dependence, edge selection, kinematic actuation, or action ranking. In the main Dec-POMDP formulation, the novelty is specifically action-centric nodes and conditioning on learned dependencies between concrete action choices, rather than merely placing agents as graph nodes (Gupta et al., 19 Feb 2026).

The main limitations are formulation-dependent. In decentralized MARL, attention over action graphs scales as $\pi_\theta(e \mid v)$04, so naive fully connected graphs become expensive for very large $\pi_\theta(e \mid v)$05 or dense action spaces. The reported failure modes include spurious graph construction, nonstationarity in mixed settings, and greater training difficulty when extremely high-order constraints require more layers or heads. Suggested mitigations include sparsifying the action graph, hierarchical or group-structured action graphs, attention regularization, curriculum learning, domain randomization, and locality-aware attention (Gupta et al., 19 Feb 2026).

In the G2PO formulation, graph construction errors arise from over-aggregation when distinct states share identical text observations, under-aggregation when formatting differences split identical states, and sensitivity of global standardization to outliers or non-stationarity. The reported implementation is in the verl-agent framework, and broader integration is left as future work (Wang et al., 22 Jun 2026).

In the ADG optimality line, the strongest guarantees assume access to an appropriate coordination graph, finite spaces, deterministic policy improvement, and in some theorems a uniqueness assumption for the argmax. A full necessary characterization of minimal ADGs is not provided, and finding optimal topological orders is related to optimal variable elimination orders and is NP-complete (Ding et al., 1 Jun 2025).

In morphology-generalized control, the compensation rule $\pi_\theta(e \mid v)$06 is motivated for small-to-moderate action dimensions under Gaussian factorized policies with slowly changing log-variance. The paper explicitly notes that large-$\pi_\theta(e \mid v)$07 settings may undermine the trust-region principle, and equalizing clipping probabilities does not imply uniform performance because per-dimension task relevance varies (Gallien, 16 Aug 2025).

In action-ranking planning, graph size grows with object count and action arity, supervision requires optimal first actions on small instances, and greedy execution without search can still get stuck despite beam ranking and cycle avoidance. The ablation removing the global node materially degrades long-range coordination, especially in Logistics (Mangannavar et al., 2024).

Taken together, these limitations indicate a broad research agenda rather than a settled design template. A plausible implication is that future AGP work will increasingly focus on graph sparsification, adaptive graph construction, partial-observability guarantees under function approximation, and hybrid schemes that preserve the structural benefits of action graphs while reducing their computational and statistical overhead.