ATT-MADDPG Framework Overview

• The paper introduces ATT-MADDPG, which extends MADDPG by integrating an attention mechanism within the centralized critic to address non-stationarity in MARL.
• It employs independent actor networks alongside a K-head attention critic that dynamically weighs teammates' actions for improved joint policy modeling.
• Empirical evaluations in cooperative navigation and packet routing tasks demonstrate enhanced stability, scalability, and coordination compared to traditional baselines.

The Attention Multi-Agent Deep Deterministic Policy Gradient (ATT-MADDPG) framework addresses the challenge of modeling and exploiting the dynamic joint policies of teammates in cooperative multi-agent reinforcement learning (MARL) environments. ATT-MADDPG extends the standard Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm by incorporating a centralized critic enhanced with an attention mechanism, enabling each agent to adapt to the evolving policies of its teammates and thereby improving cooperative behavior, stability, and scalability (Mao et al., 2018).

1. Multi-Agent Setting and Motivation

ATT-MADDPG is formulated for cooperative, distributed multi-agent Markov decision processes, specifically as Decentralized Partially Observable Markov Decision Processes (DEC-POMDPs). The setting is characterized as follows:

• There are $N$ agents, each observing $o_i$ and selecting continuous action $a_i \in \mathcal{A}_i$.
• The environment transitions from state $s$ to $s'$ via $s' \sim T(s' \mid s, \bar{a})$, with $\bar{a} = \langle a_1, ..., a_N \rangle$ the joint action.
• All agents receive a shared reward $r = R(s, \bar{a})$.
• Each agent independently learns a decentralized policy $\mu_i(o_i; \theta_i)$ to maximize expected discounted return $G = \sum_t \gamma^t r_t$.

The primary motivation is the pronounced non-stationarity in MARL due to simultaneously learning agents. Since each agent's environment includes teammates whose policies change over time, modeling the dynamic joint policy of these teammates is crucial for stable and effective learning. ATT-MADDPG provides a mechanism for agents to model and exploit the contemporaneous policies of their peers during centralized training, promising improved coordination and learning stability.

2. ATT-MADDPG Framework Architecture

ATT-MADDPG consists of two key components: independent actor networks for decentralized execution, and a centralized critic module employing a K-head attention architecture for training.

2.1 Independent Actor Networks

• Each agent $o_i$0 maintains an actor $o_i$1 that maps local observations $o_i$2 to deterministic actions.
• During execution, policies are decentralized: actors only access their own observations.

2.2 Centralized Critic with Attention

• Each agent's critic $o_i$3 receives the full set of agents’ observations and actions, giving access to the complete joint state and action for evaluation during training.
• The critic uses these inputs to estimate Q-values, mitigating non-stationarity by explicitly conditioning on teammates’ current behaviors.

2.3 Attention Mechanism

• The critic models the conditional Q-value $o_i$4 via $o_i$5 “action-conditional” Q-value heads $o_i$6 and corresponding attention weights $o_i$7.
• Let $o_i$8 be a hidden vector summarizing teammate actions $o_i$9; the unnormalized score for head $a_i \in \mathcal{A}_i$0 is $a_i \in \mathcal{A}_i$1.
• Softmax normalization yields $a_i \in \mathcal{A}_i$2 ensuring $a_i \in \mathcal{A}_i$3.
• The contextual Q-value is $a_i \in \mathcal{A}_i$4, which is then projected to a scalar Q-value via a fully connected layer.

3. Optimization Procedure and Training Dynamics

The training regimen mirrors standard actor-critic setups but with centralized-critic updates guided by attention, using an episodic or sample-based replay mechanism:

• Replay Buffer: Stores transitions $a_i \in \mathcal{A}_i$5.
• Critic Loss: The temporal-difference error for agent $a_i \in \mathcal{A}_i$6,

$a_i \in \mathcal{A}_i$7

is minimized via $a_i \in \mathcal{A}_i$8.

• Actor Update: The deterministic policy gradient is used:

$a_i \in \mathcal{A}_i$9

• Target Networks: Parameters are updated via soft updates with coefficient $s$0.
• Typical Hyperparameters: actor learning rate $s$1, critic $s$2, $s$3, $s$4, replay buffer of $s$5 transitions, batch size $s$6, number of attention heads $s$7 (typical).

4. Empirical Evaluation and Results

ATT-MADDPG was evaluated across both synthetic benchmarks and a real-world packet routing domain.

4.1 Real-World Packet Routing

• Task: Edge routers allocate flows across multiple paths to minimize maximum link utilization (MLU); reward is $s$8.
• Topologies: Small (4 routers, 8 links) and large (12 routers, Abilene-scale).
• Baselines: MADDPG, PSMADDPGV2, WCMP (rule-based), Khead-MADDPG (no attention).
• Findings: ATT-MADDPG achieved higher average rewards on both topologies, demonstrated superior scalability (performance gap grows with increasing network size), and robust performance across $s$9. Khead-MADDPG (no attention) failed to learn effective routing policies. Attention heads specialized, with higher-variance heads modeling rare but critical joint actions.

4.2 Synthetic Benchmark Tasks

Tasks included Cooperative Navigation (3 agents, 3 landmarks) and Predator-Prey (3 predators, one prey). Agents observe positional/velocity data and act in continuous velocity space with a shared reward. Performance measured as stable average episode reward.

Environment	ATT-MADDPG (K=4)	PSMADDPGV2	MADDPG	Greedy	Khead-MADDPG
Coop Navigation	–1.268	–1.586	–1.767	–2.105	–2.825
Predator-Prey	3.589	2.473	1.920	1.903	1.899

ATT-MADDPG outperformed all baselines, confirming that the attention-based critic enhances coordination in both types of cooperative tasks.

4.3 Policy Analysis

Trajectory visualizations indicated that agents, via state transitions, “signal” their intentions to each other. The attention critic effectively captured these inter-agent cues, enabling coordinated assignment to landmarks and improved group-level behavior.

5. Relation to Hierarchical and Transformer-Based Extensions

Subsequent research has investigated extensions to the ATT-MADDPG paradigm incorporating hierarchical coding via RNNs and sequence-level Transformer encoders, for instance the Hierarchical RNNs-Based Transformers MADDPG (HRTMADDPG) framework (Wei et al., 2021). HRTMADDPG augments actor and critic networks with per-agent RNN step encoders and stacked Transformer encoder layers to encode both temporal and inter-agent correlations. This hierarchical organization enables improved credit assignment and coordination in both cooperative and mixed settings, with demonstrated gains over non-attention and single-level models such as MADDPG and RMADDPG. Notably, in fully cooperative navigation, multi-layer HRTMADDPG significantly surpasses MADDPG and LSTM-based variants in test rewards.

A plausible implication is that further integrating hierarchical temporal and relational encoding architectures—e.g., by fusing RNN or graph-RNN modules with multi-agent attention critics—remains a promising direction for tackling increasingly complex MARL scenarios.

6. Scalability, Robustness, and Limitations

Experiments confirm that ATT-MADDPG achieves robust and scalable performance, with stability across a wide range of attention head numbers ($s'$0 to $s'$1). Increases in team size or network topology complexity do not degrade solution quality as rapidly as in non-attention baselines. Furthermore, the division of labor among attention heads, with some specializing in rare but critical actions, supports a degree of redundancy and flexibility against non-stationarity.

However, the explicit requirement for a centralized critic collecting all agent observations and actions during training does not eliminate communication bottlenecks in regimes with very large $s'$2 or when the communication topology is dynamic. Recent hierarchical and Transformer-based approaches seek to mitigate these issues by compressing inter-agent dependencies via local step encoding and global relational attention (Wei et al., 2021).

7. Summary and Research Impact

ATT-MADDPG represents a foundational approach to multi-agent deep reinforcement learning in cooperative settings, by addressing the dynamic nature of joint teammate policies via an explicit, learnable attention mechanism over conditional Q-value heads. Empirical results validate the efficacy of the attention-based centralized critic, establishing ATT-MADDPG as a robust, scalable baseline. Extensions using hierarchical encoding and Transformer attention suggest continued relevance for modular, expressive approaches to modeling agent interdependencies in MARL (Mao et al., 2018Wei et al., 2021).