TAAC: Team-Attention Actor-Critic

• TAAC is a centralized training and execution actor-critic algorithm that uses multi-headed attention in both actor and critic modules to enable explicit inter-agent querying.
• It introduces a conformity loss that penalizes cosine similarity of agent embeddings, encouraging diverse and complementary roles within the team.
• Empirical evaluations in simulated soccer environments demonstrate that TAAC improves win rates, Elo ratings, and tactical coordination compared to decentralized baselines.

Team-Attention-Actor-Critic (TAAC) is a centralized training and centralized execution (CTCE) actor-critic algorithm developed to enhance multi-agent collaboration in cooperative Markov games. TAAC introduces multi-headed attention mechanisms in both the actor and critic to facilitate inter-agent communication and dynamic, explicit querying of teammates at decision time. This design addresses the exponential growth of the joint-action space and introduces a penalized loss promoting diverse but complementary roles among agents, resulting in improved collaborative behaviors and task performance relative to established benchmarks (Garrido-Lestache et al., 30 Jul 2025).

1. Algorithmic Structure and Innovations

TAAC’s primary objective is to enable agents to condition their actions not only on their own observations but also on learned representations of all teammates via multi-headed attention embedded directly in the actor. Unlike prior work, such as Proximal Policy Optimization (PPO), which employs fully decentralized training/execution, or Multi-Agent Actor-Attention-Critic (MAAC), where only the critic receives attention-augmented, centralized information, TAAC’s actor also leverages full team state input and teammate representations. This enables each agent’s policy to actively “query” the state of others, structuring the local action search and accommodating the intractable joint-action space inherent in CTCE setups.

Additionally, TAAC introduces a conformity loss in the actor’s objective. This penalty, applied to the cosine similarity across per-agent attention embeddings, explicitly pushes agents toward learning distinct, role-specific representations, thereby avoiding degeneracies where agents collapse into identical, redundant behaviors. In summary, TAAC is characterized by (1) actor-critic structure under CTCE, (2) multi-headed attention in both actor and critic, and (3) a differentiable conformity penalty for role diversity (Garrido-Lestache et al., 30 Jul 2025).

2. Multi-Head Attention Architectures in Actor and Critic

Actor Module

Given the joint observation vector $\vec o = (o_1, \dots, o_n)$, each agent’s observation $o_i$ is mapped to an embedding $m_{1,i}^\pi \in \mathbb{R}^{d_{\rm model}}$. These embeddings are stacked and passed through $h_\pi$ parallel attention heads, each computing key, query, and value transformations and producing a softmax-weighted combination of teammates’ representations. The output for agent $i$ aggregates all $h_\pi$ heads as $E_i^\pi \in \mathbb{R}^{h_\pi d_v}$, serving as the agent-specific context vector. The final action distribution $\pi^{(i)}$ over $\mathcal{A}_i$ is produced by a small MLP applied to $E_i^\pi$, with parameters $o_i$0 (embeddings and attention head weights) shared across agents, enforcing permutation invariance while allowing differentiated agent behaviors.

Critic Module

The critic receives the full joint observation and action $o_i$1, embedding each agent’s $o_i$2 tuple via separate key, value, and query projections. Multi-headed attention computes inter-agent dependencies, and the resulting per-agent representations $o_i$3 are concatenated with their original embeddings for each agent $o_i$4 and passed through an MLP to produce $o_i$5, the agent-specific value estimate utilized for credit assignment during training.

This dual-attention design allows the actor and critic to distill relevant teammate and overall team information dynamically and efficiently (Garrido-Lestache et al., 30 Jul 2025).

3. Centralized Training and Execution Paradigm

TAAC operates under a CTCE framework, enabling a single controller to observe the full team state at both training and execution time. The controller factorizes the joint policy as a product of per-agent policies:

$o_i$6

where each $o_i$7 conditions the action distribution of agent $o_i$8 on the joint observation, including other agents’ states. The multi-head attention layers serve to compress and select relevant teammate information, transforming the high-dimensionality of the joint space into fixed-size, learned representations for downstream action selection.

The centralized critic operates on $o_i$9 and supplies tailored value targets and baselines for each agent’s policy update. During execution, only the actor is used, eliminating the critic and enabling efficient, decentralized action sampling by each agent from their conditional distributions, while still leveraging globally informed policies (Garrido-Lestache et al., 30 Jul 2025).

4. Penalized Conformity Loss for Role Differentiation

To ensure learned policies assign agents to complementary roles and avoid mode collapse or redundant strategies, TAAC incorporates a conformity loss applied to the cosine similarity of the post-attention embeddings $m_{1,i}^\pi \in \mathbb{R}^{d_{\rm model}}$0. The average off-diagonal similarity $m_{1,i}^\pi \in \mathbb{R}^{d_{\rm model}}$1 is defined as:

$m_{1,i}^\pi \in \mathbb{R}^{d_{\rm model}}$2

where $m_{1,i}^\pi \in \mathbb{R}^{d_{\rm model}}$3 is the cosine similarity between $m_{1,i}^\pi \in \mathbb{R}^{d_{\rm model}}$4 and $m_{1,i}^\pi \in \mathbb{R}^{d_{\rm model}}$5.

The conformity penalty takes the form:

$m_{1,i}^\pi \in \mathbb{R}^{d_{\rm model}}$6

with hyperparameters $m_{1,i}^\pi \in \mathbb{R}^{d_{\rm model}}$7 and $m_{1,i}^\pi \in \mathbb{R}^{d_{\rm model}}$8, encouraging dissimilarity in agent context vectors up to a threshold. This term is added to the actor loss, promoting specialization and structured role diversity within the team, empirically yielding improved collaboration and coverage (Garrido-Lestache et al., 30 Jul 2025).

5. Training Protocol and Optimization

TAAC’s training loop follows a standard actor-critic process augmented with the above architectural and loss terms. For each iteration:

• Sample trajectories by executing $m_{1,i}^\pi \in \mathbb{R}^{d_{\rm model}}$9 under the CTCE scheme.
• For each timestep $h_\pi$0 and agent $h_\pi$1, compute TD targets:

$h_\pi$2

• Calculate the critic loss:

$h_\pi$3

• Compute the actor loss, using a multi-agent baseline $h_\pi$4 as in COMA, plus the conformity term:

$h_\pi$5

• Simultaneously update actor and critic parameters by gradient step.

Pseudocode summary: $h_\pi$7 (Garrido-Lestache et al., 30 Jul 2025)

6. Empirical Evaluation and Collaborative Behaviors

TAAC was evaluated in a simulated 3-vs-3 continuous 2D soccer environment. Each agent perceives relative positions of all entities, the ball state, goal locations, and environment boundaries. The action space consists of 18 discrete move and kick combinations. Reward structure incentivizes exploration, coordinated ball movement, goal achievement, and penalizes clustering via a minimum-pairwise-distance component. Curriculum learning spans from passive opponents to self-play with varied spawning scenarios.

Performance metrics include win rate, goal differential, Elo rating, and collaboration-specific measures: average pairwise team distance, possession swap frequency (reflecting tactical passing), and connectivity scores (fraction of direct, unobstructed teammate links).

Empirical outcomes over 5,000 round-robin matches:

Algorithm	Elo	Win Rate	Connectivity	Possession Swaps	Avg Pairwise Distance
TAAC	1550	62%	0.82	23	4.2
PPO	1480	57%	0.65	18	5.0
MAAC	1300	40%	0.45	30	3.1

TAAC outperformed PPO (DTDE) and MAAC (CTDE) both in Elo rating and team-oriented behaviors. Notably, MAAC’s high possession swap count reflected frantic clustering, not collaborative strategy. TAAC policies maintained balanced spacing, frequent tactical passes, and high inter-agent connectivity. Figures included trajectory plots and training curves highlighting TAAC’s consistently superior collaboration and win rate (Garrido-Lestache et al., 30 Jul 2025).

7. Ablative Analysis of Attention and Conformity Penalties

Ablation studies revealed the criticality of TAAC’s design elements. Removing multi-headed attention from the actor (restricting agents to local observations) decreased Elo by approximately 40 points and reduced connectivity by 15%. Disabling the conformity loss ($h_\pi$6) resulted in symmetric, undifferentiated behaviors: agents either all pursued the ball or all remained static, with win rates dropping by 8% and pairwise distances confirming increased clustering tendencies.

Complete TAAC policies demonstrated dynamic role assignment, with agents adaptively occupying forward, support, and defensive roles, as observed in coordinated formations and rapid, yet structured, possession exchanges. On-the-fly role switching in response to ball position was observed in playback, indicating learned, context-sensitive division of labor across the team (Garrido-Lestache et al., 30 Jul 2025).

---

TAAC’s unique combination of actor-level multi-headed attention for explicit inter-agent querying, attention-based value assignment in the critic, and an embedding-level conformity penalty yields team policies that are both performant and robustly collaborative, surpassing both decentralized baselines and CTDE critics without actor attention or role-promoting constraints (Garrido-Lestache et al., 30 Jul 2025).