GPAT: Policy Improvement in Ad Hoc Multi-Agent Teams

• The paper introduces GPAT, which combines generalized policy improvement with a library of pretrained policies to enable zero-shot coordination with unseen teammates.
• It employs difference rewards to address credit assignment, mitigating variability in team dynamics and relying less on teammate behavior knowledge.
• Empirical results across diverse domains demonstrate that GPAT outperforms traditional baselines without requiring online adaptation or inference.

Generalized Policy Improvement for Ad hoc Teaming (GPAT) addresses zero-shot coordination in multi-agent systems where an agent—the learner—must collaborate with teammates whose policies are previously unseen and fixed. Rather than selecting a single robust or best-matching pretrained policy, GPAT leverages a library of pretrained policies, integrating their strengths via generalized policy improvement (GPI) and exploiting difference rewards to mitigate credit assignment and team-dynamics variability. GPAT achieves strong empirical performance in zero-shot ad hoc teaming tasks without requiring online adaptation or inference on the new team (Nigam et al., 17 Oct 2025).

1. Formalization of Ad Hoc Multi-Agent MDPs

The foundation for GPAT is the multi-agent Markov decision process (MMDP), defined as the tuple

$$(\mathcal S, \mathcal N, \{\mathcal A^i\}_{i\in\mathcal N}, P, r, \gamma)$$

where:

• $\mathcal S$: state space,
• $\mathcal N = \{1,2,\dots,n\}$: agent set,
• $\mathcal A^i$: action space for agent $i$, $\mathcal A = \prod_i \mathcal A^i$ joint action space,
• $P(s'|s,\mathbf{a})$: transition kernel,
• $r(s, \mathbf{a}, s')$: team reward,
• $\gamma \in [0,1)$: discount factor.

Each agent $i$ executes $\mathcal S$0 according to policy $\mathcal S$1, forming joint policy $\mathcal S$2 with action-value function

$\mathcal S$3

The ad hoc MMDP singles out a learner $\mathcal S$4, with teammates $\mathcal S$5 using unknown fixed policies $\mathcal S$6 ($\mathcal S$7). The ad hoc MMDP is

$\mathcal S$8

The learner must identify a policy $\mathcal S$9 that (for all $\mathcal N = \{1,2,\dots,n\}$0) maximizes

$\mathcal N = \{1,2,\dots,n\}$1

with $\mathcal N = \{1,2,\dots,n\}$2.

Zero-shot transfer constrains the agent to:

• Pretrain and perform policy evaluation on $\mathcal N = \{1,2,\dots,n\}$3 source ad hoc teams ($\mathcal N = \{1,2,\dots,n\}$4).
• Deploy a learner policy on new team $\mathcal N = \{1,2,\dots,n\}$5 with unseen teammate policies at test time, with no online updates or inference.

2. Generalized Policy Improvement and Difference Rewards

Generalized Policy Improvement (GPI)

Given a set of pretrained learner policies $\mathcal N = \{1,2,\dots,n\}$6, each with action-value $\mathcal N = \{1,2,\dots,n\}$7, the GPI action-value is

$\mathcal N = \{1,2,\dots,n\}$8

and the GPI policy is greedily defined as

$\mathcal N = \{1,2,\dots,n\}$9

Classically, if only the reward weight $\mathcal A^i$0 varies and transition dynamics remain unchanged, this construction ensures $\mathcal A^i$1, so $\mathcal A^i$2 for all $\mathcal A^i$3.

Difference Rewards

To address credit assignment, the difference reward for agent $\mathcal A^i$4 is

$\mathcal A^i$5

where $\mathcal A^i$6 is the team cumulative or immediate reward, $\mathcal A^i$7 denotes teammates’ actions, and $\mathcal A^i$8 is a baseline action (e.g., “do nothing” or sampled under $\mathcal A^i$9’s policy). For the learner ($i$0), focusing on $i$1 reduces reliance on knowledge of teammate behavior and renders learning more robust to unknown teammate policies.

3. Algorithmic Structure and Execution

GPAT comprises three stages:

A. Pretraining

• For each source team $i$2, train optimal learner policy $i$3 (computing successor features $i$4 in the linear-reward case).

B. Policy Evaluation with Difference Reward

• Define immediate difference reward:

$i$5

• For each pretrained $i$6, estimate $i$7-function with respect to $i$8:

$i$9

• For linear-reward cases, use successor features and fit $\mathcal A = \prod_i \mathcal A^i$0 via least-squares or TD-learning:

$\mathcal A = \prod_i \mathcal A^i$1

C. Zero-shot Test Execution via GPI

• On a new team $\mathcal A = \prod_i \mathcal A^i$2, do not update or infer online. For each $\mathcal A = \prod_i \mathcal A^i$3, use

$\mathcal A = \prod_i \mathcal A^i$4

and act greedily:

$\mathcal A = \prod_i \mathcal A^i$5

Algorithmic Sketch

$r(s, \mathbf{a}, s')$1

4. Experimental Results and Benchmarks

GPAT was evaluated across multiple domains:

• Cooperative Foraging: Two-agent gridworld collecting three object types; reward $\mathcal A = \prod_i \mathcal A^i$6 for counts $\mathcal A = \prod_i \mathcal A^i$7.
• Predator–Prey: Three predators, four prey (two yellow “easy,” two red “hard,” requiring two predators); analogous reward structure.
• Overcooked (Cramped Room + extra ingredient): Two-agent cooking, rewards for ingredient potting, dish management, soup serving, delivery; weights such as $\mathcal A = \prod_i \mathcal A^i$8.
• Real-World Multi-Robot: Two TurtleBot3s in $\mathcal A = \prod_i \mathcal A^i$9 grid co-foraging, with motion capture and ROS.

Baseline comparisons included:

• Oracle: Team-specific learner trained from scratch.
• Robust: Single policy trained across all source teams.
• PLASTIC: Selection of the single best pretrained policy per test team.

Metric: Interquartile mean (IQM) return over 1,000 episodes × 10 seeds, reporting 95% confidence intervals and percentage of oracle team performance.

Key Findings:

• GPAT consistently outperformed Robust and PLASTIC when the pretrained policy library contained skills relevant to the new team, notably in the first two foraging experiments, Predator–Prey, and Overcooked.
• If the library lacked any compatible skill (Foraging Exp. 3), GPAT performance degraded (to roughly 70% of optimal), while PLASTIC/Robust achieved about 80%.
• An ablation without difference rewards (“w/o DR”) showed significant performance drop, particularly an inability to switch between policies mid-episode.
• The real-robot demonstration confirmed simulator findings: TurtleBots successfully coordinated to collect complementary objects.

5. Theoretical Properties and Guarantees

GPAT builds on established theoretical results for GPI under shared dynamics. If transition kernel remains unchanged and the only variation is in linear reward weight $P(s'|s,\mathbf{a})$0, then for all $P(s'|s,\mathbf{a})$1,

$P(s'|s,\mathbf{a})$2

ensuring the GPI-constructed policy is at least as good as any pretrained policy. In the ad hoc setting, where teammate policies induce a new transition kernel at test time, this guarantee does not carry over. No novel convergence or suboptimality bounds are proven for GPAT under transition shift. The hypothesis is that difference rewards render learned $P(s'|s,\mathbf{a})$3-functions less sensitive to transition changes, preserving practical effectiveness.

6. Advantages, Limitations, and Prospective Directions

Strengths:

• Zero-shot deployment: no online adaptation upon encountering new teammates.
• Dynamic mixing of all pretrained behaviors via GPI at each time-step.
• Difference rewards effectively address credit assignment and limit sensitivity to unknown teammate dynamics.
• Demonstrated empirical superiority to robust single-policy and type-based selection baselines.

Limitations:

• If the policy library lacks any skill suitable for the new team, GPAT may underperform.
• Theoretical improvement guarantees do not generally hold under transition kernel shift induced by changing teammates.
• Estimation of difference rewards necessitates sampled rollouts or TD estimates.

Potential Extensions:

• Formal suboptimality analysis under transition shift.
• Sample-efficient few-shot updates of $P(s'|s,\mathbf{a})$4 for new teams.
• Integration with partial observability or belief-driven GPI.
• Designing communication-aware difference rewards.
• Extending to multi-learner or human–in–the–loop settings.

7. Summary Table: GPAT Algorithmic Phases and Objects

Step	Core Mathematical Objects	Key Activities
Pretraining	$P(s'|s,\mathbf{a})$5, $P(s'|s,\mathbf{a})$6	Per-team optimal policy learning
Policy Evaluation	$P(s'|s,\mathbf{a})$7, $P(s'|s,\mathbf{a})$8	Evaluate per-policy difference-Q or fit linear model
Zero-shot Test Execution	$P(s'|s,\mathbf{a})$9	Max-pooling over $r(s, \mathbf{a}, s')$0; greedy action selection

This encapsulates the formal structure, algorithmic innovations, empirical performance, theoretical assumptions, and potential avenues for future research in GPAT as presented in (Nigam et al., 17 Oct 2025).