---
title: 'GPAT: Policy Improvement in Ad Hoc Multi-Agent Teams'
url: https://www.emergentmind.com/topics/generalized-policy-improvement-for-ad-hoc-teaming-gpat
type: topic
---

# GPAT: Policy Improvement in Ad Hoc Multi-Agent Teams

Generalized Policy Improvement for Ad hoc Teaming (GPAT) addresses zero-shot coordination in multi-agent systems where an agent—the learner—must collaborate with teammates whose policies are previously unseen and fixed. Rather than selecting a single robust or best-matching pretrained policy, GPAT leverages a library of pretrained policies, integrating their strengths via generalized policy improvement (GPI) and exploiting difference rewards to mitigate credit assignment and team-dynamics variability. GPAT achieves strong empirical performance in zero-shot ad hoc teaming tasks without requiring online adaptation or inference on the new team [2510.16187].

## 1. Formalization of Ad Hoc Multi-Agent MDPs

The foundation for GPAT is the multi-agent Markov decision process (MMDP), defined as the tuple
\[
(\mathcal S, \mathcal N, \{\mathcal A^i\}_{i\in\mathcal N}, P, r, \gamma)
\]
where:
- $\mathcal S$: state space,
- $\mathcal N = \{1,2,\dots,n\}$: agent set,
- $\mathcal A^i$: action space for agent $i$, $\mathcal A = \prod_i \mathcal A^i$ joint action space,
- $P(s'|s,\mathbf{a})$: transition kernel,
- $r(s, \mathbf{a}, s')$: team reward,
- $\gamma \in [0,1)$: discount factor.

Each agent $i$ executes $a^i_t$ according to policy $\pi^i(a^i|s_t)$, forming joint policy $\pi(\mathbf{a}|s) = \prod_i \pi^i(a^i|s)$ with action-value function
\[
Q^\pi(s, \mathbf{a}) = \mathbb{E}_{P,\pi} \left[ \sum_{t=0}^{\infty} \gamma^t r(s_t,\mathbf{a}_t,s_{t+1})\mid s_0 = s, \mathbf{a}_0 = \mathbf{a} \right].
\]

The ad hoc MMDP singles out a learner $a\in \mathcal N$, with teammates $\mathcal N_u = \mathcal N \setminus \{a\}$ using unknown fixed policies $\pi^i$ ($i\in\mathcal N_u$). The ad hoc MMDP is
\[
M = (\mathcal S, \mathcal N, a, \{\mathcal A^i\}_{i\in\mathcal N}, P, r, \{\pi^i\}_{i\in\mathcal N_u}, \gamma).
\]

The learner must identify a policy $\pi^{a^*}$ that (for all $(s, a^a)$) maximizes
\[
Q^{\pi^a, \pi^{-a}}(s,a^a) = \mathbb{E} \left[ \sum_{t=0}^\infty \gamma^t r(s_t, \mathbf{a}_t, s_{t+1}) \mid s_0=s, a^a_0=a^a \right]
\]
with $\pi^{-a} = \prod_{i\in\mathcal N_u} \pi^i$.

Zero-shot transfer constrains the agent to:
- Pretrain and perform policy evaluation on $n$ source ad hoc teams ($M_1,\dots,M_n$).
- Deploy a learner policy on new team $M_{n+1}$ with unseen teammate policies at test time, with no online updates or inference.

## 2. Generalized Policy Improvement and Difference Rewards

### Generalized Policy Improvement (GPI)

Given a set of pretrained learner policies $\Pi^a = \{\pi^a_1, \dots, \pi^a_n\}$, each with action-value $Q^{\pi^a_i,\pi^{-a}_i}(s, a^a)$, the GPI action-value is
\[
Q_{\mathrm{GPI}}(s, a) = \max_{i=1,\dots,n} Q^{\pi^a_i, \pi^{-a}_i}(s, a)
\]
and the GPI policy is greedily defined as
\[
\pi_{\mathrm{GPI}}(s) \in \arg\max_{a \in \mathcal{A}^a} Q_{\mathrm{GPI}}(s, a) = \arg\max_{a}\max_i Q^{\pi^a_i, \pi^{-a}_i}(s, a).
\]

Classically, if only the reward weight $w$ varies and transition dynamics remain unchanged, this construction ensures $Q_{\mathrm{GPI}}(s, a)\ge Q^{\pi_i}(s, a)$, so $V^{\pi_{\mathrm{GPI}}}(s) \ge \max_i V^{\pi_i}(s)$ for all $s$.

### Difference Rewards

To address credit assignment, the difference reward for agent $j$ is
\[
D_j(s, \mathbf{a}) = G(s, \mathbf{a}) - G(s, (\mathbf{a}_{-j}, a'_j))
\]
where $G(s, \mathbf{a})$ is the team cumulative or immediate reward, $\mathbf{a}_{-j}$ denotes teammates’ actions, and $a'_j$ is a baseline action (e.g., “do nothing” or sampled under $j$’s policy). For the learner ($a$), focusing on $D_a$ reduces reliance on knowledge of teammate behavior and renders learning more robust to unknown teammate policies.

## 3. Algorithmic Structure and Execution

GPAT comprises three stages:

**A. Pretraining**
- For each source team $M_i$, train optimal learner policy $\pi^a_i$ (computing successor features $\psi^{\pi^a_i}(s, a)$ in the linear-reward case).

**B. Policy Evaluation with Difference Reward**
- Define immediate difference reward:
  \[
  \Delta r^a(s, \mathbf{a}, s') = r(s, \mathbf{a}, s') - r(s, (\mathbf{a}_{-a}, a'_a), s')
  \]
- For each pretrained $\pi^a_i$, estimate $Q$-function with respect to $\Delta r^a$:
  \[
  Q_{i, \Delta r^a}(s, a^a) = \mathbb{E} \Biggl[ \sum_{t=0}^{\infty} \gamma^t \Delta r^a(s_t, \mathbf{a}_t, s_{t+1}) \bigg| s_0 = s, a^a_0 = a^a, \pi^a_i \Biggr]
  \]
- For linear-reward cases, use successor features and fit $w_{\Delta r^a}$ via least-squares or TD-learning:
  \[
  Q_{i, \Delta r^a}(s, a) = \psi_i(s, a)^\top w_{\Delta r^a}.
  \]

**C. Zero-shot Test Execution via GPI**
- On a new team $M_{n+1}$, do not update or infer online. For each $a^a \in \mathcal{A}^a$, use
  \[
  Q_{\text{GPI}}^{\Delta r^a}(s, a) = \max_{i=1,\dots,n} Q_{i, \Delta r^a}(s, a)
  \]
and act greedily:
  \[
  \pi_{\text{GPAT}}(s) \in \arg\max_{a} Q_{\text{GPI}}^{\Delta r^a}(s, a).
  \]

### Algorithmic Sketch

```
Given source teams {M_1, ..., M_n}:
  Step 1: For i=1..n:
    • Train learner policy π^a_i on team M_i
    • Compute (or store) successor features ψ_i if linear

  Step 2: For i=1..n:
    • Collect rollouts under π^a_i on M_i
    • Compute Δr^a and do policy-evaluation to obtain Q_{i,Δr^a}
      (either via SF-regression or TD-learning)

  Step 3 (zero-shot on new team M_{n+1}):
    Observe state s
    For each a^a ∈ A^a, compute max_i Q_{i,Δr^a}(s,a^a)
    Pick a^a maximizing this; execute (a^a, teammates’ π⁻ᵃ_{n+1}(s))
```

## 4. Experimental Results and Benchmarks

GPAT was evaluated across multiple domains:

- **Cooperative Foraging**: Two-agent gridworld collecting three object types; reward $r = \phi^\top [1, 1, 1]$ for counts $\phi \in \mathbb{R}^3$.
- **Predator–Prey**: Three predators, four prey (two yellow “easy,” two red “hard,” requiring two predators); analogous reward structure.
- **Overcooked (Cramped Room + extra ingredient)**: Two-agent cooking, rewards for ingredient potting, dish management, soup serving, delivery; weights such as $[3, 3, 3, 5, 20]$.
- **Real-World Multi-Robot**: Two TurtleBot3s in $12\,\mathrm{ft}\!\times\!12\,\mathrm{ft}$ grid co-foraging, with motion capture and ROS.

Baseline comparisons included:
- **Oracle**: Team-specific learner trained from scratch.
- **Robust**: Single policy trained across all source teams.
- **PLASTIC**: Selection of the single best pretrained policy per test team.

**Metric:** Interquartile mean (IQM) return over 1,000 episodes × 10 seeds, reporting 95% confidence intervals and percentage of oracle team performance.

**Key Findings:**
- GPAT consistently outperformed Robust and PLASTIC when the pretrained policy library contained skills relevant to the new team, notably in the first two foraging experiments, Predator–Prey, and Overcooked.
- If the library lacked any compatible skill (Foraging Exp. 3), GPAT performance degraded (to roughly 70% of optimal), while PLASTIC/Robust achieved about 80%.
- An ablation without difference rewards (“w/o DR”) showed significant performance drop, particularly an inability to switch between policies mid-episode.
- The real-robot demonstration confirmed simulator findings: TurtleBots successfully coordinated to collect complementary objects.

## 5. Theoretical Properties and Guarantees

GPAT builds on established theoretical results for GPI under shared dynamics. If transition kernel remains unchanged and the only variation is in linear reward weight $w$, then for all $s, a$,
\[
Q^{\pi_{\mathrm{GPI}_w}}(s, a) \ge \max_i Q^{\pi_i}_w(s, a)
\]
ensuring the GPI-constructed policy is at least as good as any pretrained policy. In the ad hoc setting, where teammate policies induce a new transition kernel at test time, this guarantee does not carry over. No novel convergence or suboptimality bounds are proven for GPAT under transition shift. The *hypothesis* is that difference rewards render learned $Q$-functions less sensitive to transition changes, preserving practical effectiveness.

## 6. Advantages, Limitations, and Prospective Directions

**Strengths:**
- Zero-shot deployment: no online adaptation upon encountering new teammates.
- Dynamic mixing of all pretrained behaviors via GPI at each time-step.
- Difference rewards effectively address credit assignment and limit sensitivity to unknown teammate dynamics.
- Demonstrated empirical superiority to robust single-policy and type-based selection baselines.

**Limitations:**
- If the policy library lacks any skill suitable for the new team, GPAT may underperform.
- Theoretical improvement guarantees do not generally hold under transition kernel shift induced by changing teammates.
- Estimation of difference rewards necessitates sampled rollouts or TD estimates.

**Potential Extensions:**
- Formal suboptimality analysis under transition shift.
- Sample-efficient few-shot updates of $Q_{i, \Delta r^a}$ for new teams.
- Integration with partial observability or belief-driven GPI.
- Designing communication-aware difference rewards.
- Extending to multi-learner or human–in–the–loop settings.

## 7. Summary Table: GPAT Algorithmic Phases and Objects

| Step                     | Core Mathematical Objects          | Key Activities                           |
|--------------------------|------------------------------------|------------------------------------------|
| Pretraining              | $\pi^a_i$, $\psi_i(s, a)$          | Per-team optimal policy learning          |
| Policy Evaluation        | $Q_{i, \Delta r^a}$, $w_{\Delta r^a}$ | Evaluate per-policy difference-Q or fit linear model |
| Zero-shot Test Execution | $Q_{\text{GPI}}^{\Delta r^a}$      | Max-pooling over $Q_{i, \Delta r^a}$; greedy action selection |

This encapsulates the formal structure, algorithmic innovations, empirical performance, theoretical assumptions, and potential avenues for future research in GPAT as presented in [2510.16187].

Source: https://www.emergentmind.com/topics/generalized-policy-improvement-for-ad-hoc-teaming-gpat