---
title: Unsupervised Partner Design in MARL
url: https://www.emergentmind.com/topics/unsupervised-partner-design-upd
type: topic
---

# Unsupervised Partner Design in MARL

Searching arXiv for the cited UPD and related partner/environment design papers to ground the article in current literature.
Unsupervised Partner Design (UPD) is a population-free framework for cooperative multi-agent reinforcement learning that designs training partners online in order to improve robust ad-hoc teamwork, especially zero-shot coordination with previously unseen partners. In its canonical formulation, UPD treats the space of possible partners analogously to how unsupervised environment design treats the space of levels: rather than relying on a fixed pretrained population or a manually tuned partner generator, it generates candidate partners from the current ego policy, scores them by a learnability criterion, and trains against those that lie near the ego agent’s current learning frontier [2508.06336]. The framework is therefore best understood as a partner-space curriculum method. Related work on explicit partner modeling, such as Partner Approximating Learners (PAL), illuminates a distinct but complementary paradigm—online approximation of an encountered partner followed by simulation-based policy improvement—which is relevant to UPD as a supporting mechanism but does not itself perform partner design [1909.03868]. The broader curriculum-design context is provided by unsupervised environment design (UED), whose analyses of adaptive generator instability transfer directly to partner generation and help clarify both the promise and the failure modes of UPD-style systems [2308.10797].

## 1. Definition and problem setting

UPD was introduced for **robust ad-hoc teamwork (AHT)** in cooperative MARL, where the objective is to learn an agent that can coordinate zero-shot with **previously unseen partners** exhibiting different conventions, competence levels, or behavioral quirks [2508.06336]. The target test distribution includes strong, weak, incompatible, hard-coded, stochastic, and human partners, so the problem is not merely cooperative self-play but generalization under partner shift.

The formal setting is a **decentralised under-specified partially observable Markov decision process (Dec-UPOMDP)**. It is described as a family of environments parameterized by $\theta \in \Theta$, where each $\mathcal{M}_\theta$ is a fully specified level. Agents receive local observations, choose joint actions, obtain shared rewards, and maximize expected discounted return:
$$
J(\pi^{(1)}, \dots, \pi^{(n)}) = \mathbb{E}\left[ \sum_{t=0}^{\infty} \gamma^t R(s_t, a_t^{(1)}, \dots, a_t^{(n)})\right].
$$
In UPD, the ego policy is optimized while the training distribution over partner policies is generated and curated online [2508.06336].

The central motivation is that prior AHT approaches either depend on **large pretrained populations** or require **manual tuning** of partner-generation parameters. Population-based methods such as FCP and MEP rely on fixed sets of partners whose coverage may be expensive to obtain and maintain. Population-free work such as E3T avoids an explicit population by defining a partner as a mixture of the ego policy and a random policy,
$$
\pi_p = \epsilon \pi_r + (1 - \epsilon) \pi_{\text{ego}}, \quad \epsilon \in [0,1],
$$
but still requires manual tuning of $\epsilon$ depending on layout and evaluation scenario [2508.06336]. UPD removes both fixed-population dependence and fixed-mixture tuning by generating partners adaptively from the current ego policy and selecting them according to learnability.

This formulation implies a specific meaning of **population-free**. In the UPD paper, the term does not mean the absence of multiple partner behaviors during training; rather, it means that the method does **not require a pretrained or explicitly maintained partner population**. Partners are generated on the fly from a generator conditioned on the current ego policy [2508.06336].

## 2. Core mechanism: online partner generation and curation

UPD has two components: a **partner policy generator** $\mathcal{G}_p$ and a **learnability-based curator** that scores and selects generated partners [2508.06336]. At a high level, the training loop alternates among generating many candidate partners, evaluating them via repeated rollouts, ranking them by a learnability score, retaining the top $K$ candidates in a buffer, and training the ego agent with PPO using partners sampled from that selected buffer.

The partner generator extends E3T by replacing a fixed mixture coefficient with a sampled one. For each candidate partner, UPD samples
$$
\epsilon \sim \mathcal{U}(0,1),
$$
then forms the partner as
$$
\pi_p = \epsilon \pi_r + (1 - \epsilon)\pi_{\text{ego}}.
$$
At the extremes, $\epsilon = 0$ yields the ego policy itself and $\epsilon = 1$ yields a fully random partner. Intermediate values interpolate competence between those endpoints [2508.06336]. The key procedural difference from E3T is that UPD samples an entire range of partner competence levels and then uses learnability scoring to determine which ones are useful at the current training stage.

UPD also augments competence variation with **behavioral bias**. With probability $p_{\text{bias}}$, it samples an action-bias mask
$$
m \sim \mathrm{Dirichlet}(\alpha \cdot \mathbf{1}_A),
$$
and otherwise sets
$$
m = \mathbf{1}_A / A.
$$
The mask defines the random component $\pi_r$, so the random portion of a generated partner can exhibit persistent action preferences rather than uniform noise [2508.06336]. In the main Overcooked experiments, the paper reports $\alpha = 1.0$, partner buffer size $|\mathcal{B}| = 512$, $N=10$ evaluation rollouts per candidate, a total of $8192$ generated agents, and full buffer refresh every fourth training loop. The paper includes $p_{\text{bias}}$ in the pseudocode but does **not** report its numerical value in the experimental section [2508.06336].

The generator-curator pipeline can therefore be summarized as a dynamic curriculum over partner behaviors. Candidate partners are not retained because they are intrinsically strong or intrinsically diverse in isolation; they are retained because they are judged most learnable for the current ego policy. This suggests that UPD should be interpreted less as population synthesis in the classic MARL sense and more as adaptive partner-distribution design [2508.06336].

## 3. Learnability and the “current learning frontier”

The defining selection signal in UPD is a **variance-based learnability metric**. For a candidate partner $\pi_p$, the ego agent is rolled out multiple times with that partner, producing returns $R_1,\dots,R_N$. Learnability is then defined as
$$
\ell_{\text{var}} = \mathrm{Var}_{\tau \sim (\pi, \pi_p)} \left[R(\tau)\right], \quad \text{ where } R \in [0, R_{max}].
$$
The paper’s interpretation is that low variance indicates the pairing is either consistently failing or consistently succeeding, whereas high variance indicates unstable coordination and therefore proximity to the ego agent’s **current learning frontier** [2508.06336].

This frontier-based view is the conceptual core of UPD. Partners that are too close to self-play may be easy and thus uninformative; partners that are too random may be hopeless and also uninformative. Intermediate partners induce partial success and partial failure, producing high return variance and a stronger training signal [2508.06336]. The paper explicitly notes that $\ell_{\text{var}}$ is maximal when half the returns are $0$ and half are $R_{\max}$, directly linking the metric to curriculum-learning intuitions about intermediate difficulty.

UPD positions this learnability score as an adaptation of **sampling-for-learnability (SFL)** from UED. In binary success/failure settings, SFL uses
$$
\ell_{\text{sr}} = p(1-p),
$$
with $p$ the success rate. UPD generalizes that idea to continuous-return cooperative settings by using return variance directly [2508.06336]. The paper also discusses a Gaussian-weighted learnability metric,
$$
\ell_{\text{gauss}} = \sigma_\theta \cdot \mathcal{N}(\mu_\theta \mid \mu, \sigma^2),
$$
but does not adopt it because it downweights performance extremes, whereas ad-hoc teamwork requires sensitivity to both strong and weak partners [2508.06336].

The candidate-selection procedure is correspondingly simple. UPD generates many candidates, evaluates each with $N$ rollouts, computes $\ell = \mathrm{Var}(R_1,\dots,R_N)$, stores $(\ell,\pi_p)$ in a buffer, selects the **top $K$** partners with highest learnability, and samples from those partners during training [2508.06336]. The paper does not report moving averages, normalization, or other approximations for the standard UPD setting, implying that direct empirical return variance is the operative score.

A plausible implication is that the success of UPD depends less on the absolute realism of any one synthetic partner than on whether the generated partner distribution continually tracks the ego policy’s changing region of maximum learning progress. This interpretation is consistent with the paper’s claim that the correct partner distribution is **not stationary** over training [2508.06336].

## 4. RL formulation and integration with environment design

UPD is designed to be **RL-algorithm agnostic**. In the reported experiments, the ego policy is trained with **independent PPO (IPPO)**, but the paper states explicitly that the ego agent may be trained “using any RL method with partners sampled from the buffer” [2508.06336]. The practical role of UPD is therefore to control the **data distribution over partner policies**, not to replace the underlying policy-gradient objective.

For the main Overcooked experiments, the reported PPO configuration is: 512 environments, rollout length 400, total timesteps $5\times 10^7$, reward shaping for the first $3\times 10^7$, PPO epochs 6, minibatches 8, $\gamma = 0.99$, GAE $\lambda = 0.95$, clip $\epsilon=0.2$, entropy coefficient 0.01, value loss coefficient 1.0, and gradient-norm clip 0.5 [2508.06336]. All methods use the same recurrent actor-critic base with an encoder containing 2 embedding layers, actor and critic networks of 4 layers each, hidden size 256, GRU hidden size 256, Tanh activation, and layer normalization enabled. For E3T and UPD, the implementation also includes the partner-modeling module inherited from E3T: past five state-action pairs of the other agent are encoded, a 4-layer MLP predicts the teammate’s next action distribution, the prediction is concatenated with the ego embedding, and an auxiliary cross-entropy loss with MOA loss coefficient 1.0 is used for training [2508.06336]. The paper presents this module as implementation-specific rather than essential to the conceptual definition of UPD.

A major extension is the combination of UPD with **unsupervised environment design (UED)** to produce **Joint UPD (JUPD)**, which constructs a curriculum over the **joint partner–environment space** [2508.06336]. In each scoring step, the method samples an environment parameter $\theta \sim \Theta$, samples a partner $\pi_p \sim \mathcal{G}_p$, executes $N$ episodes in environment $\mathcal{E}(\theta)$ with $(\pi_{\text{ego}}, \pi_p)$, computes a normalized learnability score, and stores $(\ell,\pi_p,\theta)$ in the buffer. Because different levels can have different reward scales, JUPD uses the **coefficient of variation squared**:
$$
\ell_{CV^2} = \frac{\mathrm{Var}_{\tau \sim (\pi, \pi_p, \theta)} [R(\tau, \theta)]}{\left( \mathbb{E}_{\tau \sim (\pi, \pi_p, \theta)} [R(\tau, \theta)] \right)^2}.
$$
The algorithm box also writes this as
$$
\ell_{CV^2} = \text{Var}(R_1,\dots,R_N)/\text{Mean}(R_1,\dots,R_N)^2.
$$
This normalization is described as **scale-invariant** and appropriate for cross-level comparison [2508.06336].

The significance of JUPD lies in its claim to provide the first method enabling **fully unsupervised curricula over both level and partner distributions in a cooperative setting** [2508.06336]. This places UPD within the wider family of adaptive curriculum methods studied in UED. The UED literature emphasizes that when one learned component adaptively generates the training distribution for another learned component, the overall system is vulnerable to non-stationarity, entropy collapse, exploitative curricula, and co-adaptation loops [2308.10797]. These concerns are not merely adjacent to UPD; they describe the same structural problem under a different generated object.

## 5. Empirical results

The main benchmark for UPD is **Overcooked-AI**, a two-player cooperative environment with six discrete actions—up, down, left, right, interact, stay—in which agents coordinate to prepare and serve soups [2508.06336]. The paper evaluates on five standard layouts: Cramped Room (CRoom), Asymmetric Advantages (AA), Coordination Ring (CR), Counter Circuit (CC), and Forced Coordination (FC). For each method, 6 independent seeds are trained. Evaluation is performed in a true ad-hoc setting using novel partner populations that include BRDiv agents trained for self-play while minimizing cross-play compatibility, probabilistic planning agents with human-inspired heuristics, hard-coded specialists such as onion-only workers, and random or stay agents as worst-case partners [2508.06336].

Average returns across layouts are reported as follows: SP 40.4, FCP 45.2, MEP 67.8, E3T 76.8, UPD w/o bias 85.6, UPD w/o $\ell$ 90.0, and **UPD 94.4** [2508.06336]. Per layout, UPD reports 108.1 on CRoom, 181.4 on AA, 69.2 on CR, 64.5 on CC, and 48.7 on FC; the paper characterizes UPD as best overall, with CC tied-best and FC second-best but essentially tied with the best [2508.06336]. These results are presented as evidence of improved **zero-shot coordination with unseen partners**, **robustness to diverse partner types**, and ad-hoc teamwork generalization.

The paper also analyzes the selected $\epsilon$ values and reports layout-dependent adaptation. In rigid-convention layouts such as CR, CC, and FC, UPD tends toward **lower $\epsilon$**, corresponding to more competent or ego-like partners. In more flexible layouts such as CRoom and AA, it favors **higher $\epsilon$**, corresponding to more stochastic partners [2508.06336]. This supports the claim that fixed E3T-style mixture coefficients are suboptimal and that UPD learns task-dependent partner curricula.

Learnability-function ablations compare mean return, adaptive-SR, Gaussian-weighted learnability, and variance-based $\ell_{\text{var}}$. The variance-based metric performs best or equally best, although the paper notes that UPD is fairly robust to the precise learnability function provided it does not degenerate toward pathological partner choices such as self-play only [2508.06336]. Two structural ablations are also reported. **UPD w/o bias** performs well but below the full method, and **UPD w/o $\ell$** also performs strongly, suggesting that the online generator alone induces a useful implicit curriculum; nonetheless, the full method remains best [2508.06336].

For joint partner–environment generalization on the **Overcooked Generalisation Challenge (OGC)**, the reported average returns are DR-DR 49.9, CEC 23.9, SFL-E3T 44.0, and **JUPD 58.9** [2508.06336]. Per layout, JUPD reports 97.0 on CRoom, 60.0 on CR, and 17.1 on FC, outperforming all baselines across the reported layouts. The authors use these results to argue that UPD integrates effectively with UED when the agent must generalize simultaneously over partner and environment distributions [2508.06336].

## 6. Human evaluation, limitations, and relation to adjacent paradigms

The UPD paper includes a double-blind within-subjects user study in Overcooked-AI with 12 participants, ages 26–34, including 5 female participants [2508.06336]. Each participant played with four agents—SP, MEP, E3T, and UPD—across three layouts: AA, CR, and CC, with randomized order, for a total of 144 games and 36 games per method [2508.06336]. After each interaction, participants answered seven Likert-scale questions concerning enjoyment, coordination quality, adaptation, getting in the way, consistency, human-likeness, and frustration. The analysis used one-sided Wilcoxon signed-rank tests for survey questions, one-sided paired t-tests for return, and Holm–Bonferroni correction per question; aggregated preference scores were justified by $\text{Cronbach's } \alpha = 0.938$ [2508.06336]. The study reports that UPD achieved **significantly higher returns than all baselines** and was perceived as more adaptive, more human-like, a better collaborator, and less frustrating [2508.06336].

The paper also identifies several limitations. UPD is currently developed for **discrete action spaces**. Its generated diversity mainly spans competence via $\epsilon$ and low-level action biases via Dirichlet masks; it does **not** yet model richer latent preferences, goals, or intentions [2508.06336]. The appendix further notes that prioritizing the “hardest” partners fails: in Overcooked this drives selection toward near-random partners with $\epsilon \to 1$, and in OGC toward unsolvable levels, collapsing learning [2508.06336]. The exact value of $p_{\text{bias}}$ is not reported, which leaves an implementation caveat. More generally, the paper notes that artificial-agent evaluations alone do not always predict human collaboration performance [2508.06336].

These caveats align with a broader analysis from UED. The paper on stabilizing learned adversarial environment design argues that adaptive generators face generic difficulties: **non-stationarity**, **long-term credit assignment**, and **high-dimensionality**, which manifest as **entropy collapse**, **degenerate task distributions**, and learners that **fall behind** and never recover [2308.10797]. This suggests that a learned partner generator in UPD is susceptible to analogous pathologies: narrow exploitative partner types, anti-learning curricula, or collapse toward behaviorally degenerate partners. The same paper finds that stronger entropy regularization for the generator is often the most important stabilizer, and that online behavioral cloning from a stronger comparator can help when the learner is exposed to tasks it cannot yet explore [2308.10797]. These lessons are not part of the canonical UPD algorithm, but they provide a principled framework for interpreting future extensions of partner design.

A related but distinct paradigm is exemplified by **Partner Approximating Learners (PAL)**. PAL operates in mixed cooperative-competitive continuous-control settings without explicit communication, unknown partner rewards, and delayed observation of partner actions. It learns an explicit partner model $\hat{\pi}_{p_i}$ from observed state–partner-action pairs and uses that model together with known dynamics $f$ to construct an internal simulator for policy optimization [1909.03868]. Its key equations include the mixed cooperative-competitive return
$$
R_i=\sum_{k=0}^{\infty}\gamma_i^k r_i\left(x_k, \pi_{i}(x_k),\pi_{p_i}(x_k)\right),
$$
the aggregated partner control representation $u_{p_i}$, the simulated transition model
$$
\hat{x}_{\hat{k}+1} = f\left(\hat{x}_{\hat{k}}, \hat{u}_{i,\hat{k}}, \hat{\pi}_{p_i}(\hat{x}_{\hat{k}})\right),
$$
and the supervised identification loss
$$
\mathcal{L}_{\text{ID}}(\theta_p) = \mathbb{E}_{(x,u_p)\sim\mathcal{B}_{\text{ID}} \left[\|\hat{\pi}_{p_i}(x;\theta_p)-u_p\|^2\right].
$$
PAL is therefore a method for **modeling partners and adapting to them**, plus training against partner approximations, rather than a method for generating or designing diverse partners [1909.03868]. For UPD, its importance is indirect but substantial: it offers a concrete mechanism for representing and updating a partner surrogate online, something that a richer UPD pipeline could use as a subroutine.

Taken together, these strands place UPD at the intersection of ad-hoc teamwork, curriculum learning, and adaptive training-distribution design. Its distinctive contribution is to turn partner selection into an unsupervised curriculum problem over the space of possible collaborators, while related work clarifies both the algorithmic ingredients needed to stabilize such a curriculum and the complementary role of explicit partner modeling in settings where adaptation to a specific encountered partner is required [2508.06336][2308.10797][1909.03868].

Source: https://www.emergentmind.com/topics/unsupervised-partner-design-upd