Matching Multiple Experts: On the Exploitability of Multi-Agent Imitation Learning

Abstract: Multi-agent imitation learning (MA-IL) aims to learn optimal policies from expert demonstrations of interactions in multi-agent interactive domains. Despite existing guarantees on the performance of the resulting learned policies, characterizations of how far the learned polices are from a Nash equilibrium are missing for offline MA-IL. In this paper, we demonstrate impossibility and hardness results of learning low-exploitable policies in general $n$-player Markov Games. We do so by providing examples where even exact measure matching fails, and demonstrating a new hardness result on characterizing the Nash gap given a fixed measure matching error. We then show how these challenges can be overcome using strategic dominance assumptions on the expert equilibrium. Specifically, for the case of dominant strategy expert equilibria, assuming Behavioral Cloning error $ε<em>{\text{BC}}$, this provides a Nash imitation gap of $$\mathcal{O}\left(nε</em>{\text{BC}}/(1-γ)<sup>2\right)$$ for a discount factor $γ$. We generalize this result with a new notion of best-response continuity, and argue that this is implicitly encouraged by standard regularization techniques.

• The paper shows that exact state-action matching guarantees zero Nash gap only with full-state support, while state-only matching or incomplete support can permit exploitability of order Ω(1/(1−γ)).
• The paper proves that computing the tightest Nash-gap lower bound under imitation error is PPAD-hard, limiting efficient certification of how exploitable learned multi-agent policies are.
• The paper derives a tractable bound of 2nεBC/(1−γ)^2 for dominant-strategy equilibria and extends it using best-response continuity, while finding that entropy regularization can improve empirical stability.

Problem setting and motivation

Multi-agent imitation learning (MA-IL) seeks to recover good joint policies from expert demonstrations of strategic interactions. While single-agent imitation learning (SA-IL) enjoys well-understood guarantees relating imitation error to sub-optimality, the multi-agent case introduces a distinct failure mode: a learned policy may imitate the expert closely yet be severely exploitable by agents who unilaterally deviate. The paper formalizes this concern through the Nash gap—the maximum regret any player can obtain by deviating from the learned product policy—and asks when the Nash gap can be bounded consistently (vanishing as the imitation error vanishes) and tractably (computable in polynomial time from game assumptions).

The setting is $n$-player infinite-horizon Markov Games with discounted rewards in $[-1,1]$, where an expert policy $\pi^E$ is a Nash equilibrium. The learner has access to trajectories from $\pi^E$ and incurs one of three standard errors: a behavioral cloning (BC) error $\epsilon_{\text{BC}}$ on per-agent conditional policies under the expert state distribution, or occupancy-measure matching errors $\epsilon_\mu$ (state-only) and $\epsilon_\rho$ (state-action). By contrast, the Value gap always admits consistent, tractable bounds of order $\mathcal{O}(n\epsilon_{\text{BC}}/(1-\gamma)^2)$ via the performance difference lemma, or $\mathcal{O}(\epsilon_\rho/(1-\gamma))$. The paper's central contribution is showing that no analogous general result exists for the Nash gap, and characterizing exactly which structural assumptions restore it.

Impossibility results for exact measure matching

The first set of results concerns the idealized regime of zero imitation error, asking when exact matching recovers the expert Nash equilibrium. The positive anchor is a sufficiency theorem: if two policies share the same state-action occupancy measure $\rho_\pi$, they coincide on their common visited region; consequently, exact state-action matching under full-state support ($[-1,1]$0) yields $[-1,1]$1. This identifies state-action occupancy measures as sufficient statistics for equilibrium recovery on the expert's support.

Both ingredients of this guarantee are necessary, and relaxing either produces linear-in-horizon failure:

• State-only matching fails even with full-state support. The authors construct a cooperative two-player game with three states and uniform initial distribution, where every policy trivially visits all states. The expert plays $[-1,1]$2 everywhere, inducing a uniform state distribution; another policy playing $[-1,1]$3 matches $[-1,1]$4 exactly but yields $[-1,1]$5, i.e., $[-1,1]$6. The mechanism is that rewards depend on state-action pairs while state distributions do not pin down transitions. This directly undermines approaches that match only state distributions, such as score-based diffusion over states.
• State-action matching fails without full-state support. Adapting a construction of Tang et al., the authors show a two-player game where $[-1,1]$7 exactly, yet a best response routes play through unvisited states to a high-reward absorbing region, again giving Nash gap $[-1,1]$8. The dataset simply contains no information about expert behavior off its support.

Together these results show that consistency of exploitability bounds requires both full-state coverage and state-action-level matching—a substantially stronger requirement than what SA-IL theory needs.

Hardness of tractable lower bounds

Moving beyond the zero-error regime, the paper asks whether one can at least certify a best-case Nash gap given a fixed matching error $[-1,1]$9. The relevant quantity $\pi^E$0 is the minimum Nash gap achievable by any policy within $\pi^E$1 of some expert equilibrium. The main hardness theorem establishes that computing $\pi^E$2 for arbitrary bimatrix games is PPAD-hard, even when rewards and dynamics are fully known; a corollary extends this to general Markov games via one-state reductions.

The proof proceeds by reduction: first showing that finding the support of an (approximate) Nash equilibrium in bimatrix games is PPAD-complete via the indifference principle and a linear program over the support; then giving a polynomial-time algorithm that, given an oracle for $\pi^E$3, peels off actions one at a time (by perturbing payoffs with a large negative constant) to recover equilibrium supports. Since computing approximate Nash equilibria is PPAD-complete (2602.21020), so is evaluating the tight lower bound.

Two caveats are stated plainly: the hardness does not rule out analytical bounds expressed as min-max optimization problems (themselves computationally hard), and it applies specifically to tight lower bounds rather than loose certificates. Still, the implication is that any polynomial-time exploitable-gap certification would be no easier than solving for equilibria directly, foreclosing a natural approach to auditing MA-IL outputs.

Best-response continuity and positive upper bounds

The constructive half of the paper introduces $\pi^E$4-continuity of the best-response correspondence: a game is $\pi^E$5-continuous at equilibrium $\pi^E$6 if small deviations of opponents from $\pi^E$7 (measured in expected $\pi^E$8 distance under $\pi^E$9) force every best response $\pi^E$0 to remain within $\pi^E$1 of $\pi^E$2. This property is absent from SA-IL analysis and captures how sharply the induced MDP's optimal behavior changes under opponent perturbation—the phenomenon that makes the Nash gap fundamentally harder than the Value gap.

A sharp negative observation frames the landscape: even the class of games admitting consistent bounds is $\pi^E$3-continuous only for the trivial function $\pi^E$4 for all $\pi^E$5, demonstrated via a chain construction where BC error $\pi^E$6 at a rare state forces a completely different best response. At the opposite extreme, $\pi^E$7 corresponds exactly to dominant strategy equilibria (DSE), where each player's expert policy is a best response to any opponent profile.

Under DSE, the paper proves a clean bound:

$\pi^E$8

obtained by rewriting the Nash gap using dominance and applying the performance difference lemma twice, together with a factored $\pi^E$9 bound on product distributions. For $\epsilon_{\text{BC}}$0 this recovers the concentrability-based bound of Freihaut et al. with a fixed BC error—but unlike that prior work, the bound here never becomes vacuous, since it depends only on the observable BC error rather than an intractable concentrability coefficient. Inverting the bound yields a practical criterion: achieving an $\epsilon_{\text{BC}}$1-Nash equilibrium requires $\epsilon_{\text{BC}}$2.

The generalization to arbitrary $\epsilon_{\text{BC}}$3 gives:

$\epsilon_{\text{BC}}$4

so consistency reduces to $\epsilon_{\text{BC}}$5 and tractability to computability of $\epsilon_{\text{BC}}$6. The authors position $\epsilon_{\text{BC}}$7 as a spectrum of equilibrium sensitivity, with DSE and pathological games at the extremes, and argue—supported by numerical experiments on a two-player Tag-Game—that entropy regularization smooths the best-response map and empirically lowers the tightest achievable $\epsilon_{\text{BC}}$8 across noise levels. They also hypothesize that risk-aversion penalties could yield better $\epsilon_{\text{BC}}$9 functions, though this remains conjectural. All main results are extended to finite-horizon games in the appendix, with $\epsilon_\mu$0 replaced by $\epsilon_\mu$1.

Limitations and open questions

Several limitations are acknowledged explicitly. The DSE assumption is strong—dominant-strategy equilibria exclude many games of interest, including mixed-equilibrium settings like matching pennies—and the paper provides no characterization of $\epsilon_\mu$2 for broad natural game classes beyond noting that entropy regularization helps empirically. The numerical validation is confined to a single small zero-sum environment, and the authors caution that the upper bound is worst-case over $\epsilon_\mu$3-continuous classes and generally not tight for a specific game; indeed, higher temperature reduced $\epsilon_\mu$4 but was not shown to reduce the actual Nash gap. The PPAD-hardness result concerns tight lower bounds and leaves open whether loose but informative analytical certificates exist. Finally, the authors identify reachability assumptions, distribution-norm-based analyses, and data augmentation with suboptimal demonstrations (in the spirit of online IL) as candidate directions for tighter game-dependent bounds, none of which are developed here.

Conclusion

This paper delineates precisely when offline MA-IL can deliver non-exploitable policies. Exact state-action matching with full-state support suffices; dropping either condition permits linear-in-horizon exploitability even with perfect matching; certifying best-case exploitability under approximation error is PPAD-hard; and tractable, consistent upper bounds require structural assumptions on best-response continuity, with dominant-strategy experts yielding a $\epsilon_\mu$5 Nash gap guarantee. The work clarifies why naive extensions of SA-IL guarantees fail in games and supplies a concrete regularity condition—amenable to entropy regularization—under which behavioral cloning becomes a defensible equilibrium-learning method.