Multi-Agent Inverse Reinforcement Learning

• Multi-Agent IRL is a framework for deducing reward functions from collective agent interactions under various equilibrium conditions, addressing reward misspecification challenges.
• Methodologies range from game-theoretic models (e.g., Nash, mean-field) to maximum-entropy and adversarial learning techniques, enabling analysis of cooperative, competitive, and swarm systems.
• Empirical studies in robotics, urban simulations, and adversarial domains demonstrate improved reward recovery, scalability, and policy performance in complex multi-agent environments.

Multi-Agent Inverse Reinforcement Learning (IRL) studies the inverse problem of recovering reward functions, cost functions, or utility functions from the observed behavior of multiple interacting agents. In contrast to single-agent IRL, the target behavior is generated under interaction, so the inverse problem is conditioned not only on dynamics and demonstrations but also on equilibrium structure, mean-field consistency, shared constraints, or local-interaction symmetries. The literature therefore spans two-player zero-sum stochastic games (Lin et al., 2014), cooperative and general-sum Markov games (Haynam et al., 6 Mar 2025), continuous-state multi-agent MDPs (Waelchli et al., 2023), mean field games with rewards of the form $r(s,a,\mu)$ (Chen et al., 2022), swarm systems with local rewards $R(o)$ (Šošić et al., 2016), and static coordinated sensing problems in which observed actions are rationalized by Pareto-optimal utility maximization under a shared resource constraint (Snow et al., 2024).

1. Problem scope and motivation

The central motivation for multi-agent IRL is reward misspecification in environments where behavior is jointly generated. In cooperative and competitive systems alike, hand-designed rewards can be brittle, can encode incorrect trade-offs, and can produce pathological policies. In task allocation, manually tuned scalar rewards can fail to balance energy and time objectives and can lead to idling or poorly coordinated behavior; inverse reinforcement learning is used there to infer reward structures from expert demonstrations instead of hand-coding every trade-off (Yin et al., 7 Apr 2025). In general-sum games, the same issue is amplified by non-stationarity and by variance that scales with multiple agents, so that misalignment between learned and true objectives becomes harder to correct by forward RL alone (Haynam et al., 6 Mar 2025).

A second motivation is that many domains provide demonstrations but not stable behavioral theory. Urban participatory simulation is an explicit example: rather than specifying why pedestrians choose particular paths by means of many ad hoc rules, inverse reinforcement learning seeks a reward function that explains observed behavior, using data from GPS, Wi‑Fi, RFID, and participatory experiments (Suzuki, 2017). A related argument appears in real-world pedestrian crowds, where tightly coupled social interactions such as passing, intersections, swerving, and weaving render simultaneous reward learning for multiple agents intractable under standard formulations, motivating new tractable multi-agent maximum-entropy approximations (Chandra et al., 2024).

The literature does not treat all multi-agent problems as instances of one canonical model. Some papers assume homogeneous cooperative agents with a shared reward function (Waelchli et al., 2023); some recover one reward per agent in general-sum games (Haynam et al., 6 Mar 2025); some infer utility functions rather than Markovian rewards (Snow et al., 2024); and some reduce a large homogeneous swarm to a representative agent with a local observation-dependent reward (Šošić et al., 2016). This suggests that “multi-agent IRL” names a family of inverse problems unified by interaction, not a single algorithmic template.

2. Formal models and inverse problem classes

The formal setting varies with the interaction regime. Representative formulations are summarized below.

Formulation	Core definition	Interaction structure
Markov game	$$\langle n,\mathcal{S},\mathcal{A},T,\mathcal{R},p_0\rangle$$ (Haynam et al., 6 Mar 2025)	Joint state, joint action, per-agent rewards
Cooperative MAMDP	$\mathcal{M}=(\mathcal{S},\mathcal{A},r,D,N)$ (Waelchli et al., 2023)	Shared dynamics and shared reward, decentralized execution
Mean field game	$P(s' \mid s,a,\mu)$, $r(s,a,\mu)$, $\mu_t \in \Delta(S)$ (Chen et al., 2022)	Interaction through population distribution
swarMDP	$(N,\mathbb{A},T,\xi)$ with local reward $R(o)$ (Šošić et al., 2016)	Homogeneous agents, local observations
Coordinated sensing	$\arg\max_{\{\beta^i\}}\sum_i \mu^i f^i(\beta^i)$ s.t. $R(o)$0 (Snow et al., 2024)	Static Pareto-optimal coordination under shared constraint

In general-sum Markov games, the unknown object is commonly a vector of reward functions $R(o)$1, one per agent, and the demonstrations are joint trajectories $R(o)$2 generated by an expert joint policy $R(o)$3 (Haynam et al., 6 Mar 2025). In cooperative continuous-state MAMDPs, demonstrations are trajectories of multiple interacting agents in continuous state and action spaces, and the inverse problem is to jointly recover a reward network $R(o)$4 and a policy $R(o)$5 such that the induced behavior reproduces observed collective dynamics (Waelchli et al., 2023).

Mean field formulations replace explicit joint state with a representative-agent state and a mean field $R(o)$6. In that setting, the reward depends on the population state, $R(o)$7, and the mean field evolves by the McKean–Vlasov equation

$R(o)$8

so the inverse problem targets a reward $R(o)$9 that makes the demonstrated policy a best response to the demonstrated mean field (Chen et al., 2022). In swarm systems, a further structural simplification is available: under homogeneity and permutation invariance, the agent-specific local value functions coincide, allowing the multi-agent IRL problem to be reduced to single-agent IRL on local observations $$\langle n,\mathcal{S},\mathcal{A},T,\mathcal{R},p_0\rangle$$0 with reward $$\langle n,\mathcal{S},\mathcal{A},T,\mathcal{R},p_0\rangle$$1 (Šošić et al., 2016).

A distinct branch replaces sequential control with inverse optimization. In coordinated sensing, the analyst observes probe vectors $$\langle n,\mathcal{S},\mathcal{A},T,\mathcal{R},p_0\rangle$$2 and noisy joint sensing signals $$\langle n,\mathcal{S},\mathcal{A},T,\mathcal{R},p_0\rangle$$3, and seeks concave utility functions $$\langle n,\mathcal{S},\mathcal{A},T,\mathcal{R},p_0\rangle$$4 such that the observed joint actions are consistent with Pareto-optimal maximization of a weighted sum of utilities under a shared linear budget (Snow et al., 2024). This is still multi-agent IRL in the sense that utilities are recovered from coordinated behavior, but the forward model is static constrained optimization rather than an MDP or stochastic game.

3. Rationality assumptions, equilibrium concepts, and strategic interpretation

A defining issue in multi-agent IRL is what it means for demonstrated behavior to be “optimal.” In two-player zero-sum stochastic games, the natural solution concept is Nash equilibrium in minimax form. Competitive IRL in that setting directly pits expert demonstrations against Nash-equilibrium strategies and defines the inverse problem in terms of performance gaps between experts and equilibrium policies, rather than assuming that demonstrations are exactly optimal (Wang et al., 2018). Earlier Bayesian MIRL for simulated soccer likewise treats the observed bipolicy as a minimax bipolicy and imposes linear constraints encoding equilibrium conditions; the reported underperformance of single-agent IRL is attributed to its failure to capture equilibrium information in the manner possible in MIRL (Lin et al., 2014).

Not all work uses explicit game-theoretic equilibrium. In real pedestrian crowds, one formulation adopts the multi-agent maximum-entropy dynamic games framework of Mehr et al. and computes decentralized stochastic policies as a noisy mixed-Nash equilibrium, but then introduces a tractability–rationality trade-off by regularizing covariances when the exact Gaussian equilibrium computation becomes intractable (Chandra et al., 2024). In participatory urban simulation, by contrast, the multi-agent setting is treated as many agents acting in a shared environment with weak coupling; interactions such as congestion and flow direction enter the transition dynamics and state features, and the paper explicitly does not invoke Nash or correlated equilibrium concepts (Suzuki, 2017).

Mean field work exposes a sharper distinction. Population-level IRL based on reducing an MFG to an MDP over collective variables is valid only in the fully cooperative setting, because the reduction solves a mean-field social optimum rather than a general mean-field Nash equilibrium. The individual-level MFIRL formulation was proposed precisely because existing MFG-to-MDP IRL methods become invalid in non-cooperative environments (Chen et al., 2022). A complementary resolution is to replace hard best-response equilibria with entropy-regularized ones. Mean-Field Adversarial IRL introduces the Entropy-Regularized Mean-Field Nash Equilibrium (ERMFNE), in which each agent maximizes reward plus policy entropy and the resulting equilibrium induces a maximum-entropy trajectory distribution (Chen et al., 2021).

A related misconception is that multi-agent IRL always presumes complete mutual knowledge of goals. MIRL via Theory of Mind rejects that assumption: agents are modeled as maintaining posterior distributions over baseline reward profiles for their teammates and updating those beliefs from behavior, then acting according to decentralized best responses to those time-varying beliefs (Wu et al., 2023). In coordinated sensing, the equilibrium concept is different again: observed actions are rationalized by Pareto efficiency under a shared constraint, not by Nash behavior (Snow et al., 2024). Multi-agent IRL therefore depends as much on the chosen notion of rational interaction as on the reward parameterization itself.

4. Methodological families

One enduring family is feature-based IRL. The classical single-agent template of Ng and Russell is explicitly cited in the urban participatory simulation literature, where reward is treated as an unknown linear function of features and demonstrations are used to match feature expectations or otherwise recover reward parameters (Suzuki, 2017). Multi-agent maximum-entropy pedestrian IRL retains linear-in-features costs and minimizes discrepancy between feature expectations of expert and generated trajectories, but computes policies through a multi-agent dynamic game with interaction features such as distances to other agents, deviation from goal, and control effort (Chandra et al., 2024).

Bayesian and constrained formulations remain important in strategic games. In two-player zero-sum soccer, Bayesian MIRL places a Gaussian prior on the reward vector and solves a quadratic program with linear inequalities induced by minimax equilibrium conditions; the single-agent IRL comparison solves an analogous quadratic program after folding the opponent’s policy into an effective MDP (Lin et al., 2014). Competitive IRL with sub-optimal demonstrations extends this line by replacing hard optimality constraints with an objective that compares expert and Nash-equilibrium performance, while using deep neural networks for reward approximation and adversarial PPO-like training for equilibrium computation (Wang et al., 2018).

A second major family is adversarial and maximum-entropy learning. In cooperative task allocation, expert demonstrations drive a GAIL-style framework in which a generator outputs trajectory-dependent reward coefficients $$\langle n,\mathcal{S},\mathcal{A},T,\mathcal{R},p_0\rangle$$5 that modulate a base reward, and a discriminator distinguishes expert from generated trajectories (Yin et al., 7 Apr 2025). MF-AIRL extends AIRL to mean field games by conditioning the discriminator on the empirical mean field and restricting the reward-shaping ambiguity through a potential-based decomposition $$\langle n,\mathcal{S},\mathcal{A},T,\mathcal{R},p_0\rangle$$6 (Chen et al., 2021). In mean-field and continuous-control settings, adversarial learning is explicitly motivated by uncertainty in demonstrations and by the need for scalable approximations of the partition function (Chen et al., 2021, Waelchli et al., 2023).

A third family uses off-policy or critic-based reductions. IMARL combines Guided Cost Learning with ReF-ER MARL to recover a shared nonlinear reward in continuous state and action spaces, using importance sampling over both demonstrations and background trajectories collected under past policies (Waelchli et al., 2023). MAMQL instead learns one marginalized critic $$\langle n,\mathcal{S},\mathcal{A},T,\mathcal{R},p_0\rangle$$7 per agent, where the critic is averaged over the other agents’ policies; this yields a multi-agent generalization of soft-Q IRL with Boltzmann policies based on marginalized critics rather than full joint-action critics (Haynam et al., 6 Mar 2025). In online opponent modeling, Recursive Deep IRL replaces batch guided cost learning with sequential second-order Newton updates akin to an Extended Kalman Filter, enabling online recovery of an adversary’s cost function from behavior (Ghanem et al., 17 Apr 2025).

Recent work also injects structural priors into reward recovery. Attention- and graph-based models in task allocation use multi-head self-attention to encode trajectory segments and graph attention to encode agent–task relations before predicting reward coefficients (Yin et al., 7 Apr 2025). Symmetry-Guided Multi-Agent IRL formalizes data augmentation under the dihedral group $$\langle n,\mathcal{S},\mathcal{A},T,\mathcal{R},p_0\rangle$$8 and proves that symmetry-guided augmentation improves a worst-case upper bound on reward approximation error in cooperative Markov games, then instantiates the idea as a plug-in framework for MA-GAIL and MA-AIRL (Tian et al., 10 Sep 2025). In swarms, homogeneity and permutation invariance provide an even stronger reduction: the multi-agent IRL problem collapses to single-agent IRL on local observations because agent-specific value functions coincide (Šošić et al., 2016).

5. Application domains and empirical behavior

The application range is broad. Early MIRL comparisons used a simulated soccer game on a $$\langle n,\mathcal{S},\mathcal{A},T,\mathcal{R},p_0\rangle$$9 grid with 800 states and showed that MIRL recovered the underlying reward and position-dependent shooting success structure substantially better than IRL, while policies derived from MIRL-learned rewards outperformed those derived from IRL-learned rewards (Lin et al., 2014). Swarm IRL was demonstrated on the Vicsek model and an Ising model, where the learned local reward produced alignment and low-energy collective behavior comparable to expert swarm dynamics (Šošić et al., 2016). Competitive deep MIRL later scaled zero-sum IRL with sub-optimal demonstrations to a $\mathcal{M}=(\mathcal{S},\mathcal{A},r,D,N)$0 and $\mathcal{M}=(\mathcal{S},\mathcal{A},r,D,N)$1 chasing game and reported reward correlations around $\mathcal{M}=(\mathcal{S},\mathcal{A},r,D,N)$2–$\mathcal{M}=(\mathcal{S},\mathcal{A},r,D,N)$3 with the true reward across different sub-optimality levels, together with much smaller performance deterioration than Bayesian or decentralized baselines (Wang et al., 2018).

Crowd and urban applications split into conceptual and empirical strands. Participatory urban simulation introduced IRL as a way to replace brittle hand-crafted rules in multi-agent pedestrian models and argued for hybrid systems that combine domain knowledge, local knowledge, and stakeholder interaction (Suzuki, 2017). Real-world unstructured pedestrian crowds subsequently adopted a multi-agent maximum-entropy formulation and, on the dense Speedway dataset, reported first place among seven baselines with greater than $\mathcal{M}=(\mathcal{S},\mathcal{A},r,D,N)$4 improvement over single-agent IRL, while remaining competitive with large transformer-based encoder–decoder models on sparser ETH/UCY data, where it ranked third among seven baselines (Chandra et al., 2024).

Cooperative robotic and allocation domains have recently become a major testbed. In multi-agent task allocation, the attention-augmented IRL method reported higher cumulative rewards and lower time-step consumption than MAPPO and MASAC across sparse, normal, and dense reward regimes; for the normal setting with 10 agents and 40 tasks, it reported average reward $\mathcal{M}=(\mathcal{S},\mathcal{A},r,D,N)$5 and average steps $\mathcal{M}=(\mathcal{S},\mathcal{A},r,D,N)$6, versus $\mathcal{M}=(\mathcal{S},\mathcal{A},r,D,N)$7 for MAPPO and $\mathcal{M}=(\mathcal{S},\mathcal{A},r,D,N)$8 for MASAC (Yin et al., 7 Apr 2025). In general-sum games, MAMQL reported significant improvements in average reward, sample efficiency, and reward recovery, often by more than $\mathcal{M}=(\mathcal{S},\mathcal{A},r,D,N)$9–$P(s' \mid s,a,\mu)$0, across Gems, Overcooked, and a four-car highway intersection (Haynam et al., 6 Mar 2025). Symmetry-guided MIRL further showed improved sample efficiency in rendezvous, pursuit, and Vicsek tasks, and also improved cumulative distance metrics in physical Limo-robot rendezvous and pursuit experiments (Tian et al., 10 Sep 2025).

Large-population and continuous-control applications emphasize reward recovery from collective behavior. IMARL recovered about $P(s' \mid s,a,\mu)$1 of maximal return achieved by forward RL on Swimmer-v4 and Hopper-v4, about $P(s' \mid s,a,\mu)$2 on Walker2d-v4, and in the multi-agent milling task stabilized the averaged rotation at approximately $P(s' \mid s,a,\mu)$3 after about $P(s' \mid s,a,\mu)$4 million steps, matching the demonstration level (Waelchli et al., 2023). In search-and-rescue, MIRL-ToM recovered rewards that produced low feature-count differences and small Jensen–Shannon divergences from expert behavior in both known-teammate and unknown-teammate settings, while also showing that reward learning failed to converge when the baseline profile set contained only the “opposite” goal model (Wu et al., 2023).

Sensing and adversarial domains demonstrate that the same inverse problem appears outside standard path planning. In cognitive radar sensing, Wasserstein-robust inverse utility estimation preserved average Pareto-surface reconstruction error while reducing worst-case Hausdorff error from $P(s' \mid s,a,\mu)$5 to $P(s' \mid s,a,\mu)$6 over 100 Monte Carlo runs (Snow et al., 2024). In adversarial cognitive radar pursuit, Recursive Deep IRL learned the radar’s Fisher Information Matrix objective from a single trajectory and reported mean cumulative reward $P(s' \mid s,a,\mu)$7, versus $P(s' \mid s,a,\mu)$8 for GAN-GCL, $P(s' \mid s,a,\mu)$9 for GCL, and $r(s,a,\mu)$0 for GAIL (Ghanem et al., 17 Apr 2025).

6. Evaluation, limitations, and current directions

The most persistent theoretical issue is identifiability. Urban participatory simulation explicitly notes that no ground-truth reward is available and that quantitative evaluation is difficult, so models are qualitatively validated instead (Suzuki, 2017). In MFGs, identifiability is entangled with equilibrium concept: MFIRL shows that the MFG-to-MDP reduction used by earlier methods is valid only in the fully cooperative setting, so reward recovery based on population-level trajectories can be biased in non-cooperative environments (Chen et al., 2022). In static coordinated sensing, the same ambiguity appears as rationalizability of noisy data by multiple concave utilities, which is why the paper moves from naive Afriat-style reconstruction to a minimax distributionally robust objective over a Wasserstein ambiguity set (Snow et al., 2024).

A second difficulty is bounded rationality and suboptimality. Several papers reject exact optimality assumptions. ERMFNE in mean-field games introduces policy entropy to model stochastic, imperfect demonstrations (Chen et al., 2021). Competitive IRL in zero-sum stochastic games is explicitly designed for sub-optimal demonstrations and optimizes the gap between experts and Nash-equilibrium strategies instead of forcing demonstrations to be equilibria (Wang et al., 2018). MIRL-ToM goes further by assuming that agents may not even know each other’s goals, so the inverse problem must reconstruct rewards through evolving beliefs over baseline reward profiles (Wu et al., 2023). This suggests that multi-agent IRL increasingly treats “as-if optimality” as a modeling device rather than a literal behavioral assumption.

Scalability remains the main computational bottleneck. State–action spaces grow exponentially with the number of agents in explicit multi-agent models (Suzuki, 2017). Mean field games address this by replacing joint interaction with a population distribution (Chen et al., 2022), and swarMDPs exploit homogeneity to reduce the inverse problem to local observation space (Šošić et al., 2016). Other approaches rely on architectural priors and approximations: graph attention and hierarchical self-attention in task allocation (Yin et al., 7 Apr 2025), marginalized critics in general-sum games (Haynam et al., 6 Mar 2025), symmetry-guided augmentation in cooperative MIRL (Tian et al., 10 Sep 2025), and the tractability–rationality trade-off in dense pedestrian IRL (Chandra et al., 2024).

Open directions recur across papers. Task allocation highlights heterogeneity, dynamic task arrivals, partial observability, dynamic graph structures, and richer reward models beyond affine scaling as immediate extensions (Yin et al., 7 Apr 2025). MFIRL identifies continuous state and action spaces, non-equilibrium demonstrations, partial observability of the mean field, and multiple equilibria as outstanding problems (Chen et al., 2022). Robust coordinated sensing suggests extending from static budget-constrained inverse optimization to dynamic multi-agent MDPs with distributional robustness (Snow et al., 2024). A plausible implication is that the field is converging on hybrid solutions: structural reductions when symmetry or population regularity is present, adversarial or off-policy estimators when demonstrations are noisy, and explicitly belief-aware models when strategic uncertainty between agents is itself part of the behavior to be explained.