---
title: Multi-Agent Inverse Reinforcement Learning
url: https://www.emergentmind.com/topics/multi-agent-inverse-reinforcement-learning-irl
type: topic
---

# Multi-Agent Inverse Reinforcement Learning

Multi-Agent Inverse Reinforcement Learning (IRL) studies the inverse problem of recovering reward functions, cost functions, or utility functions from the observed behavior of multiple interacting agents. In contrast to single-agent IRL, the target behavior is generated under interaction, so the inverse problem is conditioned not only on dynamics and demonstrations but also on equilibrium structure, mean-field consistency, shared constraints, or local-interaction symmetries. The literature therefore spans two-player zero-sum stochastic games [1403.6822], cooperative and general-sum Markov games [2503.04679], continuous-state multi-agent MDPs [2305.10548], mean field games with rewards of the form \(r(s,a,\mu)\) [2202.06401], swarm systems with local rewards \(R(o)\) [1602.05450], and static coordinated sensing problems in which observed actions are rationalized by Pareto-optimal utility maximization under a shared resource constraint [2409.14542].

## 1. Problem scope and motivation

The central motivation for multi-agent IRL is reward misspecification in environments where behavior is jointly generated. In cooperative and competitive systems alike, hand-designed rewards can be brittle, can encode incorrect trade-offs, and can produce pathological policies. In task allocation, manually tuned scalar rewards can fail to balance energy and time objectives and can lead to idling or poorly coordinated behavior; inverse reinforcement learning is used there to infer reward structures from expert demonstrations instead of hand-coding every trade-off [2504.05045]. In general-sum games, the same issue is amplified by non-stationarity and by variance that scales with multiple agents, so that misalignment between learned and true objectives becomes harder to correct by forward RL alone [2503.04679].

A second motivation is that many domains provide demonstrations but not stable behavioral theory. Urban participatory simulation is an explicit example: rather than specifying why pedestrians choose particular paths by means of many ad hoc rules, inverse reinforcement learning seeks a reward function that explains observed behavior, using data from GPS, Wi‑Fi, RFID, and participatory experiments [1712.07887]. A related argument appears in real-world pedestrian crowds, where tightly coupled social interactions such as passing, intersections, swerving, and weaving render simultaneous reward learning for multiple agents intractable under standard formulations, motivating new tractable multi-agent maximum-entropy approximations [2405.16439].

The literature does not treat all multi-agent problems as instances of one canonical model. Some papers assume homogeneous cooperative agents with a shared reward function [2305.10548]; some recover one reward per agent in general-sum games [2503.04679]; some infer utility functions rather than Markovian rewards [2409.14542]; and some reduce a large homogeneous swarm to a representative agent with a local observation-dependent reward [1602.05450]. This suggests that “multi-agent IRL” names a family of inverse problems unified by interaction, not a single algorithmic template.

## 2. Formal models and inverse problem classes

The formal setting varies with the interaction regime. Representative formulations are summarized below.

| Formulation | Core definition | Interaction structure |
|---|---|---|
| Markov game | \(\langle n,\mathcal{S},\mathcal{A},T,\mathcal{R},p_0\rangle\) [2503.04679] | Joint state, joint action, per-agent rewards |
| Cooperative MAMDP | \(\mathcal{M}=(\mathcal{S},\mathcal{A},r,D,N)\) [2305.10548] | Shared dynamics and shared reward, decentralized execution |
| Mean field game | \(P(s' \mid s,a,\mu)\), \(r(s,a,\mu)\), \(\mu_t \in \Delta(S)\) [2202.06401] | Interaction through population distribution |
| swarMDP | \((N,\mathbb{A},T,\xi)\) with local reward \(R(o)\) [1602.05450] | Homogeneous agents, local observations |
| Coordinated sensing | \(\arg\max_{\{\beta^i\}}\sum_i \mu^i f^i(\beta^i)\) s.t. \(\alpha_t'(\sum_i \beta^i)\le 1\) [2409.14542] | Static Pareto-optimal coordination under shared constraint |

In general-sum Markov games, the unknown object is commonly a vector of reward functions \(\mathcal{R}_i(s,\mathbf{a})\), one per agent, and the demonstrations are joint trajectories \(\rho_E=\{(s,\mathbf{a},s')\}\) generated by an expert joint policy \(\boldsymbol{\pi}^E\) [2503.04679]. In cooperative continuous-state MAMDPs, demonstrations are trajectories of multiple interacting agents in continuous state and action spaces, and the inverse problem is to jointly recover a reward network \(r_\vartheta(s,a)\) and a policy \(\pi_\omega(a\mid s)\) such that the induced behavior reproduces observed collective dynamics [2305.10548].

Mean field formulations replace explicit joint state with a representative-agent state and a mean field \(\mu_t\). In that setting, the reward depends on the population state, \(r:S\times A\times \Delta(S)\to \mathbb{R}\), and the mean field evolves by the McKean–Vlasov equation
\[
\mu_{t+1}(s')=\sum_{s\in S}\mu_t(s)\sum_{a\in A}\pi_t(a\mid s)\,P(s'\mid s,a,\mu_t),
\]
so the inverse problem targets a reward \(r(s,a,\mu)\) that makes the demonstrated policy a best response to the demonstrated mean field [2202.06401]. In swarm systems, a further structural simplification is available: under homogeneity and permutation invariance, the agent-specific local value functions coincide, allowing the multi-agent IRL problem to be reduced to single-agent IRL on local observations \(o\) with reward \(R(o)\) [1602.05450].

A distinct branch replaces sequential control with inverse optimization. In coordinated sensing, the analyst observes probe vectors \(\alpha_t\) and noisy joint sensing signals \(\hat{\beta}_t^i\), and seeks concave utility functions \(f^i\) such that the observed joint actions are consistent with Pareto-optimal maximization of a weighted sum of utilities under a shared linear budget [2409.14542]. This is still multi-agent IRL in the sense that utilities are recovered from coordinated behavior, but the forward model is static constrained optimization rather than an MDP or stochastic game.

## 3. Rationality assumptions, equilibrium concepts, and strategic interpretation

A defining issue in multi-agent IRL is what it means for demonstrated behavior to be “optimal.” In two-player zero-sum stochastic games, the natural solution concept is Nash equilibrium in minimax form. Competitive IRL in that setting directly pits expert demonstrations against Nash-equilibrium strategies and defines the inverse problem in terms of performance gaps between experts and equilibrium policies, rather than assuming that demonstrations are exactly optimal [1801.02124]. Earlier Bayesian MIRL for simulated soccer likewise treats the observed bipolicy as a minimax bipolicy and imposes linear constraints encoding equilibrium conditions; the reported underperformance of single-agent IRL is attributed to its failure to capture equilibrium information in the manner possible in MIRL [1403.6822].

Not all work uses explicit game-theoretic equilibrium. In real pedestrian crowds, one formulation adopts the multi-agent maximum-entropy dynamic games framework of Mehr et al. and computes decentralized stochastic policies as a noisy mixed-Nash equilibrium, but then introduces a tractability–rationality trade-off by regularizing covariances when the exact Gaussian equilibrium computation becomes intractable [2405.16439]. In participatory urban simulation, by contrast, the multi-agent setting is treated as many agents acting in a shared environment with weak coupling; interactions such as congestion and flow direction enter the transition dynamics and state features, and the paper explicitly does not invoke Nash or correlated equilibrium concepts [1712.07887].

Mean field work exposes a sharper distinction. Population-level IRL based on reducing an MFG to an MDP over collective variables is valid only in the fully cooperative setting, because the reduction solves a mean-field social optimum rather than a general mean-field Nash equilibrium. The individual-level MFIRL formulation was proposed precisely because existing MFG-to-MDP IRL methods become invalid in non-cooperative environments [2202.06401]. A complementary resolution is to replace hard best-response equilibria with entropy-regularized ones. Mean-Field Adversarial IRL introduces the Entropy-Regularized Mean-Field Nash Equilibrium (ERMFNE), in which each agent maximizes reward plus policy entropy and the resulting equilibrium induces a maximum-entropy trajectory distribution [2104.14654].

A related misconception is that multi-agent IRL always presumes complete mutual knowledge of goals. MIRL via Theory of Mind rejects that assumption: agents are modeled as maintaining posterior distributions over baseline reward profiles for their teammates and updating those beliefs from behavior, then acting according to decentralized best responses to those time-varying beliefs [2302.10238]. In coordinated sensing, the equilibrium concept is different again: observed actions are rationalized by Pareto efficiency under a shared constraint, not by Nash behavior [2409.14542]. Multi-agent IRL therefore depends as much on the chosen notion of rational interaction as on the reward parameterization itself.

## 4. Methodological families

One enduring family is feature-based IRL. The classical single-agent template of Ng and Russell is explicitly cited in the urban participatory simulation literature, where reward is treated as an unknown linear function of features and demonstrations are used to match feature expectations or otherwise recover reward parameters [1712.07887]. Multi-agent maximum-entropy pedestrian IRL retains linear-in-features costs and minimizes discrepancy between feature expectations of expert and generated trajectories, but computes policies through a multi-agent dynamic game with interaction features such as distances to other agents, deviation from goal, and control effort [2405.16439].

Bayesian and constrained formulations remain important in strategic games. In two-player zero-sum soccer, Bayesian MIRL places a Gaussian prior on the reward vector and solves a quadratic program with linear inequalities induced by minimax equilibrium conditions; the single-agent IRL comparison solves an analogous quadratic program after folding the opponent’s policy into an effective MDP [1403.6822]. Competitive IRL with sub-optimal demonstrations extends this line by replacing hard optimality constraints with an objective that compares expert and Nash-equilibrium performance, while using deep neural networks for reward approximation and adversarial PPO-like training for equilibrium computation [1801.02124].

A second major family is adversarial and maximum-entropy learning. In cooperative task allocation, expert demonstrations drive a GAIL-style framework in which a generator outputs trajectory-dependent reward coefficients \(\alpha,\beta\) that modulate a base reward, and a discriminator distinguishes expert from generated trajectories [2504.05045]. MF-AIRL extends AIRL to mean field games by conditioning the discriminator on the empirical mean field and restricting the reward-shaping ambiguity through a potential-based decomposition \(r_\omega(s,a,\mu)+g_\phi(s',\mu')-g_\phi(s,\mu)\) [2104.14654]. In mean-field and continuous-control settings, adversarial learning is explicitly motivated by uncertainty in demonstrations and by the need for scalable approximations of the partition function [2104.14654], [2305.10548].

A third family uses off-policy or critic-based reductions. IMARL combines Guided Cost Learning with ReF-ER MARL to recover a shared nonlinear reward in continuous state and action spaces, using importance sampling over both demonstrations and background trajectories collected under past policies [2305.10548]. MAMQL instead learns one marginalized critic \(\bar{Q}_{\psi_i}(s,a_i)\) per agent, where the critic is averaged over the other agents’ policies; this yields a multi-agent generalization of soft-Q IRL with Boltzmann policies based on marginalized critics rather than full joint-action critics [2503.04679]. In online opponent modeling, Recursive Deep IRL replaces batch guided cost learning with sequential second-order Newton updates akin to an Extended Kalman Filter, enabling online recovery of an adversary’s cost function from behavior [2504.13241].

Recent work also injects structural priors into reward recovery. Attention- and graph-based models in task allocation use multi-head self-attention to encode trajectory segments and graph attention to encode agent–task relations before predicting reward coefficients [2504.05045]. Symmetry-Guided Multi-Agent IRL formalizes data augmentation under the dihedral group \(\mathcal{D}_4\) and proves that symmetry-guided augmentation improves a worst-case upper bound on reward approximation error in cooperative Markov games, then instantiates the idea as a plug-in framework for MA-GAIL and MA-AIRL [2509.08257]. In swarms, homogeneity and permutation invariance provide an even stronger reduction: the multi-agent IRL problem collapses to single-agent IRL on local observations because agent-specific value functions coincide [1602.05450].

## 5. Application domains and empirical behavior

The application range is broad. Early MIRL comparisons used a simulated soccer game on a \(4\times 5\) grid with 800 states and showed that MIRL recovered the underlying reward and position-dependent shooting success structure substantially better than IRL, while policies derived from MIRL-learned rewards outperformed those derived from IRL-learned rewards [1403.6822]. Swarm IRL was demonstrated on the Vicsek model and an Ising model, where the learned local reward produced alignment and low-energy collective behavior comparable to expert swarm dynamics [1602.05450]. Competitive deep MIRL later scaled zero-sum IRL with sub-optimal demonstrations to a \(5\times5\) and \(10\times10\) chasing game and reported reward correlations around \(0.65\)–\(0.68\) with the true reward across different sub-optimality levels, together with much smaller performance deterioration than Bayesian or decentralized baselines [1801.02124].

Crowd and urban applications split into conceptual and empirical strands. Participatory urban simulation introduced IRL as a way to replace brittle hand-crafted rules in multi-agent pedestrian models and argued for hybrid systems that combine domain knowledge, local knowledge, and stakeholder interaction [1712.07887]. Real-world unstructured pedestrian crowds subsequently adopted a multi-agent maximum-entropy formulation and, on the dense Speedway dataset, reported first place among seven baselines with greater than \(2\times\) improvement over single-agent IRL, while remaining competitive with large transformer-based encoder–decoder models on sparser ETH/UCY data, where it ranked third among seven baselines [2405.16439].

Cooperative robotic and allocation domains have recently become a major testbed. In multi-agent task allocation, the attention-augmented IRL method reported higher cumulative rewards and lower time-step consumption than MAPPO and MASAC across sparse, normal, and dense reward regimes; for the normal setting with 10 agents and 40 tasks, it reported average reward \(174.07\) and average steps \(74.06\), versus \(147.20/102.99\) for MAPPO and \(123.85/114.90\) for MASAC [2504.05045]. In general-sum games, MAMQL reported significant improvements in average reward, sample efficiency, and reward recovery, often by more than \(2\)–\(5\times\), across Gems, Overcooked, and a four-car highway intersection [2503.04679]. Symmetry-guided MIRL further showed improved sample efficiency in rendezvous, pursuit, and Vicsek tasks, and also improved cumulative distance metrics in physical Limo-robot rendezvous and pursuit experiments [2509.08257].

Large-population and continuous-control applications emphasize reward recovery from collective behavior. IMARL recovered about \(95\%\) of maximal return achieved by forward RL on Swimmer-v4 and Hopper-v4, about \(65\%\) on Walker2d-v4, and in the multi-agent milling task stabilized the averaged rotation at approximately \(0.8\) after about \(1\) million steps, matching the demonstration level [2305.10548]. In search-and-rescue, MIRL-ToM recovered rewards that produced low feature-count differences and small Jensen–Shannon divergences from expert behavior in both known-teammate and unknown-teammate settings, while also showing that reward learning failed to converge when the baseline profile set contained only the “opposite” goal model [2302.10238].

Sensing and adversarial domains demonstrate that the same inverse problem appears outside standard path planning. In cognitive radar sensing, Wasserstein-robust inverse utility estimation preserved average Pareto-surface reconstruction error while reducing worst-case Hausdorff error from \(0.9012\) to \(0.4624\) over 100 Monte Carlo runs [2409.14542]. In adversarial cognitive radar pursuit, Recursive Deep IRL learned the radar’s Fisher Information Matrix objective from a single trajectory and reported mean cumulative reward \(498.25\), versus \(392.51\) for GAN-GCL, \(249.13\) for GCL, and \(190.37\) for GAIL [2504.13241].

## 6. Evaluation, limitations, and current directions

The most persistent theoretical issue is identifiability. Urban participatory simulation explicitly notes that no ground-truth reward is available and that quantitative evaluation is difficult, so models are qualitatively validated instead [1712.07887]. In MFGs, identifiability is entangled with equilibrium concept: MFIRL shows that the MFG-to-MDP reduction used by earlier methods is valid only in the fully cooperative setting, so reward recovery based on population-level trajectories can be biased in non-cooperative environments [2202.06401]. In static coordinated sensing, the same ambiguity appears as rationalizability of noisy data by multiple concave utilities, which is why the paper moves from naive Afriat-style reconstruction to a minimax distributionally robust objective over a Wasserstein ambiguity set [2409.14542].

A second difficulty is bounded rationality and suboptimality. Several papers reject exact optimality assumptions. ERMFNE in mean-field games introduces policy entropy to model stochastic, imperfect demonstrations [2104.14654]. Competitive IRL in zero-sum stochastic games is explicitly designed for sub-optimal demonstrations and optimizes the gap between experts and Nash-equilibrium strategies instead of forcing demonstrations to be equilibria [1801.02124]. MIRL-ToM goes further by assuming that agents may not even know each other’s goals, so the inverse problem must reconstruct rewards through evolving beliefs over baseline reward profiles [2302.10238]. This suggests that multi-agent IRL increasingly treats “as-if optimality” as a modeling device rather than a literal behavioral assumption.

Scalability remains the main computational bottleneck. State–action spaces grow exponentially with the number of agents in explicit multi-agent models [1712.07887]. Mean field games address this by replacing joint interaction with a population distribution [2202.06401], and swarMDPs exploit homogeneity to reduce the inverse problem to local observation space [1602.05450]. Other approaches rely on architectural priors and approximations: graph attention and hierarchical self-attention in task allocation [2504.05045], marginalized critics in general-sum games [2503.04679], symmetry-guided augmentation in cooperative MIRL [2509.08257], and the tractability–rationality trade-off in dense pedestrian IRL [2405.16439].

Open directions recur across papers. Task allocation highlights heterogeneity, dynamic task arrivals, partial observability, dynamic graph structures, and richer reward models beyond affine scaling as immediate extensions [2504.05045]. MFIRL identifies continuous state and action spaces, non-equilibrium demonstrations, partial observability of the mean field, and multiple equilibria as outstanding problems [2202.06401]. Robust coordinated sensing suggests extending from static budget-constrained inverse optimization to dynamic multi-agent MDPs with distributional robustness [2409.14542]. A plausible implication is that the field is converging on hybrid solutions: structural reductions when symmetry or population regularity is present, adversarial or off-policy estimators when demonstrations are noisy, and explicitly belief-aware models when strategic uncertainty between agents is itself part of the behavior to be explained.

Source: https://www.emergentmind.com/topics/multi-agent-inverse-reinforcement-learning-irl