Multi-Objective Markov Game

• Multi-Objective Markov Game is a framework for multi-agent sequential decision-making where each agent receives a vector of rewards, capturing trade-offs via Pareto dominance.
• The framework links Pareto efficiency with Nash equilibria by converting multi-objective rewards into scalarized forms to compute equilibria using strategies like Online Learning via Scalarized Nash Value Iteration.
• Algorithmic approaches, including two-phase methods and weaker equilibrium notions, address computational challenges and enable adaptive planning under multiple conflicting objectives.

A Multi-Objective Markov Game (MOMG) is a formal framework for multi-agent sequential decision-making where each agent receives a vector of rewards, representing multiple objectives, at every decision point rather than a scalar reward. This structure induces intricate strategic trade-offs: each agent’s outcomes on several criteria depend on the joint actions of all agents and evolve stochastically over a Markovian state space. The MOMG framework generalizes both single-objective Markov games and single-agent multi-objective MDPs, making it highly relevant to practical multi-agent systems with complex and often conflicting goal structures (Wang, 27 Sep 2025).

1. Formal Definition and Structure

MOMGs are defined by the tuple

$$G = (\mathcal{N}, \mathcal{S}, \{\mathcal{A}_i\}_{i\in\mathcal{N}}, P, \{r_i\}_{i\in\mathcal{N}}, H),$$

where:

• $\mathcal{N}$: set of agents,
• $\mathcal{S}$: finite or infinite state space,
• $\mathcal{A}_i$: action set for agent $i$,
• $P$: transition kernel, $P(s'|s,a_1,\ldots,a_n)$,
• $r_i(s, \mathbf{a})$: $m$-dimensional vector reward for agent $i$,
• $H$: horizon (finite or infinite).

Each agent $i$ seeks to maximize its expected vectorial return—typically the discounted sum over time—across $m$ objectives: elements of $r_i(s_t, \mathbf{a}_t) \in \mathbb{R}^m$. In contrast to standard Markov games, value comparisons are generally performed under the Pareto dominance ordering, as scalar comparisons are insufficient.

Policies $\pi_i$ select actions by mapping information such as state (or observation histories, in partially observed settings) to distributions over $\mathcal{A}_i$. The jointly induced Markov process governs the evolution of system dynamics and multi-objective rewards.

2. Solution Concepts: Pareto–Nash Equilibrium and Scalarization

The central solution concept introduced for MOMGs is the Pareto-Nash Equilibrium (PNE) (Wang, 27 Sep 2025):

• A policy profile $\pi^* = (\pi_1^*, \ldots, \pi_n^*)$ is a PNE if for every agent $i$, there is no alternative policy $\pi_i$ such that the expected cumulative reward vector $V_i^{(\pi_i, \pi^*_{-i})}$ Pareto-dominates $V_i^{\pi^*}$ at the initial state.

Mathematically,

$$\nexists~\pi_i:~V_i^{(\pi_i, \pi^*_{-i})}(s_0)\succ V_i^{\pi^*}(s_0)$$

where $\succ$ denotes strict Pareto improvement.

A key result is that the PNE set in a MOMG coincides with the union over all Nash equilibria of corresponding scalarized single-objective Markov games:

$$\operatorname{PNE}(G) = \bigcup_{\lambda \in (\Delta^0_m)^n} \operatorname{NE}(G_\lambda)$$

where $\Delta^0_m$ is the interior of the probability simplex on $m$ objectives and $G_\lambda$ is the Markov game where each agent’s reward is $r_i^\lambda = \lambda_i^\top r_i$ (Wang, 27 Sep 2025).

Thus, every PNE of a MOMG can be found as a Nash equilibrium of some linearly scalarized Markov game for some strictly positive weights $\lambda$. This formally connects Pareto efficiency and individual optimality within the MOMG context.

3. Computational Complexity and Weaker Solution Notions

Even though existence of PNE is established via correspondence to scalarized game equilibria, practical computation is difficult. The set of PNE is typically large and complex; establishing whether a given policy profile is a PNE requires ruling out any Pareto improvement via unilateral deviation. Thus, enumeration or naïve optimization over all possible policies and preference profiles is computationally challenging.

To address this, the framework also proposes and analyzes more tractable, weaker equilibrium notions:

• Weak Pareto–Nash Equilibrium (WPNE): Agents cannot unilaterally strictly Pareto-improve, but weak improvement (no objective is worsened, at least one is improved) need not be excluded.
• Pareto–Correlated Equilibrium (PCE): Agents may use correlated/mediated strategies, relaxing the independence of policy choices and thereby simplifying computation.

These relaxations allow for more efficient algorithmic solutions at the expense of equilibrium stringency.

4. Algorithmic Approaches for MOMGs

Two main algorithmic techniques are developed:

A. Online Learning via Scalarized Nash Value Iteration (ONVI–MG):

• For a given preference profile $\lambda$, formulate the MOMG as a single-objective Markov game $G_\lambda$.
• Use an optimistic value iteration algorithm that computes Q-values with empirical transition and reward estimates plus upper confidence bonuses:

$$Q_h^{(i,t)}(s, a) = \min\{H, \lambda_i^\top \hat{r}_h^{(i,t)}(s, a) + \Psi_h^t(s, a) + \sum_{s'} \hat{P}_h^t(s'|s,a) U_{h+1}^{(i,t)}(s') + \Phi_h^t(s, a)\}$$

• Backward induction and Nash equilibrium computation at each stage yield a policy profile; cumulative Nash regret is used for theoretical analysis.
• Guarantees convergence to an $\varepsilon$-WPNE as the sample size increases.

B. Two-Phase, Preference-Free Algorithm:

• Phase 1 (Exploration): The environment is explored robustly, sharing sample collection across all scalarization weights.
• Phase 2 (Planning): Given a constructed empirical model, for any $\lambda$ the agents can efficiently replan—computing the corresponding NE—without further sample collection.

This decoupling enables efficient computation of the entire Pareto-Nash front and allows fast adaptation to new, possibly changing, agent preferences (Wang, 27 Sep 2025).

5. Theoretical Insights and Methodological Characterization

The union characterization of the Pareto-Nash front as

$$\mathcal{P} = \bigcup_{\lambda \in (\Delta^0_m)^n} \operatorname{NE}(G_\lambda)$$

enables systematic exploration of trade-offs and facilitates algorithm design for multi-objective multi-agent learning. Unlike scalarization-based approaches that require user-specified preferences a priori, this framework achieves a general policy set covering all strictly positive preference profiles.

Efficient regret bounds and finite-sample guarantees are obtainable under the proposed learning schemes. Critically, the two-phase algorithm ensures that, once a sufficiently accurate model is constructed, new equilibria for any preference profile can be computed without additional environment interaction. This is important for decision support in domains where preference specification is iterative or uncertain.

6. Relevance, Limitations, and Implications

MOMGs provide a unifying framework for sequential multi-agent decision problems with multiple objectives, generalizing single-objective Markov games and MOMDPs. The Pareto-Nash viewpoint formally captures multi-criteria trade-offs at the equilibrium, applicable to domains such as resource allocation, supply chain management, multi-objective negotiation, and multi-agent reinforcement learning.

Methodologically, the scalar reduction approach is powerful but computationally intense as the number of objectives and agents increases. Existence is theoretically assured, but scalable computation of the entire Pareto front remains a significant challenge—especially in real-world settings featuring many agents or high-dimensional objectives. Weaker equilibrium concepts and carefully designed learning algorithms are therefore essential.

The explicit decoupling of exploration and planning in the two-phase methodology (Wang, 27 Sep 2025) is particularly valuable in practice, supporting rapid adaptation to changing stakeholder preferences or objectives. The framework also clarifies the relation of MOMGs to other classes of multi-objective games, Markov potential games, and correlated equilibrium concepts in sequential settings, providing a foundation for further algorithmic and theoretical advancements in the field.