Repeated Multi-Objective Stackelberg Games

• Repeated multi-objective Stackelberg games are dynamic leader–follower interactions played over multiple stages with vector-valued payoffs and explicit scalarization techniques.
• They encompass models like Markov games, multi-period regulatory frameworks, and trajectory-based controls, with convergence achieved through fixed-point theory and learning algorithms.
• Applications span environmental regulation, energy tariff design, and autonomous manufacturing, offering practical insights while addressing significant computational challenges.

Repeated multi-objective Stackelberg games are leader–follower games played over multiple stages or over an infinite horizon in which the leader’s decision problem involves several objectives, either explicitly as vector-valued payoffs or costs, or implicitly through scalarized criteria such as efficiency, equity, stability, tax revenue, or environmental damage. In the literature summarized here, the repeated aspect appears as finite-horizon multi-period games, receding-horizon trajectory games, long-run average stochastic games, and infinite-horizon discounted Markov games, while the multi-objective aspect appears through vector costs, explicit upper-level bi-objectives, hierarchical internal objectives, or linear scalarization (He et al., 19 Sep 2025, Sinha et al., 2013, Kalathil et al., 2014, Srisawad et al., 20 Aug 2025).

1. Formal models and representations

A standard dynamic formulation is the Stackelberg Markov game

$$\mathcal G_S := (\mathcal S, \mathcal A_L, \mathcal A_F, P, r_L, r_F, \gamma),$$

where $\mathcal S$ is the state space, $\mathcal A_L,\mathcal A_F$ are the leader’s and follower’s action sets, $P(\cdot\mid s,a_L,a_F)$ is the transition kernel, $r_i$ are one-step rewards, and $\gamma=(\gamma_L,\gamma_F)\in[0,1)^2$ are discount factors. Under stationary policies, the interaction repeats indefinitely: the leader acts first, the follower reacts, the state evolves, and discounted rewards accumulate. Under continuity and boundedness assumptions, stationary policies suffice for optimality in infinite-horizon discounted settings (He et al., 19 Sep 2025).

Finite-horizon repeated formulations are equally important. In multi-period multi-leader–multi-follower competition, each leader and follower chooses a sequence of period decisions such as production, investment, and marketing, and periods are linked by cumulative investment and marketing as well as budget constraints. In the environmental-economics regulator–mine model, the leader chooses a tax schedule $(\tau_1,\dots,\tau_T)$, while the follower chooses extraction quantities $(q_1,\dots,q_T)$ and a technology $a$, with total tax revenue and total environmental damage evaluated over the horizon (Sinha et al., 2013, Sinha et al., 2013). A recurrent misconception is that a multi-period Stackelberg model is merely a sequence of independent stage games. In these models, cumulative investment, marketing, extraction, purification costs, and stock constraints couple periods dynamically, so the full horizon must be solved jointly rather than period by period (Sinha et al., 2013, Sinha et al., 2013).

In explicitly multi-objective stochastic formulations, the leader’s stage cost is vector-valued,

$$\underline{c} : \mathcal S \times \mathcal A \to \mathbb{R}^{K},$$

and the repeated object of interest is the average vector cost

$\mathcal S$0

The target is not a single optimal scalar payoff but a set $\mathcal S$1 that the leader tries to approach in the long run, irrespective of the follower’s strategy (Kalathil et al., 2014).

A more direct multi-objective repeated Stackelberg model uses vector-valued stage payoffs

$\mathcal S$2

with linear utilities $\mathcal S$3. In that setting the follower’s weight vector is unknown, the follower plays a deterministic best response, and the leader may manipulate the follower’s payoff by offering an additional vector $\mathcal S$4 (Srisawad et al., 20 Aug 2025).

Repeated Stackelberg interaction can also be defined at the trajectory level. In Stackelberg trajectory games, the leader and follower choose control trajectories for discrete-time LTI systems with Gaussian noise, and the game is played in receding-horizon form: at each step both agents solve horizon-$\mathcal S$5 problems, apply the first control input, observe the new state, and repeat (Ward et al., 2023).

2. Equilibrium and solution concepts

For infinite-horizon Markov settings, the central object is the stationary Stackelberg equilibrium (SSE). The follower chooses a stationary best response to the leader’s stationary policy, and the leader chooses a stationary policy that maximizes its own long-run value against that response. Under continuity and boundedness, unique best responses, and Lipschitz best-response mappings, the condition

$\mathcal S$6

implies existence and uniqueness of the SSE via Banach’s fixed-point theorem applied to the composite best-response map (He et al., 19 Sep 2025).

When followers form a continuum, the equilibrium concept becomes a stationary Stackelberg–Mean Field Equilibrium (SS-MFE), a triple $\mathcal S$7 satisfying follower optimality, mean-field consistency, and leader optimality. Existence and uniqueness again follow from contraction conditions on follower best responses, leader best responses, and the mean-field update map $\mathcal S$8 (He et al., 19 Sep 2025). This provides a repeated-game analogue of Stackelberg equilibrium in large populations.

In multi-objective settings, the solution concept depends on how objectives are aggregated. One common construction is scalarization: the leader’s vector objective $\mathcal S$9 is mapped to a scalar reward $\mathcal A_L,\mathcal A_F$0. For a fixed scalarization, existence and uniqueness results from scalar Markov-game theory apply; varying $\mathcal A_L,\mathcal A_F$1 or the weight vector traces a Pareto frontier of repeated equilibria (He et al., 19 Sep 2025). In the regulator–mine model, the leader’s upper-level objective is explicitly bi-objective,

$\mathcal A_L,\mathcal A_F$2

and the relevant outcome set is a set of Pareto Stackelberg equilibria (Sinha et al., 2013). A common point of confusion is that vector objectives eliminate the need for scalarization. In fact, several existence and uniqueness results apply only after scalarization; without scalarization, equilibrium becomes a Pareto equilibrium and may be set-valued (He et al., 19 Sep 2025).

Repeated multi-leader settings motivate asymmetric correlated notions. In general-sum multi-leader-single-follower games, a Correlated Stackelberg Equilibrium (CSE) is a distribution $\mathcal A_L,\mathcal A_F$3 over leader joint actions such that no leader can improve, up to $\mathcal A_L,\mathcal A_F$4, by any swap function $\mathcal A_L,\mathcal A_F$5 once the follower best-responds to the deviated recommendation (Yu et al., 2022). In repeated games with a learning follower, another solution notion is local rather than global: an $\mathcal A_L,\mathcal A_F$6-approximate local Stackelberg strategy is a leader mixed strategy $\mathcal A_L,\mathcal A_F$7 such that no nearby strategy $\mathcal A_L,\mathcal A_F$8 can improve the leader’s Stackelberg payoff by more than $\mathcal A_L,\mathcal A_F$9 (Ananthakrishnan et al., 26 Oct 2025).

For vector-cost stochastic games, approachability replaces equilibrium optimization by set convergence. A closed set $P(\cdot\mid s,a_L,a_F)$0 is approachable if the leader can force $P(\cdot\mid s,a_L,a_F)$1 almost surely, regardless of the follower’s strategy. For convex $P(\cdot\mid s,a_L,a_F)$2, the paper recovers Blackwell’s necessary and sufficient condition in a Stackelberg stochastic game: for every $P(\cdot\mid s,a_L,a_F)$3, the leader must be able to induce an ergodic vector cost lying in the supporting half-space of $P(\cdot\mid s,a_L,a_F)$4 at $P(\cdot\mid s,a_L,a_F)$5 under worst-case follower behavior (Kalathil et al., 2014).

3. Learning and computation

A prominent computational template is alternating best-response learning. In Stackelberg Markov games, the leader–follower interaction is implemented by a policy-iteration-like RL framework: for a given leader policy $P(\cdot\mid s,a_L,a_F)$6, compute the follower’s best response $P(\cdot\mid s,a_L,a_F)$7, for example with PPO; then update the leader via $P(\cdot\mid s,a_L,a_F)$8; and repeat until convergence. Crucially, no explicit knowledge of follower objectives is required, because the leader learns from observed follower behavior and induced state transitions (He et al., 19 Sep 2025). To stabilize best-response dynamics, the non-smooth $P(\cdot\mid s,a_L,a_F)$9 is replaced by a Boltzmann policy, and projected softmax iterates converge to an $r_i$0-approximation of the true SSE under suitable temperatures and projection radius. Entropy regularization further ensures unique and Lipschitz best responses (He et al., 19 Sep 2025).

For complex finite-horizon bilevel problems with discrete and nonlinear decisions, evolutionary bilevel optimization remains central. The multi-period multi-leader–multi-follower competition model is solved by a nested steady-state evolutionary algorithm: an outer GA optimizes leader strategies, and for each upper-level offspring an inner GA computes followers’ optimal responses (Sinha et al., 2013). In the multi-period regulator–mine model, the extended bi-objective upper level is solved with the Hybrid Bilevel Evolutionary Multi-objective Optimization algorithm, which maintains an archive of non-dominated upper-level solutions paired with lower-level optima (Sinha et al., 2013).

Distributed multi-objective repeated Stackelberg learning appears in DS2-SbPG, where each physical player hosts an internal Stackelberg game between its own objectives inside a state-based potential game. The single-leader–follower variant assigns some objectives to a leader role and others to a follower role; the stacked variant builds a hierarchy $r_i$1 of objectives and composes several local Stackelberg games. Learning uses performance maps, polynomial utility regression, and Stackelberg gradient updates, while the global game remains an SbPG under the stated locality assumptions (Yuwono et al., 2024).

Convex–concave min–max Stackelberg games with dependent strategy sets admit first-order methods as well. Plain GDA may fail to converge to a Stackelberg equilibrium, but two variants with access to optimal KKT multipliers do converge in $r_i$2 iterations to an $r_i$3-Stackelberg equilibrium, improving on algorithms with $r_i$4 complexity (Goktas et al., 2022). In Fisher markets, the resulting dynamics coincide with buyers and sellers using myopic best-response dynamics in a repeated market (Goktas et al., 2022).

In repeated multi-leader-single-follower games with noisy bandit feedback, learning CSE requires combining leader-side no-external Stackelberg-regret algorithms with follower-side best-response learning. The paper develops Hedge and EXP3 variants for full-information and bandit settings, and an $r_i$5EXP3-UCB scheme for noisy bandit feedback; via a reduction from no-external regret to no-swap regret, the empirical joint distribution converges to approximate CSE (Yu et al., 2022).

4. Multi-objective structure, trade-offs, and manipulation

Multi-objective Stackelberg structure is often encoded by scalarization. In the energy-market Markov formulation, the leader’s repeated objective can be written as

$r_i$6

with $r_i$7, and a scalar reward obtained by $r_i$8. In practice this appears as a weighted combination of efficiency, EEI-gap, and IMV terms (He et al., 19 Sep 2025). This scalarization is standard in multi-objective RL and permits existence, uniqueness, and approximation guarantees for repeated equilibria.

The regulator–mine model makes the trade-off explicit at the upper level: maximize tax revenue while minimizing environmental damage over multiple periods, subject to a lower-level extraction-and-technology problem. In the simplified single-period case, weighted-sum scalarization yields closed-form expressions for $r_i$9 and $\gamma=(\gamma_L,\gamma_F)\in[0,1)^2$0; in the realistic multi-period case, the bi-objective outcome is a Pareto front of taxation and extraction schedules (Sinha et al., 2013).

An alternative to fixed scalarization is hierarchical decomposition. DS2-SbPG replaces weighted sums of local objectives by internal Stackelberg structure: one objective or coalition of objectives acts as leader, another as follower, and in the stacked variant previous coalitions become leaders for subsequent objectives. The framework explicitly “replaces fixed scalarization weights with a hierarchical multi-stage optimization” (Yuwono et al., 2024). This suggests a different interpretation of repeated multi-objective Stackelberg games: the hierarchy need not be only between agents, but can also be internal to each agent’s objective architecture.

Vector-payoff manipulation introduces a distinct multi-objective mechanism. In the repeated manipulation model, the leader solves the Optimal Manipulation Problem

$\gamma=(\gamma_L,\gamma_F)\in[0,1)^2$1

subject to the incentive constraint

$\gamma=(\gamma_L,\gamma_F)\in[0,1)^2$2

with either unrestricted nonnegative transfers or componentwise sharing constraints (Srisawad et al., 20 Aug 2025). Under linear utilities, the optimal cost has strong structure: under (C1) it is either $\gamma=(\gamma_L,\gamma_F)\in[0,1)^2$3 or has exactly one positive component; under (C2) it is either $\gamma=(\gamma_L,\gamma_F)\in[0,1)^2$4 or saturates some payoff components and covers the remainder from one objective (Srisawad et al., 20 Aug 2025). Informative costs are those for which the manipulated payoff is strictly larger than the best-response payoff in at least one objective and strictly smaller in at least one objective, since only such offers shrink the feasible region of follower weights (Srisawad et al., 20 Aug 2025).

Approachability offers yet another multi-objective interpretation. Rather than fixing a single scalarization or a Pareto point, the leader dynamically scalarizes the vector cost using

$\gamma=(\gamma_L,\gamma_F)\in[0,1)^2$5

and solves a scalarized Stackelberg game at each current average $\gamma=(\gamma_L,\gamma_F)\in[0,1)^2$6, thereby steering the long-run average toward a target set $\gamma=(\gamma_L,\gamma_F)\in[0,1)^2$7 (Kalathil et al., 2014). This replaces “optimal trade-off selection” by “set-valued performance control.”

5. Domains and empirical findings

In electricity tariff design, the leader is a public utility or state utility commission setting time-varying rates for a heterogeneous population of prosumers. The repeated Stackelberg–mean field formulation targets economic efficiency, equity, and system stability simultaneously. In the reported simulation, with storage and RL, IMV drops by ~3 units relative to a no-learning baseline; prosumers learn to charge storage midday and discharge in evening peaks; the learned tariff assigns higher fixed charges to higher-income groups; and the EEI gap between prosumers and consumers is reduced from ~1% to ~0.7% of income while maintaining grid cost recovery and avoiding a “death spiral” scenario (He et al., 19 Sep 2025).

In autonomous decentralized manufacturing, DS2-SbPG and Stack DS2-SbPG were evaluated on the Bulk Good Laboratory Plant. All methods met demand and avoided overflow, but power consumption fell from 0.485297 kW/s for vanilla SbPG to 0.436577 kW/s for DS2-SbPG and 0.433793 kW/s for Stack DS2-SbPG, corresponding to 10.04% and 10.61% reductions. Potential values also improved, from 11.932937 for SbPG to 14.566021 for DS2-SbPG (Yuwono et al., 2024). These experiments illustrate repeated multi-objective Stackelberg control in a distributed physical system with hybrid actuation and local information.

In environmental economics, the government–mine case study provides a canonical repeated multi-objective Stackelberg model: the regulator sets a multi-period tax schedule, the mine chooses extraction quantities and technology, and the leader trades off total tax revenue against total environmental damage. The simplified model admits an analytical solution, while the realistic model requires bilevel evolutionary multi-objective optimization (Sinha et al., 2013).

In Stackelberg trajectory games, active inverse learning was demonstrated using virtual TurtleBots in Gazebo. Rather than passively observing follower trajectories, the leader chooses control inputs that maximize differences between the follower’s predicted trajectories under competing objective hypotheses. Compared with uniformly random inputs, the optimized inputs accelerate the convergence of posterior probabilities over the hypothesis set (Ward et al., 2023).

Repeated market interactions also fit the Stackelberg template. In Fisher markets, the oracle-assisted GDA formulation induces myopic best-response dynamics between buyers and sellers and converges in $\gamma=(\gamma_L,\gamma_F)\in[0,1)^2$8 iterations to an $\gamma=(\gamma_L,\gamma_F)\in[0,1)^2$9-competitive equilibrium (Goktas et al., 2022).

6. Complexity, limitations, and research directions

Several theoretical guarantees depend on scalarization, regularity, or strong structural assumptions. In scalarized Stackelberg Markov games, uniqueness follows from contraction assumptions; without scalarization, repeated multi-objective equilibrium becomes a Pareto equilibrium, uniqueness may fail, and set-valued solution concepts may be required (He et al., 19 Sep 2025). This is a central conceptual limitation rather than a technical detail.

Computational burden remains severe in general bilevel repeated games. In the 2-leader, 5-follower, 5-period model solved by nested evolutionary search, upper-level evaluations are roughly 3,500–6,300, while lower-level evaluations are about 39 million–76 million per run. The method is easily parallelizable, but the scale of nested lower-level optimization is itself a substantive obstacle (Sinha et al., 2013).

Repeated interaction with learning followers introduces a different hardness barrier. For mean-based learners, the global Stackelberg value can require an exponential number of rounds in the size of the follower’s action space, so recent work shifts to local Stackelberg equilibria. Even there, the best known smoothed-analysis PTAS runs in time polynomial in action-space size but exponential in $(\tau_1,\dots,\tau_T)$0, and an $(\tau_1,\dots,\tau_T)$1 lower bound shows that the exponential dependence on precision is unavoidable (Ananthakrishnan et al., 26 Oct 2025).

Finite-horizon repeated games are also sensitive to the number of players. For two-player repeated games, approximate Stackelberg GPAs can be computed with an optimal $(\tau_1,\dots,\tau_T)$2 rate at the expense of exponential dependence on the number of actions, or with a randomized $(\tau_1,\dots,\tau_T)$3 rate and no dependence on the number of actions. In three-player finite-horizon repeated games, however, approximating the Stackelberg value within $(\tau_1,\dots,\tau_T)$4 is NP-hard via a reduction from balanced vertex cover (Collina et al., 2022).

Modeling assumptions differ sharply across subliteratures. CSE learning in multi-leader-single-follower games assumes that the follower best-response set $(\tau_1,\dots,\tau_T)$5 is singleton for every joint leader action (Yu et al., 2022). Approachability in Stackelberg stochastic games assumes finite state and action spaces and irreducibility or at least a unichain property under every stationary policy pair (Kalathil et al., 2014). DS2-SbPG assumes locality of utilities, continuous differentiability, and adequate polynomial-regression approximation of utilities (Yuwono et al., 2024). Repeated payoff manipulation assumes a fully rational and myopic deterministic follower, known payoff matrices, and linear utility for the main theorems, with convergence established only as $(\tau_1,\dots,\tau_T)$6 and with substantial computational overhead for probabilistic feasible-region integration (Srisawad et al., 20 Aug 2025).

These constraints suggest a common research trajectory: richer repeated multi-objective Stackelberg models are becoming increasingly expressive, but their tractability still depends on regularization, scalarization, structural sparsity, or restricted feedback. A plausible implication is that future progress will continue to combine equilibrium concepts with approximate learning architectures—mean-field reductions, local-equilibrium search, distributed potential-game embeddings, and active preference elicitation—rather than rely on exact global solution of the fully general problem.