Private Markovian Equilibrium (PME)

• PME is an equilibrium concept where players use private Markovian strategies based solely on their own state and random shocks, ensuring analytical tractability in dynamic games.
• It formalizes equilibrium conditions across nonatomic, mean-field, and Stackelberg settings through Bellman-type recursions and fixed-point consistency requirements.
• Applications of PME in cyber-physical security and smart grid demand response demonstrate its utility in approximating large-scale dynamic game behaviors.

Searching arXiv for papers on "Private Markovian Equilibrium" and related mean-field / Markov game terminology. Private Markovian Equilibrium (PME) denotes an equilibrium concept for dynamic games in which each agent’s strategy is restricted to be Markovian and to depend on information that is private to that agent, together with whatever public state variables the model treats as commonly observed. In the nonatomic-game formulation of Adlakha, Johari, Weintraub, and Goldsmith, a PME is a private Markov policy $\phi$ that is optimal against the deterministic environment path generated when all players use $\phi$ (Yang, 2015). In the non-stationary mean-field-game formulation of Vasal, PME is a symmetric Markov perfect equilibrium in which the prescription $\sigma_t(a\mid \theta,z)$ depends on a player’s private type $\theta$ and the current population state $z$, with equilibrium characterized by a Bellman-type fixed point and a mean-field consistency condition (Vasal, 2019). In the Stackelberg Markov-game formulation for smart-grid demand response, PME refers to a lower-level equilibrium of users whose private states are storage levels and whose strategies are private Markovian strategies (PMS) (Huang et al., 6 Sep 2025). These formulations share the same structural idea—Markovian decision rules conditioned on private state information—but differ materially in what counts as the relevant public environment and in how equilibrium consistency is enforced.

1. Conceptual scope and terminology

The defining feature of PME is the restriction to private Markovian strategies. In the nonatomic-game model, a private (mixed) Markov policy is a measurable kernel

$\phi:S\longrightarrow\mathcal P(A),$

or equivalently a measurable function $x:S\times G\to A$, where the player uses only her own current state $s_t$ and an idiosyncratic pre-action shock $g_t$ to randomize over actions (Yang, 2015). The resulting equilibrium is “private” because the policy depends only on the player’s own state and private randomization, not on the realized states or actions of others.

In Vasal’s mean-field model, the terminology is slightly different. The game contains a large population of homogeneous players, each with a private type $x_t^i\in X$, and a commonly observed aggregate state $\phi$0 equal to the empirical distribution of private types. A Markovian strategy profile is a sequence of prescription functions $\phi$1, so the equilibrium depends on both a player’s private type and the current dynamic population state (Vasal, 2019). Here, privacy does not mean that the strategy ignores the aggregate environment; rather, it means that the individual’s private type is a primitive argument of the prescription.

In the smart-grid Stackelberg Markov game, the lower-level users have private states $\phi$2, where $\phi$3 is user $\phi$4’s storage level, and a PMS is a profile $\phi$5 with

$\phi$6

A PME is then a joint profile $\phi$7 such that no user can improve by deviating to another PMS (Huang et al., 6 Sep 2025). This suggests that PME is best understood as a family of equilibrium concepts rather than a single canonical definition: what remains invariant is the Markovian restriction together with conditioning on agent-specific private state information.

2. Formal definitions in the main model classes

In nonatomic games, the environment at date $\phi$8 is a joint distribution $\phi$9 of other players’ states and actions, the one-period payoff is

$\sigma_t(a\mid \theta,z)$0

and state evolution is governed by

$\sigma_t(a\mid \theta,z)$1

When all players use the same $\sigma_t(a\mid \theta,z)$2, the law of $\sigma_t(a\mid \theta,z)$3 is deterministic and evolves as

$\sigma_t(a\mid \theta,z)$4

Given an initial state distribution $\sigma_t(a\mid \theta,z)$5, the pair $\sigma_t(a\mid \theta,z)$6 is a PME if

$\sigma_t(a\mid \theta,z)$7

where $\sigma_t(a\mid \theta,z)$8 is the discounted expected payoff of a single agent facing the deterministic environment induced by $\sigma_t(a\mid \theta,z)$9 (Yang, 2015).

In the non-stationary mean-field-game model, a PME is a collection $\theta$0 satisfying three coupled conditions. First, the value functions $\theta$1 satisfy the Bellman optimality equation

$\theta$2

Second, once the maximizer is selected, one sets

$\theta$3

Third, the mean field evolves consistently according to

$\theta$4

For finite horizon the boundary condition is $\theta$5; for infinite horizon, the definition becomes a time-stationary fixed point (Vasal, 2019).

In the Stackelberg Markov-game setting, the lower-level user value under PMS is

$\theta$6

and $\theta$7 is a PME if, for every $\theta$8, every $\theta$9, and every alternative PMS $z$0,

$z$1

The paper also gives an equivalent Bellman-type best-response characterization against fixed $z$2 (Huang et al., 6 Sep 2025).

3. State variables, information structures, and equilibrium consistency

A central distinction across PME formulations is the object that summarizes the strategic environment. In Vasal’s model, the public environment is the mean-field state $z$3, the empirical distribution of all players’ private types. If all players use the same prescription $z$4, then the mean field updates by

$z$5

or equivalently $z$6 (Vasal, 2019). Consistency is therefore endogenous: the same prescription that is optimal at the individual level also determines the evolution of the aggregate state.

In the nonatomic-game model, the analogous object is $z$7, the deterministic law of states and actions under the common policy $z$8. Individual agents do not condition on the realized profile of others’ states or actions; instead, they optimize against the deterministic sequence $z$9 implied by mass behavior. The paper emphasizes that such equilibria “pay no attention to players’ external environments” and are therefore adoptable in settings where awareness of others’ states can be anywhere between full and non-existent (Yang, 2015).

In the smart-grid model, the public signal is renewable generation $\phi:S\longrightarrow\mathcal P(A),$0, while the private signal is the user’s storage level $\phi:S\longrightarrow\mathcal P(A),$1. The leader chooses real-time-pricing parameters $\phi:S\longrightarrow\mathcal P(A),$2, which define the price

$\phi:S\longrightarrow\mathcal P(A),$3

Users choose $\phi:S\longrightarrow\mathcal P(A),$4 subject to

$\phi:S\longrightarrow\mathcal P(A),$5

and their stage reward is

$\phi:S\longrightarrow\mathcal P(A),$6

Here the consistency requirement is not a mean-field fixed point in the sense of (Vasal, 2019); instead, it is a lower-level Markov-game equilibrium among users under the leader’s announced pricing rule (Huang et al., 6 Sep 2025).

A common misconception is to treat “private” as implying complete ignorance of aggregate variables. The cited models show otherwise. In (Yang, 2015), privacy means conditioning only on own state and private randomization. In (Vasal, 2019) and (Huang et al., 6 Sep 2025), private strategies may still depend on public state variables such as $\phi:S\longrightarrow\mathcal P(A),$7 or $\phi:S\longrightarrow\mathcal P(A),$8. What is excluded is conditioning on the full profile of other players’ private states.

4. Existence and equilibrium structure

Existence results for PME are established under different structural assumptions in each model class. In the nonatomic-game setting, assumptions (A1)–(A4) require compact metric spaces $\phi:S\longrightarrow\mathcal P(A),$9 and $x:S\times G\to A$0, continuity of $x:S\times G\to A$1, joint continuity of the transition kernel $x:S\times G\to A$2, and fixed atomless shock laws $x:S\times G\to A$3 and $x:S\times G\to A$4. Under these conditions and $x:S\times G\to A$5, there exists at least one private Markov policy $x:S\times G\to A$6 and environment path $x:S\times G\to A$7 such that $x:S\times G\to A$8 is a PME (Yang, 2015).

In Vasal’s non-stationary mean-field game, Assumption A1 requires $x:S\times G\to A$9 and $s_t$0 to be continuous in $s_t$1, with $s_t$2 bounded. Under A1, the paper states that there exists at least one solution $s_t$3 to the backward-recursive system (2.1–2.3) for finite and infinite horizon (Vasal, 2019). The proof sketch proceeds by showing that the stagewise maximization yields a nonempty, upper-hemicontinuous best-response correspondence, that the mean-field update $s_t$4 is continuous in $s_t$5, and that Kakutani’s fixed-point theorem yields existence of $s_t$6, after which backward induction constructs $s_t$7. The paper further states that under further monotonicity assumptions on $s_t$8 in $s_t$9 one can also prove uniqueness of the PME (Vasal, 2019).

In the smart-grid Stackelberg Markov game, Theorem 1 states that in the $g_t$0-stage Markov game there exists at least one pure PME $g_t$1 of the user subgame (Huang et al., 6 Sep 2025). The proof sketch uses backward induction. At stage $g_t$2, the single-stage game parameterized by $g_t$3 is an exact potential game and admits a pure equilibrium in demands $g_t$4. By eliminating dominated $g_t$5 actions, one shows $g_t$6. Folding the continuation value into stage $g_t$7 preserves the potential structure, and recursion to $g_t$8 yields a sequence of pure policies $g_t$9 that forms a pure PME (Huang et al., 6 Sep 2025).

These results indicate that PME is not tied to a single proof technique. Dynamic programming, contraction arguments, fixed-point theorems, and potential-game structure all appear, depending on whether the model is nonatomic, mean-field, or finite-horizon Markovian with private states.

5. Computation and algorithms

The computational treatment of PME differs sharply across the cited papers.

Vasal develops a backward-recursive algorithm for finite horizon $x_t^i\in X$0, with $x_t^i\in X$1 and, for each $x_t^i\in X$2 and each mean field $x_t^i\in X$3, an inner fixed-point problem for the prescription $x_t^i\in X$4. The prescription must satisfy, for each $x_t^i\in X$5,

$x_t^i\in X$6

where $x_t^i\in X$7 appears on both sides through $x_t^i\in X$8. The algorithm then sets $x_t^i\in X$9, updates the mean-field map $\phi$00, and computes

$\phi$01

For the infinite-horizon case, the paper states that one solves the stationary equation by value iteration. It also states that, at each $\phi$02 and each $\phi$03, one must solve a fixed point over the compact set $\phi$04, typically by iterating best responses or by Kakutani’s theorem; if $\phi$05 and $\phi$06 are continuous in $\phi$07, the mapping is a contraction in the infinite-horizon case, so value iteration converges; and the overall complexity is $\phi$08 (Vasal, 2019).

In the nonatomic-game treatment, computation in stationary settings proceeds by initializing a guess $\phi$09, solving the single-agent Bellman equation

$\phi$10

extracting the optimal policy $\phi$11, updating $\phi$12, and repeating until $\phi$13 is small. Under compactness and continuity, the iteration converges to $\phi$14 (Yang, 2015).

The smart-grid paper provides both a centralized and a decentralized computational route. The key theoretical step is the construction of an auxiliary Markov potential game $\phi$15, where each user’s state is only $\phi$16, action is $\phi$17, and stage payoff is

$\phi$18

The game has explicit potential

$\phi$19

Any pure ME of $\phi$20 can be found by maximizing this potential via an FIP best-response process. The paper states that the centralized FIP converges in polynomial$\phi$21 steps and reconstructs the PME in the original game by

$\phi$22

It also presents a decentralized FP + MDP procedure in which each user maintains an estimate $\phi$23 of opponents’ aggregate-demand law, solves a personal MDP, observes $\phi$24, and updates by fictitious-play averaging; under mild conditions this process empirically converges to PME (Huang et al., 6 Sep 2025).

6. Applications, finite-player interpretation, and empirical illustrations

The applications in the cited papers make clear that PME is designed for settings with dynamic private states and strategic externalities.

Vasal studies a cyber-physical security problem in which a node’s private state is binary, $\phi$25, actions are $\phi$26, and infected nodes impose a negative externality through the infected fraction $\phi$27. The state transition is

$\phi$28

and the one-stage payoff is

$\phi$29

For $\phi$30, $\phi$31, $\phi$32, and $\phi$33, the paper numerically computes the infinite-horizon equilibrium and reports three qualitative findings: healthy nodes adopt a threshold policy in which $\phi$34 is a non-decreasing function of the infected fraction $\phi$35; infected nodes almost always choose repair, $\phi$36, independent of $\phi$37; and the equilibrium value $\phi$38 decreases in $\phi$39 for both $\phi$40 (Vasal, 2019).

The smart-grid application models private battery levels in demand response. The numerical setup uses $\phi$41, $\phi$42, $\phi$43, real solar data for $\phi$44, discretized $\phi$45, transition $\phi$46 fitted from data, $\phi$47, and price parameters $\phi$48. The paper states that centralized FIP finds PME in under 100 iterations, that users’ equilibrium demands increase in $\phi$49 and respond sensibly to $\phi$50, and that the aggregator can search $\phi$51 while fixing $\phi$52 to maximize $\phi$53 (Huang et al., 6 Sep 2025).

The nonatomic-game paper supplies a different form of application-oriented significance: it proves that equilibria derived for nonatomic games can be used by large finite counterparts to achieve near-equilibrium performances. In the finite $\phi$54-player version, if every player uses the oblivious PME $\phi$55, then there exists a sequence $\phi$56, explicitly $\phi$57 under Lipschitz or bounded-difference conditions, such that the profile “all use $\phi$58” is an $\phi$59-Nash equilibrium of the $\phi$60-player game (Yang, 2015). This connects PME to approximation theory for large games: the nonatomic equilibrium provides a decentralized policy rule that remains nearly optimal in large but finite populations.

Taken together, these results show that PME serves at least three roles in current research. It is an equilibrium notion for nonatomic dynamic competition (Yang, 2015), a solution concept for non-stationary mean-field games with private types and dynamic population states (Vasal, 2019), and a tractable equilibrium restriction for lower-level strategic interaction with private storage states in Stackelberg smart-grid control (Huang et al., 6 Sep 2025). A plausible implication is that PME is most useful when the modeler seeks a compromise between strategic richness and computational or informational tractability: private state dependence is retained, while conditioning on the full joint state of all players is deliberately excluded or aggregated.