Private Markovian Strategies (PMS): Theory & Applications

• Private Markovian Strategies (PMS) are strategy classes in dynamic games where decisions depend on private information and a current, recursively updated Markovian state.
• PMS replace full-history dependence with a compressed state summary—such as beliefs, finite memory, or public-private state pairs—enabling tractable dynamic programming and equilibrium analysis.
• They are applied in dynamic Bayesian persuasion, smart-grid demand response, iterated games, and decentralized control to simplify complex stochastic optimization problems.

Private Markovian Strategies (PMS) are strategy classes in dynamic games and stochastic control in which actions depend on private information together with a current Markovian sufficient statistic, rather than on the full observable history. In the cited literature, the term appears explicitly in Stackelberg Markov games for smart-grid demand response, while closely related constructions arise in Markovian persuasion, higher-order stochastic games with Markov strategies, and decentralized control with delayed sharing. This suggests that PMS is best understood as a family of structurally similar strategy restrictions: privacy is encoded in the agent’s information set, and dynamic consistency is obtained by conditioning only on a state variable that evolves Markovianly, such as a posterior belief, a private physical state, a finite memory window, or a pair of private and common information states (Huang et al., 6 Sep 2025, Lehrer et al., 2021, Hidaka, 2015, Charalambous et al., 25 Apr 2026).

1. Core meaning and formal variants

Across the supplied sources, PMS always combines two ingredients. The first is private conditioning: the acting agent observes information that other agents do not observe. The second is Markovian conditioning: the policy depends only on a current state summary that is sufficient for continuation payoffs and updates recursively. What changes from one model class to another is the identity of that state summary.

Setting	Private information	Markovian state used by PMS
Markovian persuasion (Lehrer et al., 2021)	Sender observes current realized state $x_n$	Receiver belief over the current state, propagated by $\phi(q)=qM$
Stackelberg Markov game for demand response (Huang et al., 6 Sep 2025)	User $i$ observes own storage $b_i^t$	Current time $t$, public renewable state $e_N^t$, and own private state $b_i^t$
Iterated games with Markov strategies (Hidaka, 2015)	Own past actions and realized payoffs, or a finite memory of past outcomes	A $k$-length history window $X_t\in S^k$, or a stationary marginal in the infinite-memory limit
Delayed-sharing decentralized control (Charalambous et al., 25 Apr 2026)	Agent-specific private information $\Lambda_t^i$	Private posterior $\phi(q)=qM$0 together with common belief $\phi(q)=qM$1 or $\phi(q)=qM$2

A common consequence is that PMS replaces full-history dependence by a recursion on sufficient statistics. In persuasion, the statistic is a belief over $\phi(q)=qM$3; in the smart-grid model it is a public-private state pair; in iterated games it is an augmented Markov state or its stationary marginal; in delayed sharing it is a belief-state pair generated by Bayesian filtering. The sources therefore do not present a single canonical PMS formalism, but they do present a stable design principle: private information is localized, while dynamics are compressed into a recursively updated Markov state.

2. PMS in dynamic Bayesian persuasion

In the dynamic persuasion model of “Markovian Persuasion,” the state space is finite, $\phi(q)=qM$4, and the state $\phi(q)=qM$5 evolves as an irreducible, aperiodic Markov chain with transition matrix $\phi(q)=qM$6 and stationary distribution $\phi(q)=qM$7. The sender observes the realized state $\phi(q)=qM$8, privately randomizes a signal, and the receiver updates a posterior $\phi(q)=qM$9 by Bayes’ rule and chooses $i$0. The sender’s stage payoff at belief $i$1 is

$i$2

and the discounted objective is

$i$3

The PMS interpretation in this model is a stationary, belief-based disclosure policy: at each date, the sender conditions on the current receiver prior over the current state and implements a Bayes-plausible split of that prior into posteriors. The paper’s split set at belief $i$4 is

$i$5

The relevant Markov operator on beliefs is $i$6, and under any signaling strategy,

$i$7

Hence beliefs evolve as a controlled Markov process on $i$8: the sender first splits the current prior, then the Markov transition propagates each posterior forward (Lehrer et al., 2021).

The dynamic program is a concavification recursion. Writing $i$9,

$b_i^t$0

and equivalently

$b_i^t$1

The value function is concave, satisfies

$b_i^t$2

and obeys the two-sided bounds

$b_i^t$3

A notable no-disclosure property holds at contact points: if $b_i^t$4, then any optimal PMS pools at $b_i^t$5.

The asymptotic theory is geometric. Let $b_i^t$6 be the set of supporting hyperplanes to $b_i^t$7 at $b_i^t$8, with

$b_i^t$9

A nonempty set $t$0 is $t$1-absorbing if $t$2 for every $t$3. The main asymptotic value theorem states that there exists $t$4, independent of $t$5, such that $t$6 uniformly as $t$7, with

$t$8

Moreover, the following are equivalent: asymptotic attainment of $t$9; existence of $e_N^t$0 such that $e_N^t$1 contains an $e_N^t$2-absorbing set; and the corresponding countable absorbing-set condition. When such an absorbing set exists, a belief-Markovian PMS can keep posteriors inside a region on which $e_N^t$3 is affine, thereby achieving the maximal asymptotic value.

The same paper gives an exact formula on the absorbing domain

$e_N^t$4

where $e_N^t$5 is a maximal $e_N^t$6-absorbing set. For all $e_N^t$7 and $e_N^t$8,

$e_N^t$9

It also identifies a strong-law regime: if $b_i^t$0 contains a finite $b_i^t$1-absorbing set $b_i^t$2, then there exists a PMS such that for all $b_i^t$3,

$b_i^t$4

The two-state examples exhibit both possibilities. Under one transition matrix $b_i^t$5, the relevant contact set is not $b_i^t$6-absorbing and $b_i^t$7; under another matrix $b_i^t$8, the contact set is $b_i^t$9-absorbing, and a stationary split keeps beliefs in $k$0, yielding $k$1. In homothetic chains,

$k$2

the paper states that $k$3 for every continuous $k$4, because PMS can keep beliefs in any star-shaped set around $k$5.

3. PMS and private Markovian equilibrium in Stackelberg Markov games

In the smart-grid demand-response model, PMS is defined explicitly. The leader is an aggregator that sets real-time pricing parameters, and the followers are users who choose demand and storage actions over a finite horizon $k$6. The public state is

$k$7

with forecast error evolving as a Markov chain $k$8. User $k$9’s private state is storage

$X_t\in S^k$0

and the user observes only $X_t\in S^k$1. The action is

$X_t\in S^k$2

subject to

$X_t\in S^k$3

The public component evolves exogenously, while storage updates deterministically: $X_t\in S^k$4

A PMS for user $X_t\in S^k$5 is a sequence

$X_t\in S^k$6

so at time $X_t\in S^k$7 the user maps $X_t\in S^k$8 to a mixed action. A pure PMS is the deterministic counterpart $X_t\in S^k$9. Markovianity is strict: PMS depends on current time, public renewable state, and own private storage only, and not on the full state history or on other users’ private states. The lower-level value function under a PMS profile $\Lambda_t^i$0 is

$\Lambda_t^i$1

A profile $\Lambda_t^i$2 is a Private Markovian Equilibrium (PME) if, for every user $\Lambda_t^i$3 and initial state $\Lambda_t^i$4,

$\Lambda_t^i$5

The stage payoff couples users only through aggregate demand $\Lambda_t^i$6 entering the price

$\Lambda_t^i$7

and user $\Lambda_t^i$8’s reward is

$\Lambda_t^i$9

A central structural lemma states that for fixed $\phi(q)=qM$00, the stage game is an exact potential game; after eliminating strictly dominated actions by choosing $\phi(q)=qM$01, each user’s equilibrium demand $\phi(q)=qM$02 depends only on $\phi(q)=qM$03, not on private $\phi(q)=qM$04. This reduction is what makes pure PMS and pure PME tractable.

The existence theorem states that in the $\phi(q)=qM$05-stage lower-level Markov game there exists a pure PME constructed via backward induction. The polynomial-time computability theorem reduces the problem to an auxiliary Markov potential game $\phi(q)=qM$06 with state $\phi(q)=qM$07, action $\phi(q)=qM$08, and reduced payoff

$\phi(q)=qM$09

Its stage potential is

$\phi(q)=qM$10

The corresponding pure PMS in the original game is

$\phi(q)=qM$11

The paper presents two lower-level computational routes and an upper-level pricing search. The centralized method is a FIP algorithm on $\phi(q)=qM$12; the decentralized method is an FP+MDP algorithm on the original incomplete-information game. The FIP best response reduces to a strictly concave one-dimensional maximization, with continuous optimum

$\phi(q)=qM$13

followed by snapping to the nearest discrete choices. The FP+MDP scheme avoids explicit beliefs over other users’ private states; instead, users estimate only distributions over opponents’ aggregate demand $\phi(q)=qM$14 conditional on $\phi(q)=qM$15. This is a defining distinction from Perfect Bayesian Equilibrium.

The reported experiments use real solar generation data, 50 users, and 7 stages, with

$\phi(q)=qM$16

and forecast error $\phi(q)=qM$17 governed by

$\phi(q)=qM$18

The centralized FIP algorithm converges in finite steps, the state space depends on $\phi(q)=qM$19 rather than on $\phi(q)=qM$20, and observed convergence is reported with 50 users. In a decentralized example with 3 users and 3 stages, the NashConv metric decays toward zero and strategies stabilize after $\phi(q)=qM$21 iterations. In the upper-level grid search over $\phi(q)=qM$22, the reported best pair is $\phi(q)=qM$23 (Huang et al., 6 Sep 2025).

4. Higher-order and infinite-memory PMS in iterated games

In the framework of “Recursive Markov Process for Iterated Games with Markov Strategies,” the PMS notion corresponds to probabilistic Markov strategies that depend on finite or infinite private memory. Each player $\phi(q)=qM$24 has a finite action set $\phi(q)=qM$25, joint outcomes lie in a finite set $\phi(q)=qM$26, and a $\phi(q)=qM$27-th order private Markov strategy is described by the conditional choice probability

$\phi(q)=qM$28

Under the conditional-independence assumption,

$\phi(q)=qM$29

The induced $\phi(q)=qM$30-th order Markov process becomes first-order on the augmented state

$\phi(q)=qM$31

with transition kernel

$\phi(q)=qM$32

For finite $\phi(q)=qM$33, the stationary distribution $\phi(q)=qM$34 on $\phi(q)=qM$35 satisfies $\phi(q)=qM$36.

The computational obstruction is the exponential growth of $\phi(q)=qM$37. The paper’s main contribution is a recursive technique for computing stationary marginals without explicitly solving the full augmented chain when $\phi(q)=qM$38 is large or when $\phi(q)=qM$39. The transition matrix $\phi(q)=qM$40 is decomposed using marginalization, branching, and cycling matrices, and the $\phi(q)=qM$41-shift operator is defined as

$\phi(q)=qM$42

This operator acts on low-order marginals rather than on the full $\phi(q)=qM$43 state. The stationary $\phi(q)=qM$44-marginal is then characterized as a fixed point of the appropriate shift matrix. Under irreducibility and Cauchy convergence assumptions for the relevant sequences,

$\phi(q)=qM$45

so the stationary marginal of the infinite-order process is obtained as the limit fixed point of the shift operator. In recursive Markov processes, the stationary marginal satisfies the closed nonlinear fixed-point equation

$\phi(q)=qM$46

The paper’s learning rule is a private reinforcement-learning PMS. For action $\phi(q)=qM$47, the discounted cumulative reward over the last $\phi(q)=qM$48 periods is

$\phi(q)=qM$49

and the choice probability is softmax: $\phi(q)=qM$50 This rule uses only a player’s own past actions and realized payoffs, which is why the supplied interpretation labels it “private.”

The application is the two-player iterated prisoner’s dilemma, with $\phi(q)=qM$51. In the infinite-memory formulation, the stationary marginal $\phi(q)=qM$52 solves

$\phi(q)=qM$53

for a specific $\phi(q)=qM$54 nonlinear operator $\phi(q)=qM$55 derived from the softmax reinforcement rule. For the numerical example with payoffs $\phi(q)=qM$56, $\phi(q)=qM$57, $\phi(q)=qM$58, $\phi(q)=qM$59, $\phi(q)=qM$60, and $\phi(q)=qM$61, the reported findings are that $\phi(q)=qM$62 increases with $\phi(q)=qM$63, that $\phi(q)=qM$64 can be significant for moderate finite memory, and that the $\phi(q)=qM$65-shift approximation converges faster than the marginal extracted from the full $\phi(q)=qM$66-th order chain. In this line of work, PMS is therefore not an equilibrium concept but a stochastic-memory strategy class whose long-run implications are captured by recursive stationary-marginal analysis (Hidaka, 2015).

5. PMS as separated information-state policies under delayed sharing

In the decentralized-control framework with $\phi(q)=qM$67-step delayed sharing, PMS emerges from a separation result rather than from an equilibrium definition. There are $\phi(q)=qM$68 agents, discrete time $\phi(q)=qM$69, an unobserved state $\phi(q)=qM$70, controlled Markov kernel $\phi(q)=qM$71, and observation kernels $\phi(q)=qM$72. Agent $\phi(q)=qM$73’s information set is

$\phi(q)=qM$74

with common information

$\phi(q)=qM$75

and private information

$\phi(q)=qM$76

Actions are measurable maps

$\phi(q)=qM$77

The private information state is the posterior

$\phi(q)=qM$78

with marginal

$\phi(q)=qM$79

Its recursion is Bayesian and Markov: $\phi(q)=qM$80 The model also defines two common information states. The first is the strategy-independent common belief about the delayed state,

$\phi(q)=qM$81

which updates through a Markov operator depending only on known delayed observations and actions. The second is the strategy-dependent common belief about the current extended state,

$\phi(q)=qM$82

The key conditional Markov property is

$\phi(q)=qM$83

Thus the update depends only on the current private belief, current common information, and current action, not on the entire history. This is the formal Markovian basis for PMS.

Optimality is formulated through Person-by-Person (PbP) optimality. A strategy profile $\phi(q)=qM$84 is PbP optimal if, for each $\phi(q)=qM$85,

$\phi(q)=qM$86

The generalized DP equations define the value process for each agent with opponents’ strategies fixed at optimal responses, and the optimization is over actions $\phi(q)=qM$87, not over entire strategy spaces. The paper then establishes a separation result: optimal strategies can be taken in semi-separated, separated, or information-state form. The most compressed representation is

$\phi(q)=qM$88

These are precisely PMS in the delayed-sharing environment: policies are Markov with respect to two recursively updated sufficient statistics, one private and one common.

The paper presents this construction as a resolution of the longstanding open problem for $\phi(q)=qM$89-step delayed sharing. The claimed achievement is a DP approach that preserves the classical POMDP properties: value functions and information states depend on the actions of minimizing controls and not on their strategies, and optimal strategies are separated functionals of private and common information states. An offline backward-induction algorithm is obtained by performing DP over $\phi(q)=qM$90 or $\phi(q)=qM$91, followed by online Bayesian belief updates and action selection (Charalambous et al., 25 Apr 2026).

6. Comparative interpretation, misconceptions, and limitations

A first source of confusion is to treat PMS as a single equilibrium concept. The cited literature does not support that interpretation. In Markovian persuasion, PMS is a sender disclosure policy solving a dynamic concavification problem; in the smart-grid Stackelberg Markov game, PMS is a strategy class and PME is the corresponding equilibrium notion; in iterated games, PMS denotes stochastic learning rules whose stationary marginals are analyzed; in delayed-sharing decentralized control, PMS is the separated policy class justified by PbP dynamic programming (Lehrer et al., 2021, Huang et al., 6 Sep 2025, Hidaka, 2015, Charalambous et al., 25 Apr 2026).

A second misconception is that PMS necessarily avoids beliefs. The opposite is true in two of the four settings. Markovian persuasion is explicitly belief-centric, with the belief process satisfying $\phi(q)=qM$92, and delayed-sharing control is built around the pair of private and common posteriors $\phi(q)=qM$93 or $\phi(q)=qM$94. By contrast, the smart-grid formulation is notable precisely because PMS does not require explicit beliefs over other users’ private states; instead, decentralized learning estimates distributions over aggregate demand conditional on the public state. In the iterated-game setting, the relevant state can be private memory of payoffs rather than a posterior over hidden states.

A third misconception is that “Markovian” always means dependence on a physical state variable. The sources show four distinct meanings. It can mean dependence on a belief state $\phi(q)=qM$95 in persuasion, on $\phi(q)=qM$96 in demand response, on a $\phi(q)=qM$97-window $\phi(q)=qM$98 in iterated games, or on private and common information states in delayed sharing. What unifies these cases is not the ontology of the state variable but its recursion and sufficiency for continuation analysis.

The limits of PMS are equally model-dependent. In Markovian persuasion, the maximal asymptotic value $\phi(q)=qM$99 is not always attainable; attainability is characterized by the geometry of contact sets $i$00 and the existence of $i$01-absorbing subsets, and the two-state examples show both attainable and unattainable cases. In the smart-grid model, the existence and polynomial-time results rely on a specific structure: finite horizon, discrete state-action spaces, exogenous action-independent public-state transitions, linear consumption benefit, no explicit storage degradation cost in the baseline, and a particular RTP form; the paper states existence of pure PME but not uniqueness. In the iterated-game framework, the recursive stationary-marginal analysis relies on irreducibility and convergence conditions, and complexity remains exponential in the chosen marginal order even though the shift-operator approach avoids the full $i$02 explosion. In delayed-sharing control, the separation and DP results are derived under PbP optimality, while global team optimality may require additional conditions not covered in the supplied material.

Taken together, these papers locate PMS at the intersection of private information, recursive state compression, and dynamic optimization. The main technical question is not whether a strategy uses private information, but whether that information can be summarized by a current Markovian sufficient statistic. Where the answer is affirmative, PMS yields dynamic programs, fixed-point equations, potential-game reductions, or separation principles that are unavailable for unrestricted full-history strategies.