Randomized Equilibrium Policy

• Randomized equilibrium policy is a framework that uses explicit randomization to achieve equilibrium in dynamic systems where pure strategies may fail.
• It employs entropy regularization and stochastic control techniques to ensure exploration and the existence of stable equilibria.
• Applications include singular control, Markovian stopping games, and multi-agent reinforcement learning, offering robust strategies for complex environments.

A randomized equilibrium policy is a foundational construct in modern stochastic control, game theory, and reinforcement learning, used to capture equilibrium behavior in dynamic environments where either determinacy fails, exploration is essential, or regularization is required for existence and computation. The notion encompasses stochastic policies or control laws that, via explicit randomization, attain equilibrium objectives in settings including singular control, mean-field games, Markovian stopping problems, and stochastic Nash frameworks. This entry synthesizes the theoretical formulation, characterizations, algorithmic constructions, and application regimes of randomized equilibrium policies across principal model classes.

1. Formal Definitions and Exemplary Model Structures

A randomized equilibrium policy, in its most general form, is a measurable mapping assigning, to each state (which may be augmented with time, cumulative controls, or other features), a probability measure over the action or control space. In continuous-time singular control problems, for instance, the policy randomizes the activation of irreversible control actions to address issues such as exploration-exploitation trade-offs or to regularize singularities within the Hamilton–Jacobi–Bellman (HJB) framework (Liang et al., 2 Dec 2025).

In Markovian stopping games, a randomized equilibrium is operationalized via a state-dependent stopping probability π mapping the state to [0,1], so at each time, the agent stops with probability π(x). This randomization, as opposed to deterministic stopping, is sometimes necessary for equilibrium existence, especially in generalized Dynkin games where pure-strategy equilibria may fail to exist (Christensen et al., 2023, Christensen et al., 12 Dec 2024).

The generic setup is illustrated for a singular control problem as follows:

• The controlled process $X^{\xi}$ evolves according to $dX^{\xi}_t = \mu\,dt + \sigma\,dB_t - d\xi_t$
• The randomized policy is encoded via an auxiliary process η, representing the probability to activate a singular law at each instant. The resulting control process $(\Xi, \eta)$ yields a stochastic equilibrium policy (Liang et al., 2 Dec 2025).

2. Entropy Regularization and Exploratory Randomization

To enforce exploration and ensure the tractability of equilibrium computation, entropy regularization is often imposed on the class of randomized policies. For singular control, this leads to an objective functional of the form

$$J(\Xi, \eta) = \mathbb{E}\left[ \int_0^\infty e^{-\beta r} \left(e^{a X^{\Xi, \eta}_r} dr + c\,d\xi^{\Xi, \eta}_r\right) - \lambda \int_0^\infty e^{-\beta r} \mathcal{E}(\eta_r)\,dr\right]$$

where $\mathcal{E}(z) = z - z\ln z$ serves as an entropy penalty discouraging degenerate (deterministic) activation (Liang et al., 2 Dec 2025).

Similarly, in time-inconsistent mean–field stopping, an entropy-regularized reward

$$J_\lambda^\pi(\mu) = \sum_{k=0}^\infty \delta_\lambda(k)\,\mathbb{E}^{\mu,\pi}[r(\mu_k)\pi(\mu_k) + \lambda \mathcal{H}(\pi(\mu_k))]$$

is used to guarantee existence and stability of equilibrium relaxed (randomized) stopping rules (Yu et al., 2023).

These regularizations are not only analytical devices but are vital for practical reinforcement learning and equilibrium computation, providing unbiased exploration and improved learning performance in high-dimensional or singular environments.

3. Equilibrium Characterization in Continuous-Time and Discrete-Time Settings

The structure of randomized equilibrium policies is formalized via complementary slackness and variational inequalities (VIs), or—where applicable—via fixed-point or Bellman-type systems.

• In continuous-time singular control, equilibrium policies are characterized as solutions to a system of quasi-variational inequalities—a generalized HJB system. For irreversible reinsurance, the equilibrium trigger boundary is explicitly given by

$$\Gamma(x) = \exp\left(-\frac{\beta}{\lambda}\,\Phi(x)\right)$$

where $\Phi(x)$ is the inner value function. The auxiliary activation law is implemented via a Skorokhod-type reflection on this boundary (Liang et al., 2 Dec 2025).

• In Markovian stopping and Dynkin games (both discrete and continuous time), value functions $V$ and randomized policies $\pi$ satisfy systems of equations such as

$$V(x) = \max\{ (1-\pi^j(x))\alpha \Pi V(x) + \pi^j(x)g^i(x),\,(1-\pi^j(x))f^i(x) + \pi^j(x)h^i(x) \}$$

complemented by indifference slackness enforcing that $\pi(x) \in [0,1]$ only on regions where agent is indifferent between stopping and continuation (Christensen et al., 2023, Christensen et al., 12 Dec 2024).

Tables summarizing these forms:

Context	Equilibrium Characterization	Randomized Policy Form
Singular Control (continuous)	Extended HJB QVI (min/max, gradient, entropy)	$\eta$-randomized activation
Markov Stopping Games	Bellman–Wald equations, complementary slackness	$\pi(x)$∈0,1
Mean Field/MDP Stopping	Fixed-point for value/π, entropy-regularized	π*: S → [0,1]

4. Existence, Uniqueness, and Necessity of Randomization

The necessity and sufficiency of randomization for equilibrium attainment depend on the structure of payoffs and transition dynamics.

• Pure-strategy equilibrium may exist (and be unique) under restrictive “middle payoff” or monotonicity conditions on the reward functions—e.g., in zero-sum Dynkin games with $f \leq h \leq g$ (Christensen et al., 12 Dec 2024), or in discrete models with $h(x)=\operatorname{med}\{f(x),h(x),g(x)\}$ for all $x$ (Christensen et al., 2023).
• If these conditions fail, pure equilibria may not exist, and explicit construction of mixed/randomized equilibria is required, e.g., via additive functional representations or local-time–based randomization (Christensen et al., 12 Dec 2024).
• Generalized existence is established via fixed-point theorems (Kakutani for countable state spaces, Schauder for function spaces under regularization), ensuring at least one randomized Markovian equilibrium in broad settings (Yu et al., 2023, Christensen et al., 2023).

5. Algorithmic Realization and Reinforcement Learning Instantiations

Randomized equilibrium policies are constructible both analytically (explicit formulas for trigger surfaces or mixing rates) and algorithmically via RL-type policy iteration.

In the entropy-regularized singular control context, parameterized value functions $\Phi^\theta(x)$ and action rules are embedded within an actor–critic framework (Liang et al., 2 Dec 2025):

• The actor updates policy parameters $\theta$ using martingale-based, time-homogeneous gradient estimators;
• The critic evaluates current policy value and supplies unbiased value gradients;
• Randomization via the auxiliary $\eta$ process ensures robust exploration and unbiased estimation over non-action regions.

Pseudocode outline (as in (Liang et al., 2 Dec 2025)):

1. Simulate trajectories under current $(\Xi_{\bar x}, \Upsilon_{\bar x})$.
2. At each timestep, update $\eta$ by reflecting at $z = \Gamma^\theta(X_{t_n})$; randomize activation via a coin flip.
3. If activated, apply $\Xi_{\bar x}$; otherwise, accrue running cost.
4. Update $\theta$ via policy/value gradient step and adjust the action threshold via QVI-based policy improvement.

Empirical results demonstrate that such randomization accelerates learning and ensures convergence even when deterministic-exploit traps would impede progress (Liang et al., 2 Dec 2025).

In stochastic Nash games, randomized best-response update schemes—where only a stochastic subset of agents update at each step—are rigorously shown to converge linearly to unique equilibria, albeit with a quantitatively increased complexity exponent reflecting the cost of randomization (Lei et al., 2017).

6. Applications and Generalizations

Randomized equilibrium policy frameworks have broad applicability:

• In market-design and policy analysis, they are used to design randomization-based interventions for identification and estimation of treatment effects under equilibrium spillovers in single-market settings (Munro et al., 2021).
• In game theory, the existence and explicit construction of randomized equilibria in Dynkin games, mean-field games, and MDPs underpins the analysis of time-inconsistent preferences, mixed-strategy selection, and social-planner optimality (Yu et al., 2023, Christensen et al., 12 Dec 2024, Christensen et al., 2023).
• In multi-agent RL, reward randomization and mixture policy training (as in Reward Randomized Policy Gradient and PSRO variants) are leveraged for robust equilibrium discovery, diverse strategy generation, and avoidance of suboptimal fixed points in complex games (Tang et al., 2021, McAleer et al., 2022).

7. Significance and Research Outlook

Randomized equilibrium policies have emerged as fundamental objects for both theoretical and computational advances, resolving non-existence of pure equilibria, enabling tractable learning in singularly controlled and time-inconsistent environments, and operationalizing exploration in high-dimensional RL. They connect variational analysis, fixed-point theory, and modern statistical learning to core problems in stochastic control and dynamic games.

Recent developments include the full characterization of equilibrium randomization thresholds for singular control (Liang et al., 2 Dec 2025), the entropy-regularization paradigm for existence proofs and RL (Yu et al., 2023), and explicit solution constructions for zero-sum stopping games under general payoff orderings (Christensen et al., 12 Dec 2024). The effectiveness and necessity of randomization in learning robust, approximately optimal policies is also established in practical algorithm frameworks (Tang et al., 2021, Liang et al., 2 Dec 2025). Ongoing research explores further generalization to non-Markovian, high-dimensional, and partially observable environments, as well as extensions to multi-agent systems with complicated coupling and equilibrium selection challenges.