Entropy-Regularized Stackelberg Stopping Game

• The paper introduces an entropy regularization method that smooths the follower’s binary stop/continue decision, yielding a continuous best-response map and regular randomized equilibrium.
• It employs a discrete-time Stackelberg framework where the leader commits to a stopping strategy, leading to time inconsistency challenges that are resolved with entropy smoothing.
• The approach extends to continuous-time, mean-field, and learning formulations, offering computational algorithms and ε-equilibrium approximations for complex stopping problems.

Entropy-regularized Stackelberg stopping games are leader-follower stopping models in which the leader commits to a stopping strategy and the follower responds optimally, but the follower’s optimization is modified by an entropy term. In the discrete-time Stackelberg variant of the Dynkin game, Player 1 announces her stopping strategy first and Player 2 responds after observing it; this sequential structure produces time inconsistency for the leader, and in infinite horizon a randomized equilibrium strategy may fail to exist in the unregularized game (Zhang et al., 26 Jul 2025). Entropy regularization changes the follower’s local stop/continue problem from a discontinuous bang-bang response into a smooth logistic one, yielding a continuous best-response map and existence of a regular randomized equilibrium (Zhang et al., 26 Jul 2025). Related continuous-time, mean-field, and learning-based formulations use randomized stopping times, singular controls, occupation measures, or stopping intensities to obtain analogous smoothing, existence, and vanishing-regularization results (Dianetti et al., 23 Sep 2025, Dong, 2022, Yu et al., 15 Jan 2025).

1. Stackelberg stopping structure

In the discrete-time formulation, the underlying state process is a time-homogeneous Markov chain $X=(X_t)_{t\ge 0}$ on a finite state space $\mathbb{X}=\{1,\dots,N\}$ with transition matrix $\Pi$. The classical Dynkin game assigns stopping times $\tau$ and $\rho$ to two players and defines discounted payoffs

$$J_1(x,\tau,\rho)=\mathbb{E}_x\big[\beta^{\tau\wedge\rho}F_1(\tau,\rho)\big],\qquad J_2(x,\tau,\rho)=\mathbb{E}_x\big[\delta^{\tau\wedge\rho}F_2(\tau,\rho)\big],$$

where $F_1$ and $F_2$ distinguish the cases $\tau<\rho$, $\rho<\tau$, and $\mathbb{X}=\{1,\dots,N\}$0. The Stackelberg variant breaks the symmetry: the leader announces her stopping strategy first, and the follower chooses a best response after observing it (Zhang et al., 26 Jul 2025).

This sequential move order changes the solution concept. In finite horizon, the leader’s precommitment strategy is optimal only at the initial date, given that future selves are bound not to deviate. The equilibrium strategy is instead time-consistent: at every date, the current self has no profitable one-step deviation given the future selves’ continuation behavior. The paper’s two-period example shows that these notions need not coincide: the leader’s optimal plan at time $\mathbb{X}=\{1,\dots,N\}$1 can differ from the plan that the same leader would choose at time $\mathbb{X}=\{1,\dots,N\}$2 conditional on not having stopped. The same example also separates the Stackelberg equilibrium concept from the Nash equilibrium of the standard Dynkin game (Zhang et al., 26 Jul 2025).

A more general stochastic Stackelberg framework represents the leader’s commitment as a dynamic prescription and the follower’s response as a stagewise best response indexed by common beliefs. In that formulation, equilibrium strategies depend on current private type and current common belief rather than full history, and a backward recursive algorithm computes stagewise fixed points. This is a general Stackelberg decomposition rather than a stopping-specific theorem, but it provides a natural state-belief template for discrete-time stopping models with public actions and absorbing stopped states (Vasal, 2020).

2. Unregularized discontinuity and the role of entropy

The central obstruction in Stackelberg stopping games is the discontinuity of the follower’s best response. In the infinite-horizon model, the leader may use a path-dependent randomized stopping policy $\mathbb{X}=\{1,\dots,N\}$3, while the follower chooses either a path-dependent pair $\mathbb{X}=\{1,\dots,N\}$4 or, in the Markov restriction, state-dependent stopping probabilities $\mathbb{X}=\{1,\dots,N\}$5 and $\mathbb{X}=\{1,\dots,N\}$6. The follower’s continuation value satisfies

$\mathbb{X}=\{1,\dots,N\}$7

so the local decision is binary: stop immediately when the stopping reward dominates, otherwise continue. Small perturbations in the leader’s strategy can therefore flip the follower’s response discontinuously (Zhang et al., 26 Jul 2025).

This discontinuity propagates to the leader’s problem. The leader’s precommitment value can be written in terms of feasible follower continuation utilities $\mathbb{X}=\{1,\dots,N\}$8 and a value function $\mathbb{X}=\{1,\dots,N\}$9, but $\Pi$0 need not be continuous. As a result, the supremum of the leader’s precommitment value may fail to be attained. The paper also constructs a war-of-attrition-type example showing that, unlike in the classical Dynkin game, a randomized Markov equilibrium may fail to exist in the unregularized Stackelberg game (Zhang et al., 26 Jul 2025).

Entropy regularization is introduced precisely to remove this discontinuity. For $\Pi$1, let

$\Pi$2

The follower’s regularized objective becomes

$\Pi$3

The local maximizations are then strictly concave, and the optimal follower responses acquire closed forms: $\Pi$4

$\Pi$5

The associated continuation value becomes

$\Pi$6

The paper identifies this as a log-sum-exp smoothing of the max operator: the follower’s best response becomes continuous in the leader’s policy, and the discontinuity that prevented equilibrium existence is removed (Zhang et al., 26 Jul 2025).

3. Regular randomized equilibrium and approximation

With regularization, the leader’s value is computed against the follower’s unique regularized response. Writing

$\Pi$7

a Markov policy $\Pi$8 is a regular randomized equilibrium if, for every state $\Pi$9 and every one-step deviation $\tau$0,

$\tau$1

where

$\tau$2

Equivalently, one considers the set-valued map $\tau$3 that assigns to each $\tau$4 the set of statewise maximizers of $\tau$5. Because $\tau$6 is continuous in $\tau$7, $\tau$8 has nonempty, convex, closed values and closed graph. Kakutani’s fixed-point theorem then yields existence of a regular randomized equilibrium for every $\tau$9 (Zhang et al., 26 Jul 2025).

The regularized equilibrium is also an approximation device for the unregularized game. The follower’s extra entropy payoff is uniformly bounded by $\rho$0, and the paper states that if

$\rho$1

then the regularized equilibrium $\rho$2 yields an $\rho$3-equilibrium leader strategy for the original Stackelberg game, in the sense that the follower cannot improve by more than $\rho$4 by deviating from the regularized response pair $\rho$5 (Zhang et al., 26 Jul 2025).

The logical point is narrow but important. Entropy regularization does not prove that an exact unregularized equilibrium exists. It provides a regular randomized equilibrium for the modified game, and that equilibrium can be viewed as an approximation of the exact equilibrium when the latter is well defined. This is also why the paper emphasizes continuity of the best-response map rather than exact recovery of the original infinite-horizon Stackelberg stopping game (Zhang et al., 26 Jul 2025).

4. Continuous-time and mean-field formulations

Only the discrete-time Stackelberg stopping paper treats the leader-follower stopping game directly. Several nearby formulations provide continuous-time and large-population structures that can be used as building blocks.

Formulation	Control variable	Entropy term
Discrete-time Stackelberg stopping	Leader policy $\rho$6; follower stop probabilities $\rho$7	$\rho$8
Continuous-time randomized stopping	Stopping intensity $\rho$9	$$J_1(x,\tau,\rho)=\mathbb{E}_x\big[\beta^{\tau\wedge\rho}F_1(\tau,\rho)\big],\qquad J_2(x,\tau,\rho)=\mathbb{E}_x\big[\delta^{\tau\wedge\rho}F_2(\tau,\rho)\big],$$0
Mean-field stopping as singular control	Cumulative stopping probability $$J_1(x,\tau,\rho)=\mathbb{E}_x\big[\beta^{\tau\wedge\rho}F_1(\tau,\rho)\big],\qquad J_2(x,\tau,\rho)=\mathbb{E}_x\big[\delta^{\tau\wedge\rho}F_2(\tau,\rho)\big],$$1	$$J_1(x,\tau,\rho)=\mathbb{E}_x\big[\beta^{\tau\wedge\rho}F_1(\tau,\rho)\big],\qquad J_2(x,\tau,\rho)=\mathbb{E}_x\big[\delta^{\tau\wedge\rho}F_2(\tau,\rho)\big],$$2, canonically $$J_1(x,\tau,\rho)=\mathbb{E}_x\big[\beta^{\tau\wedge\rho}F_1(\tau,\rho)\big],\qquad J_2(x,\tau,\rho)=\mathbb{E}_x\big[\delta^{\tau\wedge\rho}F_2(\tau,\rho)\big],$$3
Major-minor mean-field stopping	Major relaxed control $$J_1(x,\tau,\rho)=\mathbb{E}_x\big[\beta^{\tau\wedge\rho}F_1(\tau,\rho)\big],\qquad J_2(x,\tau,\rho)=\mathbb{E}_x\big[\delta^{\tau\wedge\rho}F_2(\tau,\rho)\big],$$4	$$J_1(x,\tau,\rho)=\mathbb{E}_x\big[\beta^{\tau\wedge\rho}F_1(\tau,\rho)\big],\qquad J_2(x,\tau,\rho)=\mathbb{E}_x\big[\delta^{\tau\wedge\rho}F_2(\tau,\rho)\big],$$5

In the continuous-time single-agent exploratory stopping framework, stopping is encoded by a nonnegative intensity $$J_1(x,\tau,\rho)=\mathbb{E}_x\big[\beta^{\tau\wedge\rho}F_1(\tau,\rho)\big],\qquad J_2(x,\tau,\rho)=\mathbb{E}_x\big[\delta^{\tau\wedge\rho}F_2(\tau,\rho)\big],$$6, an auxiliary state $$J_1(x,\tau,\rho)=\mathbb{E}_x\big[\beta^{\tau\wedge\rho}F_1(\tau,\rho)\big],\qquad J_2(x,\tau,\rho)=\mathbb{E}_x\big[\delta^{\tau\wedge\rho}F_2(\tau,\rho)\big],$$7 evolves by $$J_1(x,\tau,\rho)=\mathbb{E}_x\big[\beta^{\tau\wedge\rho}F_1(\tau,\rho)\big],\qquad J_2(x,\tau,\rho)=\mathbb{E}_x\big[\delta^{\tau\wedge\rho}F_2(\tau,\rho)\big],$$8, and the reward functional adds the running regularizer $$J_1(x,\tau,\rho)=\mathbb{E}_x\big[\beta^{\tau\wedge\rho}F_1(\tau,\rho)\big],\qquad J_2(x,\tau,\rho)=\mathbb{E}_x\big[\delta^{\tau\wedge\rho}F_2(\tau,\rho)\big],$$9. The value function satisfies a smooth HJB equation, and the optimal intensity is

$F_1$0

After the logarithmic state transform $F_1$1, the value solves

$F_1$2

which is a penalized analogue of the classical variational inequality for optimal stopping (Dong, 2022).

In continuous-time mean-field stopping, randomized stopping times are reformulated as singular controls $F_1$3, where $F_1$4 is the conditional probability that the agent has already stopped by time $F_1$5. The entropy-regularized payoff is

$F_1$6

and equilibria exist for every $F_1$7; under Lasry-Lions monotonicity they are unique, and as $F_1$8 the regularized equilibria converge to equilibria of the unregularized game (Dianetti et al., 23 Sep 2025).

The same paper states that it does not treat Stackelberg structure directly, but it explicitly formulates an entropy-regularized bi-level extension in which a leader chooses a parameter $F_1$9, followers solve the $F_2$0-SC-MFG with $F_2$1-dependent data, and the leader optimizes against the induced equilibrium mapping $F_2$2. That proposal is presented as a natural extension rather than a proved Stackelberg theorem. It suggests that continuous-time entropy-regularized Stackelberg stopping can be cast as a leader optimization problem over follower equilibria in randomized stopping or singular-control form (Dianetti et al., 23 Sep 2025).

5. Computational and learning formulations

The discrete-time Stackelberg stopping result establishes existence of a regular randomized equilibrium, but it does not supply a general learning theorem for the leader. Computational structure instead comes from adjacent entropy-regularized stopping and Stackelberg literatures.

For continuous-time randomized stopping, policy iteration is explicit. Given a feedback intensity $F_2$3, one solves the linear PDE

$F_2$4

with

$F_2$5

and updates the policy by

$F_2$6

The iteration is monotone,

$F_2$7

and the paper proves a convergence rate involving $F_2$8 in the denominator. It also proposes a reinforcement learning algorithm based on temporal-difference error and the exact policy update $F_2$9 (Dong, 2022).

For mean-field stopping, the regularized singular-control game admits a fictitious-play algorithm. Starting from an initial control $\tau<\rho$0, one alternates between computing the best response

$\tau<\rho$1

and averaging controls,

$\tau<\rho$2

with induced measures updated by the consistency map $\tau<\rho$3. Under monotonicity and additional regularity assumptions, the iterates converge to the unique $\tau<\rho$4-SC-MFG equilibrium; under supermodularity, monotone initialization selects earliest or latest equilibria (Dianetti et al., 23 Sep 2025).

In a broader Stackelberg RL setting, the follower can be modeled as a quantal responder who solves an entropy-regularized policy optimization problem after observing the leader’s announced policy: $\tau<\rho$5 The resulting follower policy is unique and has Boltzmann form

$\tau<\rho$6

The paper develops sample-efficient online and offline algorithms based on maximum-likelihood estimation of the follower’s quantal response model and optimistic or pessimistic RL for the leader (Chen et al., 2023). This suggests a direct route to learning-based entropy-regularized Stackelberg stopping games by treating “stop” and “continue” as follower actions and by introducing an absorbing terminal state.

6. Mean-field, major-minor, and time-inconsistent extensions

Entropy-regularized Stackelberg stopping also appears in extended forms where the follower side is a population or where the leader-follower relation is interpreted dynamically.

In the major-minor mean-field game of stopping, the major player acts as a leader whose relaxed feedback control $\tau<\rho$7 affects both the dynamics and rewards of a continuum of minor players. Minor stopping problems are formulated as linear programs over occupation measures $\tau<\rho$8, while the major’s problem is regularized by Shannon differential entropy

$\tau<\rho$9

The regularized major control has Gibbs form,

$\rho<\tau$0

and regularized equilibria exist by Kakutani-Fan-Glicksberg. As $\rho<\tau$1, subsequences of regularized equilibria converge to relaxed equilibria of the original major-minor stopping game (Yu et al., 15 Jan 2025).

A different extension arises in time-inconsistent mean-field stopping with centralized stopping. There, a social planner chooses a stopping probability $\rho<\tau$2 for the whole population under non-exponential discounting, and the relevant equilibrium notion is a subgame-perfect relaxed equilibrium among temporal selves. Entropy is added both to the policy and to the discount through

$\rho<\tau$3

yielding the Gibbs rule

$\rho<\tau$4

The paper characterizes regularized equilibria as fixed points of $\rho<\tau$5, proves their existence by Schauder’s fixed-point theorem, and shows convergence to relaxed equilibria as $\rho<\tau$6 (Yu et al., 2023).

These extensions clarify two common misconceptions. First, “entropy regularization” is not a single universal object. In the stopping literature it appears as Bernoulli entropy $\rho<\tau$7 on statewise stop probabilities, as the pointwise concave term $\rho<\tau$8 on cumulative stopping probabilities, as the intensity regularizer $\rho<\tau$9, as Shannon entropy on stationary stopping policies, and as differential entropy on the leader’s control density (Zhang et al., 26 Jul 2025, Dianetti et al., 23 Sep 2025, Dong, 2022, Yu et al., 2023, Yu et al., 15 Jan 2025). Second, regularization does not eliminate the underlying time inconsistency or multiplicity issues by itself. What it does, in the proved cases, is smooth best responses, make fixed-point arguments feasible, and provide vanishing-$\mathbb{X}=\{1,\dots,N\}$00 approximations to unregularized equilibria or $\mathbb{X}=\{1,\dots,N\}$01-equilibria (Zhang et al., 26 Jul 2025, Yu et al., 2023).

The present state of the subject is therefore stratified. The discrete-time two-player Stackelberg stopping game has an explicit entropy-regularized equilibrium existence theory (Zhang et al., 26 Jul 2025). Continuous-time, mean-field, major-minor, and RL-based formulations supply closely related techniques—randomized stopping, singular control, occupation measures, Gibbs policies, fictitious play, and quantal-response learning—but some of their Stackelberg interpretations remain proposed extensions rather than established theorems (Dianetti et al., 23 Sep 2025, Chen et al., 2023).