Long-Term Average Impulse Control

• Long-term average impulse control is a framework using impulse interventions in stochastic systems to minimize average cost per unit time.
• The approach employs quasi-variational inequalities and ergodic Bellman equations to derive optimal threshold-type strategies.
• Applications span operations research, inventory management, and economics, leveraging domain truncation and renewal theory for robust analysis.

Long-term average impulse control addresses the optimization of systems governed by continuous-time stochastic dynamics in which interventions—impulses—are allowed at adaptively chosen times, subject to costs or rewards, with the objective being minimization (or maximization) of the average cost (or reward) per unit time over the infinite horizon. The mathematical formulation leads to an ergodic, or long-run, control problem for Markov processes and has seen systematic development for general Feller–Markov processes, Lévy processes, and Itô diffusions. Canonical applications include operations research, inventory theory, and ergodic stochastic control in economics and resource management.

1. General Framework and Problem Formulation

The setting consists of a state process $(X_t)$, typically a Feller–Markov process on a locally compact separable metric space $E$, evolving under its natural law except at random intervention (impulse) times $0 = \tau_0 < \tau_1 < \tau_2 < \dots$ (with $\tau_n \to \infty$ a.s.), where the state is instantaneously reset to a prescribed value $\xi_n \in U \subset E$ (typically $U$ is compact). Between impulses, $X$ follows its uncontrolled dynamics.

Costs are specified by a running cost $c : E \to [0, \infty)$, continuous and bounded on compacts, and an impulse cost $K : E \times U \to (0,\infty)$, continuous, bounded and bounded away from zero. The long-term average cost per unit time for a strategy $V = \{(\tau_n, \xi_n)\}$ starting from $x \in E$ is

$$J(x, V) = \limsup_{T \to \infty} \frac{1}{T} \mathbb{E}_x^V\left[\int_0^T c(Y_s)\,ds + \sum_{n: \tau_n \leq T} K(Y_{\tau_n^-}, \xi_n)\right],$$

where $Y_t$ is the controlled process. The optimization problem is to find $\lambda^* = \inf_{x \in E} \inf_V J(x, V)$ and a nearly optimal or optimal strategy $V^*$ attaining it (Stettner, 2022).

2. The Ergodic Bellman Equation and Quasi-Variational Inequalities

The central tool is the ergodic Bellman (quasi-variational) equation for the unknown long-run average cost $\lambda$ and a relative value function $h : E \to \mathbb{R}$: $$\lambda + h(x) = \inf_{\substack{t \ge 0 \ a \in U}} \mathbb{E}_x\left[\int_0^t c(X_s)\,ds + K(X_{t^-}, a) + h(X_t^{(a)})\right], \quad x \in E,$$ where $X_t^{(a)}$ is the process $X$ started at $X_t$ and then instantaneously reset to $a$ at time $t$. This is equivalent to a quasi-variational inequality (QVI): $$\min\left\{\lambda + \mathcal{L} h(x) - c(x),\; h(x) - \inf_{a \in U} \big(K(x,a)+h(a)\big)\right\} = 0, \quad x \in E,$$ with $\mathcal{L}$ the generator of $(X_t)$ (Stettner, 2022).

Existence and uniqueness of a pair $(\lambda, h)$ (up to an additive constant in $h$), are guaranteed for Feller–Markov processes satisfying ergodicity (existence of a unique invariant measure and solution to the Poisson equation), continuity preservation under stopped semigroups, control of exit time moments, and tightness (Stettner, 2022).

3. Domain Truncation and Approximation Techniques

Due to state space unboundedness, domain truncation is used for practical and theoretical approximation. One constructs an increasing exhaustion by relatively compact open sets $D_1 \subset D_2 \subset \cdots \uparrow E$, with associated exit times $\tau_{D_n} = \inf\{t \ge 0 : X_t \notin D_n\}$, and poses a stopped impulse control problem in each $D_n$ whereby the controller is forced to stop at the boundary and pays a penalty $v_n(X_{\tau_{D_n}})$. The stopped Bellman equation is solved in $D_n$, and as $n \to \infty$, the solutions $(\lambda_n, h_n)$ converge, respectively, to $(\lambda, h)$ of the original problem (Stettner, 2022).

$$\lambda_n + h_n(x) = \inf_{\substack{t \ge 0 \ a \in U}} \mathbb{E}_x \left[\int_0^{t \wedge \tau_{D_n}} c(X_s)\,ds + K(X_{t^-}, a) 1_{t<\tau_{D_n}} + h_n(X_{t \wedge \tau_{D_n}}^{(a)})\right],$$

with $h_n(x) = v_n(x)$ for $x \in \partial D_n$. It is shown that $\lambda_n \downarrow \lambda$ and $h_n \to h$ uniformly on compacts, and the nearly optimal policies on $D_n$ are $\varepsilon$-optimal in the full model for large enough $n$ (Stettner, 2022).

4. Structure and Verification of Optimal Policies

Optimal strategies are characterized by threshold-type feedback rules, explicitly delineated via the “action region” $\mathcal{A}$ and the “waiting region” $\mathcal{C} = E \setminus \mathcal{A}$, with

$$\mathcal{A} = \{x \in E : h(x) = \inf_{a \in U} [K(x,a) + h(a)]\}, \qquad G(x) = \inf_{a \in U}\{K(x, a) + h(a)\}.$$

The optimal strategy is: wait until the process enters $\mathcal{A}$, then apply an impulse to $a^* \in \arg\min_{a}(K(\cdot, a) + h(a))$, then restart (Stettner, 2022). This threshold (or band) form encompasses classical $(s,S)$ and multi-band control structures, depending on the cost and reward structure, and is robust to various generalizations including random effect kernels and mean field settings (Helmes et al., 2019, Helmes et al., 16 May 2025).

5. Probabilistic and Renewal-Theoretic Methods

For i.i.d.-cycle or stationary Markov impulse policies, as formalized in (Helmes et al., 2019), renewal theory provides an explicit and tractable route to long-term average costs by reduction to classical renewal-reward theory. Define the cycle as the interval between two consecutive impulses, with cycle length $\tau$ and cycle cost $C$. When cycles are i.i.d. and have finite mean and cost, the average cost is given by

$g(\pi) = \frac{\mathbb{E}[C]}{\mathbb{E}[\tau]}.$

Such renewal approaches are applicable when the policy yields independent cycles, as is the case for $(s,S)$-type policies and under renewal-theoretic stability, which is typically ensured by ergodicity and appropriate cost growth conditions (Helmes et al., 2019). This framework is extensible to complex models, including those with random effect (post-impulse state randomized according to a transition kernel) or mean field interactions (Helmes et al., 16 May 2025, Christensen et al., 2020).

6. Extensions: Multiplicative and Risk-Sensitive Criteria

Variants of the standard additive long-term average include risk-sensitive (multiplicative) objectives: $$J(x, V) = \liminf_{T \to \infty} \frac{1}{T} \ln \mathbb{E}_{(x,V)}\left[\exp\left( \int_0^T f(Y_s)\,ds + \sum_{i:\tau_i \le T} c(Y_{\tau_i^-}, \xi_i)\right)\right],$$ with associated Bellman equations characterized by nonlinear fixed-point relations and quasi-variational inequalities involving post-impulse operators of exponential type (Jelito et al., 2023). Existence and optimality rely on non-linear spectral theory (Krein–Rutman theorem), Markov process ergodicity, and approximation by compact domain and dyadic time discretizations.

For risk-sensitive performance functionals, both dyadic and continuous-time Bellman equations can be analyzed, with the existence and uniqueness of solutions established via contraction mapping principles under geometric drift and local minorisation conditions, even in unbounded settings (Jelito et al., 2019, Pitera et al., 2019).

7. Applications, Numerical Schemes, and Further Generalizations

Long-term average impulse control has applications in inventory theory, stochastic production and harvesting, queueing networks, and communications resource allocation. Band and $(s,S)$-type policies are prevalent, including in Brownian inventory with convex holding costs (Dai et al., 2011) and diffusion-driven production-inventory with control of drift and impulses (Cao et al., 2016). For Lévy processes, explicit solutions leverage scale function calculus and maximize a cycle-ratio functional in terms of auxiliary functions derived from the process generator and cost structure (Christensen et al., 2019). For problems with mean field interactions, competitive and cooperative frameworks are amenable to analytic fixed-point or Lagrangian optimization, yielding explicit threshold equilibria and demonstrating deviations between Nash and Pareto optimal solutions (Helmes et al., 16 May 2025, Christensen et al., 2020).

Convergence of policies and values under domain truncation, and equality of general discounted and undiscounted long-run optimal values for general discount kernels, are established, showing robust time-consistency and optimality of stationary Markov strategies even under general (non-exponential) discounting (Jelito et al., 2023).

The field remains active, with modern work addressing learning in unknown dynamics, exploration–exploitation trade-offs, and the implementation of nonparametric estimators to maintain near-optimality with quantifiable regret bounds (Christensen et al., 2019).