Stability of POMDPs

• Stability of POMDPs is defined by robust decision-making under partial observability, ensuring reliable performance despite model uncertainties.
• Finite-memory controllers and regularity conditions, such as weak Feller properties and Wasserstein contractivity, enable algorithmic verification of ω‐regular objectives.
• Explicit error bounds and structural decompositions guide safe, optimal policy synthesis and model reduction in practical applications.

A partially observed Markov decision process (POMDP) is a mathematical model for optimal decision-making in stochastic environments where the system’s true state is not directly observable. The stability of POMDPs refers to the robustness, recurrence, and reliability of policies in controlling such systems, both qualitatively (e.g., ensuring ω-regular objectives with probability 1) and quantitatively (guaranteeing bounded performance loss under model perturbations, approximations, or learning). Modern research has established rigorous principles and explicit conditions under which POMDPs can be stabilized—algorithmically and numerically—by leveraging finite-memory controllers, regularity properties, structural decompositions, and robust approximations. These results form the theoretical backbone for safety, verification, and control synthesis in partially observable stochastic systems.

1. Decidability and Stability via Finite-Memory Controllers

The qualitative analysis of POMDPs with general ω-regular objectives (e.g., parity, Muller, Büchi coBüchi) is undecidable in unrestricted strategy classes. However, when restricted to finite-memory strategies, both almost-sure and positive winning for parity objectives are decidable and can be algorithmically realized (Chatterjee et al., 2013). The key results are:

• Memory Structure: Finite-memory controllers are constructed by projecting the original controller’s memory into a compact representation using the “projection” operation, which aggregates memory via (i) Boolean recurrence functions (BoolRec), indicating whether the system is inside a recurrent class, and (ii) set-recurrence functions (SetRec), recording, for each state, the set of colors (priorities) seen in recurrent classes reachable from the current memory state.
• Stabilization via Recurrent Classes: The memory projection yields a strategy that, once in a recurrent class where the ω-regular condition holds (e.g., minimal recurring parity is even), remains there almost-surely, ensuring the winning objective. The “collapse” in the projection graph preserves the recurrence structure and ensures robustness even under partial observability.
• Optimal Memory Bounds and Complexity: For parity objectives with d priorities, the required controller memory does not exceed $2^{3d|S|}$ (with |S| states), and the decision/synthesis problem is EXPTIME-complete.
• Failure of Simpler Strategies: Memoryless (belief-based) strategies, or even randomized stationary ones, are provably insufficient for stability in many parity or coBüchi objectives, underscoring the necessity of appropriately augmented finite memory.

2. Regularity, Continuity, and Stability of Belief Dynamics

The existence and stability of optimal (stationary) policies for POMDPs with Borel-measurable state, observation, and action spaces depend critically on continuity properties of both transition and observation kernels (Feinberg et al., 2014, Kara et al., 9 Dec 2024). Explicitly:

• Weak Feller Property: If the transition kernel $\mathcal{T}(\cdot | x,u)$ is weakly continuous, and the observation channel $Q(\cdot|x,u)$ is continuous in total variation (not merely setwise), the filter (belief-update) kernel $\eta(\cdot|z,u)$ inherits weak continuity. This ensures applicability of dynamic programming techniques and convergence of value functions.
• Wasserstein Contractivity: Under additional compactness and Lipschitz conditions, the belief transition kernel satisfies

$$W_1(\eta(\cdot|z,u), \eta(\cdot|z',u)) \leq K_2 \, W_1(z, z')$$

for a contraction constant $K_2 < 1$, essential for bounding propagation of errors and ensuring the regularity of value functions.

• Total Variation Continuity is Essential: Examples demonstrate that setwise continuity in the observation kernel is insufficient for weak (or Wasserstein) continuity of the belief kernel, potentially resulting in instability (e.g., discontinuous jumps in filter updates).
• Controlled Filter Stability: The contraction of the filter in total variation (or Wasserstein) implies that filters initialized from different initial priors merge exponentially fast under mixing kernels, ensuring the system stabilizes to (approximately) the same sequence of beliefs regardless of initial uncertainty (Kara et al., 2020, Kara et al., 9 Dec 2024).

3. Explicit Robustness and Approximation Error Bounds

Explicit, computable bounds quantify how POMDP performance and filter dynamics degrade under perturbations or approximations of transition (T) and observation (Q) kernels (Demirci et al., 14 Aug 2025):

• Filter Kernel Error Bound: For the filter kernel η induced by (T, Q) and its approximation ηₙ induced by (Tₙ, Qₙ),

$$d_{BL}(\eta, \eta^{T_n,Q_n}) \leq 2 \left( d_{TV}(T_n,T) + d_{TV}(Q_n,Q) \right)$$

and, under a Lipschitz condition on Q,

$$d_{BL}(\eta, \eta^{T_n,Q_n}) \leq 2 \left( L_Q d_{W_1}(T_n,T) + d_{TV}(Q_n,Q) \right)$$

where $d_{BL}$ is the bounded–Lipschitz distance, $d_{TV}$ is total variation, $d_{W_1}$ is Wasserstein-1, and $L_Q$ is the Lipschitz constant of Q.

• Value Function and Performance Deviation:

$$\| J^*_{\beta}(\eta) - J^*_{\beta}(\eta^{T_n,Q_n}) \|_{\infty} \le \frac{\beta}{1-\beta} \|J_\beta\|_{Lip} \; d_{W_1}(\eta, \eta^{T_n,Q_n})$$

providing a non-asymptotic, uniform guarantee that value function error scales linearly with the filter discrepancy, itself governed by perturbations in the underlying stochastic kernels.

• Finite Model Reduction: When quantizing state or observation spaces, these bounds yield rigorous guarantees that the loss in value function and cost decays to zero as quantization becomes finer—enabling stable, justified model reduction for implementation.

4. Algorithmic and Structural Guarantees for Verification and Controller Synthesis

Qualitative stability (almost-sure satisfaction of ω-regular, safety, or reachability properties) is ensured algorithmically via:

• Projection and Collapse of Memory Structure: The projection operation on finite-memory controllers described in (Chatterjee et al., 2013) forms the core of decision procedures for ω-regular objectives—enabling EXPTIME-complete synthesis with memory bounds described above.
• Barrier Certificates and Lyapunov Functions: Translating the belief evolution into hybrid system dynamics, safety is verified by constructing Lyapunov and barrier functions whose sublevel sets are invariant under Bayesian updates. These certificates can be computed via sum-of-squares or semidefinite programs (Ahmadi et al., 2019).
• Safe-Reachability and Symbolic Synthesis: Boolean constraints on the belief space (e.g., enforcing that all reachable beliefs remain in a safe region before achieving the reachability objective) are encoded into symbolic (SMT) constraints and solved via incremental SMT solvers, ensuring that the synthesized policy meets stringent stability and safety requirements (Wang et al., 2018).

5. Necessary and Sufficient Conditions for Policy Existence and Stability

The following regularity and compactness properties are both necessary and sufficient for the existence of optimal (and stable) policies:

Condition on Model Components	Requirement for Stability/Optimality	Consequences
One-stage cost	Bounded below and K-inf-compact	Ensures minimizers exist in lifted (belief) MDP
Transition kernel	Weakly continuous (or total variation cont.)	Filter process is weak Feller
Observation kernel	Continuous in total variation	Ensures filter updates are stable
Filtering kernel	Sequential continuity (Assumption (H))	Dynamic programming is valid on belief space

Stability in this context includes:

• Convergence of Value Iteration: Validity of BeLLMan’s equations, and convergence of value iteration algorithms.
• Lower Semicontinuity of Value Functions: Ensures robustness of optimizers under system perturbations.
• Existence of Measurable Selectors: The ability to select Borel measurable optimal policies.

Counterexamples where these properties fail (e.g., only setwise-continuous Q) explicitly demonstrate the collapse of stability and policy existence (Feinberg et al., 2014).

6. Control-Theoretic and Reinforcement Learning Perspectives

Control-theoretic insight introduces Lyapunov and barrier analysis of the belief process, providing a methodology for certifying stability (invariance of safe sets and avoidance of unsafe beliefs) without needing explicit value-function computation (Ahmadi et al., 2019). From the learning perspective:

• Finite-Memory Approximations: Filter stability is leveraged to show that finite window (finite-memory) policies can achieve arbitrary closeness to the optimal, with the approximation error decaying exponentially with memory length under mixing conditions (Kara et al., 2020, Kara et al., 9 Dec 2024).
• Quantized Approximations: Wasserstein and total variation continuity guarantee that quantized finite-state approximations yield near-optimal policies with explicitly bound differences in cost and behavior (Kara et al., 9 Dec 2024).
• Reinforcement Learning: Q-learning and related schemes operating over finite-memory or quantized belief spaces converge (almost surely) to near-optimal fixed points, provided the underlying belief update satisfies the appropriate contraction/ergodicity properties, ensuring stability even under online uncertainty.

7. Implications for Practical Applications and Controller Design

The confluence of these findings makes possible the robust synthesis of controllers for real-world partially observable systems:

• Controllers designed for approximate or finite abstractions (e.g., in quantized or gridded models) are guaranteed—by explicit error bounds—to be “stable” when deployed in the true (possibly high-dimensional) system, provided the approximation errors are small. This enables principled, error-aware design in robotics, autonomous vehicles, and networked systems (Demirci et al., 14 Aug 2025).
• Stability extends to verification and model checking via techniques that project infinite systems into finite, analyzable subspaces, or by encoding stability certificates as symbolic constraints.
• The requirement for total variation or Wasserstein regularity and filter contractivity provides concrete design guidelines for system identification and observation channel engineering to ensure that practical implementations remain stable in the face of unavoidable model mismatch.

In summary, the stability of POMDPs is now quantitatively characterized across verification, synthesis, model reduction, and learning. This is achieved through a hierarchy of regularity conditions on the base model, explicit error propagation bounds, algorithmic memory constructions guaranteeing invariant and recurrent classes, and robust reinforcement learning methods, thereby ensuring that POMDP-based control systems can be constructed and operated with strong assurance of stability and bounded risk under partial observability.