Goal-Conditional POMDPs: Safe-Reachability & Planning

• Goal-Conditional POMDPs are decision-making models that integrate explicit goal conditions and safety constraints to guide actions under uncertainty.
• The framework incorporates bounded model checking and SMT-based synthesis to efficiently generate policies with formal guarantees on reaching goal states.
• Applications in robotic planning and information-gathering demonstrate significant improvements in computational efficiency and robust safety assurance.

A Goal-Conditional Partially Observable Markov Decision Process (POMDP) generalizes the standard POMDP framework by structuring decision-making under uncertainty around explicitly defined goal conditions. In this setting, an agent acts under state and observation uncertainty to either minimize expected cost or maximize the probability of reaching a goal state—potentially also satisfying strict safety constraints. Recent approaches have formalized these problems in goal-centric or safe-reachability settings, synthesizing policies with formal guarantees across all possible observation sequences, as well as introducing advanced algorithmic methods for efficient synthesis and planning (Wang et al., 2018, &&&1&&&).

1. Formal Definition and Problem Statement

A goal-based POMDP is typically defined by the tuple

$(S, A, \Omega, T, Z, b_0, G, C),$

where $S$ is a finite set of (hidden) world states, $A$ is the action set, $\Omega$ the observation set, $T$ and $Z$ are the transition and observation kernels, and $b_0$ the initial belief distribution. $G \subseteq S$ encodes the goal states (the agent's mission is complete when $s \in G$). $C(s,a)$ denotes the cost of action $a$ in state $s$ ($C(s,a)=0$ if $s \in G$). The belief state $b$ is a probability distribution over $S$, updated after an action-observation pair by

$b'(s') \propto Z(s',a,o)\sum_{s} T(s,a,s') b(s).$

In the "safe-reachability" extension, the state space $S$ is additionally partitioned into goal states $S_G$ and unsafe states $S_U$. Two thresholds are fixed:

• $\delta_{\text{reach}} \in (0,1)$: maximum goal miss probability
• $\delta_{\text{safe}} \in (0,1)$: maximum allowed probability of visiting unsafe states

The sets

$$\mathrm{Dest} = \{ b \mid \sum_{s \in S_G} b(s) \ge 1 - \delta_{\mathrm{reach}} \} \quad \text{and} \quad \mathrm{Safe} = \{ b \mid \sum_{s \in S_U} b(s) \le \delta_{\mathrm{safe}} \}$$

define belief states corresponding to goal "almost-certain" and safety satisfaction.

A trajectory $(b_0,a_1,o_1,b_1, \ldots, b_k)$ satisfies the safe-reachability objective if, for some $i \leq k$, $b_i \in \mathrm{Dest}$ and $\forall j < i$, $b_j \in \mathrm{Safe}$.

2. Goal-Constrained Belief Space and Bounded Model Checking

The conventional belief space $\Delta(S)$ is uncountably infinite. Effective policy synthesis thus requires restricting attention to a relevant subset. The "goal-constrained belief space" of horizon $h$ is defined as

$$\mathcal{B}_{\mathrm{goal}}^{(h)} = \{ b_i \mid \exists\,\sigma=(b_0,\dots,b_i) \text{ of length }i\le h \text{ satisfying the goal objective} \}.$$

This restriction is algorithmically realized by encoding the belief evolution and safety/liveness requirements as a first-order formula $\Phi_h$ over beliefs, actions, and observations. The formula $\Phi_h$ combines initial conditions, belief transitions, and a disjunction representing satisfaction of the destination and safety constraints:

$$\Phi_h := (b_0 = b_0^{\text{init}}) \wedge \bigwedge_{i=1}^h (b_i = T_B(b_{i-1},a_i,o_i)) \wedge \left( \bigvee_{i=0}^h \left[ b_i \in \mathrm{Dest} \wedge \bigwedge_{j<i} (b_j \in \mathrm{Safe}) \right] \right)$$

where $T_B$ is the deterministic belief-update operator. This encoding enables bounded model checking (BMC) approaches where only sequences satisfying safety up to goal are enumerated, facilitating orders-of-magnitude reduction in candidates considered (Wang et al., 2018).

3. Policy Objectives: Safe-Reachability vs. Expected-Reward Formulations

Traditional POMDPs maximize expected cumulative reward (or minimize expected cost):

$$V^*(b) = \max_\pi \mathbb{E}_\pi\left[\sum_{t=0}^{\infty} \gamma^t R(s_t, a_t) \mid b_0 = b \right].$$

Safe-reachability objectives enforce

$$\Pr_\pi [ \text{reach } S_G \text{ within } h ] \ge 1 - \delta_{\text{reach}}, \quad \Pr_\pi [ \text{visit } S_U \text{ at any step} ] \le \delta_{\text{safe}}$$

on every execution. This gives worst-case guarantees unattainable by reward-maximizing policies, which can be either overly conservative (never risking rare but necessary unsafe transitions) or risky (tolerating rare catastrophic failures if they make only a small contribution to expected cost). Goal-POMDP objectives (e.g., minimize expected cost-to-goal) and safe-reachability objectives are thus not interchangeable; the latter strictly enforces probabilistic pathwise specifications (Wang et al., 2018, Shani, 2024).

4. Solution Algorithms: Bounded Policy Synthesis and RTDP-BEL

Bounded Policy Synthesis (BPS): BPS searches for a finite-horizon policy of depth $\le h$ that provably meets safe-reachability objectives in all branches. Using the formula $\Phi_h$, an incremental Satisfiability Modulo Theories (SMT) solver is queried to enumerate valid plans. The BPS algorithm proceeds by proposing a candidate plan, recursively expanding branches for off-nominal observations, and blocking infeasible prefixes—yielding efficient, sound policy synthesis over large belief spaces. BPS leverages BMC encoding and solver-incrementality to reduce required solver calls and enables synthesis with worst-case guarantees (Wang et al., 2018).

RTDP-BEL (Real-Time Dynamic Programming over Beliefs): RTDP-BEL is tailored to goal-POMDPs where the agent terminates on reaching a goal belief. It simulates forward trajectories from $b_0$, applying Bellman backups only along sampled paths and using admissible heuristics for initialization:

$$V(b) \leftarrow \min_{a \in A}\left[ C(b,a) + \sum_{o \in \Omega} \Pr(o \mid b,a) V(\tau(b,a,o)) \right]$$

where $C(b,a)$ is the expected cost under belief $b$ and $\tau$ is the belief-update. On convergence, RTDP-BEL provides a policy minimizing expected cost-to-goal, exploiting structure and focused rollouts to avoid full belief space enumeration (Shani, 2024).

5. Admissible Heuristics for Goal-POMDPs: Delete-Relaxation and Value-of-Information

Efficient solution of goal-conditional POMDPs in challenging domains relies on informative heuristics. The belief-delete-relaxation, value-of-information heuristic constructs a relaxed planning graph in belief space, simulating positive effects of actions, modeling stochastic delays, and tracking information elimination via sensing. At each layer, actions are tested over the current support of the belief, and sensing is used to iteratively reduce valid states—optimistically extracting a contingent plan whose length provides an admissible heuristic $h(b)$ for RTDP-BEL rollouts (Shani, 2024).

The cost of this heuristic is $O(|\text{support}(b)| \cdot |A|^2)$, which is mitigated via caching on the valid set and most-likely state. In complex information-gathering domains (e.g., Maze, Wumpus), its use reduces the number of RTDP-BEL trajectories by one to two orders of magnitude, shrinking overall solve time by $5-10\times$ compared to cheaper MDP-based heuristics.

6. Empirical Evaluation and Practical Applications

Robotic Planning Case Study: BPS was evaluated in a simulated PR2 kitchen scavenging scenario with uncertain obstacles. Notable findings include:

• Synthesis time remains within seconds for up to $M=4$ obstacles, with incremental SMT solving yielding $2-3\times$ speedups over non-incremental approaches.
• The number of SMT calls is $\mathcal{O}(10^2)$, vastly lower than the combinatorial explosion in the full reachable belief tree ($\sim 10^{21}$ branches at depth 20).
• All synthesized policies provably satisfy the required probability-of-reach and probability-of-unsafe thresholds on every trajectory.

Heuristic-Guided RTDP-BEL: Across classic planning benchmarks (Localization, Logistics, Blocks, Maze, Wumpus), integrating the belief-delete-relaxation heuristic into RTDP-BEL reduces trajectory count and wall-clock solve time dramatically, especially when information-gathering is fundamental to task completion. In simple myopic-sensing domains, lightweight QMDP or hindsight maximum likelihood (HML) heuristics suffice, but complex sensing settings favor the structured heuristic (Shani, 2024).

7. Practical Considerations, Extensions, and Challenges

Embedding effective admissible heuristics and bounded symbolic encodings enables goal-conditional and safe-reachability POMDPs to scale to problems of realistic complexity. Key practical considerations include:

• Caching heuristic computations by belief support or most-likely state.
• Amortizing SMT solver costs via incremental scope management in BPS.
• For expensive heuristics or large support sizes, considering approximate belief tracking or support pruning.

Extensions of these frameworks accommodate non-unit costs, stochastic observations, and integration with point-based POMDP planners for improved solution quality. Domains where sensing is always necessary or belief supports are massive may pose computational challenges, making alternative factored or sampled belief representations necessary (Wang et al., 2018, Shani, 2024).

In summary, goal-conditional and safe-reachability POMDPs represent a critical evolution in planning under partial observability, providing formal methods for synthesizing robust policies with explicit trajectory-level guarantees on reaching goal states and avoiding unsafe regions, and supporting efficient policy computation through symbolic bounded verification and heuristically guided dynamic programming.