RTDP-BEL: Belief-Space Dynamic Programming

• RTDP-BEL is a planning approach for POMDPs that discretizes the belief space to efficiently estimate value functions and reduce computational overhead.
• It employs rapid Bellman backups with upper/lower bounds and heuristic initialization to guide search and achieve quick convergence.
• Extensions like B³RTDP integrate probabilistic action pruning and a convergence frontier to optimize planning performance by focusing on the most informative belief regions.

Real-Time Dynamic Programming over Beliefs (RTDP-Bel) is an asynchronous heuristic search algorithm for solving Partially Observable Markov Decision Processes (POMDPs) by operating in the belief space, which is the space of probability distributions over the discrete set of underlying hidden states. RTDP-Bel and its extensions, such as B³RTDP (“Belief Branch and Bound RTDP”), address the intractability of exact POMDP solution methods by discretizing and clustering beliefs, applying computationally efficient value backups, and leveraging upper/lower value bounds, action pruning, and rapid convergence diagnostics. These approaches have shown high efficiency and competitive solution quality compared to leading point-based solvers in POMDP domains.

1. Belief Representation and Discretization

POMDP planning requires maintaining a belief $b$, a probability vector over the state space $S$. For computational tractability, RTDP-Bel discretizes the infinite belief simplex by rounding each belief component to one of $D$ bins: $$\hat b(s) = \lceil D \cdot b(s) \rceil, \quad s \in S$$ All beliefs mapping to the same discretized vector $\hat b$ are treated identically during updates. The discretized belief $\hat b$ serves as a hash table key, enabling storage and lookup of value estimates with reduced memory overhead. This clustering mechanism ensures bounded memory and computational cost per update, at the expense of introducing discretization error.

2. Value Function Approximation and Core RTDP-Bel Algorithm

RTDP-Bel maintains a value function estimate $\hat V(b)$ for each discretized belief encountered:

• On first visit, $\hat V(b)$ is initialized using an admissible heuristic, typically a convex combination of state-based heuristics such as QMDP.
• For subsequent visits, $\hat V(b)$ is retrieved from the hash table.

The central Bellman backup is applied as: $$V(b) \gets \max_{a \in A} \Bigg[ R(b,a) + \gamma \sum_{o \in O} \Pr(o \mid b,a) V\big(\tau(b,a,o)\big) \Bigg]$$ where $S$0 is the expected one-step reward, and $S$1 is the Bayesian update to the belief after taking action $S$2 and observing $S$3. Trajectories from the initial belief $S$4 are simulated, and value backup is performed along the visited path only. This forward trial-and-backup structure enables significant sample efficiency since only reachable beliefs from $S$5 are explicitly updated (Adalgeirsson et al., 2022, Saleem et al., 30 May 2025, Shani, 2024).

3. Extensions: B³RTDP and Action Pruning

B³RTDP extends RTDP-Bel with several innovations for faster convergence and improved scalability:

Bounds on Value Functions

Each discretized belief $S$6 is associated with a lower and upper bound: $S$7 The lower bound is initialized using an admissible heuristic, while the upper bound can be set to a blind policy value or a fast informed approximation. These bounds enable detection of “solved” beliefs ($S$8) and underlie pruning and convergence machinery.

Probabilistic Action Pruning

At each belief $S$9, Q-value intervals $D$0 are maintained. Under the assumption that $D$1 is uniformly distributed within bounds, the probability that action $D$2 dominates $D$3 is computable in closed form. Any action $D$4 for which $D$5 (for a user-specified threshold $D$6) can be pruned, thus accelerating convergence by reducing unnecessary action expansions.

The Convergence Frontier

The Convergence Frontier (CF) is a dynamic collection of beliefs and associated probability mass where current uncertainty is focused. For any belief where the greedy action has provably converged, expansion is halted for that prefix, and successor frontier beliefs are created with properly weighted probabilities. The next trial is sampled from CF proportional to uncertainty, concentrating effort on open parts of the search tree.

Trials terminate when either:

• the total mass in CF drops below a small threshold $D$7 (low residual reachability), or
• the value uncertainty, weighted by probability mass, falls below $D$8.

The integration of bounds and the CF substantially reduces both the depth and breadth of necessary exploration, collapsing redundant search and focusing computation on the most informative regions of belief space (Adalgeirsson et al., 2022).

4. RTDP-Bel and B³RTDP Algorithmic Structure

The RTDP-Bel core follows a simple structure: $$\hat b(s) = \lceil D \cdot b(s) \rceil, \quad s \in S$$0 B³RTDP introduces additional data structures (upper/lower bounds, CF) and subroutines:

• pickNextBelief considers successors’ uncertainty to determine trial continuation.
• prune removes actions likely to be dominated.
• b3RTDP_Trial pushes beliefs along a trajectory, then performs backward backups and pruning.
• updateCF maintains and propagates the frontier set.

A key distinction is that B³RTDP can terminate trials early based on convergence diagnostics, while standard RTDP-Bel rolls out until trajectory termination.

5. Comparative Performance and Practical Impact

Empirical benchmarks include RockSample_7_8 and Tag, using SARSOP as a point-based POMDP baseline configured with the same QMDP lower bound. Metrics reported are Average Discounted Reward (ADR) and wall-clock planning time to convergence.

Domain	Planner	ADR	Time (ms)
RockSample_7_8	SARSOP (1,000ms)	20.35	1,000
	B³RTDP (D=15, α=0.95)	21.49	1,548
	B³RTDP (D=20, α=0.95)	21.60	3,333
Tag	SARSOP (1,000ms)	–6.38	1,000
	B³RTDP (D=15, α=0.65)	–5.80	1,786
	B³RTDP (D=10, α=0.95)	–6.06	521

B³RTDP achieves higher returns in less or comparable planning time, outperforming SARSOP and the base RTDP-Bel algorithm in on-line planning efficiency and quality. The belief clustering inherent to hash-based storage, along with the use of bounds, enables rapid collapse of the search tree as dominated strategies are detected and eliminated. The algorithm’s focus on reachable beliefs and aggressive pruning supports “anywhere–anytime” operation: policies can rapidly reach or surpass the quality of competing solvers even before full convergence (Adalgeirsson et al., 2022).

6. Extensions, Heuristics, and Related Algorithmic Developments

RTDP-Bel serves as a core component in a diversity of heuristic and lazy search POMDP solvers. The efficacy of RTDP-Bel is sensitive to the quality of the heuristic; more accurate and informative heuristics directly reduce the number of forward trajectories and accelerate convergence. The belief-space delete-relaxation heuristic (“SDR-BEL”) leverages factored domain structure to directly quantify the value of information and multi-step sensing, substantially reducing sample complexity in information-rich domains (Shani, 2024). In robotics and other domains where Bayesian belief transitions are expensive to compute, lazy evaluation variants such as Lazy RTDP-Bel postpone costly computations by using cheap, conservative Q-value estimates, only performing full belief updates for promising actions. This expedites planning by orders of magnitude without sacrificing solution quality, provided the estimator remains underestimating (Saleem et al., 30 May 2025).

7. Limitations and Applicability

RTDP-Bel and B³RTDP’s discretization and clustering induce trade-offs in solution quality versus computational and memory efficiency. The granularity of belief discretization (parameter $D$9) directly impacts approximation error and runtime. In highly deterministic or “myopic” domains where information gathering is of little value, simple state-aggregated or MDP-based heuristics may suffice; conversely, in domains with significant optional sensing or stochasticity, advanced heuristics (e.g., SDR-BEL) and aggressive pruning (as in B³RTDP) provide substantial gains (Shani, 2024). The branch-and-bound mechanisms and convergence frontier techniques in B³RTDP are most beneficial when upper/lower value bounds are tight and informative, and where early action convergence can be detected and exploited effectively.

---

RTDP-Bel and its bounded, branch-and-bound extensions such as B³RTDP constitute a scalable, anytime approach to belief-space planning in POMDPs. Their integration of belief discretization, value bounding, probabilistic action elimination, and focused exploration yields superior empirical performance in a variety of benchmark domains, especially when coupled with structurally-informed heuristics or domain-specific lazy evaluation schemes (Adalgeirsson et al., 2022, Shani, 2024, Saleem et al., 30 May 2025).