Partially Observable Reference Policy Programming (PRPP)

• Partially Observable Reference Policy Programming is an anytime solver for infinite-horizon POMDPs that combines deep sampling of future histories with iterative KL-constrained policy updates.
• It replaces explicit action optimization with a softmax parameterization of action preferences, enabling gradual and robust policy improvement without brittle commitment.
• Empirical benchmarks on tasks such as a 3D maze and Corsica rescue mission indicate success rates up to 94%, highlighting its scalability and enhanced decision-making under dynamic conditions.

Searching arXiv for the specified paper and closely related work to ground the article. Partially Observable Reference Policy Programming (PRPP, also denoted PORPP) is an anytime online approximate solver for infinite-horizon partially observable Markov decision processes (POMDPs) that combines deep sampling of future histories with gradual policy updates induced by KL regularization. In the formulation proposed in 2025, the method retains the analytically tractable structure of reference-based POMDP updates while avoiding brittle commitment to a single fixed reference policy. Instead of numerically optimizing action values at each node by explicit optimization or exhaustive enumeration, it maintains action preferences, converts them to a stochastic policy through a softmax parameterization, and improves that policy iteratively and online (Kim et al., 16 Jul 2025).

1. Conceptual position within POMDP planning

PRPP is defined on a standard infinite-horizon POMDP

$\langle S, A, O, T, Z, R, \gamma \rangle,$

where $S$ is the latent state space, $A$ the action space, $O$ the observation space, $T(s' \mid s,a)$ the transition model, $Z(o \mid s',a)$ the observation model, $R(s,a)$ a bounded reward, and $\gamma \in (0,1)$ the discount factor (Kim et al., 16 Jul 2025). As in the usual POMDP setting, the agent does not observe the true state directly and instead acts on a belief $b$, updated recursively from actions and observations.

The motivating problem is online planning under partial observability when action spaces are large, horizons are long, planning trees are deep, and the environment may be dynamic. Standard online POMDP planning is characterized in the source formulation as sampling future histories, estimating action values numerically, and then selecting the best action by explicit optimization or exhaustive enumeration over actions. PRPP is introduced precisely to alter that computational pattern while remaining compatible with deep lookahead.

A central antecedent is the family of earlier KL-regularized reference-based POMDP methods, denoted RBPOMDPs. Those methods use a reference policy and analytically tractable updates, but are described as fragile to reference-policy mis-specification because the solution is anchored to that reference policy. PRPP preserves the reference-policy formalism but replaces fixed anchoring with iterative policy improvement, where each new policy is only a gradual KL-constrained update of the previous one. This makes the method a form of reference-policy programming under partial observability rather than mere imitation of a heuristic controller.

2. Formal belief-space formulation

The belief update takes the standard Bayesian form

$$b'(s') \propto Z(o \mid s',a)\sum_{s\in S}T(s'\mid s,a)b(s), \qquad b'=\tau(b,a,o),$$

and thus the effective state of the planner is the posterior belief rather than the latent physical state (Kim et al., 16 Jul 2025).

The value function is defined on belief-action pairs: $S$0 with

$S$1

This representation places PRPP squarely in belief-space dynamic programming, but the algorithmic emphasis is not on explicit maximization over all actions at each belief. Instead, the planner works through a parametric preference representation.

A stochastic policy is represented by action preferences $S$2: $S$3 This softmax parameterization is fundamental. It converts local policy improvement into updates of preferences, which then induce action probabilities analytically. The policy therefore evolves continuously across iterations rather than through abrupt greedy replacement.

3. Reference-policy regularization and iterative improvement

For a reference policy $S$4, PRPP uses the regularized Bellman objective

$S$5

The KL term is the mechanism that enforces gradual policy change, with $S$6 controlling the role of the reference policy in the update (Kim et al., 16 Jul 2025).

This objective admits a closed-form log-sum-exp solution. Defining

$S$7

one obtains

$S$8

and the improved policy is

$S$9

The practical consequence is that the policy-improvement step is analytically tractable and does not require solving a separate numerical optimization problem at each belief node.

The exact iterative scheme is written as

$A$0

or equivalently

$A$1

This recurrence formalizes the central principle of PRPP: each iterate is a smoothed improvement over its predecessor, not a sudden greedy jump. Theoretical and empirical claims in the source both depend on this graduality.

4. Online anytime computation

PRPP is online and anytime because it interleaves planning and execution rather than computing a full offline solution before acting (Kim et al., 16 Jul 2025). At the current root history $A$2, the planner repeatedly samples a belief particle from the root, runs recursive simulations down a belief tree over histories, expands actions according to progressive widening, and updates values and preferences using sampled rollouts. When planning time expires, it chooses the action with the best sampled preference from the root, executes it, receives the resulting observation, advances the root to the corresponding child, and resamples particles at the new root.

The sampling procedure is designed to explore deeply. The implementation described in the source builds a belief tree over action-observation histories, samples state particles at nodes instead of enumerating all histories, uses a heuristic action sampler to propose promising macro-actions, uses softmax selection from current preferences to choose among available actions, rolls out recursively to a depth limit (max), and backs up sampled estimates to update preferences. If the depth exceeds the limit, the recursion returns a value heuristic.

The approximate online update at visited nodes is given by

$A$3

where $A$4 and $A$5 are Monte Carlo estimates of immediate reward and downstream value. The role of these estimates is not to support explicit argmax computation over all actions, but to perturb and refine preference values, after which the policy remains available in closed form through softmax.

The source emphasizes that this architecture is especially suited to long-horizon settings with large action spaces and dynamic environments. The more planning time is allocated, the deeper and broader the tree becomes and the better the preference estimates become; correspondingly, a small time budget yields a coarse but usable action decision, whereas a larger time budget yields deeper search and better policy quality.

5. Guarantees and approximation structure

The theoretical analysis assumes that the reachable belief space is totally bounded so that internal covering numbers are finite (Kim et al., 16 Jul 2025). Under this standing assumption, the method obtains an error statement for the exact and approximate iterative scheme in terms of accumulated approximation errors. Let $A$6 denote the approximation error at iteration $A$7, and define

$A$8

Then, if $A$9,

$O$0

The distinctive point is that the performance loss is bounded by the average of accumulated errors rather than the usual maximum single error. The source identifies this as crucial for online sampling, where sparse and noisy samples are expected and where bounds depending on a worst-case local error would be excessively pessimistic. In this reading, KL-regularized gradual improvement is not merely a stabilizing heuristic; it is the structural reason that approximation error can be averaged across iterations.

The paper also provides a sampling-based high-probability bound for belief-space approximation using an internal $O$1-covering $O$2 of the reachable belief set, with constants

$O$3

$O$4

where $O$5. The assumptions stated for the approximate result include bounded rewards, total boundedness of the reachable belief space, belief-space approximation by an internal $O$6-covering, and sampling assumptions for synchronous or asynchronous updates. In the asynchronous case, the sampler must visit relevant beliefs sufficiently often, especially those reachable under the optimal policy. The term $O$7 is identified as the irreducible approximation loss from projecting beliefs onto the covering set.

6. Empirical benchmarks and observed behavior

The empirical evaluation uses two long-horizon POMDPs with continuous states and macro-actions (Kim et al., 16 Jul 2025). In the 3D Maze with Poor Localisation, a holonomic 3D drone must reach one of two goal regions, dangerous zones carry penalty $O$8, the goal reward is $O$9, the step cost is $T(s' \mid s,a)$0, the robot starts in one of two initial positions with equal probability, and positional feedback is unavailable except at landmarks. The task horizon is about 100 steps. In the HEMS rescue mission in Corsica, a helicopter must reach two victim locations on the Cap Corse peninsula; each newly achieved objective yields $T(s' \mid s,a)$1, completing both objectives yields an additional $T(s' \mid s,a)$2, collision penalty is $T(s' \mid s,a)$3, step penalty is $T(s' \mid s,a)$4, and evolving no-fly zones change at unknown fixed time steps. The horizon is often at least 150 steps.

Benchmark	Salient properties	Horizon
3D Maze with Poor Localisation	Two possible goals, landmark-only localization, dangerous zones	About 100 steps
HEMS Rescue Mission in Corsica	Two victims, evolving NFZs, large 3D terrain	Often at least 150 steps

For both tasks, the heuristic action sampler uses an offline-generated Probabilistic Roadmap (PRM). In the maze, it samples a target landmark or goal and returns the shortest collision-free PRM path; in Corsica, it samples an unvisited victim and returns a collision-free homotopic path to it. These paths are truncated to form macro-actions. The compared methods are RefPol, which executes the heuristic reference policy without extra planning; RefSolver, the previous RBPOMDP solver using the heuristic as reference policy; and POMCP, which for fairness expands 16 macro-actions of equally spaced direction vectors. All methods are implemented in pomdp_py and Cythonized, using a desktop with 128GB RAM and an Intel Xeon Silver 4110; the discount factor is $T(s' \mid s,a)$5, and the reported tables use 100 runs per method.

On the 3D Maze, PORPP improves steadily with planning time: 71% success at 1s, 88% at 10s, and 94% at 15s. On the Corsica HEMS task, it is again the best-performing method, with 58% success at 1s and 90% at 10s, while the reward rises to about $T(s' \mid s,a)$6. The source further reports that RefPol is weaker, RefSolver performs much worse and can even degrade with more time, and POMCP is very poor in the maze and often fails almost entirely in the Corsica setting. The policy trace in the Corsica experiment indicates that PORPP adapts nontrivially to evolving NFZs, for example by descending to avoid a newly appeared NFZ and then detouring around terrain and hazards rather than taking the shortest path.

These experiments support several interpretive conclusions stated in the source: PRPP can exploit a heuristic policy without being trapped by it, gradual KL-constrained updates make online approximation robust, deep tree sampling makes long-horizon consequences accessible, and the method scales better than online baselines that rely on exhaustive action enumeration or fixed-reference commitment.

7. Scope, distinctions, and relation to broader policy programming under partial observability

PRPP should be distinguished from two nearby ideas. First, it is not equivalent to executing a heuristic reference policy. RefPol is explicitly only the reference policy without extra planning, whereas PRPP uses the reference policy as a starting point or proposal mechanism within iterative improvement. Second, it is not a generic elimination of optimization in every sense; more precisely, it avoids direct numerical optimization over actions at each node by replacing local argmax-style computation with analytically tractable softmax and log-sum-exp updates (Kim et al., 16 Jul 2025).

Its closest conceptual relation within the broader literature given here is to belief-dependent reactive control in POMDPs. An applied health-policy example casts vaccination timing, surveillance, and parameter estimation as a POMDP solved with dynamic programming, where the policy is a mapping from belief state to action and where the resulting strategy is reactive, time-varying, and informative about the value of information (Alaeddini et al., 2019). That earlier formulation shows how partial observability, belief updates, and decision-relevant information can be integrated in domain-specific control. PRPP addresses a different problem class—online approximate solution of large-scale infinite-horizon POMDPs—but it occupies the same general lineage of policy programming under partial observability: actions are selected from beliefs rather than observed states, observations are valued instrumentally through their effect on future control, and policy quality depends on how uncertainty is represented and updated.

The main practical constraints of PRPP arise from its own assumptions and approximation devices. The theory relies on total boundedness of the reachable belief space and on covering-based approximation. The online algorithm depends on the quality of the heuristic action sampler, the adequacy of progressive widening, and sufficient visitation of relevant beliefs in the asynchronous regime. A plausible implication is that the method is best matched to domains where deep but structured exploration is possible, where macro-actions can be proposed meaningfully, and where gradual policy improvement is preferable to brittle one-shot optimization.