Uncertainty-Balanced Preference Planning (UBP2)

• The paper introduces UBP2 as an active PbRL method that integrates learned world models with MPC-based planning to optimize both control and preference inference.
• It employs a unified objective combining predicted rewards, terminal values, and multiple uncertainty bonuses to explicitly manage the exploration–exploitation tradeoff.
• Empirical evaluations on Meta-World tasks demonstrate that UBP2 improves sample efficiency and robustness compared to traditional, passive preference-based methods.

Uncertainty-Balanced Preference Planning (UBP2) is a model-based preference-based reinforcement learning (PbRL) method that actively directs exploration by jointly reasoning over uncertainty in reward, dynamics, and value functions. Rather than passively collecting trajectories and learning from preferences afterward, UBP2 uses a learned world model and an MPC-style planner to choose trajectories that are simultaneously promising for control and informative for reducing epistemic uncertainty. Its central mechanism is a unified trajectory score that combines expected reward, terminal value, and uncertainty bonuses, thereby making the exploration–exploitation tradeoff explicit rather than heuristic (Nabail et al., 17 Jun 2026).

1. Formal setting in preference-based reinforcement learning

UBP2 is formulated in a standard discounted MDP,

$(\mathcal{S}, \mathcal{A}, p, r, \gamma, \rho_0),$

with unknown transition dynamics $p^*(s' \mid s,a)$, unknown true reward $r^*(s,a)$, and discount $\gamma \in (0,1)$. The control objective is to learn a policy

$$\pi^\star = \arg\max_\pi \; \mathbb{E}_{\pi,p^*}\!\left[\sum_{t=0}^{H-1}\gamma^t r^*(s_t,a_t)\right].$$

What distinguishes the setting from standard RL is that scalar rewards are not observed directly. Instead, feedback takes the form of pairwise comparisons between trajectory segments $(\tau_i,\tau_j)$, modeled with Bradley–Terry preferences: $$P(\tau_i \succ \tau_j) = \frac{\exp\big(\sum_{t=0}^{L-1} r_\theta(s_t^i,a_t^i)\big)} {\exp\big(\sum_{t=0}^{L-1} r_\theta(s_t^i,a_t^i)\big) + \exp\big(\sum_{t=0}^{L-1} r_\theta(s_t^j,a_t^j)\big)}.$$ The learned reward $r_\theta$ is only identifiable up to an equivalence class of reward functions that induce the same preference ordering, so UBP2 treats it as a proxy reward rather than claiming exact recovery of the latent true reward (Nabail et al., 17 Jun 2026).

This formulation addresses a known inefficiency of many PbRL pipelines. Existing methods are described as passive: they collect trajectories with random exploration or fixed heuristics, fit reward models from the collected preferences, and only then train a policy. UBP2 targets the sample-efficiency problem that arises when those collected trajectories are uninformative for reward learning, dynamics learning, or control improvement. A common misconception is that PbRL inefficiency is solely a reward-learning issue; UBP2 instead treats it as a joint planning-and-inference problem in which exploration should be shaped by what matters for both preference identification and downstream control (Nabail et al., 17 Jun 2026).

2. Unified uncertainty-balanced planning objective

UBP2 learns an ensemble reward model $r_\theta$, an ensemble dynamics model $p_\phi$, an ensemble value function $p^*(s' \mid s,a)$0, and a learned policy $p^*(s' \mid s,a)$1. Its key novelty is that planning is performed over imagined trajectories scored by a unified objective combining predicted cumulative reward, terminal value, reward uncertainty, dynamics uncertainty, and value uncertainty. In the notation of the method, the planner optimizes

$p^*(s' \mid s,a)$2

where $p^*(s' \mid s,a)$3, the imagined states $p^*(s' \mid s,a)$4 come from the learned dynamics model, $p^*(s' \mid s,a)$5 and $p^*(s' \mid s,a)$6 are ensemble means for reward and value, $p^*(s' \mid s,a)$7, $p^*(s' \mid s,a)$8, and $p^*(s' \mid s,a)$9 are epistemic uncertainty estimates, and $r^*(s,a)$0 control the tradeoff between exploitation and exploration (Nabail et al., 17 Jun 2026).

The decomposition is conceptually important. The $r^*(s,a)$1 term favors actions that look immediately good under the learned proxy reward; $r^*(s,a)$2 extends this to a terminal value estimate; $r^*(s,a)$3 prioritizes regions where preference-derived reward knowledge is uncertain; $r^*(s,a)$4 prioritizes model uncertainty in transition structure; and $r^*(s,a)$5 accounts for uncertainty in long-horizon continuation value. UBP2 is therefore not equivalent to a simple reward bonus. The paper explicitly distinguishes it from “add uncertainty bonus to reward” schemes by emphasizing multi-component uncertainty, trajectory-level scoring, MPC-based action selection, and preference-aware query choice (Nabail et al., 17 Jun 2026).

The uncertainty coefficients are not fixed heuristics. They are tuned online through an autotuning objective,

$r^*(s,a)$6

This design is intended to adapt the strength of uncertainty bonuses to current model behavior and policy uncertainty. A plausible implication is that UBP2 treats exploration pressure as a state of the learning system rather than as a manually fixed hyperparameterization of optimism (Nabail et al., 17 Jun 2026).

3. Planning pipeline, ensemble uncertainty, and preference querying

UBP2 follows a TD-MPC2-style model-based pipeline. During the preference-feedback phase, at each environment step the planner samples candidate action sequences, rolls them out through the learned dynamics model, evaluates each imagined trajectory with the unified optimistic score, executes only the first action of the best sequence, stores real transitions in replay, and periodically queries preference labels from trajectory segments. The planner is MPC-style and re-plans at every step. Once the preference budget is exhausted, UBP2 switches from planning to the learned policy $r^*(s,a)$7, which is trained from the learned reward and value models for fast execution (Nabail et al., 17 Jun 2026).

Ensembles are the practical mechanism for uncertainty estimation. Dynamics and value uncertainty are estimated through ensemble disagreement, while reward uncertainty is estimated using Jensen–Rényi divergence (JRD), which is used to isolate epistemic uncertainty more explicitly. The paper notes a distinction here: disagreement for dynamics and value captures total model spread, whereas JRD for reward is intended to better separate epistemic from aleatoric uncertainty. This asymmetry reflects the special status of reward in PbRL, where preference noise and representational non-identifiability complicate direct interpretation of ensemble variance (Nabail et al., 17 Jun 2026).

Preference acquisition is itself uncertainty-aware. UBP2 does not query preferences uniformly at random; it uses an optimistic preference pair selection strategy that ranks candidate trajectory pairs by a score combining predicted preference likelihood and reward-model uncertainty. For each candidate pair $r^*(s,a)$8, the method computes ensemble returns

$r^*(s,a)$9

forms an optimistic score from the ensemble mean and spread of preference probabilities, and selects the top-$\gamma \in (0,1)$0 pairs globally. The paper contrasts this with local disagreement sampling by noting that the UBP2 procedure accounts for reward value and searches globally across the replay buffer (Nabail et al., 17 Jun 2026).

The training architecture is joint. The dynamics model is trained with a consistency loss,

$\gamma \in (0,1)$1

The reward ensemble is trained with a preference loss,

$\gamma \in (0,1)$2

The value ensemble is trained by TD regression using predicted rewards, and the explicit policy is trained with an entropy-regularized objective. This joint optimization is one of the method’s defining characteristics: reward learning, model learning, value estimation, planning, and query selection are not separate modules loosely composed after the fact, but components of a single active-learning control loop (Nabail et al., 17 Jun 2026).

4. Theoretical assumptions and regret guarantees

The theory for UBP2 is built under standard GP/RKHS-style assumptions. These include continuous dynamics and bounded reward; dynamics and reward lying in RKHS spaces with bounded norms; well-calibrated learned GP models satisfying bounds of the form

$\gamma \in (0,1)$3

and

$\gamma \in (0,1)$4

a $\gamma \in (0,1)$5-function approximation error bound

$\gamma \in (0,1)$6

and, in the infinite-horizon case, an assumption that $\gamma \in (0,1)$7-uncertainty is proportional to discounted cumulative reward and dynamics uncertainty. The cleanest main result also assumes no planner suboptimality, while the appendix treats planner suboptimality explicitly (Nabail et al., 17 Jun 2026).

The central theoretical statement is an optimism lemma. For suitable coefficients $\gamma \in (0,1)$8, the true return is upper bounded by the optimistic planning objective: $\gamma \in (0,1)$9 In effect, the planning score used by UBP2 acts as an upper confidence bound over trajectories. This gives a formal interpretation of the unified score: the uncertainty bonuses are not merely exploratory incentives, but components of an optimism-based control criterion (Nabail et al., 17 Jun 2026).

The resulting regret guarantees are sublinear under the stated regularity conditions. For the finite-horizon case,

$$\pi^\star = \arg\max_\pi \; \mathbb{E}_{\pi,p^*}\!\left[\sum_{t=0}^{H-1}\gamma^t r^*(s_t,a_t)\right].$$0

For the infinite-horizon case,

$$\pi^\star = \arg\max_\pi \; \mathbb{E}_{\pi,p^*}\!\left[\sum_{t=0}^{H-1}\gamma^t r^*(s_t,a_t)\right].$$1

where

$$\pi^\star = \arg\max_\pi \; \mathbb{E}_{\pi,p^*}\!\left[\sum_{t=0}^{H-1}\gamma^t r^*(s_t,a_t)\right].$$2

The interpretation given in the paper is that regret scales with information gain in the dynamics and reward models, formalizing the claim that UBP2 is efficient when informative samples suffice to learn the environment and preference-induced proxy reward (Nabail et al., 17 Jun 2026).

5. Empirical evaluation on Meta-World

UBP2 is evaluated on 10 Meta-World manipulation tasks using proprioceptive observations and task-specific success metrics. The tasks are Door Close, Window Close, Handle Press, Coffee Button, Faucet Open, Door Open, Door Unlock, Sweep Into, Drawer Open, and Hammer. Preference feedback budgets vary by task from 500 to 10,000 comparisons. The baselines are RUNE, described as a model-free PbRL method with reward-uncertainty exploration; MRN (Meta-Reward-Net), a model-free reward-learning method with bilevel optimization; and MBP, a non-optimistic model-based PbRL baseline. Performance is reported with IQM success rate over environment steps (Nabail et al., 17 Jun 2026).

The main findings are that UBP2 reaches high success rates earlier in 9 out of 10 Meta-World tasks, achieves the best average performance and best average rank across tasks, and is generally more efficient and more robust than the model-free methods. The comparison with MBP is used to isolate the effect of uncertainty-guided planning: model-based learning alone is not enough to match the full method. The ablations further report that reward uncertainty, dynamics uncertainty, value uncertainty, and optimistic preference selection each contribute, that no single component alone is sufficient, and that the full model provides the best overall balance between performance and robustness (Nabail et al., 17 Jun 2026).

Additional empirical results refine the scope of these claims. Optimistic preference selection outperforms disagreement-based and entropy-based query selection. A longer planning horizon sometimes helps, but not universally, and the best horizon is task-dependent. UBP2 also remains effective under tighter preference budgets, indicating robustness when annotation is scarce. These findings support the method’s central thesis that active trajectory planning and active preference querying should be coordinated rather than treated as independent design choices (Nabail et al., 17 Jun 2026).

6. Relation to neighboring formulations and broader interpretations

UBP2 is closely related to other uncertainty-aware preference-planning frameworks, but its specific contribution is tied to PbRL and model-based optimistic control. The paper "Churn-Aware Recommendation Planning under Aggregated Preference Feedback" formalizes a belief-space planning problem in which a recommender chooses categories for an anonymous user drawn from a prior over latent types, receives binary feedback, updates beliefs by Bayes’ rule after positive responses, and terminates the session after a negative response. Its value function measures the expected number of likes before churn, and its theory shows that in well-separated instances optimal policies eventually converge to pure exploitation in finite time. It also proposes a branch-and-bound algorithm with upper and lower bounds, and reports performance gains over the POMDP solver SARSOP in regimes with many user types or square type-category matrices (Keinan et al., 6 Jul 2025). The connection to UBP2 is structural rather than domain-specific: both methods make exploration contingent on uncertainty, and both assign an explicit operational cost to incorrect exploratory actions. The difference is that Rec-APC models uncertainty over latent user types with hard session termination on dislike, whereas UBP2 models uncertainty over reward, dynamics, and value in an MDP with pairwise preference supervision.

A second neighboring formulation appears in "Decision-theoretic MPC: Motion Planning with Weighted Maneuver Preferences Under Uncertainty," which addresses autonomous driving by keeping multiple maneuver hypotheses alive within a receding-horizon nonlinear MPC problem rather than committing to a single maneuver homotopy upfront. The planner combines maneuver-specific objectives through weighted sums such as

$$\pi^\star = \arg\max_\pi \; \mathbb{E}_{\pi,p^*}\!\left[\sum_{t=0}^{H-1}\gamma^t r^*(s_t,a_t)\right].$$3

with weights derived from maneuver probabilities, existence probabilities, or belief about route intent, and it propagates state, existence, and prediction uncertainty into chance constraints while maintaining the feasibility of a full-braking fallback maneuver (Taş et al., 2023). This is not PbRL, but it is closely aligned with the broader design principle that preferences should be balanced under uncertainty without premature commitment. A plausible interpretation is that UBP2 belongs to a wider class of uncertainty-balanced planners in which a decision-maker evaluates alternatives by combining preference structure with uncertainty-aware optimism, although UBP2 is distinctive in grounding that idea in pairwise preference learning, ensemble world models, optimistic query selection, and regret analysis.

Two misconceptions are clarified by these comparisons. First, UBP2 is not simply a reward-learning method with active data collection; its defining mechanism is planning under joint uncertainty in reward, dynamics, and value. Second, it is not equivalent to generic optimistic exploration, because both the trajectory-planning objective and the preference-query mechanism are preference-aware and model-based. Its distinctive contribution is therefore the integration of active reward inference, active control, and explicit uncertainty balancing within a single PbRL framework (Nabail et al., 17 Jun 2026).