Reward Ensemble under Confidence (REC)

• The paper introduces a novel ensemble-based probabilistic reward model that improves preference learning by capturing uncertainty in human feedback.
• It employs Gaussian noise aggregation and uncertainty-aware predictions to stabilize reward estimates and enhance policy performance.
• REC’s ensemble resetting mechanism maintains diversity and prevents overfitting, leading to superior control in challenging acrobatic tasks.

Searching arXiv for the cited REC paper and closely related reward-uncertainty work. arXiv search query: (Merk et al., 26 Aug 2025) Reward Ensemble under Confidence Preference PPO reward uncertainty ensemble reset Reward Ensemble under Confidence (REC) is a preference-based reinforcement learning method introduced in "Learning Real-World Acrobatic Flight from Human Preferences" (Merk et al., 26 Aug 2025). It extends Preference PPO by replacing deterministic reward prediction with an ensemble-based probabilistic reward model, so that learned rewards are uncertainty-aware rather than point estimates. The method is motivated by acrobatic drone flight, where trajectory quality depends not only on task completion but also on style, smoothness, and timing, and where human feedback is noisy, sparse, subjective, and expensive. In the reported continuous powerloop task, REC achieves 88.4% of shaped-reward performance, compared to 55.2% with standard Preference PPO, and the learned policies are trained in simulation and transferred to real-world drones (Merk et al., 26 Aug 2025).

1. Origins, scope, and motivation

REC was introduced for agile drone control under human preference supervision, with a particular focus on dynamic maneuvers such as powerloops. The paper frames preference-based reinforcement learning as well-suited to objectives that are difficult to formalize or inherently subjective, and acrobatic flight is presented as a particularly challenging instance because of its complex dynamics, rapid movements, and the importance of precise execution. In this setting, standard Preference PPO learns a reward model from pairwise trajectory comparisons using a deterministic reward predictor (Merk et al., 26 Aug 2025).

The stated motivation for REC is that, in acrobatic flight, human preferences are subjective and sometimes inconsistent, feedback is sparse and expensive, the reward model can overfit to noisy labels, uncertainty in regions with little data is not explicitly modeled, and policy learning can become unstable if the reward model is overconfident. REC is introduced to address this by modeling reward uncertainty probabilistically, using an ensemble of reward networks, propagating uncertainty into preference prediction, and resetting weak ensemble members to maintain diversity and avoid collapse. The paper’s framing is that REC captures the probabilistic nature of human feedback better than a deterministic preference model, and that this improves performance in preference-based RL (Merk et al., 26 Aug 2025).

2. Baseline preference modeling and REC’s probabilistic reformulation

The baseline objective is the standard Bradley–Terry or softmax preference model used in Preference PPO. A trajectory is

$$\tau = \big((o_1,a_1),(o_2,a_2),\dots,(o_N,a_N)\big)$$

with total reward

$r(\tau)=\sum_{i=1}^N r_i.$

For two trajectories, the modeled preference is

$$\hat{p}(\tau_1 > \tau_2)=\frac{\exp(\hat{r}_1)}{\exp(\hat{r}_1)+\exp(\hat{r}_2)},$$

and the standard reward-learning objective is the cross-entropy loss

$$\text{loss}(\hat{r}) = - \sum_{(\tau_1^i,\tau_2^i,p^i)\in D}\Big[ p^i \log \hat{p}(\tau_1^i > \tau_2^i) + (1-p^i)\log \hat{p}(\tau_2^i > \tau_1^i) \Big].$$

This deterministic objective is the baseline that REC modifies (Merk et al., 26 Aug 2025).

REC changes the reward model from a deterministic scalar output to a distributional prediction at each timestep. The paper assumes

$$r(o_t,a_t)\sim \mathcal{N}\big(r_{\text{mean}(o_t,a_t)},\, r_{\text{std}(o_t,a_t)}\big),$$

and for a full trajectory,

$$r(\tau)\sim \mathcal{N}\Bigg( \sum_t r_{\text{mean}(o_t,a_t)}, \sqrt{\sum_t r_{\text{std}(o_t,a_t)}^2} \Bigg) = \mathcal{N}\big(r_{\text{mean}(\tau)},\, r_{\text{std}(\tau)}\big).$$

Preference probability is then computed with the Gaussian CDF:

$$p(\tau_1 > \tau_2) = \Phi\left( \frac{r_{\text{mean}(\tau_1)} - r_{\text{mean}(\tau_2)}} {\sqrt{r_{\text{std}(\tau_1)}^2 + r_{\text{std}(\tau_2)}^2}} \right).$$

This is the core “under confidence” mechanism: if uncertainty is high, the preference probability moves toward $0.5$; if the mean difference is large relative to uncertainty, the probability approaches $0$ or $1$ (Merk et al., 26 Aug 2025).

The corresponding training objective is the Probabilistic Cross Entropy Loss,

$r(\tau)=\sum_{i=1}^N r_i.$0

where the predicted preference probability now comes from the Gaussian reward model rather than a deterministic scalar reward. REC also adds a standard deviation target term. With target standard deviation $r(\tau)=\sum_{i=1}^N r_i.$1 and mean standard deviation across timesteps in the preference dataset $r(\tau)=\sum_{i=1}^N r_i.$2, the added loss is

$r(\tau)=\sum_{i=1}^N r_i.$3

The stated purpose is to keep predicted uncertainty calibrated to a reasonable scale, prevent degenerate ensemble uncertainty estimates, and encourage stable variance across members (Merk et al., 26 Aug 2025).

3. Ensemble structure, reward aggregation, and reset dynamics

REC uses an ensemble of MLP reward models. Each ensemble member takes $r(\tau)=\sum_{i=1}^N r_i.$4 as input and outputs a per-timestep predicted reward. The reported reward-model hyperparameters are hidden size $r(\tau)=\sum_{i=1}^N r_i.$5, number of layers $r(\tau)=\sum_{i=1}^N r_i.$6, activation $r(\tau)=\sum_{i=1}^N r_i.$7, learning rate $r(\tau)=\sum_{i=1}^N r_i.$8, and 100 epochs per retraining. The ensemble size is 5 in the flightmare or drone setting and 3 in the walker-walk DM Control setting (Merk et al., 26 Aug 2025).

A distinctive REC mechanism is uncertainty-aware reward aggregation. Rather than simply averaging ensemble predictions, the paper defines

$r(\tau)=\sum_{i=1}^N r_i.$9

where

$$\hat{p}(\tau_1 > \tau_2)=\frac{\exp(\hat{r}_1)}{\exp(\hat{r}_1)+\exp(\hat{r}_2)},$$0

The noise scale is tied to ensemble disagreement or variance, and the paper explicitly describes this as increasing reward in regions of high uncertainty, following prior work on reward uncertainty exploration. In the language used by the paper, REC is therefore both an uncertainty estimation mechanism and an uncertainty utilization mechanism (Merk et al., 26 Aug 2025).

A second mechanism is ensemble resetting. Before each reward-model retraining phase, each ensemble member is evaluated separately on newly added preference data, and the worst-performing $$\hat{p}(\tau_1 > \tau_2)=\frac{\exp(\hat{r}_1)}{\exp(\hat{r}_1)+\exp(\hat{r}_2)},$$1 members are re-initialized. The paper states that $$\hat{p}(\tau_1 > \tau_2)=\frac{\exp(\hat{r}_1)}{\exp(\hat{r}_1)+\exp(\hat{r}_2)},$$2 and that the default is $$\hat{p}(\tau_1 > \tau_2)=\frac{\exp(\hat{r}_1)}{\exp(\hat{r}_1)+\exp(\hat{r}_2)},$$3. The stated purpose is to maintain ensemble diversity and avoid all members converging to the same poor local solution (Merk et al., 26 Aug 2025).

Component	Mechanism	Stated purpose
Probabilistic reward model	Mean and standard deviation per state-action pair	Improve preference modeling and learning stability
Noisy reward aggregation	Ensemble mean plus magnitude of Gaussian noise	Increase reward in regions of high uncertainty
Ensemble resetting	Re-initialize worst-performing members before retraining	Maintain diversity and avoid collapse

4. Online preference-learning pipeline and human annotation protocol

REC is integrated into a standard online preference-based RL loop. First, trajectories are sampled from the current policy and trajectory pairs are shown to a human or a synthetic judge. Second, the annotator chooses which trajectory is better; in the human experiment, the annotator can also answer “I can’t tell”, in which case the pair is discarded and a new comparison is requested later. Third, the reward ensemble is trained on preference pairs using the probabilistic cross-entropy loss together with the standard deviation target loss, and ensemble resetting may occur before retraining. Fourth, the policy $$\hat{p}(\tau_1 > \tau_2)=\frac{\exp(\hat{r}_1)}{\exp(\hat{r}_1)+\exp(\hat{r}_2)},$$4 is optimized with RL using the learned reward model, specifically on top of PPO. Fifth, the cycle repeats. The scheduling is alternating and synchronized: reward retraining happens every time new preferences are gathered, and an epoch is the interval between two reward-retraining phases (Merk et al., 26 Aug 2025).

Preference pair selection uses three heuristics, each contributing one-third of the training pairs: Random, Ensemble disagreement, and Current with previous epoch. Ensemble disagreement uses all available trajectories, labels them under all ensemble reward predictions, and picks pairs with the most contradictory ensemble votes. The “current with previous epoch” heuristic pairs recent trajectories against ones from before the last reward update. Each trajectory is used at most once. This protocol matters because uncertainty estimation is used not only in reward learning but also in determining which comparisons are most informative (Merk et al., 26 Aug 2025).

For real-world human feedback, the setup is the same as synthetic preference training except that the labels come from a human. The paper reports 1000 preference labels, with 1064 comparisons shown because some were discarded, and a total training plus labeling time of 4 h 43 min. The policy checkpoint for deployment is chosen by the annotator rather than by reward value. These details indicate that REC was evaluated not only as a simulation method but also as an operational workflow for real-world preference collection (Merk et al., 26 Aug 2025).

5. Empirical behavior, ablations, and reported metrics

The main reported drone result is on the continuous powerloop task with 1000 synthetic preferences. In normalized terms, REC Preference PPO reaches 88.4% of shaped-reward performance, whereas standard Preference PPO reaches 55.2%. In absolute terms, the reported maximum evaluation rewards are 432.4 ± 190.8 for PPO with shaped reward, 238.9 ± 157.5 for Preference PPO, 281.0 ± 187.4 for Preference PPO + Prob. Rew. Loss, 153.9 ± 174.2 for Preference PPO + Prob. Rew. Loss + Reward Noise, and 382.4 ± 80.8 for REC, defined as Prob. Rew. Loss + Reward Noise + Reset Ensemble (Merk et al., 26 Aug 2025).

In the representative MuJoCo or DM Control walker-walk environment, the best average evaluation reward across three seeds is reported as 930.7 ± 12.0 for PPO, 719.0 ± 9.3 for Preference PPO, 777.5 ± 66.2 for + Prob. Rew. Loss, 829.4 ± 55.5 for + Prob. Rew. Loss + Rew. Noise, and 815.1 ± 43.3 for REC. The paper states that REC improves over standard Preference PPO in this continuous-control setting as well, while also noting that the exact best number is not always the maximum among the ablations because the emphasis is on stability and overall behavior (Merk et al., 26 Aug 2025).

Method	Drone continuous powerloop	walker-walk
PPO	432.4 ± 190.8	930.7 ± 12.0
Preference PPO	238.9 ± 157.5	719.0 ± 9.3
+ Prob. Rew. Loss	281.0 ± 187.4	777.5 ± 66.2
+ Prob. Rew. Loss + Reward Noise	153.9 ± 174.2	829.4 ± 55.5
REC	382.4 ± 80.8	815.1 ± 43.3

The reported evaluation metrics are maximum evaluation reward during training, mean episode reward / training reward curves, standard deviation across 3 seeds, and, in the drone setting, performance normalized as a percentage of the shaped-reward PPO baseline. The ablation sequence is explicit: Preference PPO, then + Probabilistic Reward Loss, then + Reward Noise / Gaussian exploration, then + Reset Ensemble as the final REC variant. This suggests that REC is not a single modification but a composite of probabilistic preference modeling, uncertainty-driven reward aggregation, and ensemble maintenance via resets. The same paper also highlights the limitations of manually designed rewards, reporting only 60.7% agreement with human preferences (Merk et al., 26 Aug 2025).

6. Interpretation, related confidence-aware reward modeling, and limitations

Within its own formulation, REC is an uncertainty-aware reward-learning system in two senses. Reward predictions are represented as distributions, preference probability depends on uncertainty, a standard deviation target loss is used to stabilize uncertainty magnitude, ensemble disagreement informs query selection, and reward noise uses ensemble variance. The paper explicitly notes that REC is not a full Bayesian method, but that it uses ensemble variance as a practical proxy for epistemic uncertainty. A plausible implication is that REC should be understood less as a single estimator than as an integrated design for uncertainty-sensitive preference learning (Merk et al., 26 Aug 2025).

A related theoretical treatment of reward uncertainty in RLHF argues that the right object is a distributional reward model $$\hat{p}(\tau_1 > \tau_2)=\frac{\exp(\hat{r}_1)}{\exp(\hat{r}_1)+\exp(\hat{r}_2)},$$5, and that confidence-aware ensemble rules can be understood through a single effective-reward expression. That paper states that, if REC is taken to mean a reward ensemble under confidence scheme, it is best understood as part of the same family of distributional-reward methods rather than as a distinct third principle (Hahami et al., 8 Jun 2026). A separate line of work on LLM reasoning introduces Correctness and Confidence Reward Modeling (C2RM) and is described as directly relevant to the idea behind REC because it penalizes low-confidence correct responses instead of rewarding correctness alone (He et al., 9 Nov 2025). These comparisons do not collapse the methods into one algorithm, but they place REC within a broader movement toward confidence-aware reward modeling.

The limitations reported for REC and preference-based RL in this setting are concrete. Human evaluation is time-consuming because each preference requires comparing two trajectory visualizations. Preferences are viewpoint-dependent, so camera choice can bias judgments, and rendering already imposes a design choice that may bias the notion of the maneuver. Training has high variance because acrobatic tasks are difficult, agents often failed to explore beyond an initial half-flip maneuver, and this limited statistical significance across all conditions. The authors suggest better exploration or curriculum learning as future work. Taken together, these caveats indicate that REC addresses uncertainty in reward learning, but does not remove the data-collection and exploration bottlenecks inherent in real-world preference-based control (Merk et al., 26 Aug 2025).