Refute-or-Promote Pipeline in RL

• Refute-or-Promote Pipeline is a systematic method that uses ensemble-based dynamics and statistical bottom-k averaging to compute conservative Q-value estimates for robust policy updates.
• It integrates a multi-head Q-network with ensemble models to mitigate overestimations and control model bias, enhancing both sample efficiency and performance.
• Empirical results on MuJoCo benchmarks demonstrate that this method outperforms alternatives like MBPO, MOPO, and SAC in terms of efficiency and stability under noisy conditions.

A Refute-or-Promote Pipeline in the context of reinforcement learning refers to a systematic method that uses ensembles of models and value functions to either "refute" unreliable, overly optimistic value estimations or "promote" robust, conservative policy updates. The "conservative model-based actor-critic" (CMBAC) implementation exemplifies this class by combining a probabilistic dynamics-model ensemble, per-model value functions, and a statistical operation—bottom-$k$ averaging—to produce policy updates that are robust against model bias and uncertainty. The pipeline’s primary function is to minimize the risk of policy exploitation of inaccurate model estimations while maintaining sample efficiency and high empirical performance (Wang et al., 2021).

1. Ensemble-Based Dynamics Model Construction

CMBAC initiates the pipeline by fitting a bootstrap ensemble of $N$ independent probabilistic dynamics networks for state transition prediction: $$\{\,p_{\psi_i}(s'\mid s,a)\}_{i=1}^N, \quad p_{\psi_i}(s'\mid s,a) = \mathcal{N}(\mu_{\psi_i}(s,a), \sigma_{\psi_i}(s,a))$$ This modeling step captures epistemic uncertainty and prevents overfitting to limited interaction data. To systematically probe the granularity of model discrepancy, all possible size-$M$ subsets of the ensemble are enumerated: $$\mathcal{M} = \left\{\mathcal{M}_j \subset \{1,\ldots,N\} \,\big|\, |\mathcal{M}_j| = M\right\},\quad K = \binom{N}{M}$$ Each subset $\mathcal{M}_j$ defines a "mixed" model through averaging: $$\hat{f}_j(s'|s,a) = \frac{1}{M} \sum_{i \in \mathcal{M}_j} p_{\psi_i}(s'|s,a)$$ Every mixed model induces a corresponding Q-function under policy $\pi$: $$Q_j^\pi(s,a) = Q^{\pi, \hat{f}_j}(s,a) = \mathbb{E}_{\pi, \hat{f}_j}\left[\sum_{t=0}^{\infty} \gamma^t r(S_t, A_t) \mid S_0 = s, A_0 = a\right]$$ In practice, these $K$ Q-functions are simultaneously approximated using a shared multi-head neural network $N$0, which supports efficient parallelized training and inference.

2. Conservative Cross-Model Critic: Bottom-$N$1 Averaging

The core of the pipeline lies in the computation of a conservative value estimate for a given state-action pair. For each $N$2, the $N$3 Q-value estimates are sorted ascendingly

$N$4

The system drops the top $N$5 (optimistic) heads and averages the remaining $N$6 values to yield the conservative Q-value: $N$7 This approach targets the exclusion of potentially "lucky" Q-heads that overestimate due to local model bias, mitigating the risks of catastrophic model exploitation.

Parameter $N$8 controls conservatism: increasing $N$9 (hence decreasing $$\{\,p_{\psi_i}(s'\mid s,a)\}_{i=1}^N, \quad p_{\psi_i}(s'\mid s,a) = \mathcal{N}(\mu_{\psi_i}(s,a), \sigma_{\psi_i}(s,a))$$0) yields more pessimistic estimates, while $$\{\,p_{\psi_i}(s'\mid s,a)\}_{i=1}^N, \quad p_{\psi_i}(s'\mid s,a) = \mathcal{N}(\mu_{\psi_i}(s,a), \sigma_{\psi_i}(s,a))$$1 recovers the mean. Experimental settings typically grid search over values such as $$\{\,p_{\psi_i}(s'\mid s,a)\}_{i=1}^N, \quad p_{\psi_i}(s'\mid s,a) = \mathcal{N}(\mu_{\psi_i}(s,a), \sigma_{\psi_i}(s,a))$$2 and $$\{\,p_{\psi_i}(s'\mid s,a)\}_{i=1}^N, \quad p_{\psi_i}(s'\mid s,a) = \mathcal{N}(\mu_{\psi_i}(s,a), \sigma_{\psi_i}(s,a))$$3 to balance exploration and risk.

3. Actor–Critic Update Mechanism

The refute-or-promote pipeline interacts with policy optimization via modified actor-critic updates:

• Q-network (critic) update: For each Q-head $$\{\,p_{\psi_i}(s'\mid s,a)\}_{i=1}^N, \quad p_{\psi_i}(s'\mid s,a) = \mathcal{N}(\mu_{\psi_i}(s,a), \sigma_{\psi_i}(s,a))$$4, transitions $$\{\,p_{\psi_i}(s'\mid s,a)\}_{i=1}^N, \quad p_{\psi_i}(s'\mid s,a) = \mathcal{N}(\mu_{\psi_i}(s,a), \sigma_{\psi_i}(s,a))$$5 from the model-generated buffer are used to compute targets:

$$\{\,p_{\psi_i}(s'\mid s,a)\}_{i=1}^N, \quad p_{\psi_i}(s'\mid s,a) = \mathcal{N}(\mu_{\psi_i}(s,a), \sigma_{\psi_i}(s,a))$$6

The loss minimized is

$$\{\,p_{\psi_i}(s'\mid s,a)\}_{i=1}^N, \quad p_{\psi_i}(s'\mid s,a) = \mathcal{N}(\mu_{\psi_i}(s,a), \sigma_{\psi_i}(s,a))$$7

• Policy (actor) update: Instead of the conventional soft actor-critic (SAC) Q-target, the conservative critic $$\{\,p_{\psi_i}(s'\mid s,a)\}_{i=1}^N, \quad p_{\psi_i}(s'\mid s,a) = \mathcal{N}(\mu_{\psi_i}(s,a), \sigma_{\psi_i}(s,a))$$8 is substituted:

$$\{\,p_{\psi_i}(s'\mid s,a)\}_{i=1}^N, \quad p_{\psi_i}(s'\mid s,a) = \mathcal{N}(\mu_{\psi_i}(s,a), \sigma_{\psi_i}(s,a))$$9

with corresponding sample-gradient:

$M$0

Empirically, implementation consists of removing the $M$1 highest Q-heads per sample, averaging the rest, and injecting this value into the actor’s loss.

4. High-Level Pipeline Workflow

The complete CMBAC pipeline employs the following high-level stages:

1. Initialization: Create an ensemble of $M$2 models $M$3, enumerate all size-$M$4 subsets $M$5, initialize the multi-head Q-network and policy network.
2. Epoch loop:
   • Fit all $M$6 using real environment transitions.
   • Collect additional steps under the current policy, augmenting $M$7.
   • Generate synthetic rollouts from $M$8 via randomly selected $M$9's, building the model-derived replay buffer $$\mathcal{M} = \left\{\mathcal{M}_j \subset \{1,\ldots,N\} \,\big|\, |\mathcal{M}_j| = M\right\},\quad K = \binom{N}{M}$$0.
   • For $$\mathcal{M} = \left\{\mathcal{M}_j \subset \{1,\ldots,N\} \,\big|\, |\mathcal{M}_j| = M\right\},\quad K = \binom{N}{M}$$1 gradient steps:
     • Critic update: One-step TD update for each Q-head.
     • Actor update: Compute $$\mathcal{M} = \left\{\mathcal{M}_j \subset \{1,\ldots,N\} \,\big|\, |\mathcal{M}_j| = M\right\},\quad K = \binom{N}{M}$$2 by bottom-$$\mathcal{M} = \left\{\mathcal{M}_j \subset \{1,\ldots,N\} \,\big|\, |\mathcal{M}_j| = M\right\},\quad K = \binom{N}{M}$$3 averaging; update $$\mathcal{M} = \left\{\mathcal{M}_j \subset \{1,\ldots,N\} \,\big|\, |\mathcal{M}_j| = M\right\},\quad K = \binom{N}{M}$$4 by the SAC policy gradient using this conservative critic.
3. Repeat for the experimental duration (Wang et al., 2021).

Table: Major CMBAC Configuration Parameters and Their Effects

Parameter	Range/Value	Effect on Pipeline
$$\mathcal{M} = \left\{\mathcal{M}_j \subset \{1,\ldots,N\} \,\big|\, |\mathcal{M}_j| = M\right\},\quad K = \binom{N}{M}$$5	Ensemble size	Controls epistemic uncertainty
$$\mathcal{M} = \left\{\mathcal{M}_j \subset \{1,\ldots,N\} \,\big|\, |\mathcal{M}_j| = M\right\},\quad K = \binom{N}{M}$$6	$$\mathcal{M} = \left\{\mathcal{M}_j \subset \{1,\ldots,N\} \,\big|\, |\mathcal{M}_j| = M\right\},\quad K = \binom{N}{M}$$7 to $$\mathcal{M} = \left\{\mathcal{M}_j \subset \{1,\ldots,N\} \,\big|\, |\mathcal{M}_j| = M\right\},\quad K = \binom{N}{M}$$8	Subset size; controls $$\mathcal{M} = \left\{\mathcal{M}_j \subset \{1,\ldots,N\} \,\big|\, |\mathcal{M}_j| = M\right\},\quad K = \binom{N}{M}$$9
$\mathcal{M}_j$0	$\mathcal{M}_j$1 (typical)	Directs degree of conservatism

5. Theoretical and Practical Implications

CMBAC’s refute-or-promote strategy allows approximation of a posterior distribution over $\mathcal{M}_j$2-values, capturing the full global uncertainty attributed to model bias (not limited to one-step uncertainty). By discarding the top-$\mathcal{M}_j$3 Q-heads per sample, the pipeline prevents the actor from exploiting unrealistically high $\mathcal{M}_j$4-values—a common failure mode in model-based RL—while tuning $\mathcal{M}_j$5 and $\mathcal{M}_j$6 provides a smooth trade-off between optimism and conservatism.

Distinct from robust-MDP methods employing hard $\mathcal{M}_j$7-over-model strategies (which can be overly pessimistic), bottom-$\mathcal{M}_j$8 averaging provides a softer lower-tail estimate and can be tuned for robustness as needed. Empirically, CMBAC yields more stable uncertainty quantification (see Fig. 6 in (Wang et al., 2021)) and improved resilience to injected noise compared to state-of-the-art alternatives.

6. Empirical Evaluation and Comparative Performance

CMBAC demonstrates substantial empirical gains in both sample efficiency and robustness. Across six standard MuJoCo environments (HalfCheetah, Hopper, Walker2d, Ant, Humanoid, InvertedPendulum), CMBAC outperforms:

• Model-based SAC (MBPO)
• Online MOPO (model-based penalty method)
• Off-policy model-free methods (SAC, REDQ)

On the challenging Humanoid benchmark, for example:

• MBPO achieves reward ≈$\mathcal{M}_j$9 in approximately $$\hat{f}_j(s'|s,a) = \frac{1}{M} \sum_{i \in \mathcal{M}_j} p_{\psi_i}(s'|s,a)$$0 environment steps,
• CMBAC reaches the same reward after only $$\hat{f}_j(s'|s,a) = \frac{1}{M} \sum_{i \in \mathcal{M}_j} p_{\psi_i}(s'|s,a)$$1 steps,
• SAC requires approximately $$\hat{f}_j(s'|s,a) = \frac{1}{M} \sum_{i \in \mathcal{M}_j} p_{\psi_i}(s'|s,a)$$2 steps to reach a similar performance level.

Robustness to action noise ($$\hat{f}_j(s'|s,a) = \frac{1}{M} \sum_{i \in \mathcal{M}_j} p_{\psi_i}(s'|s,a)$$3 Gaussian on Walker2d and HalfCheetah) also improves: CMBAC’s performance degrades by less than $$\hat{f}_j(s'|s,a) = \frac{1}{M} \sum_{i \in \mathcal{M}_j} p_{\psi_i}(s'|s,a)$$4, whereas MBPO and MOPO-Online show losses of $$\hat{f}_j(s'|s,a) = \frac{1}{M} \sum_{i \in \mathcal{M}_j} p_{\psi_i}(s'|s,a)$$5–$$\hat{f}_j(s'|s,a) = \frac{1}{M} \sum_{i \in \mathcal{M}_j} p_{\psi_i}(s'|s,a)$$6 (Wang et al., 2021).

7. Significance and Future Directions

The refute-or-promote pipeline, as instantiated by CMBAC, defines a general approach for value estimation under model uncertainty in reinforcement learning. This method’s flexible trade-off between optimism and conservatism allows for tailored risk management by tuning hyperparameters $$\hat{f}_j(s'|s,a) = \frac{1}{M} \sum_{i \in \mathcal{M}_j} p_{\psi_i}(s'|s,a)$$7 and $$\hat{f}_j(s'|s,a) = \frac{1}{M} \sum_{i \in \mathcal{M}_j} p_{\psi_i}(s'|s,a)$$8. CMBAC’s demonstrated improvements in sample efficiency and robustness to noise underscore the significance of ensemble-based, cross-model critics in advancing model-based RL. A plausible implication is that continued development along these lines—especially by refining the ensemble construction, bottom-$$\hat{f}_j(s'|s,a) = \frac{1}{M} \sum_{i \in \mathcal{M}_j} p_{\psi_i}(s'|s,a)$$9 stratification, and uncertainty estimation—could yield further robustness improvements in model-based policy learning under real-world confounders (Wang et al., 2021).