Batch-Constrained Bellman Backup

• Batch-Constrained Bellman Backup is a batch reinforcement learning paradigm that revises standard Bellman backups using a density-based support filter.
• It prevents extrapolation errors by zeroing out value estimates for state-action pairs that are insufficiently represented in the dataset.
• Empirical validations on CartPole and MuJoCo environments show its robustness against overestimation compared to classical batch RL methods.

Batch-Constrained Bellman Backup (BCBB) refers to a paradigm for conservative policy evaluation and improvement in batch reinforcement learning (RL), where access to data is restricted to a fixed collection of transition tuples sampled from a behavior distribution. The BCBB approach modifies standard Bellman backup operators to prevent the propagation of value estimates through state-action pairs insufficiently covered by the observed dataset. By enforcing strict support constraints derived from the empirical batch coverage, BCBB fundamentally addresses the distributional shift and overestimation problems endemic to classical batch RL methods without relying on strong concentrability assumptions (Liu et al., 2020).

1. Markov Decision Process and Batch Data Formalism

The BCBB framework is grounded in the standard discounted Markov Decision Process (MDP), defined as $M = (S, A, P, r, \gamma, \rho)$, with state space $S$, action space $A$, transition kernel $P(s' \mid s, a)$, bounded reward function $r(s, a) \in [0, R_{\max}]$, discount factor $\gamma \in [0,1)$, and initial distribution $\rho$. Policies $\pi: S \to \Delta(A)$ induce state-action value functions $Q^\pi(s, a)$ and mean values $v^\pi = \mathbb{E}_{s \sim \rho} [ V^\pi(s) ]$.

The batch setting assumes access to a finite dataset $S$0 sampled i.i.d. from some environment-aligned behavior distribution $S$1. There is no guarantee that $S$2 has significant or nonzero support over the entire $S$3; crucially, no further data collection (exploration) is permitted.

2. Classical and Conservative Bellman Operators

Traditional batch algorithms (e.g., Fitted Q-Iteration) rely on Bellman operators $S$4 (evaluation) and $S$5 (optimality), respectively,

• $S$6,
• $S$7.

This methodology allows value function propagation through any state-action pairs, regardless of their presence in $S$8. Such propagation is problematic in the batch setting, enabling erroneous bootstrapping via rarely or never observed transitions.

BCBB introduces a support-filter $S$9, where $A$0 is a density estimate of the empirical state-action visitation and $A$1 is a conservativeness threshold. The constrained operators are defined as:

• $A$2,
• $A$3.

Next-state actions with $A$4 accrue zero future value; this prohibits optimistic bootstrapping through unsupported regions.

3. Marginalized Behavior-Supported Algorithms

The BCBB methodology is embodied in two prototypical algorithms: Marginalized Behavior-Supported Policy Iteration (MBS-PI) and Marginalized Behavior-Supported Q-Iteration (MBS-QI).

Algorithm	Policy Representation	Backup Operator	Support Filtering
MBS-PI	Policy class $A$5	$A$6	$A$7 for all backups
MBS-QI	Deterministic greedy	$A$8	$A$9 for all Q-updates

In MBS-PI, policy evaluation is performed with the $P(s' \mid s, a)$0-filtered operator, followed by a policy improvement step using a filtered value function. MBS-QI applies one-step greedy fit to the $P(s' \mid s, a)$1-constrained optimality operator. Both routines require only the hyperparameter $P(s' \mid s, a)$2 to adjust the degree of conservativeness; no trust-region penalties or auxiliary constraints are introduced.

4. Theoretical Guarantees and Analysis

Let the escaping probability of a policy $P(s' \mid s, a)$3 be $P(s' \mid s, a)$4, with $P(s' \mid s, a)$5 the $P(s' \mid s, a)$6-discounted state-action occupancy. Let $P(s' \mid s, a)$7, where $P(s' \mid s, a)$8 bounds admissible marginal densities $P(s' \mid s, a)$9.

• For MBS-PI, under bounded densities, accurate density estimation, and completeness of the function class, for any policy $r(s, a) \in [0, R_{\max}]$0 with $r(s, a) \in [0, R_{\max}]$1, the result is:

$r(s, a) \in [0, R_{\max}]$2

where $r(s, a) \in [0, R_{\max}]$3 is the total variation distance between $r(s, a) \in [0, R_{\max}]$4 and $r(s, a) \in [0, R_{\max}]$5, $r(s, a) \in [0, R_{\max}]$6 is the function class, $r(s, a) \in [0, R_{\max}]$7 is the policy evaluation accuracy, and $r(s, a) \in [0, R_{\max}]$8 is the policy iteration count.

• For MBS-QI, a similar bound holds:

$r(s, a) \in [0, R_{\max}]$9

The proof leverages a construction of an auxiliary MDP $\gamma \in [0,1)$0 incorporating a zero-reward absorbing state for any unsupported $\gamma \in [0,1)$1. The fixed-point of $\gamma \in [0,1)$2 coincides with $\gamma \in [0,1)$3 in $\gamma \in [0,1)$4, where $\gamma \in [0,1)$5 is forced to redirect through the filter. This allows the error bound to depend on $\gamma \in [0,1)$6 only over the well-supported region and not a global concentrability constant.

5. Illustrative MDP Pathologies and Failure Modes

Scenarios identified by Liu et al. (2020) illustrate where classical and alternative batch RL approaches (BCQL, BEAR, SPIBB) fail and BCBB is robust:

• Rare-Transition MDP: A one-step deviation with a small probability $\gamma \in [0,1)$7 leads to a high reward but $\gamma \in [0,1)$8 is so small the dataset likely omits relevant transitions. Standard value backups, as well as BCQL and SPIBB, are misled by optimistic extrapolation or poor action-conditional density estimation. BCBB's $\gamma \in [0,1)$9 excludes these transitions entirely, precluding overvaluation.
• Combination-Lock MDP: An episodic chain where reward is only accrued at the terminal state via a unique action sequence. When the behavior is noisy, classical algorithms tend to overfit to rare transition estimates. BCBB, via its coverage constraint, guarantees improvement only in adequately supported regions, yielding optimal performance within the covered subchain.

6. Empirical Benchmarks

Empirical validation on discretized CartPole-v0 and continuous-control MuJoCo Hopper-v3 environments demonstrates the reliability and competitiveness of BCBB algorithms:

• CartPole-v0: For batch datasets collected with $\rho$0-greedy expert behavior ($\rho$1), MBS-QI achieves near-optimal control across all $\rho$2, outperforming BCQL and SPIBB in regimes with narrow or noisy coverage, and never suffering from optimistic bias due to undersampled transitions.
• MuJoCo Hopper-v3: BCQ is augmented by a $\rho$3 state filter (via ELBO-VAE), and compared with standard BCQ, offline DDPG, and Behavior Cloning. MBS-QL matches or surpasses BCQ and remains robust against extrapolation error even in late training stages, which is a typical failure mode of other batch RL methods.

Optimal $\rho$4 should retain a substantial proportion of the data support (e.g., $\rho$5), with the analysis automatically balancing conservatism and optimism accordingly.

7. Practical Significance and Limitations

BCBB presents a minimally intrusive and theoretically justified solution to extrapolation error in batch RL algorithms, requiring only a data-driven support indicator and a single threshold parameter. The method achieves competitive empirical results without reliance on explicit trust-region enforcement, optimism penalties, or stringent density ratio control assumptions.

Limitations include dependence on the quality of density estimation $\rho$6 and potential underestimation of value in sparsely explored but viable regions. A plausible implication is that BCBB is best suited for datasets with adequate coverage of the critical decision space; otherwise, value improvement is necessarily restricted to the empirical support.

Further research may analyze trade-offs between conservativeness and sample efficiency as a function of batch coverage and threshold selection (Liu et al., 2020).