Batch-Constrained Action Set for Offline RL

• Batch-Constrained Action Set is a defined subset of actions that are well-supported by historical data, ensuring reliability in offline RL.
• The approach employs behavioral cloning with probability thresholds to constrain the available actions during policy updates.
• Empirical results indicate that this method improves performance and stability in diverse domains such as robotics, healthcare, and recommendations.

A batch-constrained action set refers to the set of available actions that a learning agent—typically in the context of offline or batch reinforcement learning (RL)—can consider at any decision point, where this set is explicitly restricted to actions that are well-supported (i.e., confidently observed) in a fixed batch of logged data. This restriction is critical for stabilizing policy learning and mitigating extrapolation error when it is not possible to collect new interaction data, such as in pure offline learning scenarios or regulatory-constrained domains.

1. Formal Definitions and Construction

Let $\mathcal{B}$ denote a fixed batch of historical transition tuples $(s, a, r, s')$ generated by an unknown or suboptimal behavior policy $\pi_b$. The batch-constrained action set is constructed by fitting a state-conditioned behavioral cloning model to approximate the behavior policy using maximum-likelihood estimation or cross-entropy minimization: $p_{\mathcal{M}}(a|s; \omega) \approx \pi_b(a|s)$ with loss

$$\mathcal{L}_\omega = -\mathbb{E}_{(s,a)\sim\mathcal{B}} [ \log p_{\mathcal{M}}(a|s;\omega) ]$$

For each state $s$, the batch-constrained action set is defined as: $$A_{bc}(s) = \{ a \in \mathcal{A} : p_{\mathcal{M}}(a|s) > \beta \}$$ where $\beta$ is a user-specified probability threshold. Only actions with sufficiently high support in $\mathcal{B}$ (i.e., likely under the learned behavioral model) are permitted for policy improvement and evaluation (Garg et al., 2020).

In alternative implementations, such as in the discrete-action version of Batch-Constrained Q-learning (BCQ), the action set is defined relative to the behavioral model’s mode: $$A_\tau(s) = \{ a \in \mathcal{A} : G_\theta(a|s) / \max_{a'} G_\theta(a'|s) > \tau \}$$ with $(s, a, r, s')$0 controlling the strictness of the constraint (Fujimoto et al., 2019).

2. Algorithmic Integration in Offline RL

Batch-constrained action sets are integrated into RL algorithms by replacing standard unconstrained policy improvement steps with constrained maximizations. For instance, BCD4Rec (Batch-Constrained Distributional RL for Session-based Recommendation) performs Double-DQN-style updates, but the Q-value maximization in the Bellman update is restricted to $(s, a, r, s')$1: $(s, a, r, s')$2 with the distributional Bellman loss evaluated as

$(s, a, r, s')$3

where $(s, a, r, s')$4 is chosen according to the batch-constrained action set (Garg et al., 2020).

In BCQ–Discrete for Atari, Q-learning updates use: $(s, a, r, s')$5 thus ensuring that only state–action pairs supported by $(s, a, r, s')$6 are considered during temporal-difference backups (Fujimoto et al., 2019).

3. Extrapolation Error and Theoretical Motivation

A central motivation for batch-constrained action sets is the mitigation of extrapolation (out-of-distribution) error in offline RL. In standard Q-learning, Bellman updates can involve bootstrapping through state–action pairs $(s, a, r, s')$7 never observed in the batch, for which $(s, a, r, s')$8 is unconstrained and may become arbitrarily large. This leads to error accumulation and unstable policies.

Restricting backups to $(s, a, r, s')$9 guarantees that every $\pi_b$0 in the update is observed with significant frequency (or probability) in $\pi_b$1. Empirically, this approach prevents value blowup and results in:

• High fidelity to observed data distributions
• Reduction in popularity bias and spurious generalization
• Substantially higher click-through and buy rates in SR, or improved performance on standard batch RL benchmarks (Atari) (Garg et al., 2020, Fujimoto et al., 2019)

BCQ–Discrete achieves game scores 2–3× higher than unconstrained DQN or QR-DQN and maintains value estimates within realistic ranges, matching or exceeding behavioral policy scores within a few million gradient steps (Fujimoto et al., 2019).

4. Empirical and Algorithmic Results

Empirical validation demonstrates that batch-constrained action sets dramatically improve learning outcomes in purely offline regimes:

• BCD4Rec achieves >95% accuracy in recommending items from correct latent categories and reduces the influence of popularity bias in session-based recommendation (Garg et al., 2020).
• BCQ–Discrete on Atari delivers stable learning curves and consistently outperforms all non-constrained batch RL baselines.
• The policy improvement constraint is essential for reproducing these effects; unconstrained or weakly-constrained methods suffer from overestimation and degraded final performance (Fujimoto et al., 2019).

Pseudocode for the core update within BCD4Rec and BCQ–Discrete follows a consistent pattern: sample a minibatch from $\pi_b$2, construct the constrained set $\pi_b$3, select actions within this set, and update all parameters via gradient descent with cross-entropy or quantile-regression objectives.

5. Generalizations and Other Problem Settings

The batch-constrained action set principle generalizes across RL and bandit domains. In high-dimensional sparse linear contextual bandits with batch constraints, the family of feasible action-selection mappings $\pi_b$4 is restricted to policies that may only adapt at predefined batch boundaries. The regret lower bound under a $\pi_b$5-batch constraint quantifies how the frequency of allowable action-selection updates $\pi_b$6 impacts achievable performance: $\pi_b$7 A greedy LASSO-based algorithm achieves this minimax rate with only $\pi_b$8 batch updates, demonstrating that periodic retraining with batch-constrained action sets suffices for minimax regret (Ren et al., 2020).

The batch-constrained approach is applied in:

• Offline robotics, constraining policy updates to trajectories supported by log data
• Healthcare, restricting candidate treatments to those seen in observational records
• Broad recommendation, dialog, personalized education, and related settings

For continuous action spaces or when the density of state–action coverage varies, the constraint may be formulated via divergences, e.g.,

$\pi_b$9

or by rejection sampling to ensure sufficient overlap with batch support (Garg et al., 2020).

6. Summary Table of Batch-Constrained Action Set Approaches

Domain/Algorithm	Action Set Construction	Constraint Criterion
BCD4Rec (SR, RL)	$p_{\mathcal{M}}(a|s; \omega) \approx \pi_b(a|s)$0	Behavioral clone threshold
BCQ-Discrete (Atari)	$p_{\mathcal{M}}(a|s; \omega) \approx \pi_b(a|s)$1	Relative mode threshold
Batch Bandits (LBGL)	$p_{\mathcal{M}}(a|s; \omega) \approx \pi_b(a|s)$2: mapping fixed at batch, updated at boundaries	Batched adaptation limits

This table illustrates that, across various algorithmic instantiations, the batch-constrained action set is always defined in relation to support in a historical batch, with constraints enforced at action selection or policy update time.

7. Implications and Limitations

While no general formal convergence guarantees are provided in (Garg et al., 2020), analogous results in BCQ indicate that Bellman updates restricted to the batch support collapse the algorithm to a fixed point bounded by the representational power of the batch, providing a theoretical limit on extrapolation error. Batch-constrained action sets thus form a key methodological advance for high-stakes, offline, or resource-constrained settings where out-of-distribution action selection poses fundamental risks.

A plausible implication is that, as batch RL continues to expand to new domains, the explicit design and verification of the batch-constrained action set will become a core consideration for safe, robust offline policy learning.