Extrapolation Error in Off-Policy RL

• Extrapolation error is the systematic bias that arises when off-policy RL evaluates Q-values for underrepresented state–action pairs, leading to unreliable policy estimates.
• Traditional algorithms like DQN, DDPG, and TD3 suffer from policy collapse by overestimating Q-values on actions outside the batch data support.
• Techniques such as Batch-Constrained Q-Learning (BCQ) mitigate this error by restricting action sampling to the empirical data, ensuring stability in offline learning.

Extrapolation error in off-policy reinforcement learning denotes the systematic bias or instability that arises when a value function (typically a Q-network) is evaluated on state–action pairs that occur with low or zero probability under the data distribution present in the agent's replay buffer or batch. This error is a central impediment in offline RL, where agents must learn exclusively from a fixed, historical dataset, with no access to further environment interactions to correct invalid or unsupported estimates. The phenomenon leads to unreliable Q-values, policy collapse, and often catastrophic overestimation, undermining learning and deployment robustness in practical batch RL scenarios (Fujimoto et al., 2018, Fujimoto et al., 2019, Kim et al., 24 Sep 2025, Xi et al., 2021).

1. Formal Definition and Mechanisms of Extrapolation Error

Extrapolation error arises due to function approximation over portions of state–action space not sufficiently covered by the collected data. For any approximate MDP $M_2$ induced by empirical transitions, and true MDP $M_1$, the difference in Q-values under a policy $\pi$ is bounded by:

$$\epsilon_{s,a} = Q^\pi_{M_1}(s,a) - Q^\pi_{M_2}(s,a) \leq \sum_{n=0}^{\infty} \gamma^n \sum_{s^{(n)}, a^{(n)}} p_1(s^{(n)}|s,a; \pi) \frac{R_{\max}}{1 - \gamma} e(s^{(n)}, a^{(n)})$$

where $e(s,a)$ is the one-step $L_1$ error between transition distributions estimated from batch data and the true environment, and $N(s,a)$ is the state–action visitation count (Xi et al., 2021). When $(s,a)$ pairs are underrepresented in the batch, $e(s,a)$ becomes large, driving correspondingly high extrapolation error.

2. Manifestation in Off-Policy Algorithms

Classic off-policy deep RL algorithms (DQN, DDPG, TD3) inherently rely on maximization or bootstrapping over $Q$-values for unseen actions, causing divergence when trained with fixed batch data. This is empirically demonstrated: DDPG and DQN collapse in purely offline regimes, their value estimates spiraling and resultant policies failing to outperform mere behavioral cloning (Fujimoto et al., 2018, Fujimoto et al., 2019). The core problem is maximization over $Q(s', a')$ for arbitrary $a'$, including actions outside the support of the dataset, where function approximators may hallucinate high values.

3. Theoretical Upper Bounds and Dataset Dependence

Explicit bounds for extrapolation error have been derived (Xi et al., 2021). For sufficiently large batch size ($N$), the bound is:

$$\overline{\epsilon}_{s,a} = C \frac{R_{\max}}{1-\gamma} N^{-1/2} \Big[ (\pi_b(a|s))^{-1/2} + \gamma \mathbb{E}_{s',a'} (\pi_b(a'|s'))^{-1/2} + ... \Big]$$

where $C$ depends on domain sizes and statistical concentration parameters. Control over $\pi_b(a|s)$—the batch policy’s empirical likelihood—directly dictates the degree of error. Strictly, the error can only be bounded for transitions where $N(s,a)$ is non-negligible; penalizing or excluding unsupported actions is therefore essential.

4. Algorithmic Mitigation: Batch-Constrained Q-Learning and Beyond

Batch-Constrained Q-Learning (BCQ) offers a principled approach to suppressing extrapolation error (Fujimoto et al., 2018, Fujimoto et al., 2019, Xi et al., 2021, Kim et al., 2023, Kim et al., 24 Sep 2025). Key mechanisms:

• Generative Modeling: Fit a conditional VAE $G_\theta(s)$ on batch data; only sample candidate actions from $G_\theta(s)$, approximating the empirical support $\pi_b(a|s)$.
• Perturbation Network: Train a constrained action-correction network $\xi_\omega(s, a)$, limited in magnitude, for value-driven fine-tuning without leaving the data-manifold.
• Action Filtering/Thresholding: In discrete domains, hard thresholding selects only actions with $\frac{G(a|s)}{\max_{b} G(b|s)} > \tau$, effectively enforcing a minimum frequency constraint (Fujimoto et al., 2019, Periyasamy et al., 2023).
• Critic Update Constraint: Bellman backups, target computation, and policy selection only occur over batch-supported (or plausibly in-batch) actions.

Under BCQ, the extrapolation error bound improves to:

$$\overline{\epsilon}^{\rm BCQ}_{s,a} = C \frac{R_{\max}}{(1-\gamma)^2} (N\tau)^{-1/2}$$

with $\tau$ as the batch-constraint threshold; for $\tau > 1/|\mathcal{A}|$, BCQ strictly reduces error compared to unconstrained off-policy methods (Xi et al., 2021).

Advancements such as Frictional Q-Learning (FQL) introduce a dual constraint: actions are pulled toward the batch buffer distribution and simultaneously pushed away from an orthonormal or heterogeneous action manifold, directly inspired by physical notions of static friction; this contrastive approach exacerbates the extrapolation error for unsupported actions and improves policy robustness (Kim et al., 24 Sep 2025).

5. Empirical Observations and Quantitative Analysis

Multiple studies demonstrate the practical significance of mitigating extrapolation error:

• BCQ and its quantum variant BCQQ (utilizing VQCs) can attain superior sample efficiency and stable return profiles compared to classical batch RL, even from highly limited or noisy buffers (Periyasamy et al., 2023, Fujimoto et al., 2019). Quantum architectures exhibit pronounced robustness, achieving perfect reward from minimal random data due to effective restriction to batch-supported actions.
• In high-stakes applications (e.g., autonomous driving, joint beamforming for 5G), batch-constrained methods are favored for safety: out-of-distribution actions can trigger catastrophic failures or degrade network performance. The generative model/perturbation mechanism achieves a balance between imitation and targeted improvement while preserving system stability (Shi et al., 2021, Kim et al., 2023).
• Empirical evaluation shows discrete BCQ outperforms both unconstrained offline RL and pure behavioral policy baselines, particularly where the offline batch is diverse but not uniformly exploratory (Fujimoto et al., 2019). Mean performance remains robust, and wild value divergence typical of standard methods is absent.
• On batches with low mean episode return (poor-quality or random data), BCQ imitates suboptimal behavior, failing to sufficiently improve expected Q-values. Top-Return BCQ (TR-BCQ) remedies this by restricting learning to high-return episodes at the cost of a minor increase in extrapolation error, yielding improved policy value in practice (Xi et al., 2021).

6. Practical Implementation and Deployment Considerations

Deployment in environments with unknown or complex dynamics, as in commercial communication networks, necessitates conservative policy constraints (Kim et al., 2023). Key engineering takeaways include:

• Sample Only What is Supported: Action proposals and Bellman backups must be computed entirely over the empirical support of the dataset to obviate unreliable extrapolation.
• Generative Model Tuning: VAE/cVAE architectures must be tuned (latent dimension, KL weight, total correlation penalties) to accurately model the support without leaking to unsupported regions (Kim et al., 24 Sep 2025).
• Trade-off Between Imitation and Improvement: Perturbation bounds (e.g., $\Phi$ or $\epsilon$) mediate between strict imitation (high safety, low exploration) and value-guided improvement (wider coverage, higher reward).
• Batch Quality Monitoring: Systematic exclusion or up-weighting of high-return episodes (TR-BCQ) can address poor empirical Q-values due to suboptimal data, albeit with a secondary increase in extrapolation error.

7. Extensions, Limitations, and Future Directions

Extrapolation error remains a primary research concern in offline RL. Extensions explored include:

• Enhanced Exploration via Parameter Noise: Learnable stochastic perturbation models (noisy weights) yield richer local action diversity without sacrificing batch support (Shi et al., 2021).
• Lyapunov Function-based Safety: Enforcing a decrease in a learned Lyapunov function constrains policy exploration to safe state regions, complementing batch constraints for critical domains (autonomous driving) (Shi et al., 2021).
• Quantum Function Approximators: Variational quantum circuits exhibit improved sample and parameter efficiency in batch RL with batch constraints (Periyasamy et al., 2023), suggesting novel computational approaches to error control.

Limitations persist: BCQ and similar methods cannot surpass the unseen optimal returns if the original batch never explores highly rewarding actions (Fujimoto et al., 2019, Xi et al., 2021). Robust extrapolation error bounds and safe deployment require sufficiently diverse and representative datasets, and batch constraints must be tuned to each application to balance conservatism with policy improvement.

Continued research into contrastive generative modeling, robust estimation of batch support, and hybrid constraints (physical, statistical, or quantum-inspired) is ongoing, with the objective of further suppressing extrapolation error and maximizing the safety and efficacy of offline reinforcement learning.