Pessimistic Auxiliary Policy for Offline Reinforcement Learning

Abstract: Offline reinforcement learning aims to learn an agent from pre-collected datasets, avoiding unsafe and inefficient real-time interaction. However, inevitable access to out-ofdistribution actions during the learning process introduces approximation errors, causing the error accumulation and considerable overestimation. In this paper, we construct a new pessimistic auxiliary policy for sampling reliable actions. Specifically, we develop a pessimistic auxiliary strategy by maximizing the lower confidence bound of the Q-function. The pessimistic auxiliary strategy exhibits a relatively high value and low uncertainty in the vicinity of the learned policy, avoiding the learned policy sampling high-value actions with potentially high errors during the learning process. Less approximation error introduced by sampled action from pessimistic auxiliary strategy leads to the alleviation of error accumulation. Extensive experiments on offline reinforcement learning benchmarks reveal that utilizing the pessimistic auxiliary strategy can effectively improve the efficacy of other offline RL approaches.

• The paper presents a method that mitigates overestimation by maximizing the lower confidence bound of the Q-function, reducing estimation errors by up to 95.2%.
• It leverages ensemble-based epistemic uncertainty to guide action selection, balancing error reduction and proximity to dataset actions.
• Empirical results show normalized score improvements of 2.5% to 159.5% across benchmarks like D4RL and NeoRL-2, demonstrating robust performance.

Pessimistic Auxiliary Policy for Offline Reinforcement Learning

Motivation and Problem Statement

Offline RL circumvents the safety and efficiency risks of online interaction by relying solely on pre-collected datasets, but this paradigm introduces significant distributional shift and exacerbates function approximation errors. The principal challenge arises when the learned policy queries out-of-distribution (OOD) actions during temporal-difference (TD) updates, leading to overestimation and eventual policy degeneration. The underlying mechanism is accumulation of approximation error due to lack of corrective environmental interaction, as detailed in the theoretical derivations and simulation studies. The rate of convergence of value estimates is strongly linked to data density (Figure 1), motivating uncertainty quantification as a fundamental axis for action selection in offline RL.

Figure 1: Simulation study demonstrates the role of data density in estimation error for linear regression, highlighting increased uncertainty in regions of sparse data.

Prior Approaches and Their Limitations

Conventional approaches to mitigate OOD extrapolation error in offline RL fall into value regularization and policy constraint paradigms. Value regularization directly penalizes the value function for OOD actions (e.g., CQL, BRAC-v), while policy constraint methods enforce proximity to the behavior policy (e.g., TD3BC, AWAC). However, explicit policy constraint can restrict policy optimality and is impractical in scenarios where the behavior policy is inaccessible. Value regularization can be effective but does not provide explicit mechanisms for action selection under epistemic uncertainty. Hybrid models and transformer-based architectures exist but substantially depart from canonical RL optimization.

Pessimistic Auxiliary Policy: Construction and Theory

The paper introduces a pessimistic auxiliary policy constructed by maximizing the lower confidence bound (LCB) of the Q-function, leveraging ensemble-based epistemic uncertainty. The auxiliary policy is formulated as:

$$\mu_p = \mu + \frac{\sqrt{2\sigma}}{\| [\nabla_a Q_{LB}(s, a)]_{a=\mu} \|} [\nabla_a Q_{LB}(s, a)]_{a=\mu}$$

where $$Q_{LB}(s, \mu) = \mu_Q(s, \mu) - \beta \delta_Q(s, \mu)$$ is the lower bound, $\mu_Q$ is the mean Q approximation, and $\delta_Q$ is the standard deviation computed from the ensemble of Q networks. The design constrains the Wasserstein distance between the auxiliary and learned policy to ensure stability:

$$W(\delta(\mu_p), \delta(\mu)) = \frac{1}{2} \| \mu_p - \mu \|^2 \leq \sigma$$

This optimizes action selection towards regions with high value but low epistemic uncertainty, directly suppressing error accumulation in TD updates without manual data sparsity adjustment or additional parameterization.

Figure 2: Illustration of the lower confidence bound $Q_{LB}$ and the construction of the pessimistic auxiliary policy, which steers action selection toward regions of low uncertainty.

Integration with Offline RL Algorithms and Convergence Guarantees

The pessimistic auxiliary policy can be seamlessly integrated with policy extraction and evaluation stages in standard offline RL methods. The Bellman operator is adapted as:

$$\mathcal{T}_p Q = r(s,a) + \gamma Q_{\theta^-}(s', \pi_p(s'))$$

Theoretical analysis establishes boundedness and $\gamma$-contraction properties for $\mathcal{T}_p$, guaranteeing convergence in Banach space. Thus, the auxiliary policy maintains the convergence behavior of established operators in offline RL and does not introduce instability.

Empirical Results and Performance Analysis

Extensive evaluation on D4RL and NeoRL-2 real-world benchmarks demonstrates that pessimistic auxiliary policy delivers substantial improvements in normalized scores across MuJoCo, Adroit, and AntMaze tasks. When used as a plug-in to TD3BC (TD3PA) and Diffusion-QL (DQLPA), improvements range from 2.5% to 159.5% depending on task and environment type, with consistent gains over baselines. Real-world scenarios further corroborate generalizability, with TD3PA outperforming baselines across 7 challenging tasks.

Detailed policy analysis reveals two fundamental outcomes:

• The Q-function estimation error is significantly reduced (up to 95.2% on HalfCheetah-medium-expert).
• Action selections by the auxiliary policy are more closely aligned with dataset actions, mitigating distributional shift.

Figure 3: Empirical distances between the action executed by policy and the action in the HalfCheetah dataset show improved alignment for pessimistic auxiliary policy variants.

Figure 4: Empirical distances between the action executed by policy and the action in the AntMaze dataset highlight the reduction of OOD actions with pessimistic auxiliary policy.

Implications and Future Directions

Practically, the pessimistic auxiliary policy offers a generic, easily integrated mechanism for mitigating error accumulation in offline RL. Theoretically, it reframes action selection in offline RL as an uncertainty-aware optimization problem, leveraging ensemble methods for epistemic uncertainty quantification. This approach can be further refined by adaptive tuning of the pessimism parameter $\beta$ and neighborhood constraint $\sigma$, possibly via meta-learning or Bayesian optimization. Additionally, the framework may be extended to non-deterministic and hierarchical policies, multi-agent settings, and real-time adaptation.

Conclusion

The pessimistic auxiliary policy framework systematically addresses estimation error and overestimation in offline RL by maximizing the lower confidence bound of the Q-function for action selection. It is theoretically sound, empirically effective, and practically extensible, improving the efficacy and reliability of policy learning in offline reinforcement learning contexts (2602.23974).