SPAR: Support-Preserving Action Rectification

Abstract: Offline policy improvement faces an inherent conflict between maximizing value and fitting the data distribution. While in-sample weighted regression is stable, it suffers from over-conservatism that suppresses high-value actions in the distribution tail; conversely, gradient-based approaches often exhibit a fitting-optimization conflict of gradients, which drives the policy off the data manifold. To address this, we propose Support-Preserving Action Rectification (SPAR), which reframes global learning as a local residual rectification anchored to a frozen pure behavior cloning policy. This framework performs fine-grained fitting and local policy improvement in the residual space, thereby contracting the search space. We further introduce Latent Self-Imitation, utilizing a latent-sampling weighted-regression mechanism to address fitting-improvement gradient conflict in the residual space. Theoretically, we prove this mechanism eliminates the manifold-normal drift of standard value gradients, while extensive D4RL experiments show SPAR extracts significant gains from suboptimal baselines to achieve state-of-the-art performance.

• The paper introduces SPAR, a residual learning framework that rectifies policies by constraining updates to locally supported, empirically observed action residuals.
• It employs latent self-imitation and conservative value estimation to limit off-manifold drift, thereby overcoming the over-conservatism of classical in-support regression methods.
• Empirical evaluations on D4RL benchmarks show that SPAR achieves competitive performance in both unimodal and multimodal environments while ensuring safe deployment.

Support-Preserving Action Rectification: A Formal Analysis

Problem Statement and Motivation

Offline reinforcement learning (RL) confronts the dual challenge of maximizing policy value while conforming to the state-action support of a static, finite dataset. Classical in-support regression approaches ensure stability and deployment viability but systematically neglect rarely observed, high-return actions, hence over-conservatism. In contrast, gradient-based methods leverage Q-function optimization to enhance policy value but risk catastrophic divergence off the empirical action manifold, driven by critic approximation artifacts and lack of rigorous support constraints.

This paper introduces Support-Preserving Action Rectification (SPAR), a residual learning framework that formalizes policy improvement as fine-grained corrections anchored to a frozen behavior cloning (BC) baseline. SPAR conceptualizes offline RL as a three-stage pipeline that structurally decouples data fitting and improvement, sharply contracts the hypothesis space, and constrains exploration to the local residual support, thereby mitigating the aforementioned geometric and statistical failure modes.

Methodological Contributions

Residual Learning and Search-Space Contraction

SPAR decomposes the learned policy as a sum of a frozen BC anchor $\pi_{\text{base}}$ and a learnable residual $\Delta\pi$: $$\pi(s) = \pi_{\text{base}}(s) + \Delta\pi(s, \pi_{\text{base}}(s))$$. This parameterization restricts optimization to a neighborhood around the data manifold. The authors formally quantify contraction in the search space using a quantile-based diameter, demonstrating a polynomial-logarithmic reduction in covering number and sample complexity for certified near-optimality over the residual region—an essential analytic result for sample-efficient offline RL [(2605.27877), Thm 3.1].

Latent Self-Imitation for Residuals

To address the incompatibility between value-gradient updates and support-constrained fitting, SPAR introduces latent self-imitation: a derivative-free policy improvement mechanism wherein candidate residual actions are sampled from a latent-conditioned decoder (CVAE), evaluated with a conservative critic ensemble, and used as regression targets in a value-weighted update. This approach strictly bounds action updates within convex hulls of empirically supported residuals, achieving quadratic suppression of off-manifold drift in contrast to linear divergence under naive gradient ascent [(2605.27877), Prop. 3.3]. For unimodal tasks, SPAR-MLP employs direct residual regression; for multimodal or fragmented supports, SPAR-PROJ leverages this generative, sample-based improvement.

Conservative Value Estimation and Deployment Rectification

Stage I employs an in-sample critic ensemble using expectile regression with uncertainty-penalized lower confidence bounds, thereby minimizing OOD overestimation. Final deployment relies on a dual threshold gating mechanism at inference, only accepting residual policy output when conservatively predicted improvement over the baseline exceeds task-independent thresholds, ensuring safety across diverse environments.

Empirical Evaluation

SPAR's efficacy is benchmarked on D4RL, with comprehensive comparison to both legacy (e.g., IQL [Kostrikov et al., 2021], CQL [Kumar et al., 2020]) and state-of-the-art generative methods (e.g., Diff-QL [Wang et al., 2022], flow-based architectures). Key findings include:

• MuJoCo Locomotion: On median-replay data, SPAR-MLP attains superior or competitive normalized returns to diffusion-based and value-guided baselines, while demonstrating zero or minimal OOD drift. Transition to median-expert regimes, where residuals become multimodal, shifts the performance lead to SPAR-PROJ due to its expressiveness and robustness.
• Adroit Manipulation: In high-dimensional, narrow support tasks, SPAR-PROJ avoids the collapse and support violation typical for value-guided and naive regression methods, posting leading scores, e.g., 76.2 on Pen-Cloned.
• AntMaze Navigation: SPAR-PROJ outperforms or matches flow and diffusion baselines on sparse-reward and branching tasks, validating its ability to stably recover high-value behavior in a highly multimodal, fragmented landscape.

Notably, ablation studies indicate that each pipeline stage is essential—the method collapses without latent self-imitation or the support-preserving update. Empirical support diagnostics show that SPAR-PROJ maintains near-perfect proximity to the empirical action support, while comparable residual policies deviate substantially.

Theoretical and Practical Implications

SPAR provides a formally justified reduction in search space and an empirically validated mechanism preventing value-driven support violation—a central failure mode in offline RL. Its approach to decoupled objectives, local residual correction, and value-weighted, support-aware improvement subsumes both in-support and Q-gradient paradigms, offering a more unified and robust framework.

Practical implications are considerable for safety-critical RL deployment: residual parameterization with strong support preservation drastically reduces the risk of OOD actions at test time. The modular design allows straightforward adaptation according to empirical residual geometry, with diagnostic criteria for selecting between unimodal (MLP) and multimodal (CVAE) variants.

Future Directions

While SPAR sets a new standard for support preservation and local policy improvement in offline RL, its effectiveness depends on the representational quality of the frozen anchor policy and the reliability of the conservative critic estimation—limitations that may become pronounced in extreme sparse-reward regimes or highly nonstationary data distributions. Future research should target adaptive residual model selection, stronger theoretical guarantees for data coverage adaptation, and integration with more adaptive or data-centered safety constraints. Extending SPAR's principled framework to multi-agent or high-frequency feedback regimes may further extend its impact.

Conclusion

SPAR (Support-Preserving Action Rectification) advances offline RL by rigorously formalizing and decoupling the dual objectives of distributional fitting and value improvement. Through residual learning, latent self-imitation, and conservative deployment gating, SPAR achieves state-of-the-art results on diverse benchmark tasks while providing both theoretical and empirical guarantees against off-support collapse (2605.27877). Its approach significantly broadens the toolkit for safe, efficient offline RL and offers a foundation for further advances in support-aware policy optimization.