Support-Preserving Action Rectification (SPAR)

• The paper introduces SPAR, a method that refines a frozen behavior cloning policy with local residual corrections to improve returns while preserving dataset support.
• SPAR employs two residual parameterizations—SPAR-MLP for unimodal corrections and SPAR-PROJ with latent self-imitation for complex residual landscapes—to balance value improvement with support fidelity.
• Empirical and theoretical analyses demonstrate that SPAR reduces off-manifold drift and sample complexity, outperforming standard baselines on diverse offline RL benchmarks.

Support-Preserving Action Rectification (SPAR) is an offline reinforcement learning framework that formulates policy improvement as local residual rectification around a frozen pure behavior cloning policy, rather than as unconstrained global optimization. In the SPAR formulation, a behavior-cloned anchor provides the global approximation to the dataset action manifold, while a residual policy learns only local corrections; improvement is then driven by conservative value estimates and, in the generative variant, by latent-sampling weighted regression rather than direct value-gradient ascent. The central objective is to improve return without driving the policy off the dataset support, where critic extrapolation becomes unreliable (Zhao et al., 27 May 2026).

1. Problem setting and motivation

SPAR is defined in the standard offline RL setting over a discounted MDP

$$\mathcal{M} = (\mathcal{S}, \mathcal{A}, P, r, \gamma),$$

with a fixed dataset

$D = \{(s, a, r, s', d)\}_{i=1}^N$

collected by an unknown behavior policy $\beta(a \mid s)$. No further interaction with the environment is allowed. The target is to learn a policy $\pi(a \mid s)$ with higher return than the behavior policy while avoiding departures from the dataset support that would render value estimates unreliable (Zhao et al., 27 May 2026).

For a fixed state $s$, the support of the dataset is

$$\operatorname{supp}(\beta(\cdot \mid s)) = \{a : \beta(a \mid s) > 0\}.$$

The paper uses the broader term data manifold for the covered state-action region and its local neighborhoods where the critic is reasonably trained. In offline RL, this geometric constraint is fundamental because Q-values outside support are extrapolations; a policy that moves substantially off-manifold can exploit critic error and fail at deployment.

The method is motivated by a tension between two standard offline policy-improvement paradigms. In-support weighted regression methods such as AWR, AWAC, IQL, IDQL, and diffusion/flow-based in-sample methods are stable because they remain near the data support, but they exhibit a maximum-likelihood bias: high-density behavior actions dominate, while rare high-value tail actions can be underweighted or ignored. Gradient-based policy-improvement methods such as the CQL actor, TD3+BC actor, PLAS, Diffusion-QL, and flow actor-critic directly maximize a learned critic, but in the offline regime their gradients often have a large component normal to the data manifold, pushing the actor toward OOD regions and inducing instability. SPAR is introduced to resolve this fitting–improvement conflict by constraining improvement to a local residual space.

A common misconception is that support preservation is equivalent to pure imitation. SPAR explicitly rejects that equivalence: the critique of in-sample weighted regression is precisely that it can become overly conservative and suppress high-value actions in the distribution tail. Support preservation, in the SPAR sense, is therefore not global policy stasis; it is controlled local improvement within a support-aware neighborhood.

2. Anchored residual formulation

The SPAR policy is constructed around a frozen behavior-cloning anchor. First, a pure BC policy $\pi_{\text{base}}$ is trained on the dataset and then frozen. A residual policy outputs a local correction $\Delta a$, and the final action is

$$a_{\text{SPAR}}(s) = a_{\text{base}}(s) + G\big(\Delta a_{\theta}(s, a_{\text{base}}(s))\big),$$

where $a_{\text{base}}(s) = \pi_{\text{base}}(s)$ and $D = \{(s, a, r, s', d)\}_{i=1}^N$0 is an inference-time gate that may zero out the residual if critic-based improvement is not sufficiently reliable (Zhao et al., 27 May 2026).

Locality is defined explicitly in action space around $D = \{(s, a, r, s', d)\}_{i=1}^N$1. Let the empirical $D = \{(s, a, r, s', d)\}_{i=1}^N$2-quantile of residual magnitudes be

$D = \{(s, a, r, s', d)\}_{i=1}^N$3

SPAR then restricts the residual search region to

$D = \{(s, a, r, s', d)\}_{i=1}^N$4

with effective diameter $D = \{(s, a, r, s', d)\}_{i=1}^N$5. This contracts the search space from the full action space $D = \{(s, a, r, s', d)\}_{i=1}^N$6 to a local neighborhood around the BC anchor.

Stage I establishes the anchor and the conservative value model. The base policy is trained with pure MSE,

$D = \{(s, a, r, s', d)\}_{i=1}^N$7

An ensemble of critics and a value network is trained using in-sample expectile regression in the IQL style, with expectile $D = \{(s, a, r, s', d)\}_{i=1}^N$8 generally and $D = \{(s, a, r, s', d)\}_{i=1}^N$9 for sparse AntMaze. From the critic ensemble, SPAR constructs a lower-confidence-bound critic,

$\beta(a \mid s)$0

where $\beta(a \mid s)$1 and $\beta(a \mid s)$2 are the ensemble mean and standard deviation and $\beta(a \mid s)$3 is an uncertainty penalty. Both $\beta(a \mid s)$4 and $\beta(a \mid s)$5 are then frozen.

SPAR has two residual parameterizations. SPAR-MLP uses a deterministic residual regressor and is intended for unimodal residual geometry. SPAR-PROJ uses a CVAE residual model and is intended for multimodal or geometrically complex residuals. The paper’s implementation guidance states that SPAR-PROJ is a robust default when in doubt, whereas SPAR-MLP is a lightweight specialization when residuals are clearly unimodal.

The inference-time gate is part of the support-preserving design. For a state $\beta(a \mid s)$6, SPAR computes the base action and its conservative value $\beta(a \mid s)$7, generates candidate residual actions, and computes for each candidate $\beta(a \mid s)$8 the absolute and relative improvement,

$\beta(a \mid s)$9

The best candidate is accepted only if

$\pi(a \mid s)$0

with default thresholds $\pi(a \mid s)$1 and $\pi(a \mid s)$2. Otherwise the policy falls back to $\pi(a \mid s)$3. This makes the final rectification conditional rather than unconditional.

3. Residual objectives and Latent Self-Imitation

Residual learning in Stage II combines a fitting term and an improvement term: $\pi(a \mid s)$4 Both are weighted by a normalized advantage computed relative to the base policy,

$\pi(a \mid s)$5

The base magnitude is either exponential,

$\pi(a \mid s)$6

or uniform $\pi(a \mid s)$7. Hard filtering uses

$\pi(a \mid s)$8

whereas soft filtering uses $\pi(a \mid s)$9 (Zhao et al., 27 May 2026).

For SPAR-MLP, the residual is a deterministic network

$s$0

The fitting loss is

$s$1

The synthesized action is

$s$2

and the improvement loss is

$s$3

This remains a local value-guided update because the actor is optimized only in residual space.

For SPAR-PROJ, the residual is generated by a CVAE with encoder $s$4 and decoder $s$5. The fitting objective is a weighted ELBO,

$s$6

The KL term is left unweighted to avoid over-regularizing high-advantage samples.

The central innovation in SPAR-PROJ is Latent Self-Imitation (LSI). Instead of backpropagating $s$7 through the residual generator, LSI samples candidate latent codes $s$8, decodes residuals with a target decoder $s$9,

$$\operatorname{supp}(\beta(\cdot \mid s)) = \{a : \beta(a \mid s) > 0\}.$$0

computes normalized advantages

$$\operatorname{supp}(\beta(\cdot \mid s)) = \{a : \beta(a \mid s) > 0\}.$$1

and uses soft exponential weights

$$\operatorname{supp}(\beta(\cdot \mid s)) = \{a : \beta(a \mid s) > 0\}.$$2

The guidance loss is then a stop-gradient weighted regression,

$$\operatorname{supp}(\beta(\cdot \mid s)) = \{a : \beta(a \mid s) > 0\}.$$3

The significance of LSI is geometric. The critic supplies Q-values but not Q-gradients to the generator. Improvement is implemented by regressing toward value-weighted samples already lying on the learned residual manifold. This is the mechanism by which SPAR-PROJ is designed to remove the fitting-improvement gradient conflict that arises when direct value gradients distort the representation used for support-preserving fitting.

4. Theoretical properties

The theoretical analysis in SPAR addresses two distinct issues: statistical search-space contraction and geometric drift suppression (Zhao et al., 27 May 2026).

The contraction claim begins from the observation that residualization reduces the actor’s optimization domain from the full action set $$\operatorname{supp}(\beta(\cdot \mid s)) = \{a : \beta(a \mid s) > 0\}.$$4 to the residual ball $$\operatorname{supp}(\beta(\cdot \mid s)) = \{a : \beta(a \mid s) > 0\}.$$5. Under the assumptions that $$\operatorname{supp}(\beta(\cdot \mid s)) = \{a : \beta(a \mid s) > 0\}.$$6 is $$\operatorname{supp}(\beta(\cdot \mid s)) = \{a : \beta(a \mid s) > 0\}.$$7-Lipschitz in action and value estimates are $$\operatorname{supp}(\beta(\cdot \mid s)) = \{a : \beta(a \mid s) > 0\}.$$8-sub-Gaussian, the paper gives the informal bound

$$\operatorname{supp}(\beta(\cdot \mid s)) = \{a : \beta(a \mid s) > 0\}.$$9

where $\pi_{\text{base}}$0 is the covering number of $\pi_{\text{base}}$1 at scale $\pi_{\text{base}}$2. On simple convex domains, $\pi_{\text{base}}$3. Since $\pi_{\text{base}}$4 empirically, restricting search to $\pi_{\text{base}}$5 yields a polynomial-logarithmic reduction in sample complexity for action optimization.

This contraction introduces a localization bias,

$\pi_{\text{base}}$6

which the paper decomposes into coverage and localization parts. Under Lipschitz continuity, if

$\pi_{\text{base}}$7

then

$\pi_{\text{base}}$8

The intended implication is that if high-value in-support actions lie within the residual ball around the BC anchor, the bias is small while the gain in reliability can be substantial.

The second theoretical contribution concerns manifold-normal drift. Let $\pi_{\text{base}}$9 be a $\Delta a$0 residual manifold with tangent space $\Delta a$1 at $\Delta a$2. Any update vector $\Delta a$3 decomposes as

$\Delta a$4

For a gradient step $\Delta a$5, the distance to the manifold obeys

$\Delta a$6

so ordinary value-gradient ascent exhibits linear off-manifold drift.

LSI alters this geometry. Given sampled residuals $\Delta a$7 and weights $\Delta a$8 with $\Delta a$9, the minimizer of

$$a_{\text{SPAR}}(s) = a_{\text{base}}(s) + G\big(\Delta a_{\theta}(s, a_{\text{base}}(s))\big),$$0

is

$$a_{\text{SPAR}}(s) = a_{\text{base}}(s) + G\big(\Delta a_{\theta}(s, a_{\text{base}}(s))\big),$$1

For the two-point case, if

$$a_{\text{SPAR}}(s) = a_{\text{base}}(s) + G\big(\Delta a_{\theta}(s, a_{\text{base}}(s))\big),$$2

the maximum distance of the chord to the manifold satisfies

$$a_{\text{SPAR}}(s) = a_{\text{base}}(s) + G\big(\Delta a_{\theta}(s, a_{\text{base}}(s))\big),$$3

Accordingly, LSI updates remain within a thin tube around $$a_{\text{SPAR}}(s) = a_{\text{base}}(s) + G\big(\Delta a_{\theta}(s, a_{\text{base}}(s))\big),$$4 with quadratic deviation in chord length, whereas direct gradient ascent has first-order drift. This is the paper’s principal geometric justification for replacing critic backpropagation with latent-sampling weighted regression.

These guarantees are explicitly local. They depend on the residual manifold being sufficiently smooth, latent sampling remaining in the region where the learned residual model approximates the true residual distribution, and the conservative critic being reasonable in the local neighborhood defined by the anchor.

5. Empirical performance and ablations

SPAR is evaluated on the CORL version of D4RL across MuJoCo locomotion, AntMaze navigation, and Adroit manipulation. The base policy is always a simple BC model; the reported question is whether large gains can be extracted solely through residual rectification (Zhao et al., 27 May 2026).

The benchmark suite includes HalfCheetah, Hopper, and Walker2d in medium-replay and medium-expert settings; AntMaze umaze-diverse and large-diverse; and Adroit Pen-Cloned and Pen-Human. Baselines span gradient-based policy-improvement methods such as BCQ, TD3+BC, CQL, PLAS, and Diff-QL, and in-support learning methods such as AWAC, IQL, IDQL, LAPO, EQL, CQL-AW, and recent diffusion/flow-based methods.

Selected normalized D4RL results illustrate the regime dependence of the two SPAR variants. On Hopper-medium-replay, the base BC policy is approximately $$a_{\text{SPAR}}(s) = a_{\text{base}}(s) + G\big(\Delta a_{\theta}(s, a_{\text{base}}(s))\big),$$5, IQL is approximately $$a_{\text{SPAR}}(s) = a_{\text{base}}(s) + G\big(\Delta a_{\theta}(s, a_{\text{base}}(s))\big),$$6, Diff-QL is approximately $$a_{\text{SPAR}}(s) = a_{\text{base}}(s) + G\big(\Delta a_{\theta}(s, a_{\text{base}}(s))\big),$$7, and SPAR-MLP reaches

$$a_{\text{SPAR}}(s) = a_{\text{base}}(s) + G\big(\Delta a_{\theta}(s, a_{\text{base}}(s))\big),$$8

On HalfCheetah-medium-replay, the base is approximately $$a_{\text{SPAR}}(s) = a_{\text{base}}(s) + G\big(\Delta a_{\theta}(s, a_{\text{base}}(s))\big),$$9, Diff-QL approximately $a_{\text{base}}(s) = \pi_{\text{base}}(s)$0, and SPAR-MLP achieves

$a_{\text{base}}(s) = \pi_{\text{base}}(s)$1

On multimodal medium-expert tasks, SPAR-PROJ is the stronger variant: Hopper-medium-expert

$a_{\text{base}}(s) = \pi_{\text{base}}(s)$2

and Walker2d-medium-expert

$a_{\text{base}}(s) = \pi_{\text{base}}(s)$3

The same pattern appears in narrow or multimodal domains. On Pen-Cloned, the base is approximately $a_{\text{base}}(s) = \pi_{\text{base}}(s)$4, many advantage-weighted or constrained baselines are roughly $a_{\text{base}}(s) = \pi_{\text{base}}(s)$5, and SPAR-PROJ reaches

$a_{\text{base}}(s) = \pi_{\text{base}}(s)$6

On Pen-Human, SPAR-PROJ obtains

$a_{\text{base}}(s) = \pi_{\text{base}}(s)$7

On AntMaze large-diverse, where many methods are reported in the approximate $a_{\text{base}}(s) = \pi_{\text{base}}(s)$8 range, SPAR-PROJ achieves

$a_{\text{base}}(s) = \pi_{\text{base}}(s)$9

The ablation results clarify the mechanism. Performing self-imitation in the full action space rather than in SPAR’s anchored residual space leads to collapse on harder tasks; residual anchoring is therefore not a cosmetic detail but a structural requirement. Removing LSI or removing the Stage III gate reduces robustness and performance. Gradient-guided residuals, denoted SPAR-PLAS, still exhibit support drift and lower scores.

Support diagnostics on Pen-Cloned are particularly direct. The 95th percentile kNN support-distance ratio is approximately $D = \{(s, a, r, s', d)\}_{i=1}^N$00 for SPAR-PROJ, indicating that actions stay within the dataset boundary, whereas the gradient-guided flow residual baseline reports approximately $D = \{(s, a, r, s', d)\}_{i=1}^N$01, indicating substantial support drift. The paper also measures gradient-conflict metrics: SPAR-MLP and SPAR-PLAS show large positive directional disagreement and fitting degradation after a value-guidance update, whereas SPAR-PROJ reduces these metrics by several orders of magnitude.

The weighting and pessimism ablations show strong task dependence. Hard filtering performs well on dense MuJoCo tasks, while soft filtering is critical on sparse AntMaze and narrow Pen tasks because it preserves connectivity and bridging transitions. A temperature around $D = \{(s, a, r, s', d)\}_{i=1}^N$02 often balances focus and connectivity. Removing the uncertainty penalty by setting $D = \{(s, a, r, s', d)\}_{i=1}^N$03 can dramatically degrade performance on Pen-Cloned and AntMaze-large-diverse; the Pen-Cloned score cited in the paper drops from $D = \{(s, a, r, s', d)\}_{i=1}^N$04 to $D = \{(s, a, r, s', d)\}_{i=1}^N$05. Excessive pessimism also harms performance, particularly in AntMaze.

Implementation details are correspondingly pragmatic. The reported default design uses actor and residual MLPs with 2–3 hidden layers of 256 units and ReLU activations; CVAE encoder and decoder networks with 3 hidden layers of 256 units, latent dimension $D = \{(s, a, r, s', d)\}_{i=1}^N$06, and KL weight $D = \{(s, a, r, s', d)\}_{i=1}^N$07; and a critic ensemble of 5 Q-networks. Optimization uses Adam with learning rate $D = \{(s, a, r, s', d)\}_{i=1}^N$08, global gradient clipping at norm $D = \{(s, a, r, s', d)\}_{i=1}^N$09, and Polyak averaging with $D = \{(s, a, r, s', d)\}_{i=1}^N$10. Stage I and Stage II each run for roughly $D = \{(s, a, r, s', d)\}_{i=1}^N$11 million gradient steps.

6. Related formulations, limitations, and broader interpretation

SPAR is a specific offline RL method, but the support-preservation principle has close analogues in other RL settings. In RLVR for language-model reasoning, Anchored Policy Optimization (APO) is presented as a concrete instantiation of a Support-Preserving Action Rectification-type method. APO replaces global KL Shape Matching with Support Coverage, defining a Safe Manifold from the reference model’s top-$D = \{(s, a, r, s', d)\}_{i=1}^N$12 support and using a push-pull rectifier on negative-advantage tokens to suppress the current error while restoring mass on the anchor set; the pull term is shown to be gradient-aligned with maximizing support coverage (Wang et al., 5 Feb 2026). In sparse-reward RL with intrinsic motivation, Action-Dependent Optimality Preserving Shaping (ADOPS) converts arbitrary intrinsic rewards into an action-dependent shaping term that preserves the support of extrinsically optimal actions, even though it does not preserve the ordering among suboptimal actions and does not require PBRS-style action-independence of cumulative shaping returns (2505.12611).

These adjacent formulations help delimit what SPAR is and is not. It is not merely conservative behavior cloning, because it explicitly seeks local policy improvement. It is not unrestricted actor optimization, because both the residual parameterization and the gate are designed to prevent off-support drift. And it is not equivalent to generic reward shaping, because its rectification acts in action space around a frozen BC anchor rather than by transforming the reward function.

The limitations are correspondingly structural. SPAR depends on the quality of the base policy: if $D = \{(s, a, r, s', d)\}_{i=1}^N$13 is very poor or misaligned, the residual neighborhood may not contain good actions, and localization bias can become large. It depends on the conservative critic $D = \{(s, a, r, s', d)\}_{i=1}^N$14: if the critic is badly misestimated even within the residual neighborhood, weighting and gating can become misleading. SPAR-PROJ is more expensive at inference than SPAR-MLP because it requires multiple latent samples and critic evaluations, although the paper notes that it remains cheaper than diffusion models. Finally, the theoretical guarantees are local—they quantify search-space contraction and local drift suppression, not full offline RL performance guarantees under arbitrary distribution shift.

Taken together, these points position SPAR as a geometric and statistical answer to a specific offline RL pathology. Its defining claim is that support preservation need not imply over-conservatism if policy improvement is reformulated as local residual rectification around a frozen support-faithful anchor.