ZAPS-DA: Zero-Phase Policy Smoothing

• ZAPS-DA is a framework that decouples reward-driven learning from smooth control by training a main actor with standard RL and a separate decoupled actor via zero-phase filtered targets.
• It uses a centered Savitzky–Golay filter on replay-buffer data to generate smooth action targets, enabling the decoupled actor to produce jitter-free control without inference-time lag.
• Empirical results in driving environments show significant reductions in steering and throttle jitter with minimal reward penalties, highlighting its effectiveness in practical off-policy continuous control.

ZAPS-DA, short for Zero-Phase Action Policy Smoothing with Decoupled Actor, is a training-time architectural framework for off-policy continuous-control reinforcement learning that targets high-frequency action jitter in deployed policies without using inference-time filtering or modifying the base RL objective. It pairs an unmodified main actor, trained exactly as in the base algorithm, with a separate decoupled actor trained by supervised imitation of zero-phase filtered action targets stored in the replay buffer. The deployed policy is the decoupled actor: a feed-forward map from the current observation to a smooth action, with no action-history input and no post-processing, a mechanism termed causal distillation of a non-causal filter (Shamass, 28 May 2026).

1. Problem setting and motivation

ZAPS-DA is motivated by a practical weakness of off-policy continuous-control RL: policies can obtain high reward while producing action sequences that oscillate rapidly from one timestep to the next. The paper argues that this behavior is structural rather than incidental. In many control systems, especially vehicle control, the plant behaves like a low-pass filter, so smooth and jittery action sequences can induce nearly the same state transitions because inertia and dynamics attenuate high-frequency command variation. Since the critic evaluates state-action pairs mostly instantaneously and lacks an explicit temporal-consistency term, it exerts little or no gradient pressure toward smooth actions. The paper names this the flat Q-landscape problem: the critic’s value surface is insensitive to action smoothness, allowing the policy to oscillate freely (Shamass, 28 May 2026).

The deployment consequences are operational rather than merely aesthetic. The paper identifies actuator wear, comfort degradation, bang-bang throttle behavior, steering oscillations, and incompatibility with physical hardware as direct effects of jitter. A straightforward remedy, post-hoc low-pass filtering, attenuates oscillation but introduces phase lag. In driving tasks, delayed control can be hazardous. ZAPS-DA is positioned around the claim that one wants the smoothness associated with filtering without the lag of causal filtering, and smoothness learned without coupling it to the reward gradient (Shamass, 28 May 2026).

This framing places ZAPS-DA in a specific design space: it is not a reward redesign method and not an inference-time signal-processing wrapper. It is an architectural separation of reward learning from smooth-action synthesis.

2. Dual-actor architecture and replay-buffer workflow

The framework uses two actors with disjoint roles. The main actor $\pi_\phi$ is the ordinary RL policy and is trained with the unmodified base objective, instantiated in the paper with Soft Actor-Critic. The decoupled actor $\pi_\psi$ is a separate feed-forward policy trained only by supervised regression toward filtered action targets. The two actors do not share parameters or gradients, and the deployed policy is the decoupled actor rather than the main actor (Shamass, 28 May 2026).

The data flow proceeds through a history-aware replay construction. Environment transitions $(s_t, a_t, r_t, s_{t+1}, d_t)$ are first stored in a history buffer. Once sufficient future context is available, a zero-phase filter is applied to a centered action window to produce a filtered target $\tilde a_t$. The middle transition is then inserted into the replay buffer together with $\tilde a_t$. The main actor continues ordinary SAC training, while the decoupled actor is optimized to imitate the stored filtered targets (Shamass, 28 May 2026).

The conceptual importance of this workflow is the paper’s phrase causal distillation of a non-causal filter. The filter used during target construction is non-causal because it depends on future action samples from the centered window. Deployment, however, must remain causal. ZAPS-DA resolves this by using the non-causal filter only during training, when replay-buffer history supplies the requisite future context, and training a feed-forward policy that maps the current observation alone to the smoothed action. At inference, the filter is absent; its effect has been distilled into the decoupled actor (Shamass, 28 May 2026).

This architecture distinguishes ZAPS-DA from both post-hoc filtering and actor-regularization schemes. A plausible implication is that the method is best understood as a replay-buffer-mediated target-generation procedure combined with policy distillation, rather than as a modified control law at test time.

3. Filter construction, target generation, and imitation objective

The paper uses a Savitzky–Golay (SG) filter as the zero-phase smoother. For action dimension $d$, the centered filtered target is defined as

$$\tilde{a}_{t-m}^{(d)} = \text{SG}\left(a_{t-N+1}^{(d)},\ldots,a_t^{(d)};\,w_d,p=2\right)[m], \quad d\in\{0,\ldots,A-1\},$$

with history length $N=17$, centered index $m=\lfloor N/2\rfloor=8$, dimension-specific SG window $w_d$, and polynomial order $\pi_\psi$0. Because the filter is applied on a symmetric centered window and evaluated at the center, it has zero phase lag. The paper uses odd windows $\pi_\psi$1 (Shamass, 28 May 2026).

Boundary handling is explicit. For the first $\pi_\psi$2 steps after reset, when the history buffer is not yet full, the identity target is used: $\pi_\psi$3 At episode end, the remaining tail is flushed with one-sided polynomial fits at boundary positions. The paper notes that this slightly breaks strict zero phase at the very end but prevents terminal samples from being discarded. The filtered output is clamped into the $\pi_\psi$4 range,

$\pi_\psi$5

to avoid saturation-related issues (Shamass, 28 May 2026).

The decoupled actor is trained with a supervised imitation loss. Let $\pi_\psi$6 denote its pre-squash mean output. The imitation objective is

$\pi_\psi$7

The implemented loss is a magnitude-matched version,

$\pi_\psi$8

where $\pi_\psi$9 is a cumulative running mean over gradient steps, $(s_t, a_t, r_t, s_{t+1}, d_t)$0 is the magnitude of the base RL actor loss, and $(s_t, a_t, r_t, s_{t+1}, d_t)$1 is a numerical stabilizer. The scale $(s_t, a_t, r_t, s_{t+1}, d_t)$2 is treated as a constant in backpropagation (Shamass, 28 May 2026).

The stated purpose of magnitude matching is optimizer portability. Under Adam, constant loss scaling matters little because adaptive normalization absorbs it; under SGD, unscaled MSE gradients can be too small relative to RL-scale losses, causing distillation failure. The loss therefore serves as a zero-hyperparameter mechanism for portability across optimizer classes (Shamass, 28 May 2026).

4. Relation to post-hoc filtering and smoothness penalties

A central point of ZAPS-DA is that it is not equivalent to post-hoc smoothing. Post-hoc filtering reduces jitter by filtering the policy output during inference, but causal moving-average or low-pass filters introduce phase delay. In fast feedback tasks, this lag can materially degrade control responsiveness. ZAPS-DA instead uses zero-phase filtering during training target construction and deploys a feed-forward actor that directly emits smooth actions, so there is essentially no filter-induced lag at inference (Shamass, 28 May 2026).

It is also distinct from adding a smoothness penalty directly to the actor loss. The paper contrasts ZAPS-DA with approaches based on CAPS-style regularization or an SG-MSE penalty applied to the main policy. In that alternative formulation, smoothness and reward gradients become entangled, dual variables saturate, and it becomes difficult to determine whether performance degradation arises from RL optimization itself or from over-aggressive smoothing. ZAPS-DA avoids this by architectural separation: the main actor is trained exactly as before, and the smooth deployed actor is learned separately by imitation (Shamass, 28 May 2026).

This separation addresses a common misunderstanding that smooth control in RL must be enforced by altering the reward or actor objective. In the ZAPS-DA formulation, reward maximization and temporal smoothing are assigned to different function approximators with different losses. This suggests that the method treats smoothness as a representational target generated from experience rather than as a term to be optimized jointly with return.

5. Experimental protocol and quantitative results

The paper validates ZAPS-DA with Soft Actor-Critic (SAC) and a Savitzky–Golay filter in two driving simulators: MetaDrive and a custom Webots adaptive cruise control environment. The main actor, decoupled actor, and twin critics are all MLPs with 2 × 256 ReLU layers; actors output $(s_t, a_t, r_t, s_{t+1}, d_t)$3-squashed Gaussians, critics output scalar values, and the same architecture is used across environments. Training uses batch size 512, discount $(s_t, a_t, r_t, s_{t+1}, d_t)$4, Polyak coefficient $(s_t, a_t, r_t, s_{t+1}, d_t)$5, and Adam as the anchor optimizer, with SGD used in an ablation. Replay buffer capacity is $(s_t, a_t, r_t, s_{t+1}, d_t)$6 for MetaDrive and $(s_t, a_t, r_t, s_{t+1}, d_t)$7 for Webots. MetaDrive uses $(s_t, a_t, r_t, s_{t+1}, d_t)$8 training steps; Webots uses retuned SAC hyperparameters for higher frequency and longer episodes (Shamass, 28 May 2026).

MetaDrive is the primary evaluation environment. It runs at 10 Hz, provides 259 observation features, and uses a 2-D action consisting of steering and throttle/brake in $(s_t, a_t, r_t, s_{t+1}, d_t)$9. Its reward is described as dense and smoothness-neutral: longitudinal progress plus speed terms, with terminal rewards $\tilde a_t$0 for success, crash, and out-of-road. Webots ACC is the generalization test. It runs at 50 Hz with a 7-D observation, steering in $\tilde a_t$1 rad, and speed setpoint in $\tilde a_t$2 km/h. Its reward includes following-distance, close-following penalty, lateral alignment, yaw comfort, and speed-matching terms, with $\tilde a_t$3 on crash or lead-lost; the reward is described as mildly smoothness-friendly (Shamass, 28 May 2026).

Evaluation is paired with identical scenario rollouts and deterministic inference. MetaDrive anchor evaluation uses seed bases $\tilde a_t$4 with $\tilde a_t$5 paired episodes; the MetaDrive ablation uses seed bases $\tilde a_t$6 with $\tilde a_t$7; Webots uses seed bases $\tilde a_t$8 with $\tilde a_t$9. Statistical testing uses paired $\tilde a_t$0-tests on per-episode differences, and MetaDrive results are Bonferroni-corrected across 13 metrics (Shamass, 28 May 2026).

Jitter is quantified separately per action dimension using Variance, MAD $\tilde a_t$1, MDD $\tilde a_t$2, and JE, the MSE of action minus a moving average. Lower values indicate smoother behavior (Shamass, 28 May 2026).

In MetaDrive, under the headline SG configuration $\tilde a_t$3 with $\tilde a_t$4 at a 1.59M-step checkpoint, ZAPS-DA yields steering jitter reductions of about 14–21× across JE/MAD/MDD and throttle/speed jitter reductions of about 3–5×. All jitter metrics have $\tilde a_t$5 and survive Bonferroni correction. Completion and crash outcomes are statistically unchanged, with arrive rate $\tilde a_t$6 and crash rate $\tilde a_t$7. Reward is lower by about 6.3%, from $\tilde a_t$8 for the main actor to $\tilde a_t$9 for the decoupled actor, a difference of $d$0 with $d$1 uncorrected (Shamass, 28 May 2026).

In Webots ACC, at the 745.5K-step checkpoint, reward is essentially unchanged: main $d$2, decoupled $d$3, $d$4, or $d$5, with $d$6. Steering jitter is reduced by 8–45× depending on metric, and throttle/speed jitter by 2.7–3.8×. Again, all jitter metrics satisfy $d$7. Total task-failure rate is reduced from 2.0% to 0.7%, with 3 lead-lost cases for the main actor and 1 ego-crash for the decoupled actor (Shamass, 28 May 2026).

Taken together, the reported results show two distinct regimes. In MetaDrive, ZAPS-DA finds a trade-off point with greatly improved smoothness and modest reward cost but no statistically significant loss in completion outcomes. In Webots ACC, the same SG configuration produces what the paper describes as a Pareto improvement.

6. Ablations, implementation details, and limitations

The principal ablation concerns SG window sensitivity. The paper sweeps steering windows $d$8, $d$9, $$\tilde{a}_{t-m}^{(d)} = \text{SG}\left(a_{t-N+1}^{(d)},\ldots,a_t^{(d)};\,w_d,p=2\right)[m], \quad d\in\{0,\ldots,A-1\},$$0, $$\tilde{a}_{t-m}^{(d)} = \text{SG}\left(a_{t-N+1}^{(d)},\ldots,a_t^{(d)};\,w_d,p=2\right)[m], \quad d\in\{0,\ldots,A-1\},$$1, and a symmetric $$\tilde{a}_{t-m}^{(d)} = \text{SG}\left(a_{t-N+1}^{(d)},\ldots,a_t^{(d)};\,w_d,p=2\right)[m], \quad d\in\{0,\ldots,A-1\},$$2 case. As the steering window grows, reward cost generally increases; jitter reduction improves up to a point and then can worsen. The anchor $$\tilde{a}_{t-m}^{(d)} = \text{SG}\left(a_{t-N+1}^{(d)},\ldots,a_t^{(d)};\,w_d,p=2\right)[m], \quad d\in\{0,\ldots,A-1\},$$3 is described as the elbow of the trade-off. The throttle window is especially important: the symmetric $$\tilde{a}_{t-m}^{(d)} = \text{SG}\left(a_{t-N+1}^{(d)},\ldots,a_t^{(d)};\,w_d,p=2\right)[m], \quad d\in\{0,\ldots,A-1\},$$4 filter severely degrades performance, with reward dropping by about 23.5%, arrive rate collapsing, and crash rate rising sharply. The paper interprets this as evidence that throttle requires a narrower window because the policy uses throttle transients for reactive braking and acceleration, and over-smoothing removes necessary control information (Shamass, 28 May 2026).

A second ablation studies optimizer choice and magnitude matching in a $$\tilde{a}_{t-m}^{(d)} = \text{SG}\left(a_{t-N+1}^{(d)},\ldots,a_t^{(d)};\,w_d,p=2\right)[m], \quad d\in\{0,\ldots,A-1\},$$5 design. Under Adam, scale matching is mostly redundant. Under SGD, it is essential: without magnitude matching, the decoupled actor collapses badly, reward plummets, arrival rate collapses, and crash rate increases. This is presented as justification for running-mean magnitude normalization as a portability mechanism rather than a cosmetic change (Shamass, 28 May 2026).

The implementation notes are operationally important. The SG output is clamped into the $$\tilde{a}_{t-m}^{(d)} = \text{SG}\left(a_{t-N+1}^{(d)},\ldots,a_t^{(d)};\,w_d,p=2\right)[m], \quad d\in\{0,\ldots,A-1\},$$6 range; the history buffer uses size $$\tilde{a}_{t-m}^{(d)} = \text{SG}\left(a_{t-N+1}^{(d)},\ldots,a_t^{(d)};\,w_d,p=2\right)[m], \quad d\in\{0,\ldots,A-1\},$$7; early timesteps after reset use identity targets; end-of-episode flushing uses one-sided fits; and a per-sample Q-aware teacher selector was tried, found ineffective, and removed. The paper also remarks that longer SG windows can introduce negative side-lobes that create ripple, which helps explain non-monotonic ablation behavior (Shamass, 28 May 2026).

The limitations are explicit. Validation is restricted to SAC, driving tasks, simulation only, and Savitzky–Golay filtering. The study does not model actuator noise, sensor latency, or wear-induced drift. The authors suggest extensions to TD3/DDPG, locomotion, manipulation, alternative zero-phase filters such as Butterworth-style zero-phase processing, and real-hardware deployment (Shamass, 28 May 2026).

A common misconception would be to treat the reported gains as already established for general continuous control or physical systems. The paper does not make that claim. Its evidence base is confined to the settings above, and the broader applicability remains a stated direction rather than an evaluated result.

7. Interpretation and place within smooth-control RL

ZAPS-DA is best characterized as a distillation framework for smooth control rather than a new off-policy RL objective. The main actor remains bit-for-bit identical to a standalone SAC run when seeds and hyperparameters are matched, while the decoupled actor learns a smoothed action representation from replay-buffer targets generated by a non-causal oracle during training (Shamass, 28 May 2026).

Its central technical idea is therefore not simply “smoothing actions,” but converting a non-causal zero-phase filtering operation into a causal deployed policy through supervised imitation. That design avoids the phase lag associated with test-time filters and avoids entangling smoothness gradients with reward gradients. This suggests a broader methodological pattern: temporal control desiderata that are difficult to encode in the critic or reward may sometimes be imposed through auxiliary target-generation procedures and decoupled policy heads.

Within the scope evaluated in the paper, the empirical claim is narrow but strong: substantial reductions in steering and throttle jitter can be obtained with negligible phase lag and no inference-time filter, while preserving task-completion behavior in MetaDrive and achieving reward parity with reduced failure rate in Webots ACC (Shamass, 28 May 2026).