---
title: 'ZAPS-DA: Zero-Phase Policy Smoothing'
url: https://www.emergentmind.com/topics/zaps-da
type: topic
---

# ZAPS-DA: Zero-Phase Policy Smoothing

ZAPS-DA, short for **Zero-Phase Action Policy Smoothing with Decoupled Actor**, is a training-time architectural framework for off-policy continuous-control reinforcement learning that targets **high-frequency action jitter** in deployed policies without using inference-time filtering or modifying the base RL objective. It pairs an unmodified main actor, trained exactly as in the base algorithm, with a separate decoupled actor trained by supervised imitation of **zero-phase filtered** action targets stored in the replay buffer. The deployed policy is the decoupled actor: a feed-forward map from the current observation to a smooth action, with no action-history input and no post-processing, a mechanism termed **causal distillation of a non-causal filter** [2605.30612].

## 1. Problem setting and motivation

ZAPS-DA is motivated by a practical weakness of off-policy continuous-control RL: policies can obtain high reward while producing action sequences that oscillate rapidly from one timestep to the next. The paper argues that this behavior is structural rather than incidental. In many control systems, especially vehicle control, the plant behaves like a low-pass filter, so smooth and jittery action sequences can induce nearly the same state transitions because inertia and dynamics attenuate high-frequency command variation. Since the critic evaluates state-action pairs mostly instantaneously and lacks an explicit temporal-consistency term, it exerts little or no gradient pressure toward smooth actions. The paper names this the **flat Q-landscape problem**: the critic’s value surface is insensitive to action smoothness, allowing the policy to oscillate freely [2605.30612].

The deployment consequences are operational rather than merely aesthetic. The paper identifies actuator wear, comfort degradation, bang-bang throttle behavior, steering oscillations, and incompatibility with physical hardware as direct effects of jitter. A straightforward remedy, post-hoc low-pass filtering, attenuates oscillation but introduces **phase lag**. In driving tasks, delayed control can be hazardous. ZAPS-DA is positioned around the claim that one wants the smoothness associated with filtering **without** the lag of causal filtering, and smoothness learned **without** coupling it to the reward gradient [2605.30612].

This framing places ZAPS-DA in a specific design space: it is not a reward redesign method and not an inference-time signal-processing wrapper. It is an architectural separation of reward learning from smooth-action synthesis.

## 2. Dual-actor architecture and replay-buffer workflow

The framework uses two actors with disjoint roles. The **main actor** $\pi_\phi$ is the ordinary RL policy and is trained with the unmodified base objective, instantiated in the paper with Soft Actor-Critic. The **decoupled actor** $\pi_\psi$ is a separate feed-forward policy trained only by supervised regression toward filtered action targets. The two actors **do not share parameters or gradients**, and the deployed policy is the decoupled actor rather than the main actor [2605.30612].

The data flow proceeds through a history-aware replay construction. Environment transitions $(s_t, a_t, r_t, s_{t+1}, d_t)$ are first stored in a **history buffer**. Once sufficient future context is available, a **zero-phase filter** is applied to a centered action window to produce a filtered target $\tilde a_t$. The middle transition is then inserted into the replay buffer together with $\tilde a_t$. The main actor continues ordinary SAC training, while the decoupled actor is optimized to imitate the stored filtered targets [2605.30612].

The conceptual importance of this workflow is the paper’s phrase **causal distillation of a non-causal filter**. The filter used during target construction is non-causal because it depends on future action samples from the centered window. Deployment, however, must remain causal. ZAPS-DA resolves this by using the non-causal filter only during training, when replay-buffer history supplies the requisite future context, and training a feed-forward policy that maps the **current observation alone** to the smoothed action. At inference, the filter is absent; its effect has been distilled into the decoupled actor [2605.30612].

This architecture distinguishes ZAPS-DA from both post-hoc filtering and actor-regularization schemes. A plausible implication is that the method is best understood as a replay-buffer-mediated target-generation procedure combined with policy distillation, rather than as a modified control law at test time.

## 3. Filter construction, target generation, and imitation objective

The paper uses a **Savitzky–Golay (SG) filter** as the zero-phase smoother. For action dimension $d$, the centered filtered target is defined as
\[
\tilde{a}_{t-m}^{(d)} = \text{SG}\left(a_{t-N+1}^{(d)},\ldots,a_t^{(d)};\,w_d,p=2\right)[m], \quad d\in\{0,\ldots,A-1\},
\]
with history length $N=17$, centered index $m=\lfloor N/2\rfloor=8$, dimension-specific SG window $w_d$, and polynomial order $p=2$. Because the filter is applied on a symmetric centered window and evaluated at the center, it has **zero phase lag**. The paper uses odd windows $w \ge 5$ [2605.30612].

Boundary handling is explicit. For the first $N$ steps after reset, when the history buffer is not yet full, the identity target is used:
\[
\tilde a_t = a_t.
\]
At episode end, the remaining tail is flushed with one-sided polynomial fits at boundary positions. The paper notes that this slightly breaks strict zero phase at the very end but prevents terminal samples from being discarded. The filtered output is clamped into the $\tanh$ range,
\[
\tilde a \in [-1+\varepsilon,\,1-\varepsilon], \qquad \varepsilon = 10^{-3},
\]
to avoid saturation-related issues [2605.30612].

The decoupled actor is trained with a supervised imitation loss. Let $\mu_\psi(s)$ denote its pre-squash mean output. The imitation objective is
\[
\mathcal{L}_\text{imit}(\psi) = \mathbb{E}_{(s,\tilde a)\sim\mathcal B} \left[\left\|\tanh(\mu_\psi(s)) - \tilde a\right\|^2\right].
\]
The implemented loss is a magnitude-matched version,
\[
\mathcal{L}(\psi) = c \cdot \mathcal{L}_\text{imit}(\psi), \qquad c = \frac{\langle|\mathcal{L}_\text{actor}|\rangle} {\langle|\mathcal{L}_\text{imit}|\rangle + \varepsilon},
\]
where $\langle\cdot\rangle$ is a cumulative running mean over gradient steps, $|\mathcal{L}_\text{actor}|$ is the magnitude of the base RL actor loss, and $\varepsilon = 10^{-6}$ is a numerical stabilizer. The scale $c$ is treated as a constant in backpropagation [2605.30612].

The stated purpose of magnitude matching is optimizer portability. Under Adam, constant loss scaling matters little because adaptive normalization absorbs it; under SGD, unscaled MSE gradients can be too small relative to RL-scale losses, causing distillation failure. The loss therefore serves as a zero-hyperparameter mechanism for portability across optimizer classes [2605.30612].

## 4. Relation to post-hoc filtering and smoothness penalties

A central point of ZAPS-DA is that it is **not** equivalent to post-hoc smoothing. Post-hoc filtering reduces jitter by filtering the policy output during inference, but causal moving-average or low-pass filters introduce **phase delay**. In fast feedback tasks, this lag can materially degrade control responsiveness. ZAPS-DA instead uses zero-phase filtering during training target construction and deploys a feed-forward actor that directly emits smooth actions, so there is essentially no filter-induced lag at inference [2605.30612].

It is also distinct from adding a smoothness penalty directly to the actor loss. The paper contrasts ZAPS-DA with approaches based on CAPS-style regularization or an SG-MSE penalty applied to the main policy. In that alternative formulation, smoothness and reward gradients become entangled, dual variables saturate, and it becomes difficult to determine whether performance degradation arises from RL optimization itself or from over-aggressive smoothing. ZAPS-DA avoids this by **architectural separation**: the main actor is trained exactly as before, and the smooth deployed actor is learned separately by imitation [2605.30612].

This separation addresses a common misunderstanding that smooth control in RL must be enforced by altering the reward or actor objective. In the ZAPS-DA formulation, reward maximization and temporal smoothing are assigned to different function approximators with different losses. This suggests that the method treats smoothness as a representational target generated from experience rather than as a term to be optimized jointly with return.

## 5. Experimental protocol and quantitative results

The paper validates ZAPS-DA with **Soft Actor-Critic (SAC)** and a **Savitzky–Golay filter** in two driving simulators: **MetaDrive** and a custom **Webots adaptive cruise control** environment. The main actor, decoupled actor, and twin critics are all MLPs with **2 × 256 ReLU** layers; actors output $\tanh$-squashed Gaussians, critics output scalar values, and the same architecture is used across environments. Training uses batch size 512, discount $\gamma = 0.99$, Polyak coefficient $\tau = 0.005$, and Adam as the anchor optimizer, with SGD used in an ablation. Replay buffer capacity is $10^6$ for MetaDrive and $5\times10^5$ for Webots. MetaDrive uses $2\times10^6$ training steps; Webots uses retuned SAC hyperparameters for higher frequency and longer episodes [2605.30612].

MetaDrive is the primary evaluation environment. It runs at 10 Hz, provides 259 observation features, and uses a 2-D action consisting of steering and throttle/brake in $[-1,1]$. Its reward is described as dense and smoothness-neutral: longitudinal progress plus speed terms, with terminal rewards $\{+10,-5,-5\}$ for success, crash, and out-of-road. Webots ACC is the generalization test. It runs at 50 Hz with a 7-D observation, steering in $[-0.5,0.5]$ rad, and speed setpoint in $[0,70]$ km/h. Its reward includes following-distance, close-following penalty, lateral alignment, yaw comfort, and speed-matching terms, with $-100$ on crash or lead-lost; the reward is described as mildly smoothness-friendly [2605.30612].

Evaluation is **paired** with identical scenario rollouts and deterministic inference. MetaDrive anchor evaluation uses seed bases $\{0,50,150\}$ with $n=150$ paired episodes; the MetaDrive ablation uses seed bases $\{0,50\}$ with $n=100$; Webots uses seed bases $\{0,70,150\}$ with $n=150$. Statistical testing uses paired $t$-tests on per-episode differences, and MetaDrive results are Bonferroni-corrected across 13 metrics [2605.30612].

Jitter is quantified separately per action dimension using **Variance**, **MAD** $\big(\mathbb{E}[|a_t-a_{t-1}|]\big)$, **MDD** $\big(\mathbb{E}[|a_t-2a_{t-1}+a_{t-2}|]\big)$, and **JE**, the MSE of action minus a moving average. Lower values indicate smoother behavior [2605.30612].

In MetaDrive, under the headline SG configuration $(17,7)$ with $p=2$ at a 1.59M-step checkpoint, ZAPS-DA yields **steering jitter reductions of about 14–21× across JE/MAD/MDD** and **throttle/speed jitter reductions of about 3–5×**. All jitter metrics have $p<10^{-4}$ and survive Bonferroni correction. Completion and crash outcomes are statistically unchanged, with arrive rate $p=0.28$ and crash rate $p=0.31$. Reward is lower by about **6.3%**, from $284.34 \pm 94.96$ for the main actor to $266.54 \pm 93.26$ for the decoupled actor, a difference of $-17.8$ with $p=0.040$ uncorrected [2605.30612].

In Webots ACC, at the 745.5K-step checkpoint, reward is essentially unchanged: main $5390.5$, decoupled $5439.7$, $\Delta = +49.1$, or $+0.9\%$, with $p=0.121$. Steering jitter is reduced by **8–45×** depending on metric, and throttle/speed jitter by **2.7–3.8×**. Again, all jitter metrics satisfy $p<10^{-4}$. Total task-failure rate is reduced from **2.0% to 0.7%**, with 3 lead-lost cases for the main actor and 1 ego-crash for the decoupled actor [2605.30612].

Taken together, the reported results show two distinct regimes. In MetaDrive, ZAPS-DA finds a trade-off point with greatly improved smoothness and modest reward cost but no statistically significant loss in completion outcomes. In Webots ACC, the same SG configuration produces what the paper describes as a **Pareto improvement**.

## 6. Ablations, implementation details, and limitations

The principal ablation concerns SG window sensitivity. The paper sweeps steering windows $(9,7)$, $(13,7)$, $(17,7)$, $(21,7)$, and a symmetric $(17,17)$ case. As the steering window grows, reward cost generally increases; jitter reduction improves up to a point and then can worsen. The anchor $(17,7)$ is described as the elbow of the trade-off. The throttle window is especially important: the symmetric $(17,17)$ filter severely degrades performance, with reward dropping by about **23.5%**, arrive rate collapsing, and crash rate rising sharply. The paper interprets this as evidence that throttle requires a narrower window because the policy uses throttle transients for reactive braking and acceleration, and over-smoothing removes necessary control information [2605.30612].

A second ablation studies optimizer choice and magnitude matching in a $2\times2$ design. Under **Adam**, scale matching is mostly redundant. Under **SGD**, it is essential: without magnitude matching, the decoupled actor collapses badly, reward plummets, arrival rate collapses, and crash rate increases. This is presented as justification for running-mean magnitude normalization as a portability mechanism rather than a cosmetic change [2605.30612].

The implementation notes are operationally important. The SG output is clamped into the $\tanh$ range; the history buffer uses size $N=17$; early timesteps after reset use identity targets; end-of-episode flushing uses one-sided fits; and a per-sample Q-aware teacher selector was tried, found ineffective, and removed. The paper also remarks that longer SG windows can introduce negative side-lobes that create ripple, which helps explain non-monotonic ablation behavior [2605.30612].

The limitations are explicit. Validation is restricted to **SAC**, **driving tasks**, **simulation only**, and **Savitzky–Golay filtering**. The study does not model actuator noise, sensor latency, or wear-induced drift. The authors suggest extensions to TD3/DDPG, locomotion, manipulation, alternative zero-phase filters such as Butterworth-style zero-phase processing, and real-hardware deployment [2605.30612].

A common misconception would be to treat the reported gains as already established for general continuous control or physical systems. The paper does not make that claim. Its evidence base is confined to the settings above, and the broader applicability remains a stated direction rather than an evaluated result.

## 7. Interpretation and place within smooth-control RL

ZAPS-DA is best characterized as a **distillation framework for smooth control** rather than a new off-policy RL objective. The main actor remains bit-for-bit identical to a standalone SAC run when seeds and hyperparameters are matched, while the decoupled actor learns a smoothed action representation from replay-buffer targets generated by a non-causal oracle during training [2605.30612].

Its central technical idea is therefore not simply “smoothing actions,” but converting a **non-causal zero-phase filtering operation** into a **causal deployed policy** through supervised imitation. That design avoids the phase lag associated with test-time filters and avoids entangling smoothness gradients with reward gradients. This suggests a broader methodological pattern: temporal control desiderata that are difficult to encode in the critic or reward may sometimes be imposed through auxiliary target-generation procedures and decoupled policy heads.

Within the scope evaluated in the paper, the empirical claim is narrow but strong: substantial reductions in steering and throttle jitter can be obtained with negligible phase lag and no inference-time filter, while preserving task-completion behavior in MetaDrive and achieving reward parity with reduced failure rate in Webots ACC [2605.30612].

Source: https://www.emergentmind.com/topics/zaps-da