Action Aliasing: Representation Challenges

• Action aliasing is a phenomenon where distinct underlying signals become indistinguishable due to coarse sampling, representation learning, or tokenization.
• It appears in domains like vision-language-action models, temporal action localization, and neural audio, where it leads to ambiguities in state recognition and suboptimal decision making.
• Mitigation strategies include inverse dynamics supervision, dynamic low-pass filtering, and architectural smoothing techniques that balance alias reduction with preserving key informational content.

Searching arXiv for papers on action aliasing and closely related aliasing formulations to ground the encyclopedia entry. Action aliasing denotes a family of ambiguity phenomena in which action-relevant distinctions are lost because sampling, downsampling, tokenization, or representation learning makes different underlying sources indistinguishable. In Vision-Language-Action models, “action aliasing arises as a downstream effect, where indistinguishable state representations lead the policy to apply the same action in divergent, contextually inappropriate situations”; in temporal action localization, aliasing causes “features sampled from different sources [to become] indistinguishable and the sampled feature can’t restore to the original one”; and in neural amp modeling, nonlinearities generate harmonics that exceed the Nyquist frequency and “fold back” into the audible spectrum as aliases (Lee et al., 28 May 2026, Jin et al., 2021, Sato et al., 7 May 2025). This suggests that the term is not used as a single standardized label, but rather as a recurring description of action-relevant ambiguity induced by insufficiently discriminative observation, feature, or signal representations.

1. Conceptual scope and recurring definition

Across the recent literature, action aliasing is closely tied to the broader notion of aliasing in signal processing: high-frequency or fine-grained distinctions are lost when the effective sampling or representation is too coarse for the underlying variation. In sequential decision making, the direct formulation is representational and behavioral: “multiple underlying true states are mapped to the same observation (or representation), so a single policy action may be suboptimal for one or more of these true states” (Lee et al., 28 May 2026). In Minecraft-based deep reinforcement learning, this is framed as “perceptual aliasing,” where “many states share nearly identical visual features,” making it difficult to map visual input to optimal actions (Frazier et al., 2019).

In temporal action localization, the term is attached to temporal downsampling. The mechanism is explicitly spectral: temporal downsampling operations “often lead to the aliasing problem, due to lacking consideration of sampling rates,” and the consequence is that downsampled temporal features may become indistinguishable across different action instances (Jin et al., 2021). In vision models, the same logic appears in spatial form. Common CNN architectures “carelessly sub-sampl[e] without considering aliasing effects,” while vision transformers introduce aliasing through the “discontinuous patch-wise tokenization process,” which creates “jagged artifacts into attention maps” (Rodríguez-Muñoz et al., 2022, Qian et al., 2021).

A plausible implication is that action aliasing is best understood as an operational failure mode rather than a domain-specific object: the failure appears whenever the representation no longer preserves the distinctions needed for action selection, action localization, or action-conditioned signal reconstruction.

2. Sequential decision making, state aliasing, and policy ambiguity

The most explicit recent treatment appears in Vision-Language-Action models. There, “state aliasing occurs when visually similar but meaningfully distinct states (requiring different actions) are mapped to similar visual representations,” and “action aliasing arises as a downstream effect” of this representational collapse (Lee et al., 28 May 2026). The proposed remedy is inverse dynamics learning as an auxiliary objective that directly supervises the vision encoder. With trajectories $\tau = (l,\{(o_t,a_t)\}_{t=1}^T)$, the encoder produces $z_t = E_\phi(o_t)$ and $z_{t+k} = E_\phi(o_{t+k})$, while the inverse dynamics head predicts the action chunk $\check{a}_{t:t+k} = h_\psi(z_t,z_{t+k})$ (Lee et al., 28 May 2026). The stated purpose is to make the encoder “capture fine-grained visual distinctions that determine low-level actions.”

The same work adds pseudo time reversal, which “reverses observation sequences and negates motion actions,” thereby exposing the encoder “to a broader distribution of state transitions” and improving generalization “especially under limited demonstration data” (Lee et al., 28 May 2026). The paper emphasizes that the method “uses only standard observation-action pairs without additional annotations, and preserves the original inference pipeline at test time.” Empirically, it reports “consistent gains across diverse VLA baselines,” and the analysis goes beyond policy success: frozen-encoder probing and “state-feature alignment analysis” show that the learned features become more state-discriminative, with “higher correlation (up to 0.60 vs 0.18 for the baseline)” between feature distances and robot-state changes (Lee et al., 28 May 2026).

Minecraft-based interactive reinforcement learning addresses the same ambiguity from a different direction. The environment is described as having “high degrees of aliasing,” and the intervention is not architectural anti-aliasing but human action advice (Frazier et al., 2019). Two algorithms are studied: Feedback Arbitration and Newtonian Action Advice. Feedback Arbitration decides between exploration, exploitation, and advice according to policy confidence; Newtonian Action Advice makes advice persist for multiple timesteps through a friction parameter, so the agent can continue acting under a human-provided directional command (Frazier et al., 2019). Under visual aliasing conditions, both algorithms outperform the baseline, and “NAA converges faster than FA in all settings”; after a $90^\circ$ map rotation, NAA “quickly reconverges in <40 episodes” (Frazier et al., 2019). This suggests that action aliasing can be mitigated either by better state representation or by external supervisory signals that disambiguate aliased states online.

3. Temporal action localization and sampling-induced indistinguishability

Temporal action localization provides the most direct signal-processing formulation of action aliasing. The paper “Low Pass Filter for Anti-aliasing in Temporal Action Localization” defines the problem as a consequence of temporal downsampling without matching the sampling rate to the signal bandwidth (Jin et al., 2021). If the input sequence $x(n)$ is downsampled by a factor $N$,

$$x_p(n)=x(n)\cdot p(n)=\sum_{k=-\infty}^{\infty} x(kN)\delta(n-kN),$$

then in the frequency domain

$$X_p(\omega)=X(\omega)\otimes P(\omega)=\sum_{k=0}^{N-1} X\left(\omega-\frac{2\pi}{N}k\right).$$

The paper states that this spectral mixing causes high-frequency content to be folded into lower frequencies, making downsampled features from different sources indistinguishable (Jin et al., 2021).

The proposed solution is classical anti-aliasing by low-pass filtering before downsampling, but with a learned cutoff frequency to preserve task-relevant high-frequency information. The ideal low-pass filter is defined as

$$H(\omega)= \begin{cases} 1, & |\omega|\leq f_C \ 0, & |\omega|>f_C , \end{cases}$$

with filtered signal

$z_t = E_\phi(o_t)$0

The paper notes that the theoretically optimal cutoff is $z_t = E_\phi(o_t)$1, but also argues that high-frequency content may be important for model inference, so the cutoff should be learned dynamically (Jin et al., 2021). Concretely, “Cut-off frequencies are learnt using a conv1d(channel_in, anchor_num, 3)... That means one proposal corresponds to one LPF.”

The reported empirical effects are substantial. On THUMOS’14, “LPF + average pooling yields a performance jump from 41.87% (avg. pooling) to 49.89% mAP@0.5—an 8.02% absolute gain.” On ActivityNet 1.3, “with only 2 samples, performance improves from 65.92% to 66.27% AUC.” On Charades, applying the LPF to convolution increases “per-frame mAP from 18.4% to 22.9%” (Jin et al., 2021). The central controversy is not whether anti-aliasing helps, but how strongly it should be applied: “overly aggressive low-pass filtering” removes aliasing but damages discriminability, while “too lax a filter” leaves the model susceptible to aliasing artifacts.

4. Image models, adversarial robustness, and architectural anti-aliasing

In image classification and recognition pipelines, aliasing is increasingly treated as a structural source of fragility. “Aliasing is a Driver of Adversarial Attacks” argues that “the existence of adversarial perturbations is due in part to aliasing in neural networks,” establishes “a sufficient condition for no aliasing for general image transformations,” studies “sources of aliasing in common neural network layers,” and derives “simple modifications from first principles” to reduce it (Rodríguez-Muñoz et al., 2022). Its abstract further reports a “solid link between anti-aliasing and adversarial attacks”: reducing aliasing “already results in more robust classifiers,” and combining anti-aliasing with robust training “out-performs solo robust training on $z_t = E_\phi(o_t)$2 attacks with none or minimal losses in performance on $z_t = E_\phi(o_t)$3 attacks” (Rodríguez-Muñoz et al., 2022).

The ASAP line of work sharpens this structural diagnosis by distinguishing classical aliasing from spectral leakage artifacts. “Fix your downsampling ASAP!” argues that FLC pooling is alias-free “in theory,” but remains “prone to spectral leakage artifacts” because a hard rectangular low-pass window in the frequency domain induces ringing in the spatial domain (Grabinski et al., 2023). The proposed remedy is “aliasing and spectral artifact-free pooling,” which applies a smooth Hamming window with

$z_t = E_\phi(o_t)$4

before FLC-style pooling (Grabinski et al., 2023). The reported outcomes are architectural rather than task-specific: ASAP yields downsampled signals with a power spectrum “most similar to the original image,” improves corruption robustness “by over 3%” on CIFAR-10, produces gains that “exceed 10% in some architectures” on ImageNet, and can avoid “catastrophic overfitting” during FGSM adversarial training while outperforming FLC under stronger attacks (Grabinski et al., 2023).

Vision transformers exhibit a distinct aliasing source. “Blending Anti-Aliasing into Vision Transformer” attributes aliasing to the “discontinuous patch-wise tokenization process,” which introduces “jagged artifacts into attention maps” (Qian et al., 2021). The Aliasing-Reduction Module is inserted “immediately after self-attention and before merging with the skip connection,” where it smooths attention maps by a fixed Gaussian filter, a learnable convolutional filter, or a learned combination of a pre-defined filter bank (Qian et al., 2021). The paper reports consistent ImageNet-1k gains across several backbones:

Model	Baseline Top-1	+ ARM
Swin-T	81.2	82.0
DeiT-S	79.8	80.7
DeiT-B	81.8	82.4

The same paper reports better data efficiency, including “42.64% → 45.81%” on 10% of ImageNet data for Swin-T, and a reduction in ImageNet-C mean corruption error from “60.7” to “59.8” (Qian et al., 2021). This suggests that action aliasing in downstream tasks can originate far upstream, in spatial resampling and tokenization choices that systematically erase or distort discriminative signal content.

5. Neural audio models and spectral formulations of aliasing

In neural amp modeling, aliasing is defined spectrally and is tied directly to nonlinear activation functions. “Aliasing Reduction in Neural Amp Modeling by Smoothing Activations” states that activation-induced aliasing is a “key limitation” of neural network-based black-box modeling for analog audio hardware, because nonlinear functions generate harmonics that exceed the Nyquist frequency and fold back into the audible band as aliases (Sato et al., 7 May 2025). The paper’s central finding is that “activation functions with smoother curves tend to achieve lower ASR values,” and that this reduction can be achieved “without a substantial increase in ESR.”

Two of the activation forms discussed are

$z_t = E_\phi(o_t)$5

and

$z_t = E_\phi(o_t)$6

The parameter $z_t = E_\phi(o_t)$7 acts as a stretch factor; “larger $z_t = E_\phi(o_t)$8 yields a smoother, more linear function” for CustomTanh, while large $z_t = E_\phi(o_t)$9 in Snake yields more locally linear behavior (Sato et al., 7 May 2025). The paper introduces the Aliasing-to-Signal Ratio as a linear metric for quantifying aliasing energy relative to harmonic energy, and studies “124 activation variants” in a global search (Sato et al., 7 May 2025).

The reported trade-off is explicit. “Smooth, non-gated activations reduce aliasing,” but “smoother, more linear activations decrease aliasing (ASR) but can lead to higher modeling error (ESR)” (Sato et al., 7 May 2025). At one end, “False_CustomTanh_32” yields “ASR of 0.001284” with “ESR = 0.0896”; at the other, the “Best ESR model (True_SELU)” achieves “ESR = 0.0106, but much higher ASR (0.0091)” (Sato et al., 7 May 2025). Spectrum plots reportedly show aliasing harmonics “suppressed by 10–20 dB” for smooth or stretched activations. In this setting, action aliasing is not a policy-level ambiguity but a signal-level distortion in models whose outputs are themselves action-conditioned or behaviorally meaningful.

6. Common mitigation principles, adjacent signal-processing frameworks, and persistent misconceptions

Across these domains, anti-aliasing is not a single recipe. In temporal action localization it is low-pass filtering with dynamically learned cutoff frequencies; in CNN downsampling it is alias-free or artifact-free pooling; in vision transformers it is smoothing of attention maps; in neural audio it is smoothing or stretching activation functions; in VLA models it is inverse dynamics supervision; and in Minecraft reinforcement learning it is persistent human action advice (Jin et al., 2021, Grabinski et al., 2023, Qian et al., 2021, Sato et al., 7 May 2025, Lee et al., 28 May 2026, Frazier et al., 2019). This suggests that the unifying principle is not any particular filter, but the preservation or recovery of distinctions that matter for action.

The broader aliasing literature reinforces the same point. In time-resolved flow data, derivative information “can be used to detect aliasing and to turn the ill-posed problem of removing aliasing from data into a well-posed problem, yielding a prediction of the true spectrum,” while spatial filtering can remove aliasing in convective systems (Karban et al., 2022). In near-field array processing, sub-Nyquist spatial sampling creates “spurious peaks” in the ambiguity function, and recent work introduces “Critical Antenna Elements (CAEs)” and the “Non-Contributive Zone (NCZ)” to analyze how geometry affects the resolution-aliasing trade-off (Sambon et al., 2 Feb 2026). Although these studies are not framed as action aliasing in sequential decision making, they show the same structural pathology: ambiguity emerges when sampling or representation is insufficient for the underlying bandwidth.

Several misconceptions follow from treating aliasing too narrowly. First, action aliasing is not reducible to one architecture class or one learning paradigm; the literature reports it in reinforcement learning, robot manipulation, temporal localization, CNNs, transformers, and neural audio models. Second, anti-aliasing is not always synonymous with aggressive low-pass suppression: both TAL and neural amp modeling explicitly report a trade-off between aliasing reduction and preservation of informative high-frequency content (Jin et al., 2021, Sato et al., 7 May 2025). Third, eliminating classical aliasing may still leave other artifacts; ASAP’s distinction between aliasing and spectral leakage is a concrete example (Grabinski et al., 2023). The contemporary view, therefore, is that action aliasing is best analyzed as a problem of representation sufficiency under sampling and discretization constraints, with mitigation strategies chosen according to where the ambiguity first enters the pipeline.