Vertical Jitter Encoding (VJE)

• Vertical Jitter Encoding (VJE) is a method that extracts high-frequency vertical vibration signals from quadruped locomotion using frequency-domain filtering.
• It employs a first-order Butterworth high-pass filter and camera-pose encoding to separate jitter from low-frequency motion, ensuring controllable background dynamics in generated videos.
• Integrating VJE within QuaDreamer enhances panoramic video fidelity and improves metrics like PTrack, LPIPS, SSIM, and PSNR for robotic video synthesis.

Searching arXiv for the cited paper to ground the article and confirm metadata. tool call: arxiv_search({"query":"id:(Wu et al., 4 Aug 2025) OR \"QuaDreamer: Controllable Panoramic Video Generation for Quadruped Robots\"","max_results":5,"sort_by":"submittedDate","sort_order":"descending"}) Vertical Jitter Encoding (VJE) is a frequency-domain method introduced in "QuaDreamer: Controllable Panoramic Video Generation for Quadruped Robots" for isolating and encoding the characteristic high-frequency vertical vibrations induced by quadruped locomotion into camera-pose features used to condition a latent video diffusion model (Wu et al., 4 Aug 2025). Within QuaDreamer, VJE is intended to make quadruped-specific background jitter controllable despite scarce high-quality panoramic training data, while remaining compatible with motion disentanglement through the Scene-Object Controller (SOC) and distortion-aware refinement through the Panoramic Enhancer (PE) (Wu et al., 4 Aug 2025).

1. Problem setting and motivation

QuaDreamer is presented as the first controllable panoramic video generation engine tailored for quadruped robots, motivated by the observation that panoramic cameras capture comprehensive 360-degree environmental data but are difficult to support with large, clean, annotated training sets because of kinematic constraints and complex sensor calibration challenges (Wu et al., 4 Aug 2025). The paper identifies a specific bottleneck: quadruped robots exhibit unique vertical vibration patterns during locomotion due to gait cycles, body compliance, and ground contact dynamics, and this motion interacts unfavorably with panoramic sensors because wide FoV cameras are inherently sensitive to geometric distortions and stitching or synchronization artifacts (Wu et al., 4 Aug 2025).

VJE is introduced to capture these vertical vibration characteristics by extracting controllable vertical signals through frequency-domain feature filtering and converting them into high-quality prompts via a camera encoder (Wu et al., 4 Aug 2025). In the formulation given for QuaDreamer, this serves three stated purposes: explicit control of background jitter in generated videos, separation of high-frequency vertical jitter from low-frequency object-relative motion, and enrichment of data generation when real panoramic training data is limited (Wu et al., 4 Aug 2025).

The broader system context is equally important. QuaDreamer generates controllable panoramic videos from a single panoramic image, object motion trajectories, and a vibration signal; VJE supplies the vibration-conditioned component of that pipeline (Wu et al., 4 Aug 2025). This suggests that VJE is not a standalone motion estimator, but a conditioning mechanism designed for generative control in robot-perspective panoramic video synthesis.

2. Signal definition and frequency-domain filtering

In the current design, VJE derives a vertical motion proxy directly from video rather than from an IMU or proprioceptive sensor stream (Wu et al., 4 Aug 2025). The paper states that the vertical pixel coordinate of a persistent foreground entity, described as the data collector and typically a human present in the scene, is tracked as a time series of bounding-box $y$-coordinates. Formally, the raw signal is

$x(t) \equiv y_{\text{human}}(t).$

VJE decomposes this signal into low-frequency and high-frequency components through spectral analysis, with the high-frequency component interpreted as robot-induced vertical jitter and the low-frequency component interpreted as slow relative displacement between the collector and the robot (Wu et al., 4 Aug 2025). Let

$X(\omega)=\mathcal{F}\{x(t)\}.$

The method then applies a first-order Butterworth high-pass filter with cutoff $f_c=0.3$ Hz and order $n=1$, using the frequency response

$H(f)=\frac{(f/f_c)^{2n}}{1+(f/f_c)^{2n}}.$

Filtering in the frequency domain yields

$S(\omega)=H(\omega)\,X(\omega),$

and the inverse transform returns the time-domain signal

$s(t)=\mathcal{F}^{-1}\{S(\omega)\}.$

In the paper’s notation, the vertical jitter signal is

$$y_w(t)=\mathcal{F}^{-1}\left[\mathcal{F}\big(y_{\text{human}}(t)\big)\cdot H(f)\right].$$

The cutoff $0.3$ Hz is reported to separate slow object drift from vertical jitter typical of quadruped locomotion (Wu et al., 4 Aug 2025). The text further notes that dominant jitter frequencies align with gait cycles and that trot pacing generally yields higher-frequency vertical oscillation than walk, although the paper does not explicitly parameterize by gait labels (Wu et al., 4 Aug 2025). A common control-oriented harmonic parameterization is also described,

$x(t) \equiv y_{\text{human}}(t).$0

but the paper explicitly states that an explicit harmonic control head is not part of QuaDreamer’s implementation (Wu et al., 4 Aug 2025).

3. Pose-feature construction and conditioning representation

After filtering, VJE converts the vertical jitter signal into camera-pose features. For frame $x(t) \equiv y_{\text{human}}(t).$1, world coordinates are defined as

$x(t) \equiv y_{\text{human}}(t).$2

with

$x(t) \equiv y_{\text{human}}(t).$3

Camera intrinsics are written as

$x(t) \equiv y_{\text{human}}(t).$4

and camera coordinates are computed by

$x(t) \equiv y_{\text{human}}(t).$5

The camera pose is then converted into a Plücker embedding $x(t) \equiv y_{\text{human}}(t).$6, and background jitter features are produced with a frozen CameraCtrl encoder:

$x(t) \equiv y_{\text{human}}(t).$7

These features are the “high-quality prompts” supplied by VJE for subsequent conditioning (Wu et al., 4 Aug 2025).

The paper distinguishes two conditioning spaces. In geometric space, VJE supports position-aware diffusion through camera-pose features; in latent space, it supports attention-based integration of control signals (Wu et al., 4 Aug 2025). This separation is central to the stated controllability objective, because the filtered signal is not injected merely as a scalar temporal curve but as a pose-conditioned representation consumable by the generative backbone.

The paper also provides a camera-pose description under jitter:

$x(t) \equiv y_{\text{human}}(t).$8

where $x(t) \equiv y_{\text{human}}(t).$9 is the VJE signal, typically injected as vertical translation (Wu et al., 4 Aug 2025). For equirectangular projection of spherical coordinates $X(\omega)=\mathcal{F}\{x(t)\}.$0,

$X(\omega)=\mathcal{F}\{x(t)\}.$1

The paper states that vertical jitter alters the mapping of scene rays to $X(\omega)=\mathcal{F}\{x(t)\}.$2, inducing systematic vertical shifts and distortions that PE compensates via SSM and FFC (Wu et al., 4 Aug 2025).

4. Interaction with SOC, PE, and the latent diffusion backbone

QuaDreamer uses a Latent Diffusion Model with the standard forward and training formulations

$X(\omega)=\mathcal{F}\{x(t)\}.$3

and

$X(\omega)=\mathcal{F}\{x(t)\}.$4

where $X(\omega)=\mathcal{F}\{x(t)\}.$5 contains conditioning signals including VJE-derived pose features (Wu et al., 4 Aug 2025). The paper states that no extra frequency-domain losses are introduced; controllability is instead validated at evaluation time with the proposed PTrack metric (Wu et al., 4 Aug 2025).

Within SOC, motion is decomposed into a background motion field and an object motion field (Wu et al., 4 Aug 2025). For the background branch, temporal attention over $X(\omega)=\mathcal{F}\{x(t)\}.$6 captures dependencies of the camera trajectory and is added pixel-wise to image latents to strengthen background control, while 3D convolutions produce spatiotemporal features

$X(\omega)=\mathcal{F}\{x(t)\}.$7

For the object branch, bounding boxes are Fourier-embedded as

$X(\omega)=\mathcal{F}\{x(t)\}.$8

with visibility-mask gating

$X(\omega)=\mathcal{F}\{x(t)\}.$9

where $f_c=0.3$0 is a learnable zero embedding for occlusions (Wu et al., 4 Aug 2025). Fusion is written as

$f_c=0.3$1

and a gated self-attention mechanism integrates $f_c=0.3$2 with input visual features (Wu et al., 4 Aug 2025). The ablation statement is explicit: removing the fusion with $f_c=0.3$3, that is, not using VJE features, causes loss of jitter controllability (Wu et al., 4 Aug 2025).

PE addresses panoramic distortions and detail fidelity through a dual-stream architecture (Wu et al., 4 Aug 2025). Spatial structure correction uses State Space Models, injected symmetrically before the first downsampling and after the last upsampling:

$f_c=0.3$4

Frequency-texture refinement uses Fast Fourier Convolution residual blocks in intermediate layers:

$f_c=0.3$5

with $f_c=0.3$6 split at a $f_c=0.3$7 local/global ratio (Wu et al., 4 Aug 2025). In this architecture, VJE supplies the controllable jitter signal, SOC disentangles background and object motion, and PE compensates for the panoramic distortions that such jitter can exacerbate.

5. Training protocol, controllability, and reported results

The training objective includes VJE-derived background jitter features $f_c=0.3$8 via PoseEncoder and SOC’s fused background-object features $f_c=0.3$9 via gated attention (Wu et al., 4 Aug 2025). During training, gradients propagate through the SOC fusion and attention mechanisms to learn how to express jitter controllably in the generated panoramas, while the PoseEncoder remains frozen to reduce training complexity (Wu et al., 4 Aug 2025). The implementation details reported for the full system are: Stable Video Diffusion as the base model, training for 320 epochs (200k steps) on a single A6000 48G, batch size 2, FP16, learning rate $n=1$0 with 750-step warm-up, and DPM-Solver with 30 sampling steps (Wu et al., 4 Aug 2025).

To quantify controlled jitter fidelity against real videos, the paper defines PTrack using CoTracker3 (Wu et al., 4 Aug 2025). For tracked trajectories $n=1$1, temporal variances are

$n=1$2

A jitter-only subset is then selected by percentile thresholding:

$n=1$3

The final score is

$n=1$4

with

$n=1$5

The reported results on QuadTrack are summarized below (Wu et al., 4 Aug 2025).

Setting	Reported result
QuaDreamer vs. TrackDiffusion	LPIPS reduced by 3.68%, SSIM increased by 3.46%, PSNR improved by 1.68%
QuaDreamer vs. TrackDiffusion	PTrack decreased from 25.2496 to 14.1744
OmniTrack + QuaDreamer sequences vs. “Real Only”	HOTA improved by 10.1%, MOTA improved by 14.8%
Adding SOC	MOTA increased by 6.74%, PTrack decreased by 36.76%
Adding PE	FVD reduced by 6.36%
With VJE filtering vs. without filtering	PTrack improved from 17.4361 to 14.1744

These figures are used in the paper to support three linked claims: VJE improves jitter controllability, the full QuaDreamer system improves image and video quality, and generated sequences can serve as training data for the quadruped robot’s panoramic visual perception model, specifically enhancing multi-object tracking in 360-degree scenes (Wu et al., 4 Aug 2025).

6. Scope, constraints, and common points of confusion

A frequent point of confusion concerns the source of control signals. In the present system, VJE does not use IMU-derived posture signals; it uses video-derived motion from the vertical coordinates of the collector’s bounding box, while future work proposes using IMU to obtain robot posture and broader motion control (Wu et al., 4 Aug 2025). Another point of confusion concerns the nature of control: QuaDreamer allows users to adjust the jitter signal, for example by amplitude scaling or cutoff selection, before encoding, but an explicit harmonic control head is not part of the implementation (Wu et al., 4 Aug 2025).

The paper also states several constraints. Extreme gait speeds or off-nominal terrains may shift jitter spectra; very low sampling rates or aliasing could reduce controllability; and panoramic boundary regions remain harder to control because of distortion discontinuity (Wu et al., 4 Aug 2025). Reported failure modes include over-smoothing of jitter at extreme speeds, artifacts near panorama seams, and mismatch under unusual gaits or erratic body dynamics (Wu et al., 4 Aug 2025). PE alleviates boundary-related issues but does not fully eliminate them (Wu et al., 4 Aug 2025).

The listed extensions are correspondingly narrow and concrete: incorporate IMU-based posture signals such as pitch, roll, and vertical components; broaden to non-vertical motion components; adapt to other robot morphologies such as biped and hexapod systems; and support non-panoramic cameras or RGB-X inputs including depth, IR, and event modalities (Wu et al., 4 Aug 2025). This suggests that VJE, as currently formulated, should be understood as a quadruped-oriented vertical-motion conditioning mechanism rather than a general-purpose full-body robot dynamics representation.