Vertical Jitter Encoding (VJE)

• Vertical Jitter Encoding (VJE) is a method that extracts high-frequency vertical vibration signals from quadruped locomotion using frequency-domain filtering.
• It employs a first-order Butterworth high-pass filter and camera-pose encoding to separate jitter from low-frequency motion, ensuring controllable background dynamics in generated videos.
• Integrating VJE within QuaDreamer enhances panoramic video fidelity and improves metrics like PTrack, LPIPS, SSIM, and PSNR for robotic video synthesis.

Searching arXiv for the cited paper to ground the article and confirm metadata. tool call: arxiv_search({"^^^^^^^^2query2 OR \2^^^^^^^^"QuaDreamer: Controllable Panoramic Video Generation for Quadruped Robots\"","max_results":5,"sort_by":"submittedDate","sort_order":"descending"}) Vertical Jitter Encoding (VJE) is a frequency-domain method introduced in "QuaDreamer: Controllable Panoramic Video Generation for Quadruped Robots" for isolating and encoding the characteristic high-frequency vertical vibrations induced by quadruped locomotion into camera-pose features used to condition a latent video diffusion model (&&&^^^^2query2^^^^&&&). Within QuaDreamer, VJE is intended to make quadruped-specific background jitter controllable despite scarce high-quality panoramic training data, while remaining compatible with motion disentanglement through the Scene-Object Controller (SOC) and distortion-aware refinement through the Panoramic Enhancer (PE) (&&&^^^^2query2^^^^&&&).

^^^^2id:(Wu et al., 4 Aug 2025) OR \2^^^^. Problem setting and motivation

QuaDreamer is presented as the first controllable panoramic video generation engine tailored for quadruped robots, motivated by the observation that panoramic cameras capture comprehensive 36^^^^2query2^^^^-degree environmental data but are difficult to support with large, clean, annotated training sets because of kinematic constraints and complex sensor calibration challenges (&&&^^^^2query2^^^^&&&). The paper identifies a specific bottleneck: quadruped robots exhibit unique vertical vibration patterns during locomotion due to gait cycles, body compliance, and ground contact dynamics, and this motion interacts unfavorably with panoramic sensors because wide FoV cameras are inherently sensitive to geometric distortions and stitching or synchronization artifacts (&&&^^^^2query2^^^^&&&).

VJE is introduced to capture these vertical vibration characteristics by extracting controllable vertical signals through frequency-domain feature filtering and converting them into high-quality prompts via a camera encoder (&&&^^^^2query2^^^^&&&). In the formulation given for QuaDreamer, this serves three stated purposes: explicit control of background jitter in generated videos, separation of high-frequency vertical jitter from low-frequency object-relative motion, and enrichment of data generation when real panoramic training data is limited (&&&^^^^2query2^^^^&&&).

The broader system context is equally important. QuaDreamer generates controllable panoramic videos from a single panoramic image, object motion trajectories, and a vibration signal; VJE supplies the vibration-conditioned component of that pipeline (&&&^^^^2query2^^^^&&&). This suggests that VJE is not a standalone motion estimator, but a conditioning mechanism designed for generative control in robot-perspective panoramic video synthesis.

2. Signal definition and frequency-domain filtering

In the current design, VJE derives a vertical motion proxy directly from video rather than from an IMU or proprioceptive sensor stream (&&&^^^^2query2^^^^&&&). The paper states that the vertical pixel coordinate of a persistent foreground entity, described as the data collector and typically a human present in the scene, is tracked as a time series of bounding-box PRESERVED_PLACEHOLDER_^^^^2query2^^^^-coordinates. Formally, the raw signal is

PRESERVED_PLACEHOLDER_^^^^2id:(Wu et al., 4 Aug 2025) OR \2^^^^

VJE decomposes this signal into low-frequency and high-frequency components through spectral analysis, with the high-frequency component interpreted as robot-induced vertical jitter and the low-frequency component interpreted as slow relative displacement between the collector and the robot (&&&^^^^2query2^^^^&&&). Let

$X(\omega)=\mathcal{F}\{x(t)\}.$

The method then applies a first-order Butterworth high-pass filter with cutoff $f_c=0.3$ Hz and order $n=1$, using the frequency response

$H(f)=\frac{(f/f_c)^{2n}}{1+(f/f_c)^{2n}}.$

Filtering in the frequency domain yields

$S(\omega)=H(\omega)\,X(\omega),$

and the inverse transform returns the time-domain signal

$s(t)=\mathcal{F}^{-1}\{S(\omega)\}.$

In the paper’s notation, the vertical jitter signal is

$$y_w(t)=\mathcal{F}^{-1}\left[\mathcal{F}\big(y_{\text{human}}(t)\big)\cdot H(f)\right].$$

The cutoff $0.3$ Hz is reported to separate slow object drift from vertical jitter typical of quadruped locomotion (&&&^^^^2query2^^^^&&&). The text further notes that dominant jitter frequencies align with gait cycles and that trot pacing generally yields higher-frequency vertical oscillation than walk, although the paper does not explicitly parameterize by gait labels (&&&^^^^2query2^^^^&&&). A common control-oriented harmonic parameterization is also described,

PRESERVED_PLACEHOLDER_^^^^2id:(Wu et al., 4 Aug 2025) OR \2query2^^^^

but the paper explicitly states that an explicit harmonic control head is not part of QuaDreamer’s implementation (&&&^^^^2query2^^^^&&&).

3. Pose-feature construction and conditioning representation

After filtering, VJE converts the vertical jitter signal into camera-pose features. For frame PRESERVED_PLACEHOLDER_^^^^2id:(Wu et al., 4 Aug 2025) OR \2id:(Wu et al., 4 Aug 2025) OR \2^^^^, world coordinates are defined as

PRESERVED_PLACEHOLDER_^^^^2id:(Wu et al., 4 Aug 2025) OR \22^^^^

with

PRESERVED_PLACEHOLDER_^^^^2id:(Wu et al., 4 Aug 2025) OR \2^^^^3

Camera intrinsics are written as

PRESERVED_PLACEHOLDER_^^^^2id:(Wu et al., 4 Aug 2025) OR \2^^^^4

and camera coordinates are computed by

PRESERVED_PLACEHOLDER_^^^^2id:(Wu et al., 4 Aug 2025) OR \2^^^^5

The camera pose is then converted into a Plücker embedding PRESERVED_PLACEHOLDER_^^^^2id:(Wu et al., 4 Aug 2025) OR \2^^^^6, and background jitter features are produced with a frozen CameraCtrl encoder:

PRESERVED_PLACEHOLDER_^^^^2id:(Wu et al., 4 Aug 2025) OR \2^^^^7

These features are the “high-quality prompts” supplied by VJE for subsequent conditioning (&&&^^^^2query2^^^^&&&).

The paper distinguishes two conditioning spaces. In geometric space, VJE supports position-aware diffusion through camera-pose features; in latent space, it supports attention-based integration of control signals (&&&^^^^2query2^^^^&&&). This separation is central to the stated controllability objective, because the filtered signal is not injected merely as a scalar temporal curve but as a pose-conditioned representation consumable by the generative backbone.

The paper also provides a camera-pose description under jitter:

PRESERVED_PLACEHOLDER_^^^^2id:(Wu et al., 4 Aug 2025) OR \2^^^^8

where PRESERVED_PLACEHOLDER_^^^^2id:(Wu et al., 4 Aug 2025) OR \2^^^^9 is the VJE signal, typically injected as vertical translation (&&&^^^^2query2^^^^&&&). For equirectangular projection of spherical coordinates $X(\omega)=\mathcal{F}\{x(t)\}.$^^^^2query2^^^^,

$X(\omega)=\mathcal{F}\{x(t)\}.$^^^^2id:(Wu et al., 4 Aug 2025) OR \2^^^^

The paper states that vertical jitter alters the mapping of scene rays to $X(\omega)=\mathcal{F}\{x(t)\}.$2, inducing systematic vertical shifts and distortions that PE compensates via SSM and FFC (&&&^^^^2query2^^^^&&&).

4. Interaction with SOC, PE, and the latent diffusion backbone

QuaDreamer uses a Latent Diffusion Model with the standard forward and training formulations

$X(\omega)=\mathcal{F}\{x(t)\}.$3

and

$X(\omega)=\mathcal{F}\{x(t)\}.$4

where $X(\omega)=\mathcal{F}\{x(t)\}.$5 contains conditioning signals including VJE-derived pose features (&&&^^^^2query2^^^^&&&). The paper states that no extra frequency-domain losses are introduced; controllability is instead validated at evaluation time with the proposed PTrack metric (&&&^^^^2query2^^^^&&&).

Within SOC, motion is decomposed into a background motion field and an object motion field (&&&^^^^2query2^^^^&&&). For the background branch, temporal attention over $X(\omega)=\mathcal{F}\{x(t)\}.$6 captures dependencies of the camera trajectory and is added pixel-wise to image latents to strengthen background control, while 3D convolutions produce spatiotemporal features

$X(\omega)=\mathcal{F}\{x(t)\}.$7

For the object branch, bounding boxes are Fourier-embedded as

$X(\omega)=\mathcal{F}\{x(t)\}.$8

with visibility-mask gating

$X(\omega)=\mathcal{F}\{x(t)\}.$9

where $f_c=0.3$^^^^2query2^^^^ is a learnable zero embedding for occlusions (&&&^^^^2query2^^^^&&&). Fusion is written as

$f_c=0.3$^^^^2id:(Wu et al., 4 Aug 2025) OR \2^^^^

and a gated self-attention mechanism integrates $f_c=0.3$2 with input visual features (&&&^^^^2query2^^^^&&&). The ablation statement is explicit: removing the fusion with $f_c=0.3$3, that is, not using VJE features, causes loss of jitter controllability (&&&^^^^2query2^^^^&&&).

PE addresses panoramic distortions and detail fidelity through a dual-stream architecture (&&&^^^^2query2^^^^&&&). Spatial structure correction uses State Space Models, injected symmetrically before the first downsampling and after the last upsampling:

$f_c=0.3$4

Frequency-texture refinement uses Fast Fourier Convolution residual blocks in intermediate layers:

$f_c=0.3$5

with $f_c=0.3$6 split at a $f_c=0.3$7 local/global ratio (&&&^^^^2query2^^^^&&&). In this architecture, VJE supplies the controllable jitter signal, SOC disentangles background and object motion, and PE compensates for the panoramic distortions that such jitter can exacerbate.

5. Training protocol, controllability, and reported results

The training objective includes VJE-derived background jitter features $f_c=0.3$8 via PoseEncoder and SOC’s fused background-object features $f_c=0.3$9 via gated attention (&&&^^^^2query2^^^^&&&). During training, gradients propagate through the SOC fusion and attention mechanisms to learn how to express jitter controllably in the generated panoramas, while the PoseEncoder remains frozen to reduce training complexity (&&&^^^^2query2^^^^&&&). The implementation details reported for the full system are: Stable Video Diffusion as the base model, training for 32^^^^2query2^^^^ epochs (2^^^^2query2query2^^^^k steps) on a single A6^^^^2query2query2query2^^^^ 48G, batch size 2, FP^^^^2id:(Wu et al., 4 Aug 2025) OR \2^^^^6, learning rate $n=1$^^^^2query2^^^^ with 75^^^^2query2^^^^-step warm-up, and DPM-Solver with 3^^^^2query2^^^^ sampling steps (&&&^^^^2query2^^^^&&&).

To quantify controlled jitter fidelity against real videos, the paper defines PTrack using CoTracker3 (&&&^^^^2query2^^^^&&&). For tracked trajectories $n=1$^^^^2id:(Wu et al., 4 Aug 2025) OR \2^^^^, temporal variances are

$n=1$2

A jitter-only subset is then selected by percentile thresholding:

$n=1$3

The final score is

$n=1$4

with

$n=1$5

The reported results on QuadTrack are summarized below (&&&^^^^2query2^^^^&&&).

Setting	Reported result
QuaDreamer vs. TrackDiffusion	LPIPS reduced by 3.68%, SSIM increased by 3.46%, PSNR improved by ^^^^2id:(Wu et al., 4 Aug 2025) OR \2^^^^.68%
QuaDreamer vs. TrackDiffusion	PTrack decreased from 25.2496 to ^^^^2id:(Wu et al., 4 Aug 2025) OR \2^^^^4.^^^^2id:(Wu et al., 4 Aug 2025) OR \2^^^^744
OmniTrack + QuaDreamer sequences vs. “Real Only”	HOTA improved by ^^^^2id:(Wu et al., 4 Aug 2025) OR \2query2^^^^.^^^^2id:(Wu et al., 4 Aug 2025) OR \2^^^^%, MOTA improved by ^^^^2id:(Wu et al., 4 Aug 2025) OR \2^^^^4.8%
Adding SOC	MOTA increased by 6.74%, PTrack decreased by 36.76%
Adding PE	FVD reduced by 6.36%
With VJE filtering vs. without filtering	PTrack improved from ^^^^2id:(Wu et al., 4 Aug 2025) OR \2^^^^7.436^^^^2id:(Wu et al., 4 Aug 2025) OR \2^^^^ to ^^^^2id:(Wu et al., 4 Aug 2025) OR \2^^^^4.^^^^2id:(Wu et al., 4 Aug 2025) OR \2^^^^744

These figures are used in the paper to support three linked claims: VJE improves jitter controllability, the full QuaDreamer system improves image and video quality, and generated sequences can serve as training data for the quadruped robot’s panoramic visual perception model, specifically enhancing multi-object tracking in 36^^^^2query2^^^^-degree scenes (&&&^^^^2query2^^^^&&&).

6. Scope, constraints, and common points of confusion

A frequent point of confusion concerns the source of control signals. In the present system, VJE does not use IMU-derived posture signals; it uses video-derived motion from the vertical coordinates of the collector’s bounding box, while future work proposes using IMU to obtain robot posture and broader motion control (&&&^^^^2query2^^^^&&&). Another point of confusion concerns the nature of control: QuaDreamer allows users to adjust the jitter signal, for example by amplitude scaling or cutoff selection, before encoding, but an explicit harmonic control head is not part of the implementation (&&&^^^^2query2^^^^&&&).

The paper also states several constraints. Extreme gait speeds or off-nominal terrains may shift jitter spectra; very low sampling rates or aliasing could reduce controllability; and panoramic boundary regions remain harder to control because of distortion discontinuity (&&&^^^^2query2^^^^&&&). Reported failure modes include over-smoothing of jitter at extreme speeds, artifacts near panorama seams, and mismatch under unusual gaits or erratic body dynamics (&&&^^^^2query2^^^^&&&). PE alleviates boundary-related issues but does not fully eliminate them (&&&^^^^2query2^^^^&&&).

The listed extensions are correspondingly narrow and concrete: incorporate IMU-based posture signals such as pitch, roll, and vertical components; broaden to non-vertical motion components; adapt to other robot morphologies such as biped and hexapod systems; and support non-panoramic cameras or RGB-X inputs including depth, IR, and event modalities (&&&^^^^2query2^^^^&&&). This suggests that VJE, as currently formulated, should be understood as a quadruped-oriented vertical-motion conditioning mechanism rather than a general-purpose full-body robot dynamics representation.