QuaDreamer: Panoramic Video for Quadruped Robots

• QuaDreamer is a controllable framework that generates high-quality panoramic videos by simulating quadruped dynamics and addressing sensor calibration challenges.
• The system integrates Vertical Jitter Encoding, Scene-Object Controller, and Panoramic Enhancer to manage motion signals and correct wide-FoV distortions in 360° scenes.
• Empirical evaluations demonstrate that QuaDreamer improves metrics like LPIPS, PSNR, and SSIM, enhancing synthetic data quality for downstream multi-object tracking tasks.

Searching arXiv for the specified papers and closely related context. QuaDreamer is a controllable panoramic video generation framework for quadruped robots, introduced as “the first panoramic data generation engine specifically designed for quadruped robots.” Its central motivation is the scarcity of high-quality panoramic training data for quadruped embodied perception, a limitation attributed to inherent kinematic constraints and complex sensor calibration challenges. The system is designed to mimic the motion paradigm of quadruped robots and to generate realistic panoramic videos that can be used as training data for downstream tasks, including panoramic visual perception and multi-object tracking in 360-degree scenes (Wu et al., 4 Aug 2025).

1. Problem Setting and Scope

QuaDreamer is situated at the intersection of embodied AI, panoramic imaging, and controllable video generation. The paper argues that panoramic cameras are suitable for quadruped robots because they capture comprehensive 360-degree environmental data for surrounding perception and interaction with complex environments. At the same time, the development of robust panoramic perception systems is constrained by the lack of sufficiently diverse and high-quality training corpora tailored to quadruped platforms (Wu et al., 4 Aug 2025).

The framework addresses this data bottleneck by synthesizing controllable panoramic videos rather than only enhancing or reconstructing observed imagery. This design choice distinguishes QuaDreamer from panoramic restoration systems whose objective is to recover a high-quality image from degraded optical capture. A plausible implication is that QuaDreamer targets synthetic-data generation as an infrastructure layer for embodied learning, not merely image-quality improvement.

The paper frames the problem around three panorama-specific and platform-specific difficulties. First, quadruped locomotion produces distinctive vertical vibration patterns that must be represented if generated videos are to match robot motion statistics. Second, controllable generation must separate object motion from background jitter in a way compatible with 360-degree scenes. Third, wide-FoV panoramic imagery introduces severe distortions that degrade both local detail and global structural continuity.

2. System Composition

QuaDreamer is organized around three named components: Vertical Jitter Encoding (VJE), the Scene-Object Controller (SOC), and the Panoramic Enhancer (PE) (Wu et al., 4 Aug 2025).

Component	Function	Stated purpose
VJE	Vertical Jitter Encoding	Extracts controllable vertical signals through frequency-domain feature filtering and provides high-quality prompts
SOC	Scene-Object Controller	Manages object motion and boosts background jitter control through the attention mechanism
PE	Panoramic Enhancer	Handles panoramic distortions in wide-FoV video generation

VJE is introduced to capture the unique vertical vibration characteristics exhibited during quadruped locomotion. The paper states that it extracts controllable vertical signals through frequency-domain feature filtering and converts them into prompts suitable for the generator. This places locomotion-induced jitter not as incidental noise but as an explicit control variable.

SOC is proposed to facilitate high-quality panoramic video generation under jitter signal control. Its stated role is to manage object motion and improve background jitter control through the attention mechanism. The paper does not provide, in the supplied material, an explicit cross-attention equation for SOC, but it clearly positions SOC as the control module responsible for scene dynamics.

PE addresses the visual degradation specific to wide-FoV panoramic generation. The paper motivates it by separating two failure modes: recovery of fine local textures and preservation of global structural continuity across the entire 360-degree image, especially near distorted boundary regions. Within the overall system, PE is therefore the quality-restoration and geometry-correction component embedded inside the generator rather than an external post-processing stage.

3. Panoramic Enhancer Architecture

PE is described as a dual-stream module built on an encoder–decoder architecture. Its two streams are a frequency-texture refinement stream and a spatial-structure correction stream, and the paper explicitly states that they operate in phased collaboration rather than as isolated branches (Wu et al., 4 Aug 2025).

The spatial-structure correction stream is implemented with State Space Model (SSM) modules inserted symmetrically in the encoder–decoder hierarchy: before the first downsampling layer and after the final downsampling or at the late decoding stage. Its function is to model long-range geometric dependencies via multi-directional scanning. In panoramic imagery, where equirectangular or wide-FoV projections create discontinuities and edge distortions, this stream is tasked with suppressing distortions by propagating structural information across distant spatial locations. The conceptual targets are continuity of object shapes across distorted regions, consistency of large scene layout, and smooth geometric transitions across panoramic boundaries.

The frequency-texture refinement stream is implemented with residual blocks containing Fast Fourier Convolution (FFC) in the intermediate layers after several downsampling stages. Its purpose is to recover periodic textures, high-frequency details, and fine-grained local appearance while reducing grid artifacts, blur, and resolution-sensitive degradation. The feature partition is explicitly specified as

$25\% \text{ local}, \qquad 75\% \text{ global}.$

This allocation reflects the panoramic setting, where global receptive fields are emphasized because distortions and structural relations have large spatial extent.

The paper describes the interaction between the two streams architecturally. The SSM-enhanced early encoder regularizes features before they enter the frequency-refinement core; after the FFC-based enhancement restores texture and periodic detail, a second SSM injection in the decoder reinforces geometric continuity during reconstruction. The resulting data flow is

$$\text{Input} \rightarrow \text{SSM} \rightarrow \text{Downsampling} \rightarrow \text{FFC residual refinement} \rightarrow \text{Upsampling} \rightarrow \text{SSM} \rightarrow \text{Output}.$$

This assigns early and late stages to global structure and the bottleneck to local detail restoration.

4. Mathematical Formulation and Architectural Specifics

For an encoder feature tensor

$D \in \mathbb{R}^{B \times C \times H \times W},$

the SSM-enhanced output is given as

$$D^{*}[b,c,x,y] = \frac{1}{N} \sum_{d \in \mathrm{scan}} F_{S6}\!\left(S_d(D[b,c,x,y])\right),$$

where $N$ is the number of scan directions, $S_d(\cdot)$ is the directional scanning function, and $F_{S6}(\cdot)$ is the S6 transformation from the SSM formulation (Wu et al., 4 Aug 2025). This equation formalizes the aggregation of multi-directional structural information to produce a geometry-consistent feature representation.

The FFC residual update is defined as

$X_{\text{out}} = X + FFC\bigl(FFC(X_l, X_g)\bigr),$

with

$X = (X_l, X_g), \qquad X_l : X_g = 25\% : 75\%.$

The formulation indicates a residual structure with two stacked FFC operations refining a local/global feature decomposition. The paper interprets this as a mechanism for recovering periodic structure and high-frequency texture while avoiding grid artifacts.

PE is trained jointly with the full latent diffusion system rather than through a dedicated texture or structure loss. The paper explicitly states that no PE-specific texture loss, structural consistency loss, or separately named PE objective is defined. Instead, optimization is governed by the standard diffusion noise-prediction loss

$$\mathcal{L}(\theta) = \mathbb{E}_{t \sim u(1,T),\, \epsilon_t \sim \mathcal{N}(0,I)} \left[ \left\| \epsilon_t - \epsilon_\theta(z_t; t, y) \right\|^2 \right].$$

The forward diffusion process is

$$\text{Input} \rightarrow \text{SSM} \rightarrow \text{Downsampling} \rightarrow \text{FFC residual refinement} \rightarrow \text{Upsampling} \rightarrow \text{SSM} \rightarrow \text{Output}.$$0

The paper also gives the modular interpretation

$$\text{Input} \rightarrow \text{SSM} \rightarrow \text{Downsampling} \rightarrow \text{FFC residual refinement} \rightarrow \text{Upsampling} \rightarrow \text{SSM} \rightarrow \text{Output}.$$1

where $$\text{Input} \rightarrow \text{SSM} \rightarrow \text{Downsampling} \rightarrow \text{FFC residual refinement} \rightarrow \text{Upsampling} \rightarrow \text{SSM} \rightarrow \text{Output}.$$2 denotes the Panoramic Enhancer parameters embedded inside the diffusion backbone.

Architecturally, PE uses an encoder–decoder backbone with an initial feature extraction stage, three successive downsampling stages, intermediate processing layers, and three-stage upsampling reconstruction. The paper does not specify exact numerical channel widths, kernel sizes, head counts, state dimensions, or the numerical scan count $$\text{Input} \rightarrow \text{SSM} \rightarrow \text{Downsampling} \rightarrow \text{FFC residual refinement} \rightarrow \text{Upsampling} \rightarrow \text{SSM} \rightarrow \text{Output}.$$3. It also states that PE itself is not described as transformer-based: attention is discussed elsewhere in QuaDreamer, particularly in SOC, whereas PE relies on SSM modules, FFC residual blocks, and multiscale encoder–decoder processing.

5. Empirical Evaluation

The paper provides both comparison results and ablations that support the contribution of PE to panoramic video quality (Wu et al., 4 Aug 2025).

A comparison particularly relevant to PE is between SVD and QuaDreamer* (the model without control modules). The reported values are:

Method	FVD	LPIPS	PSNR	SSIM
SVD	831.31	0.2711	14.50	0.3874
QuaDreamer*	830.92	0.2669	14.51	0.3893

Because QuaDreamer* excludes control modules, the paper attributes the quality gain in part to the enhanced panoramic modeling, especially PE. The improvement is modest but consistent across LPIPS, PSNR, and SSIM.

For TrackDiffusion versus full QuaDreamer, the paper reports that LPIPS decreases by $$\text{Input} \rightarrow \text{SSM} \rightarrow \text{Downsampling} \rightarrow \text{FFC residual refinement} \rightarrow \text{Upsampling} \rightarrow \text{SSM} \rightarrow \text{Output}.$$4, SSIM increases by $$\text{Input} \rightarrow \text{SSM} \rightarrow \text{Downsampling} \rightarrow \text{FFC residual refinement} \rightarrow \text{Upsampling} \rightarrow \text{SSM} \rightarrow \text{Output}.$$5, and PSNR improves by $$\text{Input} \rightarrow \text{SSM} \rightarrow \text{Downsampling} \rightarrow \text{FFC residual refinement} \rightarrow \text{Upsampling} \rightarrow \text{SSM} \rightarrow \text{Output}.$$6. The numerical results are:

Method	FVD	LPIPS	PSNR	SSIM
TrackDiffusion	887.31	0.2714	14.27	0.3815
QuaDreamer	895.70	0.2614	14.51	0.3947

Although FVD rises slightly, the authors explain this as a side effect of increased control complexity. The perceptual and structural metrics improve, which the paper treats as more direct evidence of image-quality gains.

The most direct quantitative evidence for PE comes from the ablation study. The baseline setting with no SOC and no PE reports

$$\text{Input} \rightarrow \text{SSM} \rightarrow \text{Downsampling} \rightarrow \text{FFC residual refinement} \rightarrow \text{Upsampling} \rightarrow \text{SSM} \rightarrow \text{Output}.$$7

while the PE-only setting reports

$$\text{Input} \rightarrow \text{SSM} \rightarrow \text{Downsampling} \rightarrow \text{FFC residual refinement} \rightarrow \text{Upsampling} \rightarrow \text{SSM} \rightarrow \text{Output}.$$8

The paper emphasizes that introducing PE yields a significant FVD improvement, specifically a decrease of $$\text{Input} \rightarrow \text{SSM} \rightarrow \text{Downsampling} \rightarrow \text{FFC residual refinement} \rightarrow \text{Upsampling} \rightarrow \text{SSM} \rightarrow \text{Output}.$$9 relative to baseline. LPIPS, PSNR, and SSIM also improve. The qualitative findings reported in the paper further state that PE helps resolve geometric distortion and detail degradation in wide-FoV generation and that, in blurry scenes caused by real robot jitter and long exposure, QuaDreamer produces cleaner outputs and reduces blur in generated data.

6. Downstream Use and Relation to Panoramic Enhancement Research

The generated video sequences are not presented only as synthetic demonstrations. The paper states that they can serve as training data for a quadruped robot’s panoramic visual perception model and that doing so enhances the performance of multi-object tracking in 360-degree scenes (Wu et al., 4 Aug 2025). This places QuaDreamer within a data-engineering lineage in which generative models are used to improve downstream embodied perception.

A common misconception would be to treat QuaDreamer solely as a panoramic enhancement network. The paper does not support that characterization. QuaDreamer is a controllable panoramic video generation engine whose PE module is one subsystem inside a broader latent diffusion framework. Another possible misconception is that PE is trained with explicit texture or structural consistency losses; the paper explicitly states that it is not, and that optimization is implicit through the global denoising objective.

In the broader panoramic-imaging literature, QuaDreamer’s PE can be contrasted with the PSF-aware restoration framework “Minimalist and High-Quality Panoramic Imaging with PSF-aware Transformers” (Jiang et al., 2023). That work addresses panoramic image recovery from a minimalist optical system using a dense PSF map and a PSF-aware Aberration-image Recovery Transformer, whereas QuaDreamer’s PE addresses distortions arising in wide-FoV video generation through SSM-based spatial correction and FFC-based frequency refinement. This suggests two distinct panoramic-enhancement paradigms: one grounded in optical degradation inversion and one embedded inside controllable generative video synthesis.

The source code and model weights for QuaDreamer are stated to be publicly available at the project repository associated with the paper. A plausible implication is that the framework is intended not only as a model for one benchmark, but as reusable infrastructure for panoramic synthetic-data generation in quadruped robotics.