---
title: 'PanFlow: Decoupled Panoramic Video Generation'
url: https://www.emergentmind.com/topics/panflow
type: topic
---

# PanFlow: Decoupled Panoramic Video Generation

Searching arXiv for the PanFlow paper and a related base model mentioned in the provided data.
PanFlow is a panoramic video generation method designed for motion-controllable synthesis in equirectangular \(360^\circ\) video. It addresses two central difficulties in this setting: the entanglement of large camera rotations with residual scene motion, and the requirement of loop consistency at panorama seams and poles. The method decomposes the input optical flow into a rotational component induced by spherical camera motion and a derotated residual flow, conditions a diffusion model on the residual component, and restores the full motion by re-applying the accumulated rotation. It also introduces spherical noise warping to preserve motion continuity across panorama boundaries, and is trained on a motion-rich panoramic video dataset with frame-level pose and flow annotations. The approach is presented in "PanFlow: Decoupled Motion Control for Panoramic Video Generation" [2512.00832].

## 1. Problem Setting and Motivation

Panoramic video generation differs from standard perspective video synthesis because the camera’s field of view wraps around and requires exact loop consistency at the left/right seam and at the poles. Realistic dynamics in this domain arise from a mixture of large camera rotations, camera translations, and object motions, rather than from local motion alone [2512.00832].

The method is motivated by three limitations identified for prior diffusion-based panoramic image and video generators. First, loop inconsistencies occur when motion near the panorama boundary causes visible jumps as content reappears on the opposite side. Second, motion is entangled: perspective models treat optical flow, including both camera and object motion, as a single conditioning signal, so large camera rotations can overwhelm the denoiser and lead to blurred or misaligned motion. Third, explicit motion control is limited: methods such as 360DVD or DynamicScaler may accept a coarse panning directive but cannot faithfully reproduce complex user-provided flow fields [2512.00832].

PanFlow addresses these issues by decoupling analytically the camera-rotation component in the spherical domain from the residual motion, and by warping diffusion noise on the sphere so that the model respects motion loops across seams. This suggests a division of labor in which analytically tractable global rotation is handled outside the denoiser, while learned generation is concentrated on residual motion and appearance preservation.

## 2. Spherical Motion Decoupling

The method takes as input an equirectangular panorama \(I_0\), a target optical flow sequence \(\{\mathbf f_t\}\), and optionally a text prompt, and produces panorama frames \(\{I_t\}\) that follow the specified motion, maintain loop consistency, and preserve photorealistic appearance [2512.00832].

Its first stage is motion decoupling. A \(360^\circ\) panorama is modeled as an equirectangular projection of the unit sphere, with pixel coordinates parameterized by latitude \(\phi \in \left[-\tfrac{\pi}{2}, \tfrac{\pi}{2}\right]\) and longitude \(\theta \in [-\pi,\pi]\). A pure camera rotation \(\mathbf R \in SO(3)\) induces a spherical rotation flow
$$
\mathbf f_r(\theta,\phi)=\Pi\bigl(\mathbf R\,\mathbf x(\theta,\phi)\bigr)-\Pi\bigl(\mathbf x(\theta,\phi)\bigr),
$$
where \(\Pi(\cdot)\) reprojects a 3D direction onto image coordinates. In implementation, this is expressed analytically on the sphere as
$$
\mathbf f_r(\mathbf x)=\mathbf R^\top \mathbf x-\mathbf x,\qquad \mathbf x\in\mathbb S^2,
$$
followed by conversion of the 3D offset to a 2D displacement in image space via the equirectangular Jacobian [2512.00832].

Given an observed or user-provided optical flow field \(\mathbf f\), the method extracts the rotation matrix \(\mathbf R\), for example from SLAM poses or by accumulating differential rotations, and computes the derotated residual flow as
$$
\mathbf f_d=\mathbf f-\mathbf f_r.
$$
Conditioning the diffusion model on \(\mathbf f_d\) instead of \(\mathbf f\) removes large content-agnostic rotational motion from the network’s input. The paper states that this reduces the network’s motion modeling burden to translation plus object motion only [2512.00832].

After denoising, full motion is recovered by re-rotating the generated derotated frame \(\tilde I_t\) according to the cumulative rotation \(\mathbf R_t\):
$$
I_t(\mathbf x)=\text{Bilinear}\bigl(\tilde I_t,\mathbf R_t^\top \mathbf x\bigr).
$$
This restores the original camera dynamics after residual motion has been synthesized in the derotated frame.

## 3. Spherical Noise Warping and Loop Consistency

The second major component is spherical noise warping. Rather than embedding the full flow field directly, PanFlow warps Gaussian latent noise according to the derotated flow and uses the warped noise as the motion condition for diffusion. For panoramas, this operation must propagate noise coherently across the left/right seam and at the poles [2512.00832].

Let \(q_{t-1}\in \mathbb R^{H\times W\times C}\) denote the previous latent noise map, and let \(\mathbf f_d\) denote the derotated 2D flow at the latent resolution. The next noise map is defined by
$$
q_t(i,j)=q_{t-1}\bigl(\hat i(i,j),\hat j(i,j)\bigr),
$$
where the wrapped coordinates are
$$
\hat i(i,j)=
\begin{cases}
-i, & i<0,\\[3pt]
i, & 0\le i<H,\\[3pt]
2(H-1)-i, & i\ge H,
\end{cases}
\qquad
\hat j(i,j)=
\begin{cases}
(j+\tfrac W2)\bmod W, & i<0 \text{ or } i\ge H,\\[3pt]
j\bmod W, & 0\le i<H.
\end{cases}
$$
This operation folds coordinates that move beyond the top or bottom edges back by reflection and wraps horizontal coordinates by modulo. In the terminology of the paper, this enforces pole consistency and seam consistency directly in the latent motion-conditioning process [2512.00832].

The significance of this construction is that loop consistency is not treated purely as a post hoc correction. Instead, the diffusion transformer is trained on latents that already satisfy spherical boundary conditions, so the generator learns motion trajectories that close across seams. A plausible implication is that this makes panoramic continuity a property of the learned generation process rather than an after-the-fact stitching heuristic.

## 4. Architecture and Objective Functions

PanFlow builds on CogVideoX-5B-I2V, described in the paper as an image-conditioned video diffusion transformer. Its structure includes a pretrained 3D-VAE with encoder \(\mathcal E\) and decoder \(\mathcal D\), mapping each frame to a latent grid of spatial tokens with \((H{=}8,\;W{=}90)\) and temporal depth of 49 frames, and a transformer-based denoiser \(\epsilon_\theta\) that receives the noisy latent sequence \(z_t\), warped-noise condition \(q_t\), optional text embeddings from CLIP, and optional image embeddings from the first frame [2512.00832].

Two modifications are introduced to adapt the model to panoramas. First, LoRA adapters of rank 128 are inserted on all self- and cross-attention layers, enabling efficient fine-tuning in 2 days on eight A100s without destroying the pretrained weights. Second, circular padding is applied inside \(\mathcal E\) and \(\mathcal D\): every 2D convolution over the \(W\) dimension is padded by wrapping columns from the opposite side, which guarantees pixel-space continuity at the seam [2512.00832].

The training procedure also includes latent rotation. During each diffusion step, the entire latent grid is randomly rotated by \(\theta=40^\circ\) longitudinally before tokenization and rotated back afterward. The stated purpose is to expose the network to seams at different horizontal positions and reduce seam artifacts [2512.00832].

Training uses the standard denoising diffusion objective together with seam and pole consistency regularizers. The denoising term is
$$
L_{\rm DDPM}=\mathbb E_{t,z_0,\epsilon}\Bigl\|\epsilon-\epsilon_\theta\bigl(\sqrt{\bar\alpha_t}z_0+\sqrt{1-\bar\alpha_t}\epsilon,\; t,\; q_t\bigr)\Bigr\|^2.
$$
The seam-consistency loss is
$$
L_{\rm seam}=\mathbb E_z\bigl\|\mathcal D(z)_{:,0}-\mathcal D(z)_{:,W-1}\bigr\|_1,
$$
and the pole-consistency loss is
$$
L_{\rm pole}=\mathbb E_z\bigl\|\mathcal D(z)_{0,:}-\mathcal D(z)_{H-1,:}\bigr\|_1.
$$
The total loss is
$$
L=L_{\rm DDPM}+\lambda_{\rm seam}L_{\rm seam}+\lambda_{\rm pole}L_{\rm pole},
$$
with \(\lambda_{\rm seam}=\lambda_{\rm pole}=0.1\) in practice [2512.00832].

## 5. Dataset Curation and Training Data

To support training, the work curates a large-scale panoramic video dataset from 360-1M. The final dataset contains **150 000 clips** and is explicitly described as motion-rich, with frame-level camera poses \(\{\mathbf R_t\}\) and optical flows \(\{\mathbf f_t\}\) [2512.00832].

The curation pipeline consists of several filtering and preprocessing stages:

| Step | Procedure | Purpose stated in the data |
|---|---|---|
| 1 | Format check via SSIM across left/right halves and circular masks | Remove stereo and fisheye clips |
| 2 | Scene segmentation using PySceneDetect + OpenVSLAM tracking-lost signals | Cut soft fades |
| 3 | Pose extraction with OpenVSLAM | Obtain frame-level \(\mathbf R_t\) |
| 4 | Watermark detection with LAION-5B-WatermarkDetection | Drop high-watermark clips |
| 5 | Flow filtering via Farneback optical flow | Discard low-magnitude clips \((<2\) pixels) |

The resulting clips have lengths of 3–10 s [2512.00832]. This training corpus is central to the method because the proposed decomposition requires frame-level pose and flow information. A plausible implication is that the dataset is not only a source of supervision for synthesis quality, but also a structural prerequisite for learning motion-conditioned panoramic generation under explicit rotational decoupling.

## 6. Empirical Performance and Applications

The reported evaluation uses a held-out 100-clip test set from 360-1M and a 100-clip subset of WEB360. PanFlow is compared against 360DVD, DynamicScaler, HoloTime, MotionClone, and Go-With-The-Flow on Fréchet Video Distance, Fréchet Image Distance, CLIP-similarity to prompt, optical-flow EPE to ground truth, LPIPS, PSNR, SSIM, and Q-Align human-style aesthetic and video quality [2512.00832].

On 360-1M, the paper gives the example that PanFlow achieves **FVD = 298** versus **477** for Go-With-The-Flow, **EPE = 2.01** versus **3.30**, and **SSIM = 0.723** versus **0.622** [2512.00832]. Qualitatively, the method is reported to produce sharper textures on ground, sky, and architectural details; perfect seam closure without jumps at \(0^\circ/360^\circ\); accurate large panning, including \(120^\circ\) panning in 3 s; and better geometry, with buildings and horizons remaining straight under roll [2512.00832].

Ablation studies isolate the contribution of the main components. Removing decoupling raises EPE by approximately \(30\%\). Disabling spherical warping doubles the seam MSE. Disabling LoRA fine-tuning raises FID from 23 to 40 [2512.00832]. These results support the paper’s claim that motion decoupling, spherical warping, and panoramic adaptation of the backbone each contribute materially to performance.

The paper also presents two application settings. In **motion transfer**, a source panorama video provides \(\{\mathbf f_t^{\rm src}\}\), and a target static panorama \(I_0^{\rm tgt}\) supplies appearance; the output preserves the appearance or genus of the target while following the exact motion of the source. In **video editing**, an edited first frame \(I_0'\), created via text-driven inpainting or external image models, is combined with the original or specified flow sequence to generate an edited \(360^\circ\) video retaining the same motion pattern [2512.00832].

## 7. Limitations, Interpretation, and Future Directions

The paper identifies three principal strengths: fine-grained control over arbitrary flow fields including large rotations, exact loop consistency enforced at training time through spherical noise warping and padding, and a simple analytic decoupling of rotation flow that reduces network complexity [2512.00832].

It also states several limitations. The method relies on accurate rotation estimation, whether from SLAM or flow accumulation, and performance may degrade under very low parallax or textureless scenes. The current resolution is \(480\times 960\), and scaling to \(4\text{K}\) is described as nontrivial [2512.00832].

The future directions listed in the paper are tightly aligned with these limitations. They include learning a self-supervised derotation network to predict \(\mathbf R\) directly from a single panorama and thereby remove the SLAM dependency; extending the method to higher resolution, specifically \(4\text{K}\times 2\text{K}\), and longer duration videos exceeding 10 s; and jointly modeling depth and optical flow for more accurate 3D-consistent editing [2512.00832].

Within the broader landscape of panoramic generation, PanFlow can therefore be understood as a method that redefines motion control in spherical video as a partially analytic problem rather than a wholly learned one. This suggests a methodological trend toward hybrid systems in which geometric factors with known structure, such as camera rotation on \(\mathbb S^2\), are separated from the residual components delegated to diffusion-based synthesis.

Source: https://www.emergentmind.com/topics/panflow