WorldWeaver: Long-Horizon Video Generation

• The paper introduces a unified approach for long-sequence video generation by jointly modeling RGB frames and perceptual cues such as depth and optical flow to mitigate temporal drift.
• WorldWeaver employs a shared spatio-temporal latent representation with segmented noise scheduling and a memory bank to maintain structural consistency over extended sequences.
• Experimental results demonstrate that integrating depth as a stable signal reduces drift significantly (from 0.15 to 0.07), improving both video fidelity and motion consistency.

WorldWeaver is a framework for long video generation that jointly models RGB frames and perceptual conditions within a unified long-horizon modeling scheme. It is introduced to address a central failure mode of long-sequence generative video modeling: structural and temporal inconsistency caused by accumulated errors when models rely predominantly on RGB signals. The framework combines RGB, depth, and optical flow in a shared latent representation, uses a memory bank in which depth is treated as more resistant to drift than RGB, and applies segmented noise scheduling for training prediction groups. The reported result is reduced temporal drift and improved video fidelity in both diffusion- and rectified flow-based models (Liu et al., 21 Aug 2025).

1. Problem setting and conceptual motivation

The motivating problem is that generative video modeling has made significant strides, yet ensuring structural and temporal consistency over long sequences remains a challenge. Current methods are described as predominantly relying on RGB signals, which leads to accumulated errors in object structure and motion over extended durations. WorldWeaver addresses this by jointly predicting perceptual conditions and color information from a unified representation, with the stated effect of significantly enhancing temporal consistency and motion dynamics (Liu et al., 21 Aug 2025).

The framework is organized around a specific intuition: if a model is trained to reproduce both color and geometric or motion cues, it is forced to learn stable scene structure through depth and temporal dynamics through flow. The paper states that this dramatically reduces drift over very long sequences. A plausible implication is that WorldWeaver treats long-horizon generation less as isolated frame synthesis and more as structured world-state evolution constrained by multiple perceptual channels.

2. Unified latent representation and model architecture

WorldWeaver takes three synchronized input modalities:

$$X\in\mathbb{R}^{F\times 3\times H\times W},\qquad D\in\mathbb{R}^{F\times3\times H\times W},\qquad O\in\mathbb{R}^{F\times3\times H\times W},$$

where $X$ denotes RGB frames, $D$ depth maps repeated to three channels, and $O$ optical-flow encoded as RGB (Liu et al., 21 Aug 2025).

A shared spatio-temporal VAE encoder $\mathcal{E}$ maps each modality into a latent tensor:

$$z_x = \mathcal{E}(X),\quad z_d = \mathcal{E}(D),\quad z_o = \mathcal{E}(O),$$

with each latent of shape $\mathbb{R}^{f\times c\times h\times w}$. These latent tensors are concatenated along the channel dimension to form the joint representation

$$\mathbf{z} = [\,z_x,\;z_d,\;z_o\,]\in\mathbb{R}^{f\times (3c)\times h\times w}.$$

The generative backbone is either a Diffusion Transformer or a flow-matching network with extended input and output layers. It consumes the noised latent $\mathbf{z}_t$ and predicts the denoising velocity for all three modalities simultaneously. The output head $u^+(\mathbf{z}_t,t;\theta)$ is split into three predicted latents,

$X$0

which are then decoded by the VAE decoder back into RGB frames, depth maps, and flow fields. Architecturally, the central design choice is not merely multimodal conditioning, but joint latent prediction under a shared spatio-temporal representation.

3. Mathematical formulation and training mechanics

The paper formulates WorldWeaver in a flow-matching setting. Given a clean video latent $X$1 data and noise $X$2, the interpolation is

$X$3

The standard flow-matching loss is

$X$4

For the unified RGB-depth-flow setting, the noised latent is written as $X$5, and the model predicts velocities for all three streams at once:

$X$6

The joint training objective is

$X$7

Two additional mechanisms define the long-horizon training procedure. First, memory-bank noise injection distinguishes short-term and long-term memory. For short-term memory, the noise level is set to zero for all channels. For long-term memory, depth channels are kept clean while RGB and flow channels are noised to a small degree $X$8. During training $X$9, and at inference $D$0. The paper notes that, in the diffusion convention, a higher $D$1 means less noise injected (Liu et al., 21 Aug 2025).

Second, segmented noise scheduling is defined over frame-groups. Let $D$2 be the number of frame-groups. One samples

$D$3

Then for group $D$4,

$D$5

and $D$6 and $D$7 are looked up from the 1000-step scheduler. During inference, groups are denoised from highest to lowest noise in a streaming fashion. The reported role of this mechanism is to mitigate drift while reducing computational cost.

4. Depth as a drift-resistant signal

A central empirical claim of WorldWeaver is that depth is more resistant to drift than RGB. The paper supports this through a “static image repeat” test in which the same image frame is fed 20 times. Under RGB-only prediction, the sequence warps and degrades rapidly, whereas depth-only predictions remain structurally precise (Liu et al., 21 Aug 2025).

The quantitative drift metric is defined as $D$8, the absolute difference in per-frame image-quality between the first $D$9 and last $O$0. In this test, the RGB-only baseline has $O$1, depth-only has $O$2, and joint RGB+depth has $O$3. On real long-horizon generation, adding depth reduces $O$4 from $O$5.

These measurements constrain interpretation rather tightly. They indicate not merely that depth is a useful auxiliary cue, but that it functions as a more stable carrier of structural context over long horizons. This suggests that the memory bank is not only a retrieval mechanism; it is also a modality-selective stability mechanism in which clean depth acts as a persistent geometric anchor.

5. Benchmark results on long-horizon generation

The paper reports results in two settings: an open-source comparison on 20–30 s videos from general data, and a controlled comparison on the DROID robotic-arm dataset (Liu et al., 21 Aug 2025).

In the open-source comparison, the reported Table 1 excerpt is:

• StreamingT2V: Subject 79.34, Background 84.32, Image 0.47, Smoothness 0.73, Drift 0.23
• CausVid (Wan): Subject 84.71, Background 88.43, Image 0.54, Smoothness 0.82, Drift 0.16
• SkyReels-V2 1.3B: Subject 87.05, Background 90.07, Image 0.61, Smoothness 0.86, Drift 0.11
• Magi 4.5B: Subject 88.71, Background 90.32, Image 0.62, Smoothness 0.89, Drift 0.09
• Ours: Subject 87.34, Background 90.49, Image 0.59, Smoothness 0.87, Drift 0.07

The stated conclusion is that the model achieves the lowest drift and the best background consistency.

In the controlled comparison on the DROID robotic-arm dataset, the base model is CogVideoX-2B, and all systems are trained and inferenced on 16 A100s using 20 s clips. The reported Table 2 excerpt is:

• History Diffusion: 89.78, 91.92, 0.57, 0.68, 0.09, $O$6 steps
• Rolling Diffusion: 88.74, 91.09, 0.58, 0.70, 0.05, $O$7 steps
• Diffusion Forcing: 88.67, 91.23, 0.61, 0.72, 0.06, $O$8 steps
• Ours (RGB-only): 89.57, 91.51, 0.55, 0.67, 0.15, $O$9 steps
• Ours (RGB+perception): 90.92, 92.39, 0.60, 0.75, 0.07, $\mathcal{E}$0 steps

The stated interpretation is that joint perception yields the best consistency and competitive motion, at half or fewer training steps. The contrast between the RGB-only and RGB+perception variants is especially informative because it isolates the effect of perceptual channels within the same framework.

6. Ablation studies and modality contributions

The ablation study on short 5 s videos evaluates the impact of individual perceptual cues. The reported Table 3 values are (Liu et al., 21 Aug 2025):

• RGB alone: Subject 92.72 %, Background 94.76 %, Dynamic 0.62, Smoothness 0.74
• +Depth: 94.87 %, 95.14 %, 0.70, 0.82
• +Segmentation: 94.01 %, 94.83 %, 0.65, 0.70
• +Flow: 92.48 %, 94.22 %, 0.74, 0.81
• Depth+Flow: 94.76 %, 95.23 %, 0.73, 0.85

The paper’s conclusion is that depth gives the largest boost to consistency, while flow is complementary for motion. This division of labor is consistent with the framework’s design rationale: depth contributes structural persistence, and flow contributes dynamic continuity.

A separate ablation varies the long-term memory noise level $\mathcal{E}$1, both with and without perceptual conditioning. Without cues, lowering $\mathcal{E}$2, meaning adding more noise, reduces drift from $\mathcal{E}$3 but also hurts consistency. With cues, the model is reported as robust: drift runs as low as $\mathcal{E}$4 and consistency remains high at approximately $\mathcal{E}$5–$\mathcal{E}$6 %. This suggests that perceptual conditioning stabilizes the trade-off between memory corruption and long-horizon coherence, allowing the system to tolerate noise in long-term memory without a comparable collapse in consistency.

7. Limitations and proposed extensions

The paper identifies several limitations. Very small objects still sometimes vanish, and depth cannot fully capture tiny geometry. As video length grows beyond 20 s, drift and failure rates slowly climb. Only depth, flow, and segmentation are studied, and the authors note that other modalities, such as surface normals and material cues, might help further (Liu et al., 21 Aug 2025).

The proposed future directions are correspondingly geometric and hierarchical. One direction is discovering richer perceptual signals, such as 3D point clouds, to tighten geometric constraints. Another is scaling to multi-minute or hour-long videos via more sophisticated memory hierarchies. A third is combining the framework with explicit physics engines or symbolic dynamics for improved long-term physical realism.

Taken together, these limitations and directions clarify the scope of WorldWeaver. The framework addresses long-horizon drift by enriching representation and memory with perceptual structure, but it does not claim to solve all forms of long-term video consistency. The reported evidence instead places it as a multimodal latent modeling strategy in which depth-centered perception functions as the primary mechanism for preserving world structure over extended generation horizons.