The Seriality Gap in Video Diffusion Models

Abstract: When one ball strikes another, then another, video models should predict the consequences of each bounce. In controlled experiments on multi-ball hard-sphere dynamics, we find that the performance of standard bidirectional video diffusion degrades as the causal chain lengthens, even when provided more denoising steps. In a length-matched single-ball control, where ball-ball interactions are absent, the degradation largely disappears, isolating dependent-event structure rather than video length as the cause. Across intervention studies, methods that increase effective serial computation improve performance disproportionately, including autoregressive/blockwise generation and architectural depth. We identify this pattern as the seriality gap: a mismatch between tasks requiring growing serial computation and video diffusion models whose denoising loop does not provide scalable serial compute. We then prove that, for deterministic video prediction, denoising steps do not add serial computation beyond the backbone, indicating a structural obstacle for video diffusion on serial reasoning and simulation tasks.

• The paper reveals a seriality gap where bidirectional diffusion models struggle with deterministic, serial propagation in multi-ball dynamics.
• Empirical findings show that increased denoising steps do not close the gap, while autoregressive setups maintain significantly lower prediction errors.
• Theoretical analysis connects the gap to limits of fixed-depth transformer backbones, underlining the need for deeper architectures or explicit serial computation.

The Seriality Gap in Video Diffusion Models

Overview

"The Seriality Gap in Video Diffusion Models" (2607.13031) systematically investigates the computational limitations of bidirectional diffusion-based video models in simulating deterministic physical processes that require serial propagation of dependent events. The paper introduces a controlled hard-sphere dynamics task as a minimal environment to dissect these limitations and empirically establishes a performance gap—termed the "seriality gap"—between bidirectional diffusion models and inference paradigms with more explicitly serial computation (autoregressive/blockwise models). Theoretical analysis is provided to show that repeated denoising steps do not address this gap for deterministic settings, and the core limitation persists even under substantial increases to compute or architectural width, with backbone depth and temporal factorization being the most effective remedies.

Task Design: Isolating Serial Reasoning

Hard-sphere dynamics in a frictionless box is chosen as a diagnostic environment where prediction complexity is directly determined by the chain length of collision events:

• Multi-ball setting ($n>1$): Each collision changes the state governing subsequent interactions, creating a causally linked chain that must be resolved sequentially.
• Single-ball control ($n=1$): Dynamics admit a closed-form solution for any rollout length, and no intermediate events need to be propagated.

The experimental setup allows for holding visual content fixed while varying the demand for serial computation by adjusting video length and ball count.

Figure 1: (a) Schematic differentiation between serial and non-serial prediction in hard-sphere dynamics; (b) Question posed on the ability of diffusion models to accurately predict as serial complexity grows.

Empirical Characterization of the Seriality Gap

The paper reports five core empirical findings, supported by rigorous ablations and analyses:

1. Performance Degradation with Serial Complexity: Bidirectional diffusion models' accuracy decays as the chain of dependent events lengthens, despite increased inference compute as video length grows. In contrast, autoregressive schemes exhibit nearly flat local prediction error across horizons.
2. Control: No Gap where Seriality Is Absent: When ball–ball interaction is removed (single-ball setting), the degradation in error with video length vanishes and model performance becomes nearly identical across bidirectional and autoregressive models.
3. Temporal Serialization Improves Performance: Blockwise and autoregressive factorization in inference monotonically improves local error, interpolating from bidirectional (Block-11) through staggered block sizes down to fully autoregressive (Block-1).
4. Denoising Steps Provide No Scalable Serial Computation: Increasing the number of denoising steps in bidirectional diffusion does not close the gap; performance plateaus and the gap persists.
5. Backbone Depth is Critical: Depth scaling (with fixed width and compute budgets) confers disproportionately more benefit than width scaling, up to practical optimization limits.

Figure 2: Charted empirical evidence—(top left) local error as chain length increases, (top right) effect of denoising steps, (bottom left) depth vs. width scaling, (bottom right) blockwise to autoregressive performance interpolation.

Qualitative analysis reveals state-consistency failures (e.g., tunneling, ball-count violations, color-swaps) highly prevalent in bidirectional generations for high-velocity, long-horizon settings, but strongly suppressed in autoregressive outputs.

Figure 3: Representative qualitative failure modes—bidirectional diffusion shows physical inconsistencies, such as splitting and merging of objects, color identity swaps, or invalid motion.

Theoretical Analysis: Why Bidirectional Diffusion Fails on Serial Chains

The theoretical section formalizes the intuition using computational complexity notions:

• Fixed-depth transformer backbones are bounded by parallel circuit depth ($\mathsf{TC}^0$ class), suited to problems with closed-form, highly parallel solutions.
• Physical simulation with dependent events (as in multi-ball collisions) is $\mathsf{P}$-complete—requiring inherently serial computations that cannot be efficiently parallelized.

The diffusion denoising loop is shown to not provide additional serial computation for deterministic settings. If the score function matches ground truth, the unique trajectory can be recovered in a single evaluation regardless of trajectory length, and further denoising iterations only consume more parallel compute without adding serial depth. Thus, the backbone’s depth and inference factorization (e.g., autoregression) must match the problem’s serial dependency structure.

Figure 4: Illustration of bidirectional denoising vs. inference paradigm—blockwise and autoregressive factorization add scalable serial computation and improve error as block size decreases.

This result extends, with caveats, beyond hard-sphere dynamics to other tasks (puzzles, mazes) where serial dependency graphs may not align with frame order, making autoregression an incomplete remedy and highlighting the necessity for depth or alternative inference.

Robustness, Human Evaluation, and Limitations

Extensive experiments rule out confounds from optimization (batch size, initialization), randomness (training seeds), and pretraining. Blockwise and autoregressive factorization robustly offer lower error, and human preference studies corroborate that physical plausibility differences in output are salient and aligned with rollout error metrics.

Figure 5: Paired human preference judgements; autoregressive and blockwise outputs perceived as more physically plausible compared to bidirectional outputs.

A preliminary extension to stochastic settings (vibrating-wall dynamics) shows that the qualitative pattern of the seriality gap persists, although the theoretical guarantees for deterministic dynamics do not strictly apply.

Figure 6: Rollout-5 error on deterministic segments in stochastic hard-sphere dynamics, showing persistent seriality gap.

Practical and Theoretical Implications

The work’s implications are twofold:

• Practical: Current bidirectional diffusion models are structurally unable to propagate long chains of dependent events at scale unless backbone depth or inference factorization are aligned to the task’s serial complexity; increasing denoising steps or architectural width is insufficient.
• Theoretical: Diffusion models, under the score-matching formalism, converge in $O(1)$ serial steps regardless of problem size for deterministic tasks, fundamentally limiting their applicability as world simulators unless serial computation is explicitly provisioned.

Future directions include developing iterative generative objectives that preserve and accumulate intermediate computation across steps, or exploring non-score-matching architectures for scalable serial reasoning.

Conclusion

This paper rigorously demonstrates the mismatch between problem demands for serial computation and the serial compute provisions in bidirectional diffusion models for deterministic video prediction tasks. The seriality gap persists regardless of increases in parallel compute or denoising steps; addressing it requires backbone depth scaling or inference paradigms with explicit serial structure, such as blockwise or autoregressive generation. The theoretical framework clarifies the boundaries of what diffusion models can compute efficiently, and the diagnostic task design isolates the computational bottleneck that may underlie wider failures of video models in visual reasoning and simulation. This work informs both practical model engineering and the theoretical design of generative approaches targeting complex sequential prediction.