FlowBlending: Stage-Aware Multi-Model Sampling for Fast and High-Fidelity Video Generation (2512.24724v1)

Abstract: In this work, we show that the impact of model capacity varies across timesteps: it is crucial for the early and late stages but largely negligible during the intermediate stage. Accordingly, we propose FlowBlending, a stage-aware multi-model sampling strategy that employs a large model and a small model at capacity-sensitive stages and intermediate stages, respectively. We further introduce simple criteria to choose stage boundaries and provide a velocity-divergence analysis as an effective proxy for identifying capacity-sensitive regions. Across LTX-Video (2B/13B) and WAN 2.1 (1.3B/14B), FlowBlending achieves up to 1.65x faster inference with 57.35% fewer FLOPs, while maintaining the visual fidelity, temporal coherence, and semantic alignment of the large models. FlowBlending is also compatible with existing sampling-acceleration techniques, enabling up to 2x additional speedup. Project page is available at: https://jibin86.github.io/flowblending_project_page.

• The paper introduces a stage-aware sampling strategy that allocates model capacity based on the denoising stage to ensure both global structure and fine detail.
• It employs an LSL (Large-Small-Large) scheduling strategy, achieving nearly baseline quality with up to 1.65× faster runtime and a 57.35% reduction in FLOPs.
• Systematic ablation and velocity divergence analysis validate that early and late stages are capacity-sensitive, while the intermediate phase permits a smaller model without quality loss.

FlowBlending: Stage-Aware Multi-Model Sampling for Efficient Video Diffusion

Introduction and Motivation

The computational expense of diffusion-based video generation has escalated with the increasing size of state-of-the-art models. While visual fidelity and temporal coherence have improved substantially, these advances are accompanied by high inference cost due to deep, multi-step denoising with large-capacity neural architectures. Existing acceleration approaches—such as step reduction via advanced solvers or step distillation into a compact model—largely treat all timesteps as equally demanding in terms of model capacity. This uniform treatment is typically suboptimal, especially for large-scale models available in both full and reduced-parameter variants (e.g., LTX-Video 13B/2B, WAN2.1 14B/1.3B).

The core hypothesis of "FlowBlending: Stage-Aware Multi-Model Sampling for Fast and High-Fidelity Video Generation" (2512.24724) is that model capacity is not uniformly important throughout the denoising trajectory. Rather, the early and late stages are empirically found to be capacity-sensitive—dictating global structure/motion and fine detail/artifact suppression, respectively—while the intermediate regime admits substantial capacity reduction.

Stage-Sensitive Model Allocation

The FlowBlending paradigm leverages the temporal non-uniformity of capacity requirements by introducing an LSL (Large-Small-Large) scheduling strategy:

• Early stage: Large model, to ensure robust formation of global structure and prompt alignment.
• Intermediate stage: Small model, since velocity predictions and produced latents are largely indistinguishable from the large model.
• Late stage: Large model, to enhance detail and suppress perceptually critical artifacts.

This insight is confirmed through systematic ablation, with sharp degradations observed when the early or late stages lack sufficient capacity, and relative invariance to capacity reductions in the intermediate phase.

Figure 1: Model capacity in early denoising—large model in early steps (LSS) yields global structure and motion close to the all-large baseline (LLL); early-stage use of the small model (SLL, SSS) results in misalignment and temporal inconsistency.

Figure 2: Late stage ablation indicates LSL schedule restores detail and reduces flicker compared to LSS, producing quality nearly indistinguishable from LLL at lower cost.

Methodology: Determining Stage Boundaries

Determining optimal stage boundaries is posed as a data-driven process. For the early boundary, semantic similarity (DINO [caron2021emerging], CLIP [radford2021learning]), low-level similarity (LPIPS [zhang2018unreasonable], PSNR), and structural alignment to LLL are tracked as the switch point from large to small model is varied.

Figure 3: DINO similarity vs. early boundary—sharp semantic degradation past the 96% similarity mark guides the optimal transition from large to small model.

The late-stage boundary is selected to minimize FID, with a V-shaped trade-off curve indicating a clear sweet spot for reintroducing the large model to finalize denoising.

Figure 4: FID as a function of late-stage boundary, confirming optimality for reintroducing the large model before final steps.

Empirical Analysis: Velocity Divergence and Capacity Sensitivity

A velocity divergence analysis, measuring cosine and $\ell_2$ distances between large and small model velocity fields at each denoising step, reveals a characteristic U-shaped profile:

• High variance in early phase (global structure/motion)—capacity-sensitive;
• Low divergence in middle phase—capacity-insensitive, suitable for substitution by a smaller model;
• High mean divergence in late phase—capacity-sensitive for detail/artifact suppression.

  Figure 5: U-shaped velocity divergence across denoising steps, supporting large-model allocation in early/late stages and small model for the intermediate phase.

Quantitative Performance and Efficiency

On LTX-Video and WAN 2.1, FlowBlending achieves FID/FVD/Aesthetic/Background/Subject/Motion scores within margin of the large-only baseline, while decreasing runtime by up to $1.65\times$ and reducing FLOPs by $57.35\%$ without retraining or architecture changes.

Figure 6: Pareto frontier analysis in FID/FLOPs space across all schedule variants; LSL (yellow star) yields large-model-level quality with significantly reduced compute.

Compatibility and Generalization

FlowBlending is orthogonal to commonly adopted sampling accelerators and model distillation techniques:

• Combinable with DPM++/ODE solvers for further reduction in NFE (Figure 7).
• Applicable to small models obtained via step distillation (Figure 8).

  Figure 7: LSL schedule is compatible with accelerated DPM++ solvers—preserves visual quality compared to all-large, contrasting with artifacts from SSS.

  

  Figure 8: Swapping in step-distilled small models in the LDL schedule reproduces LLL performance, whereas the DDD schedule with only the small models degrades.

Extensive schedule sweeps on both the PV and VBench datasets reaffirm the robustness of the LSL configuration across architectures, datasets, and quality metrics (Figures 15–18). LSL consistently occupies top ranks in FID, FVD, VBench Aesthetic/Subject Consistency, and Motion Smoothness.

Figure 9: LSL near-optimal across FID/FVD in LTX-Video PV sweep.

Figure 10: LSL maintains superiority under VBench evaluation ordering.

Figure 11: Same trend for WAN 2.1—LSL outperforms other allocations in FID/FVD.

Figure 12: VBench confirms robustness of LSL in WAN 2.1.

Theoretical and Practical Implications

From a theoretical perspective, the findings reinforce the staged nature of denoising in diffusion models and challenge the prevailing assumption of uniform capacity requirements. The observed U-shaped divergence raises questions about corresponding dynamics in image generation and the generality of this principle across modalities.

Practically, FlowBlending provides a training-free, architecture-agnostic paradigm for reducing inference cost in high-fidelity video generation settings. Since the large and small models are typically available in released codebases, this method can be seamlessly integrated into existing pipelines and further compounded with conventional acceleration methods.

The requirement to estimate stage boundaries per-model and configuration is a current limitation; research into automatic, model-agnostic stage-detection or dynamic scheduling algorithms could close this gap.

Conclusion

FlowBlending sets forth a principled approach for capacity-aware inference in video diffusion models by exploiting stage-specific capacity sensitivity. The LSL schedule—large model in early/late phases, small model in the middle—preserves large-model generation quality while effecting substantive reductions in runtime and computational overhead. This work has direct implications for scalable deployment of high-quality video generation models and invites further exploration of stage-wise capacity modulation, both in video and broader generative contexts.

Reference: "FlowBlending: Stage-Aware Multi-Model Sampling for Fast and High-Fidelity Video Generation" (2512.24724)