Vectorized Timestep Adaptation (VTA)

• Vectorized Timestep Adaptation (VTA) is a paradigm that replaces a scalar diffusion timestep with a per-frame vector to address rigid synchronization in video diffusion.
• It utilizes both score-based SDE and flow-matching formulations to assign independent noise schedules, enabling diverse tasks like image-to-video generation and interpolation.
• Empirical results demonstrate that VTA improves video generation quality and efficiency while preserving base-model capabilities through adaptive strategies like PTSS and lightweight tuning.

Vectorized Timestep Adaptation (VTA) is a video-diffusion conditioning paradigm in which the conventional clip-level scalar diffusion timestep is replaced by a vector of per-frame timesteps, allowing different frames in the same video to occupy different positions along the noising or denoising trajectory. In the literature, the core idea first appears as the vectorized timestep variable (VTV) in the Frame-Aware Video Diffusion Model (FVDM), and is later developed in PUSA as a non-destructive adaptation of a pretrained text-to-video foundation model. Under this formulation, ordinary synchronized video generation remains a special case obtained by setting all frame timesteps equal, while image-to-video generation, interpolation, start-end conditioning, video extension, and continued text-to-video generation become expressible through frame-wise timestep configurations rather than task-specific architectures (Liu et al., 2024, Liu et al., 22 Jul 2025).

1. From scalar synchronization to frame-aware temporal states

The central motivation for VTA is the claim that standard video diffusion models inherit an image-style scalar timestep and therefore force all frames to share the same noise level and denoising pace. Both FVDM and PUSA identify this as a structural mismatch for video, because video frames can play different semantic and temporal roles: some may be fixed conditioning frames, some may be partially constrained, and others may need to evolve freely. The reported failure mode is “rigid temporal synchronization,” which is especially restrictive for image-conditioned generation, interpolation, transition, and extension tasks (Liu et al., 2024, Liu et al., 22 Jul 2025).

In FVDM, this critique is stated as a limitation of models that “treat a video as a monolithic entity.” The proposed remedy is to make the denoiser frame-aware by exposing an explicit per-frame temporal state. In PUSA, the same idea is reinterpreted as an adaptation mechanism for a pretrained Wan2.1-T2V-14B model: rather than relearning video synthesis, the adaptation changes temporal conditioning so that some frames can be frozen and others denoised. A key property emphasized in PUSA is reversibility at the interface level: if all frame timesteps are set equal, the model reduces back to the original scalar-timestep behavior. This is the basis for the paper’s characterization of VTA as “non-destructive.”

The conceptual significance of this transition is that temporal modeling is no longer limited to cross-frame correlation under a shared global diffusion state. Instead, time itself becomes a structured conditioning signal indexed by frame position. This suggests that VTA is not merely a sampling heuristic, but a reformulation of the state space on which video diffusion is defined.

2. Mathematical formulations

Two closely related mathematical views appear in the literature. FVDM presents a score-based SDE formulation in which the timestep becomes a vector-valued function

$$\bm{\tau}(t) = [\tau^{(1)}(t), \tau^{(2)}(t), \ldots, \tau^{(N)}(t)]^{\top},$$

so each frame has its own diffusion-time coordinate. For the video

$$\mathbf{X} = [\mathbf{x}^{(1)}, \mathbf{x}^{(2)}, \ldots, \mathbf{x}^{(N)}]^{\top},$$

the forward process is written as

$$d\mathbf{X} = \bm{U}(\mathbf{X}, \bm{\tau}(t))\,dt + \bm{\Sigma}(\bm{\tau}(t))\,d\mathbf{W},$$

where $\bm{\Sigma}(\bm{\tau}(t))$ is diagonal across frames, encoding independent frame-wise noise schedules. Under the DDPM parameterization, each frame uses

$$\bm{\mu}(\mathbf{x}^{(i)}, \tau^{(i)}(t)) = -\frac{1}{2}\beta(\tau^{(i)}(t))\mathbf{x}^{(i)}, \qquad \sigma(\tau^{(i)}(t)) = \sqrt{\beta(\tau^{(i)}(t))}.$$

The reverse process and score network are then conditioned on the full vectorized timestep rather than a scalar clip time (Liu et al., 2024).

PUSA instead gives a flow-matching formulation. For a video with $N$ frames,

$$\bm{\tau} = [\tau^1, \tau^2, \ldots, \tau^N]^{\top}, \qquad \tau^i \in [0,1].$$

Let the clean video be

$$\mathbf{X}_0 = [\mathbf{x}_0^1, \ldots, \mathbf{x}_0^N]^{\top}$$

and the prior noise sample be

$$\mathbf{X}_1 = [\mathbf{x}_1^1, \ldots, \mathbf{x}_1^N]^{\top}.$$

Each frame follows its own linear interpolation path,

$$\mathbf{x}^i_{\tau^i} = (1-\tau^i)\mathbf{x}_0^i + \tau^i\mathbf{x}_1^i,$$

and the whole clip is

$$\mathbf{X} = [\mathbf{x}^{(1)}, \mathbf{x}^{(2)}, \ldots, \mathbf{x}^{(N)}]^{\top},$$0

The model is a video-level velocity field

$$\mathbf{X} = [\mathbf{x}^{(1)}, \mathbf{x}^{(2)}, \ldots, \mathbf{x}^{(N)}]^{\top},$$1

with target

$$\mathbf{X} = [\mathbf{x}^{(1)}, \mathbf{x}^{(2)}, \ldots, \mathbf{x}^{(N)}]^{\top},$$2

For the linear interpolation path, that target is independent of $$\mathbf{X} = [\mathbf{x}^{(1)}, \mathbf{x}^{(2)}, \ldots, \mathbf{x}^{(N)}]^{\top},$$3 and of $$\mathbf{X} = [\mathbf{x}^{(1)}, \mathbf{x}^{(2)}, \ldots, \mathbf{x}^{(N)}]^{\top},$$4, which is one of the paper’s simplifying observations (Liu et al., 22 Jul 2025).

Taken together, these formulations define VTA as a per-frame temporal coordinate system. The SDE view emphasizes independent frame-wise corruption and score estimation, while the flow-matching view emphasizes per-frame asynchronous positions on a shared denoising path.

3. Conditioning mechanisms, objectives, and training regimes

The architectural intervention introduced by VTA is concentrated in timestep conditioning rather than in a wholesale redesign of the denoiser. In FVDM, the backbone is Latte. The scalar timestep input shape $$\mathbf{X} = [\mathbf{x}^{(1)}, \mathbf{x}^{(2)}, \ldots, \mathbf{x}^{(N)}]^{\top},$$5 is replaced with a vectorized input of shape $$\mathbf{X} = [\mathbf{x}^{(1)}, \mathbf{x}^{(2)}, \ldots, \mathbf{x}^{(N)}]^{\top},$$6, sinusoidal positional encoding maps it to $$\mathbf{X} = [\mathbf{x}^{(1)}, \mathbf{x}^{(2)}, \ldots, \mathbf{x}^{(N)}]^{\top},$$7, and the resulting embeddings condition the transformer through adaptive layer norm zero (adaLN-Zero), affecting both attention and MLP blocks. In PUSA, the scalar timestep embedding module is modified to accept $$\mathbf{X} = [\mathbf{x}^{(1)}, \mathbf{x}^{(2)}, \ldots, \mathbf{x}^{(N)}]^{\top},$$8, producing frame-specific timestep embeddings that are projected into per-frame modulation parameters—scale, shift, and gate—inside DiT blocks. The paper conceptually writes

$$\mathbf{X} = [\mathbf{x}^{(1)}, \mathbf{x}^{(2)}, \ldots, \mathbf{x}^{(N)}]^{\top},$$9

PUSA is fine-tuned from the pretrained Wan T2V model and uses LoRA for lightweight adaptation (Liu et al., 2024, Liu et al., 22 Jul 2025).

The training objectives differ by framework. FVDM uses a vectorized score-matching objective conditioned on $$d\mathbf{X} = \bm{U}(\mathbf{X}, \bm{\tau}(t))\,dt + \bm{\Sigma}(\bm{\tau}(t))\,d\mathbf{W},$$0. PUSA starts from the scalar flow-matching objective

$$d\mathbf{X} = \bm{U}(\mathbf{X}, \bm{\tau}(t))\,dt + \bm{\Sigma}(\bm{\tau}(t))\,d\mathbf{W},$$1

and extends it to Frame-Aware Flow Matching (FAFM),

$$d\mathbf{X} = \bm{U}(\mathbf{X}, \bm{\tau}(t))\,dt + \bm{\Sigma}(\bm{\tau}(t))\,d\mathbf{W},$$2

Its Frobenius norm expands to a sum of framewise squared errors.

A recurring issue is combinatorial explosion. FVDM addresses it with Probabilistic Timestep Sampling Strategy (PTSS): with probability $$d\mathbf{X} = \bm{U}(\mathbf{X}, \bm{\tau}(t))\,dt + \bm{\Sigma}(\bm{\tau}(t))\,d\mathbf{W},$$3, distinct timesteps are sampled for each frame, and with probability $$d\mathbf{X} = \bm{U}(\mathbf{X}, \bm{\tau}(t))\,dt + \bm{\Sigma}(\bm{\tau}(t))\,d\mathbf{W},$$4, one sampled timestep is copied across all frames. PUSA reports that, by leveraging the pretrained Wan T2V prior, it can train with fully randomized vectorized timesteps, effectively $$d\mathbf{X} = \bm{U}(\mathbf{X}, \bm{\tau}(t))\,dt + \bm{\Sigma}(\bm{\tau}(t))\,d\mathbf{W},$$5, rather than the mixed sync/async curriculum used in from-scratch FVDM-style training. This difference is central to later claims about efficiency and preservation of base-model capability.

4. Task parameterization and inference behavior

A major practical consequence of VTA is that many downstream video tasks can be expressed directly as timestep configurations. In FVDM, standard video generation corresponds to

$$d\mathbf{X} = \bm{U}(\mathbf{X}, \bm{\tau}(t))\,dt + \bm{\Sigma}(\bm{\tau}(t))\,d\mathbf{W},$$6

which recovers synchronized denoising. Image-to-video generation is specified by fixing the first frame,

$$d\mathbf{X} = \bm{U}(\mathbf{X}, \bm{\tau}(t))\,dt + \bm{\Sigma}(\bm{\tau}(t))\,d\mathbf{W},$$7

Video interpolation fixes both endpoints,

$$d\mathbf{X} = \bm{U}(\mathbf{X}, \bm{\tau}(t))\,dt + \bm{\Sigma}(\bm{\tau}(t))\,d\mathbf{W},$$8

Long video generation is described by setting the first $$d\mathbf{X} = \bm{U}(\mathbf{X}, \bm{\tau}(t))\,dt + \bm{\Sigma}(\bm{\tau}(t))\,d\mathbf{W},$$9 frames to timestep $\bm{\Sigma}(\bm{\tau}(t))$0 and the remaining frames to regular noise, thereby conditioning on the end of the previous clip. The paper also suggests schedules such as

$\bm{\Sigma}(\bm{\tau}(t))$1

for image-to-video, illustrating that vectorized timesteps function as a task specification interface rather than only as a training trick (Liu et al., 2024).

PUSA reports analogous behavior in a pretrained foundation-model setting. Because some frames can be kept at or near zero noise while others evolve, the model supports image-to-video, start-end frame generation, video extension, and video completion/transition without task-specific retraining, while still retaining text-to-video generation. Its image-to-video inference uses Euler integration over diffusion noise levels: the first frame is initialized from the encoded input image, the remaining frames are initialized as Gaussian noise, and the first frame’s noise level remains zero throughout the solver trajectory. The update is

$\bm{\Sigma}(\bm{\tau}(t))$2

with $\bm{\Sigma}(\bm{\tau}(t))$3, ensuring that the first frame stays fixed (Liu et al., 22 Jul 2025).

This task-level flexibility directly addresses a common misconception that video diffusion must be specialized per task through separate fine-tuning. In the VTA literature, the claim is instead that conditioning over frame-wise temporal states can unify multiple tasks within one model family.

5. Empirical results and mechanistic interpretations

The empirical record reported for vectorized timesteps spans both from-scratch frame-aware models and lightweight adaptations of pretrained video generators. In FVDM, evaluations on FaceForensics, SkyTimelapse, UCF101, and Taichi-HD report FVD values of $\bm{\Sigma}(\bm{\tau}(t))$4, $\bm{\Sigma}(\bm{\tau}(t))$5, $\bm{\Sigma}(\bm{\tau}(t))$6, and $\bm{\Sigma}(\bm{\tau}(t))$7, compared with Latte at $\bm{\Sigma}(\bm{\tau}(t))$8, $\bm{\Sigma}(\bm{\tau}(t))$9, $$\bm{\mu}(\mathbf{x}^{(i)}, \tau^{(i)}(t)) = -\frac{1}{2}\beta(\tau^{(i)}(t))\mathbf{x}^{(i)}, \qquad \sigma(\tau^{(i)}(t)) = \sqrt{\beta(\tau^{(i)}(t))}.$$0, and $$\bm{\mu}(\mathbf{x}^{(i)}, \tau^{(i)}(t)) = -\frac{1}{2}\beta(\tau^{(i)}(t))\mathbf{x}^{(i)}, \qquad \sigma(\tau^{(i)}(t)) = \sqrt{\beta(\tau^{(i)}(t))}.$$1. On the FaceForensics ablation with model scale B and DDIM-50, the paper reports that $$\bm{\mu}(\mathbf{x}^{(i)}, \tau^{(i)}(t)) = -\frac{1}{2}\beta(\tau^{(i)}(t))\mathbf{x}^{(i)}, \qquad \sigma(\tau^{(i)}(t)) = \sqrt{\beta(\tau^{(i)}(t))}.$$2 consistently performs best, reaching FVD $$\bm{\mu}(\mathbf{x}^{(i)}, \tau^{(i)}(t)) = -\frac{1}{2}\beta(\tau^{(i)}(t))\mathbf{x}^{(i)}, \qquad \sigma(\tau^{(i)}(t)) = \sqrt{\beta(\tau^{(i)}(t))}.$$3 at 200k steps, compared with baseline Latte at $$\bm{\mu}(\mathbf{x}^{(i)}, \tau^{(i)}(t)) = -\frac{1}{2}\beta(\tau^{(i)}(t))\mathbf{x}^{(i)}, \qquad \sigma(\tau^{(i)}(t)) = \sqrt{\beta(\tau^{(i)}(t))}.$$4. Larger models perform better, with XL achieving FVD $$\bm{\mu}(\mathbf{x}^{(i)}, \tau^{(i)}(t)) = -\frac{1}{2}\beta(\tau^{(i)}(t))\mathbf{x}^{(i)}, \qquad \sigma(\tau^{(i)}(t)) = \sqrt{\beta(\tau^{(i)}(t))}.$$5. The reported qualitative findings are more coherent sequences, smoother transitions, more realistic motion, and better preservation of identity or content (Liu et al., 2024).

In PUSA, VTA is linked to efficiency claims under a pretrained-foundation-model regime. The paper reports a VBench-I2V total score of $$\bm{\mu}(\mathbf{x}^{(i)}, \tau^{(i)}(t)) = -\frac{1}{2}\beta(\tau^{(i)}(t))\mathbf{x}^{(i)}, \qquad \sigma(\tau^{(i)}(t)) = \sqrt{\beta(\tau^{(i)}(t))}.$$6, compared with $$\bm{\mu}(\mathbf{x}^{(i)}, \tau^{(i)}(t)) = -\frac{1}{2}\beta(\tau^{(i)}(t))\mathbf{x}^{(i)}, \qquad \sigma(\tau^{(i)}(t)) = \sqrt{\beta(\tau^{(i)}(t))}.$$7 for Wan-I2V-14B, together with I2V Subject Consistency $$\bm{\mu}(\mathbf{x}^{(i)}, \tau^{(i)}(t)) = -\frac{1}{2}\beta(\tau^{(i)}(t))\mathbf{x}^{(i)}, \qquad \sigma(\tau^{(i)}(t)) = \sqrt{\beta(\tau^{(i)}(t))}.$$8 versus $$\bm{\mu}(\mathbf{x}^{(i)}, \tau^{(i)}(t)) = -\frac{1}{2}\beta(\tau^{(i)}(t))\mathbf{x}^{(i)}, \qquad \sigma(\tau^{(i)}(t)) = \sqrt{\beta(\tau^{(i)}(t))}.$$9, I2V Background Consistency $N$0 versus $N$1, Motion Smoothness $N$2, and Dynamic Degree $N$3. The headline resource claim is surpassing Wan-I2V-14B with $N$4 of the training cost ($N$5) and $N$6 of the dataset size (4K versus $N$7M samples; detailed discussion gives 3,860 samples). The paper further reports strong results with only 10 inference steps and training convergence around 900 iterations (Liu et al., 22 Jul 2025).

Mechanistic analyses are used to explain why asynchronous frame-wise timesteps do not collapse model behavior. PUSA identifies the fundamental concern as a combinatorial explosion of timestep configurations, citing around $N$8 possible configurations for 16 frames. Its answer is that the model does not learn video generation from scratch: it starts from a strong pretrained text-to-video prior and learns new temporal control behavior. The attention-map analysis reports that the Wan T2V baseline has mostly diagonal attention maps, Wan-I2V alters the attention structure heavily, and PUSA strengthens attention to the first frame while leaving much of the original structure intact. The parameter-drift analysis similarly reports that Wan-I2V changes many parameters substantially, including the text encoder and cross-attention, whereas PUSA changes far fewer parameters, mostly in self-attention blocks associated with temporal dynamics. This suggests that, in the reported regime, VTA behaves as a targeted temporal interface layered on top of an existing generative prior rather than as a wholesale model rewrite.

6. Related methods, limitations, and terminological ambiguity

The phrase “Vectorized Timestep Adaptation” is not uniform across all adjacent literature, and several neighboring concepts must be distinguished. In video diffusion, the closest antecedent is FVDM’s “vectorized timestep variable,” which is effectively the same underlying idea expressed in score-based form. By contrast, ART and ART-RL study timestep allocation for diffusion sampling as a continuous-time control problem over the global sampling clock. ART learns a time warp $N$9 with control $$\bm{\tau} = [\tau^1, \tau^2, \ldots, \tau^N]^{\top}, \qquad \tau^i \in [0,1].$$0, so a uniform grid in the learned clock induces a nonuniform scalar schedule in diffusion time. The paper explicitly states that it does not discuss VTA, and the safest interpretation is that ART is conceptually similar in its goal of adapting timestep placement, but different in formulation, objective, and theoretical machinery. A plausible implication is that ART addresses global schedule allocation along a diffusion trajectory, whereas VTA addresses frame-wise asynchronous temporal state within a video clip (Huang et al., 2 Jul 2026).

A second source of confusion is terminological spillover from other domains. In reinforcement learning, TIDBD adapts a vector of per-feature TD step-sizes $$\bm{\tau} = [\tau^1, \tau^2, \ldots, \tau^N]^{\top}, \qquad \tau^i \in [0,1].$$1 by stochastic meta-descent. That work is relevant only at the level of the general idea of vectorized adaptation; it is not a video-diffusion method. In hardware compilation, “VTA” denotes the Versatile Tensor Accelerator, an FPGA-based matrix accelerator with Load, Compute, and Store modules, and is entirely unrelated to timestep parameterization in generative modeling. These acronym collisions matter because “VTA” alone is semantically overloaded across current arXiv usage (Kearney et al., 2018, Faure-Gignoux et al., 27 Apr 2026).

Within video diffusion itself, the principal open issue remains tractability over the large asynchronous state space. FVDM handles this with PTSS, which mixes synchronized and asynchronous training states, while PUSA argues that a pretrained text-to-video prior makes fully randomized vectorized timesteps feasible. Another recurrent concern is whether task adaptation destroys the base model’s original capability. The literature explicitly positions vectorized timesteps as a remedy to catastrophic forgetting in fine-tuning-based pipelines: FVDM emphasizes zero-shot adaptability, and PUSA emphasizes non-destructive adaptation and retained text-to-video generation. The resulting picture is that VTA is best understood as a frame-wise temporal conditioning formalism whose effectiveness depends strongly on whether it is trained from scratch or grafted onto a strong pretrained video prior.