TI2V-Zero: Zero-Shot Text-to-Video Synthesis

• TI2V-Zero is a zero-shot, tuning-free framework for text-conditioned image-to-video synthesis that preserves the subject’s visual details while dynamically following prompt-driven actions.
• It leverages a frozen, pretrained diffusion model with a ‘repeat-and-slide’ conditioning strategy and DDPM inversion to ensure temporal coherence and high-fidelity appearance.
• Empirical results on datasets like MUG and UCF-101 demonstrate substantial improvements in metrics (e.g., lower FVD scores), highlighting its practical potential for diverse video synthesis tasks.

TI2V-Zero is a zero-shot, tuning-free framework for text-conditioned image-to-video (TI2V) synthesis. Its central innovation is enabling a frozen, pretrained text-to-video (T2V) diffusion model to generate videos that (i) faithfully preserve the subject and visual details of a user-provided image and (ii) follow the dynamics or action described in a natural language prompt—without any back-propagation, parameter updates, or architectural modifications. This approach eliminates the need for dedicated training on video–text datasets or architecture-specific finetuning, allowing broad deployment of TI2V models with drastically reduced computational and engineering overhead. TI2V-Zero employs a “repeat-and-slide” conditioning strategy, an autoregressive generation paradigm, DDPM inversion for latent initialization, and a resampling operator for visual detail preservation. These mechanisms collectively support zero-shot generalization to tasks such as video infilling, prediction, and the synthesis of long, temporally coherent videos (Ni et al., 2024).

1. Task Definition and Zero-Shot Conditioning Paradigm

TI2V-Zero addresses the canonical TI2V synthesis task: Given a starting image $x^0$ and a text prompt $y$, synthesize a video $$\hat{\mathbf{x}} = \langle x^0, \hat{x}^1, \dots, \hat{x}^M \rangle$$ such that the generated sequence dynamically follows the semantic intent of $y$ while preserving the visual attributes (e.g., identity, pose, background) of $x^0$. Unlike previous approaches—which either train custom models on large paired datasets or introduce extra modules that require finetuning—TI2V-Zero operates exclusively at inference time by injecting the reference image directly into the denoising steps of a pretrained T2V diffusion backbone. The model weights remain frozen, and no model- or sample-specific optimization steps are performed.

This design enables broad practical deployment, as it circumvents the typical computational and data collection bottlenecks associated with task-specific training. The zero-shot, tuning-free nature is particularly suited to open-domain scenarios with unconstrained visual and semantic variation (Ni et al., 2024).

2. Pretrained Diffusion Backbone and Conditioning Process

TI2V-Zero leverages a latent diffusion model (LDM) trained for text-to-video generation. Let $\mathcal{E}, \mathcal{D}$ denote the autoencoder encoder/decoder for mapping RGB videos to/from their latent representations, and let $\epsilon_\theta$ denote the U-Net-based DDPM noise predictor. Inputs to TI2V-Zero are encoded:

$z_0^i = \mathcal{E}(x^i),\quad i=0, \dots, K,$

for $(K+1)$-frame latent representations. The forward noising process is

$$q(\mathbf{z}_t\,|\,\mathbf{z}_{t-1}) = \mathcal{N}\bigl(\mathbf{z}_t;\, \sqrt{1-\beta_t}\,\mathbf{z}_{t-1},\, \beta_t I\bigr),$$

with reverse denoising via

$y$0

Classifier-free guidance is employed, interpolating the predicted noise between conditional and unconditional prompts (Ni et al., 2024).

3. Repeat-and-Slide Conditioning and Autoregressive Generation

Autoregressive generation in TI2V-Zero is orchestrated by a frame queue and a “repeat-and-slide” update. The initial queue $y$1 is populated by encoding $y$2 multiple times. For each frame $y$3, TI2V-Zero:

• Initializes the latent variables’ first $y$4 channels using a DDPM-inversion of the current queue, anchoring initialization to plausible latent states.
• Runs denoising steps, at each step directly overwriting the first $y$5 slots with Gaussian noised versions of the current queue.
• After denoising, decodes only the last latent $y$6 via the decoder $y$7.

The queue is updated by sliding: $y$8. This process recurses for $y$9 frames. The explicit autoregressive architecture imposes strong temporal continuity by conditioning new frames on recent outputs, while simultaneously ensuring each frame inherits visual detail and context from the conditioning image (Ni et al., 2024).

4. DDPM Inversion and Resampling for Temporal and Visual Fidelity

TI2V-Zero addresses two critical challenges: maintaining temporal coherence and high-fidelity appearance. The former is achieved via DDPM inversion, where the current queue of $$\hat{\mathbf{x}} = \langle x^0, \hat{x}^1, \dots, \hat{x}^M \rangle$$0 latent frames is projected through the forward (noising) process to obtain initialization samples for denoising, rather than using independent pure noise. This anchors the generation within the plausible distribution of the pretrained model, which empirically leads to stable, consistent motion and background structure.

To further improve fidelity, a resampling operator may be applied within each reverse step. This consists of repeated denoise–re-noise mini-loops (akin to RePaint), which regularize and reinforce fine image details such as identity and complex backgrounds. Empirical ablations confirm that 2–4 inner resampling rounds are sufficient to capture and preserve hard-to-reconstruct image details as generation progresses (Ni et al., 2024).

5. Extensions to Infilling, Prediction, and Long-Range Synthesis

The queue-based design of TI2V-Zero generalizes naturally beyond the single-image TI2V setting. If the initial queue $$\hat{\mathbf{x}} = \langle x^0, \hat{x}^1, \dots, \hat{x}^M \rangle$$1 is seeded from multiple ground-truth images, the framework can perform:

• Video infilling: Given keyframes at beginning and end, predict plausible transitions by holding corresponding frames in the queue during generation.
• Video prediction: Seed with the first $$\hat{\mathbf{x}} = \langle x^0, \hat{x}^1, \dots, \hat{x}^M \rangle$$2 frames and forecast future dynamics.
• Long-form synthesis: Continue the repeat-and-slide loop over hundreds of frames; empirical results demonstrate preservation of scene identity and background geometry over extended sequences (e.g., 128 frames).

This flexibility demonstrates that the TI2V-Zero approach is not limited to clip-length generation but is structurally suited to arbitrary-length open-domain video synthesis (Ni et al., 2024).

6. Experimental Results and Empirical Significance

TI2V-Zero has been evaluated on both domain-specific (MUG facial expressions, UCF-101 actions) and open-domain video generation tasks. Main quantitative results include:

Dataset	Baseline (FVD)	TI2V-Zero (FVD)
MUG	1095	180 (with 4× resample)
UCF-101	590	477

TI2V-Zero also records several-hundred-point improvements in subject- and text-conditioned FVD (sFVD, tFVD) over prior open-domain TI2V methods. Qualitative evaluation shows the framework preserves input identity and background, produces motion adhering to the prompt semantics, and reduces temporal drift and detail loss compared to strong baselines (e.g., DynamiCrafter). All results are achieved in a strictly zero-shot, tuning-free regime (Ni et al., 2024).

7. Limitations and Prospects

TI2V-Zero inherits certain limitations inherent to the frozen T2V diffusion backbone: generation artifacts and blurriness in complex scenes track back to pretrained model capacity. Frame-by-frame inference entails high computational cost, although acceleration using faster samplers (e.g., DPM-Solver, progressive distillation) is proposed as a mitigation. Residual flicker or motion blur can be addressed in post with video deflickering or deblurring networks. Prospective research directions include adapting the method to next-generation video diffusion architectures, integrating stronger motion priors, and exploring test-time patch-wise editing for even higher identity and detail fidelity. The zero-shot, plug-and-play nature makes these explorations immediately tractable and generally applicable (Ni et al., 2024).