Zo3T: Zero-shot 3D-Aware I2V Generation

• Zo3T is a zero-shot framework that synthesizes 3D-aware videos using trajectory-guided test-time training without offline fine-tuning.
• It leverages DepthPro for monocular depth estimation, 3D-Aware Kinematic Projection, and an ephemeral LoRA adapter to ensure motion accuracy and photorealism.
• Zo3T outperforms both supervised and zero-shot baselines on metrics like FID, FVD, and ObjMC while preserving appearance consistency and temporal continuity.

Zo3T, short for Zero-shot 3D-Aware Trajectory-Guided image-to-video generation via Test-Time Training, is a framework for trajectory-guided image-to-video (I2V) generation that synthesizes videos from an input image and user-specified object or region trajectories while preserving appearance, motion accuracy, photorealism, and temporal coherence. It operates in a zero-shot regime—without fine-tuning the backbone on trajectory-annotated data—and instead performs test-time training (TTT) by optimizing the latent state and an ephemeral LoRA adapter on each individual test instance. The method is built on Stable Video Diffusion (SVD) and uses DepthPro for monocular depth estimation, combining 3D-Aware Kinematic Projection, Trajectory-Guided Test-Time LoRA, Guidance Field Rectification, and Fourier Orthogonal Recomposition into a single inference-time control pipeline (Zhang et al., 8 Sep 2025).

1. Problem setting and conceptual scope

Zo3T addresses the setting in which the input consists of an initial image $I_0$, together with one or more user-specified trajectories for controlled entities. For each controlled entity $b \in \{1,\dots,M\}$, the user provides an initial bounding box $\mathcal{B}_{b,0}$ and a 2D trajectory

$$\mathcal{T}_{2D}^b = \{(u_{b,k}, v_{b,k})\}_{k=1}^{N_f},$$

where $(u_{b,k}, v_{b,k})$ denotes the target center position in frame $k$. The generative objective is to synthesize a latent video $\{z_k\}_{k=0}^{N_f-1}$, later decoded to frames, such that the controlled region follows the prescribed path while its identity and appearance remain consistent with the first frame (Zhang et al., 8 Sep 2025).

The framework is explicitly formulated as zero-shot and test-time adaptive rather than supervised retraining. The underlying SVD backbone remains frozen; only per-example optimization is performed during inference. This distinguishes Zo3T from training-based trajectory-control systems that require large trajectory-annotated datasets and substantial optimization budgets. A common misconception is that “zero-shot” implies purely feed-forward inference. In Zo3T, zero-shot instead means that no task-specific offline fine-tuning is performed; the method still carries out substantial instance-specific optimization at test time (Zhang et al., 8 Sep 2025).

The design target is not merely motion transfer in 2D image coordinates. Zo3T is motivated by three coupled constraints: 3D perspective consistency, on-manifold latent evolution, and accurate adherence to user trajectories. The paper argues that purely 2D control ignores depth-dependent scaling, while direct latent editing can misalign manipulated states with the denoiser’s learned noise distribution, causing texture collapse, identity loss, flickering, and broader fidelity degradation (Zhang et al., 8 Sep 2025).

2. Relation to prior trajectory-controlled I2V methods

The method is positioned against both training-based and zero-shot baselines. Training-based systems such as DragNUWA, DragAnything, LeviTor, and MotionCtrl are described as relying on fine-tuning video diffusion models on trajectory-annotated data. The cited limitations are expensive training, reliance on scarce annotations, frequent operation at low resolutions such as $320\times576$, overfitting toward motion adherence at the expense of object integrity, and reduced flexibility for unseen motion patterns or new domains (Zhang et al., 8 Sep 2025).

Among these, LeviTor and ObjCtrl-2.5D are highlighted for using 3D cues, but with different trade-offs. LeviTor leverages 3D trajectory-oriented training yet requires costly training on extensive masked datasets. ObjCtrl-2.5D treats object motion as camera motion in a 2.5D sense, which the paper states restricts complex scenes, especially those involving multi-object motion or independent foreground-background dynamics (Zhang et al., 8 Sep 2025).

Zero-shot approaches are divided into two broad categories. The first performs latent-space manipulation, typically through feature consistency losses and attention-based alignment across frames. The stated failure mode is “hard editing” of $\mathbf{z}_t$, which can push the state off the learned diffusion manifold and produce artifacts. The second category alters initial or intermediate noise to encode motion priors, but such methods are characterized as generally 2D, not 3D-aware, and coarse in fine-grained spatial consistency (Zhang et al., 8 Sep 2025).

This positioning motivates Zo3T’s central claim: trajectory control should be implemented as a soft, co-adaptive process that aligns geometric priors, latent states, and denoising dynamics rather than forcing motion by rigid latent overrides. A plausible implication is that Zo3T treats motion control as a constrained inference problem over the pre-trained model rather than as a separate retraining objective (Zhang et al., 8 Sep 2025).

3. Framework architecture and generative pipeline

Zo3T operates on Stable Video Diffusion, which is described as a latent video diffusion model with a VAE encoder, a U-Net-style denoiser $\epsilon_\theta(\mathbf{z}_t, t, c)$, and a decoder back to pixel space. The method loads official pre-trained SVD weights and does not retrain the backbone (Zhang et al., 8 Sep 2025).

Its high-level pipeline has four guided components followed by standard sampling. The first is 3D-Aware Kinematic Projection, which uses DepthPro to infer monocular depth from $b \in \{1,\dots,M\}$0, then converts 2D trajectories into perspective-aware affine transformations and frame-dependent masks. The second is Trajectory-Guided Test-Time LoRA, in which an ephemeral LoRA adapter is injected into the U-Net and jointly optimized with the latent state at selected denoising steps. The third is Guidance Field Rectification (GFR), which optimizes the conditional guidance field through a one-step lookahead objective. The fourth is Fourier Orthogonal Recomposition (FOR), which merges low-frequency content from optimized states with high-frequency content from the original states in order to preserve textural fidelity (Zhang et al., 8 Sep 2025).

The paper describes the overall inference trajectory as follows: at selected timesteps, latent and LoRA optimization are performed; then the guidance field is rectified; FOR is applied to regularize the resulting optimized signals; and standard DDIM/DDPM denoising resumes for later steps. The output is a 14-frame, $b \in \{1,\dots,M\}$1 video (Zhang et al., 8 Sep 2025).

Component	Function	Mechanism
3D-Aware Kinematic Projection	Perspective-consistent motion prior	DepthPro, affine transforms, time-varying masks
Trajectory-Guided Test-Time LoRA	Local model adaptation during inference	Joint latent and LoRA optimization with regional feature consistency loss
Guidance Field Rectification	Better denoising direction toward target motion	One-step lookahead optimization of $b \in \{1,\dots,M\}$2
Fourier Orthogonal Recomposition	Fidelity-preserving regularization	Low-frequency optimized content plus original high-frequency content

The architecture is notable for distributing control across geometry, representation, and sampler dynamics rather than localizing all control in a single latent-editing stage. This suggests a modular view of trajectory-guided generation in which different failure modes are handled at different layers of the diffusion process (Zhang et al., 8 Sep 2025).

4. Core algorithmic mechanisms

4.1 3D-Aware Kinematic Projection

Zo3T begins from a depth map

$b \in \{1,\dots,M\}$3

computed by DepthPro on the initial frame. For an object trajectory with center pixel coordinates $b \in \{1,\dots,M\}$4, the approximate depth is read directly from the map: $b \in \{1,\dots,M\}$5 Using a pinhole camera approximation, Zo3T defines the frame-relative scaling factor as

$b \in \{1,\dots,M\}$6

The affine transform mapping the initial bounding box $b \in \{1,\dots,M\}$7 to frame $b \in \{1,\dots,M\}$8 is

$b \in \{1,\dots,M\}$9

This yields frame-dependent masks

$\mathcal{B}_{b,0}$0

These masks vary both in position and scale, thereby encoding a pseudo-3D motion prior rather than a fixed-size 2D box (Zhang et al., 8 Sep 2025).

The paper argues that this matters not primarily because it dramatically improves the motion-control metric, but because it makes the subsequent feature consistency losses physically meaningful: a moving object that changes depth should not be compared against mismatched 2D crops of fixed size. The ablation reported for removing 3D projection shows only a modest change in ObjMC, but a larger qualitative decline in realism due to shape distortion under depth change (Zhang et al., 8 Sep 2025).

4.2 Trajectory-Guided Test-Time LoRA

At selected denoising steps, Zo3T injects an ephemeral LoRA adapter into the SVD U-Net and jointly optimizes the latent $\mathcal{B}_{b,0}$1 and the LoRA weights $\mathcal{B}_{b,0}$2. The adapted parameters are denoted $\mathcal{B}_{b,0}$3, and the optimization objective is

$\mathcal{B}_{b,0}$4

The core loss is a regional feature consistency loss: $\mathcal{B}_{b,0}$5 Here, $\mathcal{B}_{b,0}$6 is a selected feature map at layer $\mathcal{B}_{b,0}$7 and frame $\mathcal{B}_{b,0}$8, $\mathcal{B}_{b,0}$9 is the detached reference from frame 1, and $$\mathcal{T}_{2D}^b = \{(u_{b,k}, v_{b,k})\}_{k=1}^{N_f},$$0 is a Gaussian heatmap that focuses the loss toward the object interior. Because projected masks vary in size, the cropped feature regions are resized to a uniform size before comparison (Zhang et al., 8 Sep 2025).

The central idea is co-adaptation. Rather than forcing $$\mathcal{T}_{2D}^b = \{(u_{b,k}, v_{b,k})\}_{k=1}^{N_f},$$1 alone to satisfy the motion constraint, the denoiser is locally adapted so that its internal representation becomes compatible with the manipulated latent. The paper describes this as “soft editing,” intended to reduce manifold deviation. Ablation indicates that removing TTT causes the largest degradation among the four main components, with a shift from FID 74.83 / FVD 197.63 / ObjMC 12.74 in the full system to FID 89.36 / FVD 219.53 / ObjMC 14.24 without TTT adaptation (Zhang et al., 8 Sep 2025).

4.3 Guidance Field Rectification

In classifier-free guidance, the conditional control direction is

$$\mathcal{T}_{2D}^b = \{(u_{b,k}, v_{b,k})\}_{k=1}^{N_f},$$2

Standard CFG uses this direction directly. Zo3T instead optimizes it. Let $$\mathcal{T}_{2D}^b = \{(u_{b,k}, v_{b,k})\}_{k=1}^{N_f},$$3 denote a one-step DDIM solver that predicts $$\mathcal{T}_{2D}^b = \{(u_{b,k}, v_{b,k})\}_{k=1}^{N_f},$$4 from the current latent using a candidate guidance field $$\mathcal{T}_{2D}^b = \{(u_{b,k}, v_{b,k})\}_{k=1}^{N_f},$$5. Zo3T defines a one-step kinematic cost

$$\mathcal{T}_{2D}^b = \{(u_{b,k}, v_{b,k})\}_{k=1}^{N_f},$$6

The guidance field is then optimized by gradient descent starting from the original CFG difference: $$\mathcal{T}_{2D}^b = \{(u_{b,k}, v_{b,k})\}_{k=1}^{N_f},$$7 producing $$\mathcal{T}_{2D}^b = \{(u_{b,k}, v_{b,k})\}_{k=1}^{N_f},$$8, and the rectified conditional noise is reconstructed as

$$\mathcal{T}_{2D}^b = \{(u_{b,k}, v_{b,k})\}_{k=1}^{N_f},$$9

The loss uses pooled features rather than pixel-wise alignment, which the paper states encourages more organic motion and avoids excessive rigidity (Zhang et al., 8 Sep 2025).

4.4 Fourier Orthogonal Recomposition

FOR is used to preserve the diffusion model’s original high-frequency texture content while importing optimized low-frequency structural changes. The paper describes this as a Fourier-domain recomposition that acts as a strong regularizer and prevents optimization artifacts from dominating textural detail. Its ablation indicates that removing FOR leaves ObjMC nearly unchanged but markedly worsens generative quality, with FID 95.45 and FVD 221.09 relative to the full system’s FID 74.83 and FVD 197.63 (Zhang et al., 8 Sep 2025).

5. Optimization protocol and empirical evaluation

The optimization is confined to the early-to-mid denoising window $(u_{b,k}, v_{b,k})$0 in a 50-step DDPM/DDIM schedule. At each such timestep, the TTT stage optimizes the latent state $(u_{b,k}, v_{b,k})$1 and LoRA weights $(u_{b,k}, v_{b,k})$2, and the GFR stage optimizes the guidance field $(u_{b,k}, v_{b,k})$3. The backbone parameters $(u_{b,k}, v_{b,k})$4 remain frozen throughout. The optimizer is Adam, with learning rates chosen via ablation: 0.25 for the latent, 0.01 for the LoRA weights, and 0.05 for the guidance field (Zhang et al., 8 Sep 2025).

The evaluation setup uses Stable Video Diffusion as the backbone and generates 14-frame videos at $(u_{b,k}, v_{b,k})$5. For comparison with supervised baselines operating at $(u_{b,k}, v_{b,k})$6, generated outputs are downsampled to $(u_{b,k}, v_{b,k})$7 for evaluation. The motion-control benchmark is based on the open-source set in DragAnything and augmented by mirroring most trajectories, producing 143 challenging samples. For FID and FVD, real videos are drawn from VIPSeg, generated videos are resized to $(u_{b,k}, v_{b,k})$8, and FVD is computed using ViCLIP features (Zhang et al., 8 Sep 2025).

The reported metrics include FID, FVD, Object Motion Consistency (ObjMC), and VBench metrics for Subject Consistency, Background Consistency, Aesthetic Quality, and Imaging Quality. ObjMC is defined as the average per-frame Euclidean distance between target and observed trajectories, where the observed object path is estimated using CoTracker. The authors explicitly note that ObjMC can be misleading when distorted objects remain trackable; they therefore emphasize qualitative analysis and a user study alongside the metric (Zhang et al., 8 Sep 2025).

Method	Setting	Key metrics
DragNUWA	Supervised	FID 126.31; FVD 251.04; ObjMC 10.84
DragAnything	Supervised	FID 119.07; FVD 266.42; ObjMC 11.64
LeviTor	Supervised	FID 79.41; FVD 207.23; ObjMC 12.06
FreeTraj	Zero-shot	FID 92.12; FVD 230.21; ObjMC 31.63
ObjCtrl-2.5D	Zero-shot	FID 81.06; FVD 212.17; ObjMC 18.72
SG-I2V	Zero-shot	FID 79.36; FVD 209.53; ObjMC 14.43
Zo3T	Zero-shot	FID 74.83; FVD 197.63; ObjMC 12.74

On the same benchmark, Zo3T attains the highest reported VBench scores: Subject Consistency 0.9760, Background Consistency 0.9682, Aesthetic Quality 0.6779, and Imaging Quality 0.6820. The paper’s interpretation is that Zo3T outperforms all zero-shot baselines across all metrics and surpasses supervised methods in FID and FVD while maintaining ObjMC values close to supervised systems (Zhang et al., 8 Sep 2025).

The user study involves 6 expert evaluators and 50 videos, with comparisons against three strong baselines on imaging fidelity, motion coherence, trajectory accuracy, and overall quality. Zo3T receives 51.7\% of all preference votes across pairwise comparisons (Zhang et al., 8 Sep 2025).

6. Limitations, computational profile, and broader significance

Zo3T’s gains are accompanied by substantial inference cost. On an NVIDIA A100 80GB, with 576×1024 resolution, 14 frames, and 50 DDPM/DDIM steps, using TTT and GFR from $(u_{b,k}, v_{b,k})$9 to $k$0, the average runtime is approximately 175 seconds per video, with peak GPU memory around 50 GB. The paper attributes the memory cost primarily to backpropagation through the U-Net at high resolution. It notes that optimized attention implementations such as FlashAttention or SageAttention could reduce this overhead (Zhang et al., 8 Sep 2025).

The method’s limitations are also explicit. It relies on monocular depth and a simple affine projection, so complex 3D interactions, strong occlusions, or self-occlusions are not perfectly modeled. Experiments are limited to 14-frame sequences, and longer videos would increase computation and may accumulate errors. The control interface is region-based, using bounding boxes rather than articulated body or mesh control, so highly articulated motion lies outside the paper’s scope. Depth estimation errors from DepthPro may produce incorrect scaling, although the authors state that the system is somewhat robust to such errors (Zhang et al., 8 Sep 2025).

A second misconception concerns the meaning of “3D-aware.” Zo3T does not perform full 3D reconstruction or explicit scene geometry modeling. Its 3D awareness is a pseudo-3D perspective prior derived from monocular depth and affine box scaling. This is sufficient to improve physical plausibility in the tested setting, but it is not equivalent to explicit 3D scene reasoning (Zhang et al., 8 Sep 2025).

Within the broader research landscape, Zo3T sits at the intersection of diffusion-based video generation, 3D-aware generative control, and test-time adaptation. Its specific contribution is to show that ephemeral adapters, 3D-aware masking, and guidance-field optimization can be combined into a zero-shot control mechanism that is competitive with or superior to supervised trajectory-controlled video generators on the reported benchmarks. A plausible implication is that future controllable video systems may increasingly shift from dataset-level retraining toward more powerful inference-time adaptation strategies, especially where controllability, flexibility, and distributional robustness must be balanced on a per-instance basis (Zhang et al., 8 Sep 2025).