---
title: 'Zo3T: Zero-shot 3D-Aware I2V Generation'
url: https://www.emergentmind.com/topics/zo3t
type: topic
---

# Zo3T: Zero-shot 3D-Aware I2V Generation

Zo3T, short for **Zero-shot 3D-Aware Trajectory-Guided image-to-video generation via Test-Time Training**, is a framework for **trajectory-guided image-to-video (I2V) generation** that synthesizes videos from an input image and user-specified object or region trajectories while preserving appearance, motion accuracy, photorealism, and temporal coherence. It operates in a **zero-shot** regime—without fine-tuning the backbone on trajectory-annotated data—and instead performs **test-time training (TTT)** by optimizing the latent state and an ephemeral LoRA adapter on each individual test instance. The method is built on **Stable Video Diffusion (SVD)** and uses **DepthPro** for monocular depth estimation, combining **3D-Aware Kinematic Projection**, **Trajectory-Guided Test-Time LoRA**, **Guidance Field Rectification**, and **Fourier Orthogonal Recomposition** into a single inference-time control pipeline [2509.06723].

## 1. Problem setting and conceptual scope

Zo3T addresses the setting in which the input consists of an initial image \(I_0\), together with one or more user-specified trajectories for controlled entities. For each controlled entity \(b \in \{1,\dots,M\}\), the user provides an initial bounding box \(\mathcal{B}_{b,0}\) and a 2D trajectory
\[
\mathcal{T}_{2D}^b = \{(u_{b,k}, v_{b,k})\}_{k=1}^{N_f},
\]
where \((u_{b,k}, v_{b,k})\) denotes the target center position in frame \(k\). The generative objective is to synthesize a latent video \(\{z_k\}_{k=0}^{N_f-1}\), later decoded to frames, such that the controlled region follows the prescribed path while its identity and appearance remain consistent with the first frame [2509.06723].

The framework is explicitly formulated as **zero-shot** and **test-time adaptive** rather than supervised retraining. The underlying SVD backbone remains frozen; only per-example optimization is performed during inference. This distinguishes Zo3T from training-based trajectory-control systems that require large trajectory-annotated datasets and substantial optimization budgets. A common misconception is that “zero-shot” implies purely feed-forward inference. In Zo3T, zero-shot instead means that no task-specific offline fine-tuning is performed; the method still carries out substantial **instance-specific optimization at test time** [2509.06723].

The design target is not merely motion transfer in 2D image coordinates. Zo3T is motivated by three coupled constraints: **3D perspective consistency**, **on-manifold latent evolution**, and **accurate adherence to user trajectories**. The paper argues that purely 2D control ignores depth-dependent scaling, while direct latent editing can misalign manipulated states with the denoiser’s learned noise distribution, causing texture collapse, identity loss, flickering, and broader fidelity degradation [2509.06723].

## 2. Relation to prior trajectory-controlled I2V methods

The method is positioned against both **training-based** and **zero-shot** baselines. Training-based systems such as DragNUWA, DragAnything, LeviTor, and MotionCtrl are described as relying on fine-tuning video diffusion models on trajectory-annotated data. The cited limitations are expensive training, reliance on scarce annotations, frequent operation at low resolutions such as \(320\times576\), overfitting toward motion adherence at the expense of object integrity, and reduced flexibility for unseen motion patterns or new domains [2509.06723].

Among these, LeviTor and ObjCtrl-2.5D are highlighted for using 3D cues, but with different trade-offs. LeviTor leverages 3D trajectory-oriented training yet requires costly training on extensive masked datasets. ObjCtrl-2.5D treats object motion as camera motion in a 2.5D sense, which the paper states restricts complex scenes, especially those involving multi-object motion or independent foreground-background dynamics [2509.06723].

Zero-shot approaches are divided into two broad categories. The first performs **latent-space manipulation**, typically through feature consistency losses and attention-based alignment across frames. The stated failure mode is “hard editing” of \(\mathbf{z}_t\), which can push the state off the learned diffusion manifold and produce artifacts. The second category alters initial or intermediate noise to encode motion priors, but such methods are characterized as generally 2D, not 3D-aware, and coarse in fine-grained spatial consistency [2509.06723].

This positioning motivates Zo3T’s central claim: trajectory control should be implemented as a **soft, co-adaptive process** that aligns geometric priors, latent states, and denoising dynamics rather than forcing motion by rigid latent overrides. A plausible implication is that Zo3T treats motion control as a constrained inference problem over the pre-trained model rather than as a separate retraining objective [2509.06723].

## 3. Framework architecture and generative pipeline

Zo3T operates on **Stable Video Diffusion**, which is described as a **latent video diffusion model** with a VAE encoder, a U-Net-style denoiser \(\epsilon_\theta(\mathbf{z}_t, t, c)\), and a decoder back to pixel space. The method loads official pre-trained SVD weights and does not retrain the backbone [2509.06723].

Its high-level pipeline has four guided components followed by standard sampling. The first is **3D-Aware Kinematic Projection**, which uses DepthPro to infer monocular depth from \(I_0\), then converts 2D trajectories into perspective-aware affine transformations and frame-dependent masks. The second is **Trajectory-Guided Test-Time LoRA**, in which an ephemeral LoRA adapter is injected into the U-Net and jointly optimized with the latent state at selected denoising steps. The third is **Guidance Field Rectification (GFR)**, which optimizes the conditional guidance field through a one-step lookahead objective. The fourth is **Fourier Orthogonal Recomposition (FOR)**, which merges low-frequency content from optimized states with high-frequency content from the original states in order to preserve textural fidelity [2509.06723].

The paper describes the overall inference trajectory as follows: at selected timesteps, latent and LoRA optimization are performed; then the guidance field is rectified; FOR is applied to regularize the resulting optimized signals; and standard DDIM/DDPM denoising resumes for later steps. The output is a **14-frame, \(576\times1024\)** video [2509.06723].

| Component | Function | Mechanism |
|---|---|---|
| 3D-Aware Kinematic Projection | Perspective-consistent motion prior | DepthPro, affine transforms, time-varying masks |
| Trajectory-Guided Test-Time LoRA | Local model adaptation during inference | Joint latent and LoRA optimization with regional feature consistency loss |
| Guidance Field Rectification | Better denoising direction toward target motion | One-step lookahead optimization of \(\mathbf{d}_t\) |
| Fourier Orthogonal Recomposition | Fidelity-preserving regularization | Low-frequency optimized content plus original high-frequency content |

The architecture is notable for distributing control across geometry, representation, and sampler dynamics rather than localizing all control in a single latent-editing stage. This suggests a modular view of trajectory-guided generation in which different failure modes are handled at different layers of the diffusion process [2509.06723].

## 4. Core algorithmic mechanisms

### 4.1 3D-Aware Kinematic Projection

Zo3T begins from a depth map
\[
M_D = \mathcal{D}_{\text{depth}}(I_0),
\]
computed by DepthPro on the initial frame. For an object trajectory with center pixel coordinates \(\mathbf{p}_{b,k} = (u_{b,k}, v_{b,k})\), the approximate depth is read directly from the map:
\[
d_{b,k} = M_D(\mathbf{p}_{b,k}), \qquad d_{b,0} = M_D(\mathbf{p}_{b,0}).
\]
Using a pinhole camera approximation, Zo3T defines the frame-relative scaling factor as
\[
\sigma_k = \frac{d_0}{d_k}.
\]
The affine transform mapping the initial bounding box \(\mathcal{B}_0\) to frame \(k\) is
\[
\mathbf{A}_k =
\begin{pmatrix}
\sigma_k & 0 & u_k - \sigma_k u_0 \\
0 & \sigma_k & v_k - \sigma_k v_0 \\
0 & 0 & 1
\end{pmatrix}.
\]
This yields frame-dependent masks
\[
\mathcal{M}_k(\mathbf{p}) = [\mathbf{p} \in \mathbf{A}_k \mathcal{B}_0].
\]
These masks vary both in position and scale, thereby encoding a pseudo-3D motion prior rather than a fixed-size 2D box [2509.06723].

The paper argues that this matters not primarily because it dramatically improves the motion-control metric, but because it makes the subsequent feature consistency losses physically meaningful: a moving object that changes depth should not be compared against mismatched 2D crops of fixed size. The ablation reported for removing 3D projection shows only a modest change in ObjMC, but a larger qualitative decline in realism due to shape distortion under depth change [2509.06723].

### 4.2 Trajectory-Guided Test-Time LoRA

At selected denoising steps, Zo3T injects an **ephemeral LoRA adapter** into the SVD U-Net and jointly optimizes the latent \(\mathbf{z}_t\) and the LoRA weights \(\Delta\theta_{\text{LoRA}}\). The adapted parameters are denoted \(\theta' = \theta \oplus \Delta\theta_{\text{LoRA}}\), and the optimization objective is
\[
(\mathbf{z}_t^*, \theta_{\text{LoRA}}^*)
=
\operatorname{argmin}_{\mathbf{z}_t, \theta_{\text{LoRA}}}
\mathcal{J}_{\text{TTT}}(\mathbf{z}_t, \theta').
\]
The core loss is a **regional feature consistency loss**:
\[
\mathcal{J}_{\text{TTT}}
=
\sum_{b=1}^{M}
\sum_{l \in \mathcal{L}} w_l
\sum_{k=2}^{N_f}
\left\|
\mathcal{G}_b \odot
\left(
F_{l,k}[\mathcal{M}_{b,k}]
-
(F_{l,1})_{\text{frozen}}[\mathcal{M}_{b,1}]
\right)
\right\|_F^2.
\]
Here, \(F_{l,k}\) is a selected feature map at layer \(l\) and frame \(k\), \((F_{l,1})_{\text{frozen}}\) is the detached reference from frame 1, and \(\mathcal{G}_b\) is a Gaussian heatmap that focuses the loss toward the object interior. Because projected masks vary in size, the cropped feature regions are resized to a uniform size before comparison [2509.06723].

The central idea is **co-adaptation**. Rather than forcing \(\mathbf{z}_t\) alone to satisfy the motion constraint, the denoiser is locally adapted so that its internal representation becomes compatible with the manipulated latent. The paper describes this as “soft editing,” intended to reduce manifold deviation. Ablation indicates that removing TTT causes the largest degradation among the four main components, with a shift from **FID 74.83 / FVD 197.63 / ObjMC 12.74** in the full system to **FID 89.36 / FVD 219.53 / ObjMC 14.24** without TTT adaptation [2509.06723].

### 4.3 Guidance Field Rectification

In classifier-free guidance, the conditional control direction is
\[
\mathbf{d}_t = \bm{\epsilon}_t^{\text{cond}} - \bm{\epsilon}_t^{\text{uncond}}.
\]
Standard CFG uses this direction directly. Zo3T instead optimizes it. Let \(\mathcal{P}(\mathbf{z}_t^*, \mathbf{d})\) denote a one-step DDIM solver that predicts \(\mathbf{z}_{t-1}\) from the current latent using a candidate guidance field \(\mathbf{d}\). Zo3T defines a one-step **kinematic cost**
\[
\mathcal{J}_{\text{guide}}(\mathbf{d})
=
\frac{1}{2}
\sum_{b=1}^{M}
\sum_{k=2}^{N_f}
\int_{\mathcal{B}_{b,k}}
\mathcal{G}_b(\mathbf{p})
\left\|
\text{Pooling}\bigl(\mathcal{P}(\mathbf{z}_t^*, \mathbf{d})_k[\mathcal{B}_{b,1}]\bigr)
-
\text{Pooling}\bigl(\mathcal{P}(\mathbf{z}_t^*, \mathbf{d})_1[\mathcal{B}_{b,1}]\bigr)_{\text{frozen}}
\right\|_2^2
\, d\mathbf{p}.
\]
The guidance field is then optimized by gradient descent starting from the original CFG difference:
\[
\frac{\partial \mathbf{d}}{\partial \tau}
=
- \nabla_{\mathbf{d}} \mathcal{J}_{\text{guide}}(\mathbf{d}),
\]
producing \(\mathbf{d}_t^*\), and the rectified conditional noise is reconstructed as
\[
\bm{\epsilon}_t^{\text{cond,opt}}
=
\bm{\epsilon}_t^{\text{uncond}} + \mathbf{d}_t^*.
\]
The loss uses **pooled** features rather than pixel-wise alignment, which the paper states encourages more organic motion and avoids excessive rigidity [2509.06723].

### 4.4 Fourier Orthogonal Recomposition

FOR is used to preserve the diffusion model’s original high-frequency texture content while importing optimized low-frequency structural changes. The paper describes this as a Fourier-domain recomposition that acts as a strong regularizer and prevents optimization artifacts from dominating textural detail. Its ablation indicates that removing FOR leaves ObjMC nearly unchanged but markedly worsens generative quality, with **FID 95.45** and **FVD 221.09** relative to the full system’s **FID 74.83** and **FVD 197.63** [2509.06723].

## 5. Optimization protocol and empirical evaluation

The optimization is confined to the early-to-mid denoising window \(t \in \{45,\dots,30\}\) in a **50-step DDPM/DDIM schedule**. At each such timestep, the TTT stage optimizes the latent state \(\mathbf{z}_t\) and LoRA weights \(\Delta\theta_{\text{LoRA}}\), and the GFR stage optimizes the guidance field \(\mathbf{d}\). The backbone parameters \(\theta\) remain frozen throughout. The optimizer is **Adam**, with learning rates chosen via ablation: **0.25** for the latent, **0.01** for the LoRA weights, and **0.05** for the guidance field [2509.06723].

The evaluation setup uses **Stable Video Diffusion** as the backbone and generates **14-frame videos at \(576\times1024\)**. For comparison with supervised baselines operating at \(320\times576\), generated outputs are downsampled to \(320\times576\) for evaluation. The motion-control benchmark is based on the open-source set in DragAnything and augmented by mirroring most trajectories, producing **143 challenging samples**. For FID and FVD, real videos are drawn from **VIPSeg**, generated videos are resized to \(256\times256\), and **FVD is computed using ViCLIP features** [2509.06723].

The reported metrics include **FID**, **FVD**, **Object Motion Consistency (ObjMC)**, and VBench metrics for **Subject Consistency**, **Background Consistency**, **Aesthetic Quality**, and **Imaging Quality**. ObjMC is defined as the average per-frame Euclidean distance between target and observed trajectories, where the observed object path is estimated using **CoTracker**. The authors explicitly note that ObjMC can be misleading when distorted objects remain trackable; they therefore emphasize qualitative analysis and a user study alongside the metric [2509.06723].

| Method | Setting | Key metrics |
|---|---|---|
| DragNUWA | Supervised | FID 126.31; FVD 251.04; ObjMC 10.84 |
| DragAnything | Supervised | FID 119.07; FVD 266.42; ObjMC 11.64 |
| LeviTor | Supervised | FID 79.41; FVD 207.23; ObjMC 12.06 |
| FreeTraj | Zero-shot | FID 92.12; FVD 230.21; ObjMC 31.63 |
| ObjCtrl-2.5D | Zero-shot | FID 81.06; FVD 212.17; ObjMC 18.72 |
| SG-I2V | Zero-shot | FID 79.36; FVD 209.53; ObjMC 14.43 |
| Zo3T | Zero-shot | FID 74.83; FVD 197.63; ObjMC 12.74 |

On the same benchmark, Zo3T attains the highest reported VBench scores: **Subject Consistency 0.9760**, **Background Consistency 0.9682**, **Aesthetic Quality 0.6779**, and **Imaging Quality 0.6820**. The paper’s interpretation is that Zo3T outperforms all zero-shot baselines across all metrics and surpasses supervised methods in FID and FVD while maintaining ObjMC values close to supervised systems [2509.06723].

The user study involves **6 expert evaluators** and **50 videos**, with comparisons against three strong baselines on imaging fidelity, motion coherence, trajectory accuracy, and overall quality. Zo3T receives **51.7\%** of all preference votes across pairwise comparisons [2509.06723].

## 6. Limitations, computational profile, and broader significance

Zo3T’s gains are accompanied by substantial inference cost. On an **NVIDIA A100 80GB**, with **576×1024** resolution, **14 frames**, and **50 DDPM/DDIM steps**, using TTT and GFR from \(t=45\) to \(t=30\), the average runtime is approximately **175 seconds per video**, with peak GPU memory around **50 GB**. The paper attributes the memory cost primarily to backpropagation through the U-Net at high resolution. It notes that optimized attention implementations such as **FlashAttention** or **SageAttention** could reduce this overhead [2509.06723].

The method’s limitations are also explicit. It relies on **monocular depth** and a **simple affine projection**, so complex 3D interactions, strong occlusions, or self-occlusions are not perfectly modeled. Experiments are limited to **14-frame** sequences, and longer videos would increase computation and may accumulate errors. The control interface is **region-based**, using bounding boxes rather than articulated body or mesh control, so highly articulated motion lies outside the paper’s scope. Depth estimation errors from DepthPro may produce incorrect scaling, although the authors state that the system is somewhat robust to such errors [2509.06723].

A second misconception concerns the meaning of “3D-aware.” Zo3T does not perform full 3D reconstruction or explicit scene geometry modeling. Its 3D awareness is a **pseudo-3D perspective prior** derived from monocular depth and affine box scaling. This is sufficient to improve physical plausibility in the tested setting, but it is not equivalent to explicit 3D scene reasoning [2509.06723].

Within the broader research landscape, Zo3T sits at the intersection of **diffusion-based video generation**, **3D-aware generative control**, and **test-time adaptation**. Its specific contribution is to show that **ephemeral adapters**, **3D-aware masking**, and **guidance-field optimization** can be combined into a zero-shot control mechanism that is competitive with or superior to supervised trajectory-controlled video generators on the reported benchmarks. A plausible implication is that future controllable video systems may increasingly shift from dataset-level retraining toward more powerful inference-time adaptation strategies, especially where controllability, flexibility, and distributional robustness must be balanced on a per-instance basis [2509.06723].

Source: https://www.emergentmind.com/topics/zo3t