SEE4D: Pose-Free 4D Generation

• SEE4D is a pose-free 4D generation framework that converts a single handheld monocular clip into synchronized multi-view videos.
• It employs depth-guided warping alongside auto-regressive video inpainting to manage occlusions and dynamic content without manual pose annotations.
• The method uses diffusion-based inpainting with spatiotemporal transformers and overlapping processing to ensure high-quality synthesis for VR replay and film applications.

SEE4D is a pose-free 4D generation framework for turning a single, hand-held monocular clip into a time-aligned set of videos from multiple viewpoints, yielding a spatiotemporal representation that can be fused into a dynamic 4D scene for free-view VR playback. The method is formulated as a trajectory-to-camera system rather than a trajectory-to-trajectory system: instead of predicting an output camera path, it renders to a bank of synchronized, fixed virtual views and uses auto-regressive video inpainting to complete occluded or missing content. Its central claim is that rendering to fixed virtual cameras separates camera control from scene modeling, avoids manually annotated camera poses, and improves stability and generalization on dynamic, in-the-wild videos (Lu et al., 30 Oct 2025).

1. Problem setting and conceptual shift

SEE4D addresses 4D generation from casual videos, a setting in which the input is unposed, noisy, monocular, and often contains dynamic content and occlusions. In this regime, frame-accurate 6-DoF camera poses are labor-intensive to annotate and brittle for in-the-wild videos, while dense 3D ground truth for dynamic scenes is often unavailable. The paper therefore treats pose labels and explicit 3D supervision as practical bottlenecks rather than prerequisites (Lu et al., 30 Oct 2025).

The method is positioned against prior warp-then-inpaint and trajectory-to-trajectory pipelines. Those systems take an input camera path and synthesize a different output path, which the paper characterizes as an entanglement of camera motion and scene dynamics: the model must jointly preserve temporal behavior while changing viewpoint. The stated consequences are more difficult training and inference, a bias toward staying near the source trajectory, and increased sensitivity to depth errors.

SEE4D replaces that formulation with a trajectory-to-camera formulation. The system predicts synchronized videos for a set of fixed virtual cameras instead of an output trajectory. This decoupling is the organizing idea of the framework: camera control is the choice of which views to synthesize or traverse between, while scene modeling is the task of maintaining content over time, including motion and disocclusion. A common misunderstanding is to equate “pose-free” with the absence of camera parameterization. In SEE4D, virtual cameras still use standard pinhole intrinsics and extrinsics; what is removed is the dependence on manually annotated pose labels during training and conditioning (Lu et al., 30 Oct 2025).

2. Virtual camera bank and depth-guided rendering

The virtual-camera representation is explicit. Each virtual camera is parameterized by intrinsics $K$ and extrinsics $T \in SE(3)$, where $T$ encodes rotation and translation from world to camera. During inference, SEE4D renders the input to a set of virtual cameras with fixed intrinsics and extrinsics. Users specify source-to-target view pairs, and the pipeline spline-interpolates intermediate virtual cameras to bridge the gap (Lu et al., 30 Oct 2025).

Depth-guided warping provides the structured conditioning for the inpainting model. With per-pixel depth $D$, camera intrinsics $K$, and a relative transform $T \in SE(3)$, the forward projection from a source pixel $\mathbf{u}=(u,v)$ to the target pixel $\mathbf{u}_w$ is

$$\mathbf{u}_w= \pi\bigl( \mathbf{K}\mathbf{T} [\mathbf{D}(\mathbf{u})\mathbf{K}^{-1}[u,v,1]^{T};1] \bigr),$$

where $\pi([u,v,w]^T)=[u/w,v/w]^T$.

The rendering stage uses bilinear splatting of source pixels to sub-pixel targets, producing a warped frame $T \in SE(3)$0 and a mask $T \in SE(3)$1 that flags off-screen projections and z-buffer visibility conflicts. This pair is intended to tell the inpainting model which regions are reliable, because they are supported by the warp, and which must be hallucinated, because they are masked. The paper explicitly emphasizes that SEE4D implements “realistically synthesized warped images” containing tearing, stretching, and occlusion artifacts typical of monocular depth warping under viewpoint changes (Lu et al., 30 Oct 2025).

The role of the fixed virtual camera bank is methodological as much as geometric. Because the system renders to synchronized fixed views, it focuses on producing temporally aligned clips per view, while the user or a downstream system controls which views to render. This suggests a division of labor in which view planning is externalized and the generative model concentrates on geometry-aware completion.

3. View-conditional video inpainting and geometry priors

The generative core is a diffusion-based view-conditional video inpainting model. Videos are compressed with a VAE encoder $T \in SE(3)$2, denoised in latent space with a U-Net/Transformer backbone, and decoded with the VAE decoder. The conditioning variables are the source latent $T \in SE(3)$3 and a warp/mask tensor $T \in SE(3)$4, rather than explicit camera poses. The paper’s training objective is the standard latent-space denoising loss

$T \in SE(3)$5

The backbone includes a lightweight spatiotemporal transformer added to a multi-view U-Net. Two stated components are frame-time embeddings, implemented as sinusoidal codes injected into residual paths, and spatial–temporal attention operating jointly across spatial tokens and neighboring frames. The paper attributes cross-frame consistency to this design (Lu et al., 30 Oct 2025).

A central training device is Realistic Warp Synthesis, which is described as a four-step augmentation scheme that simulates test-time artifacts:

1. Depth prediction for target frames with a monocular estimator to obtain $T \in SE(3)$6 and lift $T \in SE(3)$7 to a point cloud.
2. Random scene re-pose: segment foreground (BEN), pick an origin at the largest object’s center region, rotate by a random axis in $T \in SE(3)$8 and translate with $T \in SE(3)$9, giving synthetic transform $T$0.
3. Forward projection: render the transformed point cloud with z-buffer and bilinear splatting to obtain warped image $T$1 and depth $T$2, introducing realistic tearing/stretching.
4. Back-projection with pose jitter: warp reversely with $T$3 and a jittered $T$4, producing misaligned but informative $T$5 and a z-buffer-derived $T$6 capturing occlusions and off-screen regions.

This procedure is used to teach the network to respect geometric cues where the warp is reliable and to inpaint disocclusions coherently across time and views. The paper frames this as learning a robust geometry prior without explicit 3D annotations (Lu et al., 30 Oct 2025).

The conditioning is further modified by a Noise-Adaptive Condition. The stated purpose is to prevent overfitting to unreliable warps by increasing noise on the warp latent relative to the target latent. The blend is modulated by mask density

$T$7

a time-dependent weight $T$8, and a scale $T$9. The text specifies that reliable warps contribute more early, while unreliable warps are down-weighted over time to favor refinement from the target latent.

4. Spatiotemporal autoregressive inference

SEE4D factors inference into spatial recursion and temporal recursion. For spatial traversal, the source-to-target viewpoint shift is decomposed into $D$0 small hops along a spline over extrinsics, and intrinsics can also vary if needed. At each hop, the current frames are warped and inpainted to the next virtual camera. The paper states that depth is re-estimated and scale-aligned between hops with a global solver as in Shape-of-Motion/See3D, with the goal of reducing drift across wide baselines (Lu et al., 30 Oct 2025).

For temporal extension, the video is processed in windows of length $D$1 with overlap $D$2. For window $D$3, the last $D$4 clean latents from window $D$5 are prepended to the current noisy latents, denoised, and the window then slides forward. The overlap acts as an anchor preserving motion and appearance continuity across boundaries.

The pipeline therefore traverses virtual-camera splines while extending videos with overlapping windows. The explicit rationale is bounded per-step complexity: each step processes a manageable spatial hop and a fixed-length temporal window, so complexity grows roughly with window length and feature-map size rather than sequence duration. This is the basis for the paper’s claim that the method scales to long sequences and wide baselines (Lu et al., 30 Oct 2025).

The inference loop is given at pseudocode level. Inputs are the source video $D$6, target virtual camera parameters $D$7, window length $D$8, overlap $D$9, number of spatial hops $K$0, and a spline $K$1. At each spatial hop, and then for each temporal window, the pipeline performs depth estimation, warping to the intermediate pose, latent encoding, construction of the noise-adaptive condition, denoising over $K$2 frames with overlap reuse, latent decoding, sequence update, and depth re-estimation for the next window. The output is a temporally aligned video at the target camera, repeated for each target view in the bank.

5. Training protocol and empirical performance

The model is trained on 170K synchronized multi-view dynamic clips from synthetic datasets. SynCamMaster contributes 34K clips from 3.4K scenes filmed by ten static cameras, and ReCamMaster contributes 136K clips from 13.6K scenes captured by nine moving and one fixed camera. Source and target clips are sampled per scene so that the model learns cross-view conditioning without explicit pose labels (Lu et al., 30 Oct 2025).

The reported optimization configuration is: initialization from See3D weights; resolution $K$3; 16-frame sequences; batch size 16; 10K iterations; learning rate $K$4; bfloat16 precision; classifier-free guidance with 0.1 probability of dropping visual conditions; and compute on 8 NVIDIA A800-80GB GPUs for approximately 48 hours. Inference uses a DDIM sampler with the same guidance strategy.

Evaluation is conducted on two benchmarks. The first is 4D reconstruction on the iPhone dataset, where a monocular source is paired with a synchronized fixed-camera target and performance is measured by PSNR, SSIM, and LPIPS. The second is cross-view video generation on 200 in-the-wild WebVid clips with randomized static target viewpoints such as pan, tilt, and hemispherical arcs, evaluated with VBench metrics including subject consistency, background consistency, temporal flicker, motion smoothness, image quality, and aesthetic quality (Lu et al., 30 Oct 2025).

On iPhone 4D reconstruction, the average metrics reported for SEE4D are PSNR 14.56, SSIM 0.442, and LPIPS 0.492. The paper compares these with TrajectoryCrafter at 14.24 / 0.417 / 0.519 and ReCamMaster at 12.02 / 0.387 / 0.566, and states that per-scene gains are consistent across Apple, Block, Paper, Spin, and Teddy. On VBench generation, SEE4D reports subject consistency 92.18, background consistency 94.63, temporal flicker 96.66, motion smoothness 97.87, image quality 53.15, and aesthetic quality 41.35.

The qualitative findings are summarized in terms of sharper geometry, stable parallax, reduced flicker in reconstruction, and smooth occlusion-aware texture synthesis with seamless transitions in cross-view generation. The reported failure patterns of baselines are blur, bleed-through, or jitter under depth noise and wide baselines.

6. Ablations, limitations, and practical significance

The ablation study is designed to isolate the framework’s stated causal components. Replacing Realistic Warp Synthesis with simpler schemes from prior work causes large drops, with PSNR falling from 14.56 to 11.93–12.48. Removing the Noise-Adaptive Condition reduces PSNR from 14.56 to 13.47 and SSIM from 0.442 to 0.411, while increasing LPIPS from 0.492 to 0.508. Removing temporal modules from the Spatial–Temporal Backbone degrades performance to PSNR 10.66, SSIM 0.347, and LPIPS 0.685. In the inference pipeline, removing spatial hops yields PSNR 13.58 and LPIPS 0.518, while removing temporal overlap yields PSNR 13.02 and LPIPS 0.517. The paper treats both spatial decomposition and overlapping windows as indispensable for high-quality synthesis (Lu et al., 30 Oct 2025).

The limitations are also explicit. Fast motion, thin structures, and severe disocclusions can expose monocular depth errors, causing tearing in the warp and challenging inpainting. Extremely wide baselines or extreme camera paths increase off-screen regions and reduce conditioning reliability; spline hops mitigate but do not remove this difficulty. Highly reflective or translucent materials and repeated patterns can degrade depth estimation and weaken geometry priors. Although the system is pose-free, it still relies on depth estimators and segmentation, so failures in upstream cues propagate into warps and masks.

These constraints help clarify what SEE4D is and is not. It does not eliminate geometry from the pipeline; rather, it shifts geometry from explicit pose supervision to depth-warp and mask conditioning. It does not remove camera control; it relocates it to a bank of fixed virtual cameras and spline traversal. A plausible implication is that the framework is especially suited to settings where camera labels are unavailable but monocular depth cues remain informative.

The practical implications described in the paper are multi-angle replay and re-framing for VR and film, robot manipulation through view augmentation for grasp planning, autonomous driving through augmented dash-cam coverage, and interactive gaming through fly-around sequences. The paper also states that SEE4D produces synchronized fixed-view videos that downstream 4D reconstruction, including dynamic Gaussian splatting, can fuse into dense space-time representations. In that sense, SEE4D functions both as a generative model and as an intermediate representation for practical 4D world modeling from casual videos (Lu et al., 30 Oct 2025).