Uni4D: 4D Modeling from Monocular Video

• Uni4D is a 4D modeling framework that defines dynamic scene reconstruction by jointly recovering static geometry, dynamic point trajectories, and camera motion from monocular videos.
• It employs a staged optimization pipeline that integrates outputs from segmentation, tracking, and depth models through static bundle adjustment and non-rigid motion estimation.
• The method is training-free and modular, leveraging pretrained vision models to achieve competitive performance in pose estimation, depth accuracy, and dynamic trajectory recovery.

Searching arXiv for exact-title and closely related "Uni4D" papers to ground the article. Uni4D denotes a multi-stage optimization framework for 4D modeling from a single casual monocular video, introduced in “Uni4D: Unifying Visual Foundation Models for 4D Modeling from a Single Video” (Yao et al., 27 Mar 2025). In this usage, “4D” refers to joint recovery of static scene structure, dynamic geometry, camera trajectory, and dense 3D motion over time from an ordinary RGB video. The method is explicitly training-free: it does not retrain or fine-tune its constituent pretrained models, but instead combines their outputs through staged geometric optimization. The same name has also been used for later and distinct systems, including a self-supervised framework for point cloud videos (Zuo et al., 7 Apr 2025) and a retrieval-grounded framework for controlled 4D generation (Xu et al., 25 Dec 2025). In the literature on monocular dynamic-scene reconstruction, however, Uni4D most commonly refers to the single-video dynamic modeling system of (Yao et al., 27 Mar 2025).

1. Definition and scope

Uni4D addresses 4D modeling from a single casual monocular video: given a handheld or casually captured RGB sequence of a dynamic scene, it reconstructs a temporally coherent representation consisting of static scene geometry, dynamic point trajectories, and camera parameters (Yao et al., 27 Mar 2025). The method targets scenes with moving objects, occlusions, unknown camera intrinsics, large viewpoint changes, and depth ambiguities, where classical SfM or SLAM assumptions are violated by non-rigid or independently moving content.

Its output is scene-centric and point-based. The reconstructed scene is represented as

$$\mathcal{P} = \{ \mathbf{P}_\mathrm{static}, \mathcal{P}_\mathrm{dyn}\},$$

where $\mathbf{P}_\mathrm{static}$ is a time-invariant static point cloud and

$$\mathcal{P}_\mathrm{dyn} = \{\mathbf{p}_k \in \mathbb{R}^{T\times 3}\}_{k}$$

is a set of dynamic 3D point trajectories in world coordinates (Yao et al., 27 Mar 2025). Camera parameters are

$\mathcal{C} = (\mathcal{T}, \mathbf{K}),$

with poses

$$\mathcal{T} = \{\boldsymbol{\xi}_t \in \mathbb{SE}(3)\}_{t=0}^T,$$

and a shared intrinsic matrix $\mathbf K$ whose optimized parameters are $f_x$ and $f_y$ (Yao et al., 27 Mar 2025).

This representation distinguishes Uni4D from canonical-template or neural-field methods. It does not infer a canonical deformable object with a learned warp field, nor does it train a scene-specific radiance field. Its central claim is instead that multiple pretrained vision foundation models already provide strong but partial observations of the same latent 4D world, and that these can be unified through explicit energy minimization (Yao et al., 27 Mar 2025).

2. Foundation-model inputs and scene representation

Uni4D is organized around three pretrained visual signals: dynamic masks, 2D point trajectories, and monocular depth with initial intrinsics (Yao et al., 27 Mar 2025). These are treated as pseudo-observations, initialization cues, and filtering constraints in subsequent optimization.

For dynamic object discovery and segmentation, the system uses RAM to identify semantic classes present in the video, GPT-4o to filter those classes into dynamic versus static/background categories, Grounding-SAM to segment dynamic objects on keyframes, and DEVA to propagate and track the segments over time, producing the dynamic masks

$\mathcal{M}=\{\mathbf{M}_t\}_{t=0}^T$

(Yao et al., 27 Mar 2025). These masks partition the scene into regions that will supervise static bundle adjustment and regions that will supervise dynamic non-rigid bundle adjustment.

For motion correspondences, Uni4D uses CoTracker3, run bidirectionally on a dense grid every 10 frames, to obtain 2D tracklets

$$\mathcal{Z}=\{\mathbf{Z}_k \in \mathbb{R}^{T\times 2}\}_{k=0}^K$$

with visibility only where tracking succeeds (Yao et al., 27 Mar 2025). These trajectories are central: static tracklets constrain camera and static structure, while dynamic tracklets are lifted into 3D motion trajectories.

For depth and initial intrinsics, Uni4D uses UniDepthV2 to produce per-frame depth maps

$\mathbf{P}_\mathrm{static}$0

and an initial intrinsic matrix $\mathbf{P}_\mathrm{static}$1 (Yao et al., 27 Mar 2025). These predictions are not final outputs; rather, they initialize stage-1 camera estimation and later support depth-aligned densification.

A notable feature of the framework is its modularity. The paper emphasizes that no retraining or fine-tuning is required, and supplementary ablations swap depth estimators and trackers to illustrate that improved components can, in principle, be dropped into the same optimization pipeline (Yao et al., 27 Mar 2025). This suggests a systems-level interpretation of “unifying visual foundation models”: unification occurs through explicit geometry rather than through a newly trained end-to-end network.

3. Multi-stage optimization pipeline

Uni4D proceeds in four broad steps: extracting foundation-model cues, initializing camera poses, refining static geometry and camera with bundle adjustment, estimating dynamic 3D trajectories with non-rigid bundle adjustment, and finally densifying the result by aligning and fusing depth maps into a dense 4D reconstruction (Yao et al., 27 Mar 2025). The authors explicitly motivate this divide-and-conquer structure by the nonconvexity of the full problem and by the observation that erroneous dynamic evidence can degrade pose estimation if the variables are optimized jointly throughout.

The first optimization stage estimates camera parameters only. It uses static-region tracklets together with predicted depth to enforce cross-frame reprojection consistency: $\mathbf{P}_\mathrm{static}$2 This is applied over frame pairs within a temporal sliding window of 5 frames (Yao et al., 27 Mar 2025). Once initialized, the depth maps are unprojected into a common world frame to provide an initial scene structure.

The second stage jointly refines camera poses and static geometry with a static bundle-adjustment objective plus a temporal camera-motion prior: $\mathbf{P}_\mathrm{static}$3 This stage uses only static tracklets, as determined by the dynamic masks, and is the point at which the rigid backbone of the scene is stabilized (Yao et al., 27 Mar 2025).

The third stage estimates dynamic 3D trajectories while freezing the camera: $\mathbf{P}_\mathrm{static}$4 Freezing the camera is an explicit design choice. The authors report that allowing dynamic optimization to continue updating pose makes the system less robust, because erroneous dynamic evidence can corrupt the recovered camera trajectory (Yao et al., 27 Mar 2025).

The final stage densifies the reconstruction by aligning predicted monocular depth with the optimized point-based solution, yielding dense per-pixel depth and a denser 4D point-cloud reconstruction (Yao et al., 27 Mar 2025). This densification is not an afterthought: it is the mechanism that converts the semi-dense track-based optimization result into a dense geometric output.

4. Energy formulation and regularization

The overall optimization is expressed as an energy over camera, static structure, and dynamic structure: $\mathbf{P}_\mathrm{static}$5 (Yao et al., 27 Mar 2025). Each term has a specific geometric role.

The static bundle-adjustment term is

$\mathbf{P}_\mathrm{static}$6

where $\mathbf{P}_\mathrm{static}$7 is a static world-space point corresponding to track $\mathbf{P}_\mathrm{static}$8, $\mathbf{P}_\mathrm{static}$9 is the camera pose, and $$\mathcal{P}_\mathrm{dyn} = \{\mathbf{p}_k \in \mathbb{R}^{T\times 3}\}_{k}$$0 is a visibility indicator (Yao et al., 27 Mar 2025). This is a standard reprojection objective adapted to dynamic scenes by excluding tracklets in dynamic regions.

The dynamic non-rigid bundle-adjustment term is

$$\mathcal{P}_\mathrm{dyn} = \{\mathbf{p}_k \in \mathbb{R}^{T\times 3}\}_{k}$$1

Here each track is associated with a full 3D trajectory $$\mathcal{P}_\mathrm{dyn} = \{\mathbf{p}_k \in \mathbb{R}^{T\times 3}\}_{k}$$2 rather than a single static point (Yao et al., 27 Mar 2025). Because this lifting is underconstrained from monocular supervision alone, dynamic regularization is required.

The motion prior decomposes as

$$\mathcal{P}_\mathrm{dyn} = \{\mathbf{p}_k \in \mathbb{R}^{T\times 3}\}_{k}$$3

where the ARAP term is

$$\mathcal{P}_\mathrm{dyn} = \{\mathbf{p}_k \in \mathbb{R}^{T\times 3}\}_{k}$$4

and the temporal smoothness term is

$$\mathcal{P}_\mathrm{dyn} = \{\mathbf{p}_k \in \mathbb{R}^{T\times 3}\}_{k}$$5

(Yao et al., 27 Mar 2025). The first encourages local rigidity among neighboring dynamic tracks; the second suppresses temporal jitter.

Camera regularization is handled separately via

$$\mathcal{P}_\mathrm{dyn} = \{\mathbf{p}_k \in \mathbb{R}^{T\times 3}\}_{k}$$6

with normalized second-order penalties

$$\mathcal{P}_\mathrm{dyn} = \{\mathbf{p}_k \in \mathbb{R}^{T\times 3}\}_{k}$$7

$$\mathcal{P}_\mathrm{dyn} = \{\mathbf{p}_k \in \mathbb{R}^{T\times 3}\}_{k}$$8

This prior is designed to penalize abrupt pose changes without overpenalizing genuinely large camera motions (Yao et al., 27 Mar 2025).

5. Static–dynamic decomposition and dense motion recovery

A defining property of Uni4D is that it treats static–dynamic decomposition as a prerequisite for reliable 4D modeling rather than as a by-product of reconstruction. Dynamic masks generated by RAM, GPT-4o, Grounding-SAM, and DEVA partition the scene into $$\mathcal{P}_\mathrm{dyn} = \{\mathbf{p}_k \in \mathbb{R}^{T\times 3}\}_{k}$$9 for static content and $\mathcal{C} = (\mathcal{T}, \mathbf{K}),$0 for dynamic content (Yao et al., 27 Mar 2025). Static tracklets supervise the rigid stages of optimization; dynamic tracklets are deferred until camera and static structure have been stabilized.

This decomposition is binary at the track level. Although the masks may originate from multiple segmented objects, the optimization itself does not model multiple explicit object identities. All dynamic points are pooled into $\mathcal{C} = (\mathcal{T}, \mathbf{K}),$1, and the ARAP prior is imposed through KNN neighborhoods among these tracks rather than through object-level kinematic models (Yao et al., 27 Mar 2025). This makes the formulation class-agnostic, but it also implies that the method does not perform explicit object-centric scene graph inference.

Dense 3D motion tracking is recovered by lifting dense 2D tracklets into world-space trajectories under the optimized camera and motion priors (Yao et al., 27 Mar 2025). This differs from both canonical-deformation methods and dense scene-flow field estimation. Uni4D does not estimate a volumetric motion field over space, nor does it infer a learned latent deformation model. Instead, it recovers explicit world-coordinate point trajectories whose temporal coherence is enforced by reprojection, ARAP, and smoothness.

The densification stage further aligns monocular depth to these optimized trajectories. For a pixel $\mathcal{C} = (\mathcal{T}, \mathbf{K}),$2, the local scale correction is

$\mathcal{C} = (\mathcal{T}, \mathbf{K}),$3

and the final dense depth is

$\mathcal{C} = (\mathcal{T}, \mathbf{K}),$4

(Yao et al., 27 Mar 2025). The paper states that $\mathcal{C} = (\mathcal{T}, \mathbf{K}),$5 contains the 3 nearest point trajectories in 3D to the unprojection of $\mathcal{C} = (\mathcal{T}, \mathbf{K}),$6, with weights $\mathcal{C} = (\mathcal{T}, \mathbf{K}),$7. This yields a tracklet-aware correction of temporally inconsistent monocular depth while avoiding interpolation across object boundaries.

A plausible implication is that Uni4D’s densification stage is doing more than smoothing depth: it uses the optimized sparse 4D structure as a global geometric scaffold to calibrate dense but noisy per-frame predictions.

6. Empirical performance, ablations, and limitations

Uni4D is evaluated on camera pose estimation and video depth. On Sintel, it reports $\mathcal{C} = (\mathcal{T}, \mathbf{K}),$8, $\mathcal{C} = (\mathcal{T}, \mathbf{K}),$9, and $$\mathcal{T} = \{\boldsymbol{\xi}_t \in \mathbb{SE}(3)\}_{t=0}^T,$$0, while the known-intrinsics variant Uni4D* improves to $$\mathcal{T} = \{\boldsymbol{\xi}_t \in \mathbb{SE}(3)\}_{t=0}^T,$$1 (Yao et al., 27 Mar 2025). On TUM-dynamics it reports $$\mathcal{T} = \{\boldsymbol{\xi}_t \in \mathbb{SE}(3)\}_{t=0}^T,$$2, and on Bonn $$\mathcal{T} = \{\boldsymbol{\xi}_t \in \mathbb{SE}(3)\}_{t=0}^T,$$3 (Yao et al., 27 Mar 2025). For video depth, under per-sequence scale+shift on Sintel it achieves Abs Rel $$\mathcal{T} = \{\boldsymbol{\xi}_t \in \mathbb{SE}(3)\}_{t=0}^T,$$4 and $$\mathcal{T} = \{\boldsymbol{\xi}_t \in \mathbb{SE}(3)\}_{t=0}^T,$$5; on Bonn $$\mathcal{T} = \{\boldsymbol{\xi}_t \in \mathbb{SE}(3)\}_{t=0}^T,$$6 and $$\mathcal{T} = \{\boldsymbol{\xi}_t \in \mathbb{SE}(3)\}_{t=0}^T,$$7; on KITTI $$\mathcal{T} = \{\boldsymbol{\xi}_t \in \mathbb{SE}(3)\}_{t=0}^T,$$8 and $$\mathcal{T} = \{\boldsymbol{\xi}_t \in \mathbb{SE}(3)\}_{t=0}^T,$$9 (Yao et al., 27 Mar 2025).

The paper emphasizes that these results are competitive or state of the art among joint pose-and-depth methods, particularly on real dynamic videos, and that learning-based VO baselines can generalize less well from synthetic to real scenes (Yao et al., 27 Mar 2025). Qualitatively, Uni4D is reported to produce cleaner static geometry, fewer trailing artifacts, and better moving-object reconstructions than CasualSAM and MonST3R (Yao et al., 27 Mar 2025).

The ablations support the staged design. On Sintel, stage 1 alone yields $\mathbf K$0, whereas stage 2 alone from weak initialization fails badly with $\mathbf K$1, and the full method reaches $\mathbf K$2 (Yao et al., 27 Mar 2025). A depth-consistency ablation reports that UniDepth self-consistency on Sintel improves from SC $\mathbf K$3 to $\mathbf K$4, and from $\mathbf K$5 to $\mathbf K$6 after Uni4D optimization (Yao et al., 27 Mar 2025). Ablations also indicate that CoTracker3 and DEVA are effective design choices, and that mask quality is a major bottleneck: using ground-truth segmentation improves dynamic-region depth further (Yao et al., 27 Mar 2025).

The method is not without limitations. It depends heavily on the quality of frozen segmentation, tracking, and depth models (Yao et al., 27 Mar 2025). Supplementary failure cases include incomplete or incorrect dynamic masks, trailing pixels around thin structures, poor camera pose, and incorrect dynamic trajectories (Yao et al., 27 Mar 2025). Large dynamic objects dominating the frame can break pose estimation, as in Sintel failure cases such as Cave 2 and Temple 3 (Yao et al., 27 Mar 2025). The final output is also a point cloud rather than a watertight mesh or photorealistic neural renderer (Yao et al., 27 Mar 2025).

The reported runtime is about 5 minutes for a 50-frame video on an RTX A6000, with 600 iterations per sliding window in stage 1, 2000 iterations in stage 2, and 1000 iterations in stage 3 (Yao et al., 27 Mar 2025). This places Uni4D in the category of test-time optimization systems rather than feedforward reconstruction networks.

7. Broader usage of the name and conceptual significance

The name “Uni4D” is not unique to (Yao et al., 27 Mar 2025). “Uni4D: A Unified Self-Supervised Learning Framework for Point Cloud Videos” (Zuo et al., 7 Apr 2025) uses the term for self-supervised representation learning on 4D point cloud videos, with a self-disentangled masked autoencoder, geometry tokens, latent tokens, and alignment losses for action recognition, gesture recognition, and action segmentation. “A Three-Level Alignment Framework for Large-Scale 3D Retrieval and Controlled 4D Generation” (Xu et al., 25 Dec 2025) uses “Uni4D” for a retrieval-grounded multimodal system that couples large-scale text-to-3D retrieval with controlled 4D generation. These systems are conceptually distinct from the monocular dynamic-scene modeling framework of (Yao et al., 27 Mar 2025).

Within the specific context of 4D modeling from monocular video, Uni4D is best understood as a methodological statement as much as a reconstruction system. Its central thesis is that modern pretrained models for segmentation, tracking, and monocular depth already encode much of the information required for dynamic scene understanding, and that the missing ingredient is not necessarily another large trained model but a geometry-aware optimization layer that unifies them (Yao et al., 27 Mar 2025). This places Uni4D between classical geometric vision and contemporary foundation-model reuse.

A common misconception is to treat Uni4D as a learned end-to-end 4D foundation model. In the terminology of (Yao et al., 27 Mar 2025), this would be inaccurate. The system is explicitly training-free, optimization-based, and modular. Another possible misunderstanding is to equate it with canonical-space dynamic reconstruction methods such as Ub4D (Johnson et al., 2022). Uni4D does not use a canonical SDF and deformation field; it reconstructs static world geometry plus dynamic world-coordinate point trajectories. The difference is structural rather than superficial.

Taken in that sense, Uni4D is a representative instance of a broader research trend: composing specialized frozen experts under explicit scene optimization rather than retraining monolithic networks. This suggests a durable implication for 4D vision. If stronger segmentation, depth, or tracking models become available, the Uni4D pipeline can in principle absorb those gains without revising its geometric formulation (Yao et al., 27 Mar 2025).