Drag4D: Interactive 4D Scene and Motion Generation

• Drag4D is an interactive 4D framework that generates detailed 3D scenes from text, integrating object insertion and animation in a unified pipeline.
• It employs a three-stage process featuring enhanced text-to-3D reconstruction, physics-aware position learning, and part-augmented motion-conditioned video diffusion.
• The system ensures spatial and temporal coherence across views by reusing shared semantic features, achieving consistent geometry, semantics, and motion control.

Drag4D is an interactive framework for text-driven 3D scene generation that integrates controllable object motion into a unified 4D pipeline. It turns a text description into a high-quality 3D scene, inserts a user-specified object reconstructed from a single image, and animates that object along a user-defined 3D trajectory while maintaining spatial and temporal coherence across views. The system is organized as a three-stage pipeline: enhanced text-to-3D background reconstruction, 3D copy-and-paste with physics-aware object position learning, and part-augmented, motion-conditioned multiview video diffusion for temporal animation (Kang et al., 26 Sep 2025).

1. System definition and problem setting

Drag4D is designed around three coupled difficulties in controllable 4D generation. First, text-driven scene generation methods often struggle with multi-view consistency when outpainting unseen regions; panoramic approaches improve coverage but can still leave holes or artifacts in novel viewpoints. Second, trajectory-conditioned video generation is typically formulated in 2D, lacks multi-view consistency, and is not readily integrated with a text-generated 3D scene. Third, inserting a target object into a generated scene introduces alignment problems such as incorrect scale, floating, and collisions unless physics priors are imposed (Kang et al., 26 Sep 2025).

The central proposal is a unified three-stage architecture. Stage 1 strengthens text-to-3D background generation by training 2D Gaussian Splatting jointly on panoramic base images and inpainted novel views. Stage 2 reconstructs a full 360° object mesh from a single reference image and composes it into the scene using a 3D copy-and-paste procedure with physics-aware position learning. Stage 3 animates the spatially aligned object along a user-defined 3D path using a part-augmented, motion-conditioned multiview video diffusion model. The framework therefore links scene synthesis, object insertion, and trajectory-conditioned animation within one calibrated coordinate system.

A key architectural choice is the reuse of semantic features across stages. DINO and DIFT features are embedded in the 2D-GS representation during scene construction and then reused for part-aware motion conditioning. This shared representation supplies a consistent prior for both composition and animation. A plausible implication is that Drag4D treats geometry, semantics, and motion not as isolated subproblems, but as mutually constraining components of a single 4D generation process.

2. Enhanced text-to-3D background reconstruction

The first stage begins with a text-to-360° diffusion model that generates a panorama $I_p$. Metric3D provides depth $D_p$ and normals, enabling point-cloud initialization through inverse equirectangular projection,

$P = E^{-1}(I_p, D_p).$

For a panorama pixel at spherical angles $(\theta, \phi)$, the 3D direction is

$$d(\theta, \phi) = [\cos \theta \cos \phi, \sin \theta, \cos \theta \sin \phi],$$

and the corresponding 3D point is

$X = D_p(\theta, \phi)\, d(\theta, \phi).$

Base perspective images $I_i$ are then created by projecting the point cloud with shared intrinsic $K$ and extrinsics $E_i$,

$I_i = \Phi(P, K, E_i).$

To enlarge viewpoint coverage, augmented extrinsics $D_p$0 displace cameras off the panoramic center:

$D_p$1

Because these augmented views exhibit holes and boundary artifacts due to depth instability, the method extracts a mask $D_p$2 using a depth gradient threshold and inpaints uncertain regions with a high-resolution rectified-flow diffusion model:

$D_p$3

The reconstruction model is 2D Gaussian Splatting. Each 2D Gaussian primitive $D_p$4 has image-plane mean $D_p$5, covariance $D_p$6, color $D_p$7 or spherical harmonics, and opacity $D_p$8. Its footprint weight is

$D_p$9

Rendering uses front-to-back compositing,

$P = E^{-1}(I_p, D_p).$0

$P = E^{-1}(I_p, D_p).$1

The formulation is derived from 3D-GS by dropping the third row and column of the projected covariance $P = E^{-1}(I_p, D_p).$2, where $P = E^{-1}(I_p, D_p).$3 is world-to-camera and $P = E^{-1}(I_p, D_p).$4 is the Jacobian of projection (Kang et al., 26 Sep 2025).

Optimization combines base and augmented supervision. The total objective is

$P = E^{-1}(I_p, D_p).$5

Here $P = E^{-1}(I_p, D_p).$6 includes a photometric reconstruction term, depth distortion, and depth-normal consistency regularizers. For augmented views, the rendered view $P = E^{-1}(I_p, D_p).$7 is mixed with the inpainted result using depth-normal similarity $P = E^{-1}(I_p, D_p).$8 in the masked regions:

$P = E^{-1}(I_p, D_p).$9

Depth-normal similarity downweights noisy areas and is, in practice, the cosine similarity of normals modulated by depth agreement. The stage also distills DINO and DIFT features into splats under SAM2 masks through $(\theta, \phi)$0.

This stage is intended to improve multi-view consistency, reduce unseen-view distortions, and produce dense geometry suitable for later compositing and animation. The reported implementation uses LoRA fine-tuning and super-resolution for panorama generation, Metric3D v2 for depth and normals, a geometric uncertainty mask with a cosine similarity threshold around $(\theta, \phi)$1, and a 2D-GS training schedule of 4000 iterations.

3. 3D copy-and-paste and physics-aware placement

The second stage inserts a user-specified object into the generated scene. Starting from a reference image $(\theta, \phi)$2 and foreground mask $(\theta, \phi)$3, InstantMesh reconstructs a full 360° textured mesh using a multi-view diffusion prior, Zero123++, to generate six fixed-view images and a transformer-based sparse-view reconstructor. The mesh is then cleaned with standard mesh post-processing, including outlier removal and watertightness fixes, and materials are estimated from multi-view textures (Kang et al., 26 Sep 2025).

Scene composition is initialized by aligning the mesh to the user’s 3D bounding box $(\theta, \phi)$4, defined by center, dimensions, and rotation. A floor plane is detected from the Stage 1 scene point cloud, and the object is roughly placed on that plane. The principal refinement mechanism is physics-aware position learning, a differentiable optimization over the object’s 6-DoF pose and optional scale. Let $(\theta, \phi)$5 be the object point cloud, $(\theta, \phi)$6 the scene point cloud, and $(\theta, \phi)$7, $(\theta, \phi)$8 the corresponding normals. The collision and gravity losses are

$(\theta, \phi)$9

$$d(\theta, \phi) = [\cos \theta \cos \phi, \sin \theta, \cos \theta \sin \phi],$$0

with the total physics term

$$d(\theta, \phi) = [\cos \theta \cos \phi, \sin \theta, \cos \theta \sin \phi],$$1

Here $$d(\theta, \phi) = [\cos \theta \cos \phi, \sin \theta, \cos \theta \sin \phi],$$2 is a small contact distance, approximately $$d(\theta, \phi) = [\cos \theta \cos \phi, \sin \theta, \cos \theta \sin \phi],$$3, $$d(\theta, \phi) = [\cos \theta \cos \phi, \sin \theta, \cos \theta \sin \phi],$$4 is gravitational acceleration, $$d(\theta, \phi) = [\cos \theta \cos \phi, \sin \theta, \cos \theta \sin \phi],$$5 is a mass per vertex or uniform weight, and $$d(\theta, \phi) = [\cos \theta \cos \phi, \sin \theta, \cos \theta \sin \phi],$$6 is the floor height. Gradients are obtained analytically through autograd, and Adam with learning rate $$d(\theta, \phi) = [\cos \theta \cos \phi, \sin \theta, \cos \theta \sin \phi],$$7 updates the pose. Rotation is limited to plausible ranges, and an optional stability term can penalize high center of mass over small support regions.

After pose optimization, the object is composited with the background through a joint 2D-GS optimization. The composited point cloud initializes a new splat representation, and background occluded regions are hallucinated with SDS applied only to background splats:

$$d(\theta, \phi) = [\cos \theta \cos \phi, \sin \theta, \cos \theta \sin \phi],$$8

Instance-level DINO features are reused for the background, while new DINO features are distilled for the foreground object. This refinement adjusts opacity and spherical harmonics coefficients for natural blending while using SDS on occlusions to keep the scene faithful.

Within the architecture, this stage addresses a common failure mode of object insertion pipelines: plausible geometry alone does not guarantee plausible placement. Drag4D’s formulation explicitly targets contact, collision avoidance, and gravity-consistent support, rather than treating composition as a purely visual alignment problem.

4. Part-augmented motion-conditioned multiview video diffusion

The final stage animates the composed object along a user-defined 3D trajectory $$d(\theta, \phi) = [\cos \theta \cos \phi, \sin \theta, \cos \theta \sin \phi],$$9. For each camera view $X = D_p(\theta, \phi)\, d(\theta, \phi).$0 with calibration $X = D_p(\theta, \phi)\, d(\theta, \phi).$1, the trajectory is projected into the image plane:

$X = D_p(\theta, \phi)\, d(\theta, \phi).$2

This yields multiview pairs $X = D_p(\theta, \phi)\, d(\theta, \phi).$3, where $X = D_p(\theta, \phi)\, d(\theta, \phi).$4 is the first-frame image rasterized from the composited 2D-GS and $X = D_p(\theta, \phi)\, d(\theta, \phi).$5 is the 2D trajectory in that view. The paper uses eight views, with azimuths $X = D_p(\theta, \phi)\, d(\theta, \phi).$6, $X = D_p(\theta, \phi)\, d(\theta, \phi).$7, $X = D_p(\theta, \phi)\, d(\theta, \phi).$8, $X = D_p(\theta, \phi)\, d(\theta, \phi).$9 and elevations $I_i$0, $I_i$1 (Kang et al., 26 Sep 2025).

The motion model is a part-augmented, motion-conditioned video diffusion model called Local-Global DragAnything. For each view, the inputs are the first-frame image and the projected trajectory. Instance-level DIFT features are pooled within the foreground mask. Part-level features are obtained by applying k-means clustering to DINOv2 features within the instance mask to define part masks. ControlNet encodes trajectory heatmaps and entity features, while the first-frame latent $I_i$2 and encoded trajectory features are injected during denoising.

The instance-only diffusion objective is

$I_i$3

where $I_i$4 is the instance-level entity feature and $I_i$5 is the Gaussian heatmap following $I_i$6. Drag4D augments this with part-level features $I_i$7 and heatmaps $I_i$8:

$I_i$9

Training uses VIPSeg videos with instance masks, and part masks are created by clustering DINOv2 features within the mask.

The stated purpose of part-aware conditioning is to reduce local motion hallucination, especially for articulated objects, where global instance features alone can produce local drift or deformation. Multiview consistency is encouraged not by an explicit additional loss, but by sharing the first-frame entity features across views, using identical part segmentation logic on rasterized multiview frames from the 2D-GS, and conditioning each view on the 2D projection of the same 3D trajectory. The composited splat features thereby act as a shared prior across views. This is a notable design distinction: synchronized multiview behavior is induced through shared conditioning and calibration rather than through a separately defined cross-view optimization term.

5. Evaluation, datasets, and empirical behavior

Evaluation is performed on two datasets. Drag4D-30 contains 30 long-text prompts for 3D scene generation together with four object-centric images—teddy bear, batman, rabbit, and robot—for insertion and animation. VIPSeg is used both to train the motion-conditioned video models and to evaluate alignment (Kang et al., 26 Sep 2025).

The reported metrics cover image quality, novel-view quality, and motion alignment. For image and novel-view quality, the paper uses CLIP-Score, CLIP-IQA with Sharp, Colorful, and Quality submetrics, PSNR, and SSIM. For motion alignment, it reports FVD, ObjMC, and adherence measures on VIPSeg, alongside multiview CLIP-Score and CLIP-IQA on Drag4D-30.

For Stage 1 on Drag4D-30, Drag4D is compared with LucidDreamer and SceneDreamer360. LucidDreamer reports CLIP 0.656, Sharp 0.961, Colorful 0.603, and Quality 0.704. SceneDreamer360 reports CLIP 0.773, Sharp 0.970, Colorful 0.760, Quality 0.736, PSNR 24.59, and SSIM 0.857. Drag4D reports CLIP 0.782, Sharp 0.973, Colorful 0.740, Quality 0.747, PSNR 25.74, and SSIM 0.885. For Stage 3 on VIPSeg, reproduced DragAnything obtains FID 34.45, FVD 288.68, PSNR 18.41, SSIM 0.57, and ObjMC 19.9, whereas LG-DragAnything reports FID 32.79, FVD 272.02, PSNR 19.02, SSIM 0.59, and ObjMC 17.6. On Drag4D-30 multiview videos, DragAnything reports CLIP 0.805, Quality 0.48, and Colorful 0.71, while LG-DragAnything reports CLIP 0.814, Quality 0.51, and Colorful 0.75.

The ablations for object-scene composition vary SDS usage, normal consistency, joint training with or without background, and rasterization. The best settings—SDS plus normal consistency plus joint training—yield CLIP approximately 0.775, Sharp approximately 0.976, and alignment error approximately 0.528, where lower alignment error is better. Qualitatively, the paper reports that joint training on inpainted augmented views reduces unseen-view artifacts, physics-aware placement removes floating and collisions, and part-aware conditioning reduces local motion hallucinations while improving trajectory adherence and cross-view consistency.

These results support the paper’s narrower claims. They do not establish universal robustness; rather, they indicate that the combined design improves reconstruction quality, placement plausibility, and motion alignment on the stated benchmarks.

6. Relation to prior work, limitations, and future directions

Drag4D is positioned against three neighboring research areas. In text-driven 3D generation, LucidDreamer maps outpainted RGB-D to point clouds and then optimizes 3D-GS, while SceneDreamer360 and HoloDreamer generate panoramas and reconstruct with 3D-GS. Drag4D differs by using 2D-GS trained jointly on base and inpainted augmented views with adaptive weighting. In 3D composition, Layout-your-3D composes objects via 2D blueprints, and CG3D and GraphDreamer enforce physical plausibility or scene graphs. Drag4D instead uses differentiable physics-aware losses—collision and gravity—together with joint splat optimization and SDS. In motion control, DragAnything controls instance motion from 2D trajectories with global features only; Drag4D extends it with part-aware conditioning and multiview synchronization using shared splat features and consistent trajectory projections (Kang et al., 26 Sep 2025).

Several common misconceptions are directly addressed by the system design and the stated limitations. Drag4D is not a full 3D latent-control motion generator; trajectory conditioning is 2D per view rather than full 3D latent control, which can reduce adherence in edge cases. It is also not a general physical simulator: it handles single-object motion, and multi-object interactions, occlusions, and complex support relations beyond a single plane require further work. Non-Lambertian effects and relighting are out of scope, so object textures under complex scene lighting are not fully modeled. Failure cases include ambiguity for motion parallel to the view and hard fast motions.

The future directions named in the paper are extensions of the same three-stage logic. These include multi-object motion with collision avoidance and inter-object constraints, differentiable physics integration for contact dynamics and stability, closed-loop trajectory editing with feedback from rendered multiview consistency and adherence metrics, and 3D trajectory conditioning at the model level through mechanisms such as 3D curve control fields. The stated applications are content creation for AR/VR, film previz, education, and interactive media with user-controlled 4D scenes. The paper also notes an ethical consideration: synthetic content controls must consider misuse and should ensure transparency and consent for any real-world likeness in inserted objects.