4DAnyone: Create Anyone in 4D from a Casual Monocular Video

Abstract: We present 4DAnyone, a framework for reconstructing 4D humans from an uncalibrated monocular video by generating reconstruction-grade multiview-consistent videos and lifting them into 4D Gaussian Splatting (4DGS). Existing camera-controlled video diffusion models synthesize plausible novel-view videos but fail to maintain consistency when scaled to the tens of target views required for 4DGS reconstruction. We identify this failure as a bounded-attention-context problem: when target views exceed the capacity of a single DiT forward pass, they must be split into groups, exposing two coupled bottlenecks. On the reference-context side, conditioning on all previously generated views grows as $O(N)$, weakening cross-view appearance guidance. On the target-context side, disjoint groups cannot directly exchange information, causing global structural drift. 4DAnyone addresses both bottlenecks with two complementary designs: Reference Context Packing (RCP) compresses growing reference views into a fixed-length mixed-resolution context with $O(1)$ reference-context complexity, while Target Context Routing (TCR) rotates target-view groupings during denoising to share context across groups at high-noise steps and stabilize details at low-noise steps. We further build the MVGameHuman dataset using our in-house game engine and combine it with light-stage and in-the-wild video datasets for training. Experiments on DNA-Rendering and DyMVHumans show that 4DAnyone outperforms prior methods in both novel-view video quality and downstream 4DGS reconstruction, with robust in-the-wild generalization. See our project page for video results and source code: https://4danyone.github.io.

• The paper introduces a skeleton-conditioned video diffusion pipeline that generates synchronized target-view videos from casual monocular footage before reconstructing a dynamic 4D Gaussian Splatting avatar.
• The method combines multiview attention, Recurrent Context Packing, and Target Context Routing, with TCR improving PSNR from 22.2079 to 22.6294 and reducing LPIPS from 0.1964 to 0.1906 in the reported sweep.
• The paper reduces skeleton-regression error from 14.0 mm to 3.5 mm, but performance remains vulnerable to human-motion estimation failures, loose garments, nonrigid motion, and diffusion inconsistencies across views.

Problem formulation and contribution

“4DAnyone: Create Anyone in 4D from a Casual Monocular Video” (2608.20335) addresses monocular-to-4D human reconstruction by treating novel-view synthesis as a conditioned multi-view video-generation problem rather than relying exclusively on explicit parametric human models or per-sequence optimization. Given a casual monocular video, the method generates synchronized videos from multiple target viewpoints and subsequently reconstructs a dynamic 4D Gaussian Splatting (4DGS) representation. The resulting representation supports free-viewpoint rendering of the subject over time.

The central design choice is to condition video diffusion on a 3D-aware human skeleton extracted from the input sequence. This condition provides an explicit geometric scaffold for pose and viewpoint transformation, while the source video supplies appearance, clothing, motion texture, and scene information. The generated multi-view videos are then processed by FreeTimeGS (2608.20335) to obtain the final dynamic Gaussian representation. The method therefore decomposes the problem into three coupled stages: human-motion recovery and skeleton rendering, multi-view video generation, and 4DGS reconstruction.

The paper’s claim is not that monocular video intrinsically contains sufficient geometric information for exact reconstruction. Instead, it uses learned priors from large-scale video diffusion and synthetic multi-view data to complete unobserved views. This makes the quality of the result dependent on both the reliability of the human-motion estimator and the learned distribution of appearance and motion.

Architecture and multi-view generation

4DAnyone is built by fine-tuning Wan2.2-TI2V-5B (2608.20335). Its video diffusion backbone is extended with multiview self-attention, recurrent context packing (RCP), and Target Context Routing (TCR). The multiview attention mechanism rearranges latent tokens so that tokens from different viewpoints at the same temporal instant attend directly to one another. Temporal self-attention handles within-view dynamics, while multiview attention establishes cross-view information exchange. Both modules share the basic architecture and initialization of the pretrained temporal attention layers, allowing temporal coherence learned by Wan2.2 to serve as the initialization for cross-view consistency learning.

RCP provides a mechanism for incorporating multiple source and reference views at different spatial compression ratios. The standard Wan patchification uses a spatial stride of two, while the $2\times$ and $4\times$ RCP branches use spatial strides of four and eight. These branches produce one-fourth and one-sixteenth as many spatial tokens, respectively. Their convolutional kernels are initialized by spatially tiling the pretrained kernel and dividing by the corresponding area ratio, preserving activation variance during transfer learning. This design permits the model to use one, four, or eight source views without making the computational cost grow proportionally with the number of conditioning images.

TCR addresses the practical problem that all target views cannot necessarily be denoised jointly at the target resolution. The target cameras are partitioned into groups and routed dynamically across denoising steps. During the early and intermediate diffusion steps, the grouping changes through a sliding or otherwise reordered camera schedule, allowing information to propagate between groups. During the final denoising steps, the grouping becomes fixed and contiguous, stabilizing local refinement. The paper reports that this routing strategy improves generation consistency relative to fixed grouping.

The switching-time sweep provides a direct numerical characterization of this effect:

Fraction of final steps with fixed grouping	Sliding steps	PSNR	SSIM	LPIPS
1.00	0	22.2079	0.7880	0.1964
0.50	10	22.4575	0.7925	0.1933
0.25	15	22.6093	0.7955	0.1912
0.20	16	22.6294	0.7963	0.1906
0.00	20	22.6414	0.7971	0.1903

The improvement is substantial between fixed routing and predominantly sliding routing: PSNR increases from 22.2079 to 22.6294 at the selected operating point, while LPIPS decreases from 0.1964 to 0.1906. However, the sweep also shows diminishing returns. Continuing to increase the number of sliding steps produces only marginal changes, and some individual metrics fluctuate non-monotonically. The default $t_s/T=0.2$ is consequently selected at the beginning of the reported saturation regime rather than at the absolute endpoint.

Skeleton conditioning and human-motion recovery

The conditioning signal is obtained by running GVHMR (2608.20335) on the monocular input video. GVHMR estimates a ground-aligned SMPL-X sequence, from which the method extracts 70 3D keypoints in the Goliath vocabulary. For the diffusion condition, the authors retain body, foot, and palm-level hand keypoints, including the wrist and five knuckles per hand, while excluding facial and individual finger joints. The skeleton is rendered as a depth-buffered RGB sequence with body-part-specific colors. Camera-space depth determines pixelwise visibility ordering, making the representation insensitive to absolute scale and global translation.

A sparse SMPL-X-to-Goliath70 regressor is trained as a convex combination of nearby SMPL-X vertices. This is preferable to assigning each keypoint to its nearest vertex because it reduces local discretization error. On held-out scenes, the learned regressor achieves a mean keypoint error of 3.5 mm, compared with 14.0 mm for the nearest-vertex baseline. The approximately fourfold reduction is important because skeleton errors are not merely local conditioning noise: they can be propagated consistently to every generated target view.

The skeleton encoder comprises ten 3D convolutional layers, followed by a $1\times1\times1$ projection into the diffusion-token dimensionality. It performs $32\times$ spatial and $4\times$ temporal downsampling. Its final projection is zero-initialized, so the conditioning branch initially contributes no residual and preserves the behavior of the pretrained Wan2.2 model. Three copies of the first frame are prepended to match the temporal arrangement used by the Wan2.2 VAE. This initialization strategy reduces the risk that the new geometric pathway destabilizes the pretrained video prior.

The staged training schedule reflects the different roles of pose, appearance, and background. Stage 1 uses foreground-only DNA-Rendering data and independently samples the source frame range with probability 0.2. This deliberately separates the temporal evidence used for pose from the temporal evidence used for appearance, encouraging the model to follow the skeleton while preserving identity and clothing. Stage 2 introduces background-bearing synthetic multi-view data, including MVGameHuman and SynCamVideo. Stage 3 adds monocular Pexels and TedTalk videos and removes finger keypoints, whose monocular detections are considered too noisy. The model is therefore trained to infer fine hand appearance from the source reference rather than to reproduce unreliable explicit finger conditions.

Training data and computational design

MVGameHuman is a major component of the training regime. It contains 38,000 synchronized multi-view human videos rendered at $2560\times1440$, with 318 actors observed by 24 virtual cameras per sequence. The dataset varies actors, clothing, motion, lighting, scenes, and backgrounds.

Figure 1: Representative synchronized MVGameHuman views spanning actors, clothing, motion, lighting, scenes, and backgrounds.

The use of a synthetic synchronized multi-view dataset addresses a principal difficulty in training cross-view video models: naturally occurring monocular video rarely provides temporally aligned observations from multiple calibrated viewpoints. MVGameHuman supplies direct supervision for view correspondence, while DNA-Rendering and DyMVHumans support evaluation and additional training diversity. The sampling policy maintains approximately constant token counts by trading target-camera count against sequence length, for example using configurations such as $6\times41$, $4\times61$, and $1\times121$ target-camera frames.

Training is performed at $4\times$0 resolution with a learning rate of $4\times$1 and an LPIPS weight of 0.25. The three stages require approximately 0.5, 1, and 1.5 days, respectively, on 128 H20-3E GPUs. Training is reported to converge stably with 32 or more GPUs, although 128 GPUs are used for acceleration. To control the memory cost of perceptual losses over long clips, the method samples body-aware crops with probabilities of 0.2 for the full body, 0.2 for the face, 0.1 for the left hand, 0.1 for the right hand, and 0.4 for uniform image regions. This allocation prioritizes regions with high perceptual and identity sensitivity without eliminating global image supervision.

Reconstruction pipeline and efficiency

At inference, the method first preprocesses the monocular sequence with GVHMR and renders the multi-view skeleton conditions. It generates four uniformly spaced reference views in a single round, then generates 16 target views in four-view groups using TCR and a fixed RCP context. This hierarchical arrangement reduces the need to generate every camera view independently while retaining a stable source-and-reference context.

The generated videos are converted into a 4DGS model using FreeTimeGS. Coarse geometry is initialized through space carving from predicted foreground masks, after which the Gaussian primitives are optimized for 50,000 iterations using Adam. The reconstruction uses sequences of 16 cameras and 121 frames. The final model is therefore not a direct output of the diffusion network: it is a representation fitted to generated observations, and reconstruction quality depends on both the consistency of those observations and the optimization behavior of FreeTimeGS.

The reported inference cost is approximately two minutes for GVHMR and skeleton rendering on a single RTX 4090, seven minutes for generating four 121-frame videos with 20 denoising steps on one H20 GPU, and 30 minutes for 4DGS optimization on one RTX 4090. The authors report that reducing diffusion sampling from 20 to 10 denoising steps produces minimal quality degradation under the strong skeleton conditioning. Thus, the generation stage can be reduced in practice, although the paper does not provide a complete quantitative speed–quality curve for this reduction.

The method also supports multi-GPU generation. A 16-camera layer uses four GPUs with four target cameras per GPU; two layers use eight GPUs and cover 32 cameras; and three layers use eight GPUs with six target cameras per GPU, covering 48 cameras. More densely sampled camera layers are intended for complex garments or extreme motions where sparse view coverage is insufficient.

Single-image extension

The paper demonstrates that the monocular-video interface can be extended to single-image input by chaining Wan-Animate (2608.20335) with 4DAnyone. Wan-Animate first synthesizes a subject-specific monocular video driven by a motion sequence. 4DAnyone then converts that video into synchronized target-view videos, and FreeTimeGS reconstructs the resulting dynamic Gaussian avatar.

Figure 2: A single image is animated into a monocular source video, expanded into synchronized multi-view videos, and reconstructed as a 4D Gaussian avatar.

This experiment is technically significant because it isolates the role of the input modality. 4DAnyone itself remains a monocular-video-to-multiview system; identity preservation and source-motion synthesis are delegated to Wan-Animate. Consequently, artifacts introduced during image-to-video animation can be inherited by every target view and by the final 4DGS reconstruction. The result demonstrates compositionality of the pipeline, but does not establish that the complete image-to-4D system is robust to arbitrary single-image inputs.

Evaluation protocol and empirical evidence

The evaluation uses ten DNA-Rendering scenes and three DyMVHumans scenes. Each sequence contains 16 approximately uniformly distributed cameras and 98 frames. One camera is used as the source, target views are generated across the camera array, selected views provide RCP references, and held-out views are reserved for consistency evaluation. The remaining target views are used for 4DGS optimization. This protocol distinguishes generation quality from reconstruction consistency and tests whether the generated videos support a coherent dynamic representation.

The comparisons include MV-Performer, TrajectoryCrafter, ReCamMaster, and 4DAnyone. MV-Performer and TrajectoryCrafter are evaluated using their officially released models, while ReCamMaster is fine-tuned under the same datasets and training settings and equipped with the same RCP and TCR infrastructure. This controlled ReCamMaster comparison is intended to isolate the contribution of explicit skeleton-geometric conditioning from camera-parameter conditioning.

The supplied paper content does not include the principal benchmark result tables, so the relative performance of 4DAnyone against these baselines cannot be stated numerically from the available material. The reported TCR ablation and the 3.5 mm keypoint-regression result are quantitative, but they should not be conflated with end-to-end benchmark superiority. In particular, the evaluation protocol gives some baselines favorable geometric inputs: MV-Performer uses Depth-Anything-3 depth with ground-truth camera parameters, and TrajectoryCrafter uses depth alignment based on camera-aligned reference estimates. Any end-to-end comparison must therefore be interpreted in light of these preprocessing choices.

Limitations and open questions

The method’s strongest limitation is its dependence on HMR accuracy. The diffusion model can produce a coherent result even when the estimated pose is imperfect, but coherence does not imply correctness. When GVHMR misinterprets an unusual pose, the error is propagated to all generated views. The paper’s en-pointe example is explicit: flat-foot pose estimation causes every target view to inherit the incorrect foot configuration.

Figure 3: Failure cases caused by skeleton-insensitive loose garments and incorrect HMR estimation of an en-pointe pose.

Loose and highly nonrigid garments expose a second limitation. A skeleton is informative about articulated body motion but does not adequately describe fabric that extends far from the body or exhibits independent dynamics. In the reported flowing-fabric case, the garment is generated inconsistently across viewpoints, producing a degraded 4DGS reconstruction. This indicates that the method’s geometric prior is human-pose-centric rather than a general dynamic-scene representation.

The training distribution also imposes an important assumption. MVGameHuman provides extensive synchronized multi-view supervision, but it is generated by an in-house game engine. The paper does not establish how well the learned appearance and garment priors transfer to real-world illumination, sensor artifacts, nonhuman objects, or subjects outside the synthetic actor distribution. The addition of Pexels and TedTalk data improves monocular diversity, but these datasets do not replace synchronized real multi-view supervision.

Finally, the reconstruction stage optimizes FreeTimeGS on generated rather than observed target views. This makes the final representation susceptible to systematic diffusion errors, including texture hallucination, view-dependent inconsistencies, and temporal drift. An open question is whether explicit multi-view geometric losses or jointly optimized 4D representations could correct such errors rather than merely fit them.

Conclusion

4DAnyone presents a structured monocular-video-to-4D pipeline based on 3D-aware skeleton conditioning, multiview video diffusion, dynamic target-camera routing, and 4DGS reconstruction. Its principal technical contributions are the integration of multiview attention into a pretrained video diffusion transformer, RCP for scalable multi-source conditioning, TCR for cross-group consistency, and a staged training strategy that separates pose, appearance, background, and monocular-data supervision. The reported TCR sweep improves generation consistency from 22.2079 to 22.6294 PSNR and from 0.1964 to 0.1906 LPIPS at the selected operating point, while the learned keypoint regressor reduces error from 14.0 mm to 3.5 mm. The method remains constrained by HMR failures and by the limited expressiveness of skeleton conditioning for loose garments and independent nonrigid motion. Its main unresolved technical question is how to combine explicit human geometry with appearance- and deformation-aware representations that remain reliable when the input motion estimate or garment dynamics fall outside the training distribution.