4DAnyone: From Monocular Video to Dynamic 4D Humans
This presentation explains how 4DAnyone transforms a single casual video of a person into a complete 4D representation that can be viewed from any angle at any moment. Rather than relying solely on geometric reconstruction, the method treats multi-view synthesis as a learned video generation problem conditioned on human skeletal motion. By combining video diffusion models with explicit 3D pose scaffolding and novel attention mechanisms, the system generates synchronized views from multiple cameras and reconstructs them into dynamic Gaussian avatars. The approach demonstrates how large-scale synthetic training data and careful architectural design enable monocular-to-4D reconstruction without per-sequence optimization.Script
A single handheld video clip contains far less geometric information than dozens of synchronized cameras, yet this paper shows how to reconstruct a full 4D human avatar from exactly that casual monocular input.
The authors treat this as a conditioned video generation problem, not pure reconstruction. They extract a 3D skeleton from the input using an existing pose estimator, render that skeleton from new viewpoints, and condition a video diffusion model to generate what those cameras would see. The generated multi-view videos are then optimized into a dynamic Gaussian Splatting representation.
The diffusion backbone extends a pretrained video model with multiview self-attention, so tokens from different camera angles at the same time instant attend directly to one another. Recurrent context packing lets the model incorporate multiple reference views at different spatial compression ratios without scaling cost linearly with the number of inputs, while Target Context Routing dynamically regroups cameras across denoising steps to propagate information before locking groups for final refinement.
The camera routing strategy delivers measurable improvements. When all denoising steps use fixed grouping, PSNR is 22.21. Switching to predominantly sliding groups with only the final 20 percent fixed raises it to 22.63 and drops perceptual error from 0.196 to 0.191 LPIPS. Beyond that point the metrics saturate, so the authors select the beginning of the plateau rather than chasing diminishing returns.
The method inherits the limitations of its skeleton estimator. When the pose tracker misreads an en-pointe stance as flat feet, every generated view reproduces that error. Loose garments present a different challenge: skeletons describe articulated bodies well but provide little information about fabric that moves independently, leading to view-inconsistent generation and degraded reconstructions.
4DAnyone shows that conditioning video diffusion on explicit 3D human motion can complete unobserved views without per-scene optimization, but its success depends on both reliable pose estimation and training distributions that cover real-world appearance. To dive deeper into this work and generate your own video explanations of the latest research, visit EmergentMind.com.
