---
title: '4DAnyone: Monocular Video to 4D Human Avatars'
url: https://www.emergentmind.com/topics/4danyone
type: topic
---

# 4DAnyone: Monocular Video to 4D Human Avatars

4DAnyone is a framework for reconstructing dynamic, viewable human avatars from casual monocular RGB video. It generates temporally synchronized, multiview-consistent videos from an uncalibrated input sequence and lifts those videos into a time-varying 4D Gaussian Splatting (4DGS) representation. The method addresses the failure of conventional camera-controlled video diffusion models to maintain identity, appearance, geometry, and motion consistency when generating the tens of target views required for 4D reconstruction. Its principal contributions are Reference Context Packing (RCP), which compresses reference views into a fixed-length mixed-resolution context, and Target Context Routing (TCR), which changes target-view groupings during denoising to enable information exchange across groups. The framework is described in “4DAnyone: Create Anyone in 4D from a Casual Monocular Video” [2608.20335].

## 1. Problem definition and objectives

4DAnyone takes a casual monocular video

$$
\mathbf{I}^{s}_{1:F}=\{I^s_t\}_{t=1}^{F},
$$

captured by an unknown source camera $s$. The system generates synchronized videos

$$
\left\{\hat{\mathbf{I}}^{v}_{1:F}\right\}_{v=1}^{V}
$$

for virtual target cameras $v$, which are subsequently used to optimize a dynamic 4DGS model.

The target representation must satisfy several simultaneous requirements:

1. Each target video should be individually realistic.
2. All target views should depict the same person and motion.
3. Corresponding frames should remain synchronized.
4. Appearance, geometry, and clothing should remain consistent across views.
5. The generated views should be sufficiently coherent for downstream 4DGS reconstruction.

This problem is more demanding than ordinary novel-view synthesis. A monocular video does not directly reveal the subject’s back or sides, and casual capture introduces camera motion, unknown intrinsics and extrinsics, motion blur, occlusion, changing backgrounds, and inaccurate pose estimates. The framework therefore combines temporal evidence, learned human priors, explicit 3D pose conditioning, cross-view attention, and a reconstruction stage.

The output is not merely a collection of independently plausible videos. Multiview disagreements that are visually tolerable in isolation can produce severe artifacts when used jointly to reconstruct a dynamic 3D representation. 4DAnyone consequently treats cross-view consistency as a primary objective rather than as a secondary property of video generation.

## 2. Bounded-context failure in multiview video diffusion

4DAnyone is built around a bounded-attention-context analysis of multiview video diffusion. Video diffusion transformers have finite token and attention budgets. If the source video and all previously generated target views are retained at full resolution, the reference context grows as more views are produced. A direct attention computation has approximately quadratic complexity,

$$
\mathcal{O}\left((N_{\mathrm{ref}}+N_{\mathrm{tgt}})^2\right),
$$

where $N_{\mathrm{ref}}$ and $N_{\mathrm{tgt}}$ are the reference and target token counts.

Because of GPU-memory constraints, target views are generally divided into groups. This produces two coupled bottlenecks.

### Reference-context growth

Suppose target groups are generated sequentially as $G_1,G_2,\ldots,G_K$. If all generated groups are retained as full-resolution references, the context after group $k$ is approximately

$$
\mathcal{C}_{k}
=
\left[
\mathbf{I}^{s},
\hat{\mathbf{I}}^{G_1},
\ldots,
\hat{\mathbf{I}}^{G_k}
\right].
$$

Its token count grows according to

$$
|\mathcal{C}_{k}|
\propto
|\mathbf{I}^{s}|+
\sum_{i=1}^{k}
|\hat{\mathbf{I}}^{G_i}|.
$$

Memory and attention cost consequently increase with the number of generated groups. Discarding older views avoids this growth but removes global appearance coverage. Later views may then depend only on nearby or recently generated views, increasing the risk of identity drift, inconsistent clothing, and missing backside details.

### Target-group isolation

If target views are processed in disjoint groups, views within a group can communicate through attention, but views in different groups cannot directly exchange information. A fixed grouping also prevents views that never co-occur from communicating, even when they are adjacent in azimuth or observe complementary portions of the subject.

The resulting group-boundary discontinuities can include changes in clothing texture, face or hair appearance, limb geometry, and hallucinated unseen surfaces. These errors are especially damaging to 4DGS because the reconstruction stage assumes that corresponding observations describe a coherent dynamic subject.

RCP and TCR address these two failure modes separately. RCP controls the size of the reference context, while TCR propagates information between target groups over denoising time.

## 3. Architecture and human-geometry conditioning

4DAnyone fine-tunes Wan2.2-TI2V-5B and augments it with a skeleton-conditioning branch, multiview self-attention, RCP, TCR, and multiscale patchification for compressed reference views.

### 3D-aware skeleton conditioning

The input monocular video is first processed by GVHMR to estimate a ground-aligned SMPL-X mesh sequence. A sparse vertex-to-keypoint regressor converts the SMPL-X meshes into 70 Goliath-vocabulary 3D keypoints. The retained representation includes body keypoints, foot keypoints, and palm-level hand keypoints consisting of the wrist and five knuckles per hand. Face landmarks and individual finger joints are omitted during inference because monocular detections of these structures are unreliable.

For target camera $v$, a 3D keypoint $X_{t,j}$ is projected using

$$
p^{v}_{t,j}=\Pi_v(X_{t,j}),
$$

where $\Pi_v$ is the target-camera projection. Camera-space depth $z^{v}_{t,j}$ is used for z-buffering. The resulting depth-buffered, body-part-colored RGB skeleton video is invariant to global scale and translation in depth because it uses relative depth ordering.

The skeleton encoder $g_{\phi}$ produces a residual at the DiT token resolution:

$$
\mathbf{r}^{v}
=
g_{\phi}\left(\mathbf{S}^{v}_{1:F}\right),
\qquad
\mathbf{z}^{v}_{\mathrm{cond}}
=
\mathbf{z}^{v}_{\mathrm{noisy}}
+
\mathbf{r}^{v}.
$$

Its stated implementation contains 10 Conv3D layers, with five strided and five non-strided layers, SiLU activations, channels progressing as

$$
3\rightarrow16\rightarrow32\rightarrow64\rightarrow128\rightarrow256\rightarrow d,
$$

strided kernels of size $(3,4,4)$, $32\times$ spatial and $4\times$ temporal downsampling, and a final $1\times1\times1$ projection. The final projection is zero-initialized so that the new branch initially contributes zero and preserves pretrained Wan behavior. Three copies of the first frame are prepended before VAE encoding to match Wan2.2’s temporal encoding convention, which maps $4n+1$ input frames to $n+1$ latent frames.

Each Goliath keypoint is represented as a convex combination of nearby SMPL-X vertices:

$$
\hat{\mathbf{k}}_j
=
\sum_{m\in\mathcal{N}_j}
w_{jm}\mathbf{v}_m,
\qquad
w_{jm}\geq0,
\qquad
\sum_m w_{jm}=1.
$$

The support vertices are fixed to nearby SMPL-X vertices and the weights are learned with Smooth-L1 loss. The reported held-out mean error is 3.5 mm, compared with 14.0 mm for a nearest-vertex baseline.

### Multiview self-attention

Wan2.2 already provides temporal self-attention. 4DAnyone introduces multiview self-attention with the same architecture and initially copied temporal-attention weights. For latent tensors with frame, view, spatial-token, and channel dimensions, the tokens are rearranged into

$$
(f,\;vhw,\;d),
$$

allowing tokens from different views at the same time step to attend to one another. Temporal and multiview attention jointly exchange information across time, views, and spatial locations.

Initializing multiview attention from temporal-attention weights uses temporal coherence as the starting point for learning cross-view consistency. This design supplies explicit communication across cameras without requiring all target views to be processed simultaneously.

## 4. Reference Context Packing

Reference Context Packing addresses the growth of the reference context. The system uses the source video and a small set of generated reference views as appearance evidence. Instead of retaining all views at full resolution, RCP converts them into a fixed-length, mixed-resolution context.

Given a raw reference set

$$
\mathcal{R}
=
\left\{
\mathbf{I}^{s},
\hat{\mathbf{I}}^{r_1},
\ldots,
\hat{\mathbf{I}}^{r_M}
\right\},
$$

RCP produces

$$
\tilde{\mathcal{R}}
=
\operatorname{Pack}(\mathcal{R}),
$$

with token count approximately independent of the number of target views:

$$
|\tilde{\mathcal{R}}|
=
\mathcal{O}(1)
$$

with respect to the number of target views. The fixed complexity depends on the selected reference budget and resolution; it does not imply independence from those fixed design choices.

### Mixed-resolution patchification

RCP uses three patchification scales:

- $1\times$: standard Wan patchification with Conv3D kernel and stride $(1,2,2)$;
- $2\times$: kernel and stride $(1,4,4)$, producing one-quarter as many spatial tokens;
- $4\times$: kernel and stride $(1,8,8)$, producing one-sixteenth as many spatial tokens.

The $2\times$ and $4\times$ layers are initialized by spatially tiling the pretrained $(1,2,2)$ kernel and dividing by the corresponding area ratios, 4 and 16, to preserve activation variance.

The design retains some references at higher resolution while representing additional views more compactly. Training varies the number of source views:

- one source view uses $1\times$ patchification;
- four source views use $2\times$ patchification;
- eight source views use a combination of $1\times$, $2\times$, and $4\times$ patchification.

Higher-resolution tokens preserve local appearance and identity cues, whereas lower-resolution views provide broader angular coverage at reduced token cost.

At inference, four approximately uniformly distributed reference views are generated from the source video in one round. The packed reference context is then held fixed while the remaining target views are generated. For DNA-Rendering, the reference cameras are $[04,16,28,40]$; for DyMVHumans, they are $[04,19,38,54]$.

RCP can therefore be viewed as a transformation from a growing full-resolution context to a fixed-length mixed-resolution memory. Its role is to preserve global appearance evidence without allowing reference-context complexity to scale with the number of target groups.

## 5. Target Context Routing and multiview generation

Target Context Routing addresses target-group isolation by changing which cameras are grouped together during diffusion denoising.

Let the target views be ordered by azimuth and initially partitioned into groups of four:

$$
\mathcal{G}^{(0)}
=
\{G^{(0)}_1,\ldots,G^{(0)}_K\}.
$$

At denoising step $n$, TCR applies a permutation or circular shift before regrouping:

$$
\mathcal{G}^{(n)}
=
\operatorname{Group}\left(\pi_n(\mathcal{V})\right),
$$

where $\mathcal{V}$ is the target-view set and $\pi_n$ changes with denoising step $n$.

A view can consequently attend to different neighboring or distant views at different denoising steps. Repeated attention propagates information across the complete view set without requiring all cameras to appear in one attention window.

### High-noise routing

During early denoising, latent videos are highly corrupted and contain limited reliable local detail. The principal objective is to establish global structure and cross-view agreement. TCR uses dynamic groupings during this phase so that views from different groups can exchange information over time.

The reported routing variants are:

- **Sliding**: a cumulative one-position circular shift at each dynamic step;
- **Random**: a deterministic permutation with seed $42+n$;
- **Strided**: views reordered with stride $4-(n\bmod4)$ before grouping.

### Low-noise stabilization

During later denoising, the samples contain detailed texture, edges, hands, faces, and clothing patterns. Continuously changing group membership can destabilize these details. TCR therefore switches to fixed contiguous groups during the final denoising steps.

The switching parameter is $t_s/T$, where $T$ is the total number of denoising steps. The default is

$$
\frac{t_s}{T}=0.2.
$$

With 20 denoising steps, this produces 16 dynamic-routing steps followed by four fixed-group steps. The routing schedule is

$$
\pi_n=
\begin{cases}
\pi_{\mathrm{dynamic}}(n),
& n<(1-t_s/T)T,\\
\pi_{\mathrm{fixed}},
& n\geq(1-t_s/T)T.
\end{cases}
$$

The TCR sweep reports the following consistency metrics:

| $t_s/T$ | Dynamic steps | PSNR | SSIM | LPIPS |
|---:|---:|---:|---:|---:|
| 1.00 | 0 | 22.2079 | 0.7880 | 0.1964 |
| 0.50 | 10 | 22.4575 | 0.7925 | 0.1933 |
| 0.20 | 16 | 22.6294 | 0.7963 | 0.1906 |
| 0.00 | 20 | 22.6414 | 0.7971 | 0.1903 |

Dynamic routing improves consistency over fixed grouping, with gains largely saturating around $t_s/T=0.2$. Fully dynamic routing produces slightly higher reported metrics in this sweep, but the default schedule uses late fixed grouping to stabilize local details.

### Interaction between RCP and TCR

RCP and TCR operate along different dimensions:

- RCP compresses the reference-view dimension;
- TCR propagates information between target groups over denoising time.

Together they maintain bounded per-step attention cost while providing broader all-view communication. The approximate cost is

$$
\mathcal{O}\left(
\left(
N_{\mathrm{packed\ ref}}
+
N_{\mathrm{group}}
\right)^2
\right),
$$

where both terms are determined by fixed reference and group budgets rather than by the total number of target views.

The inference process generates four approximately uniformly distributed reference views in a single diffusion round. Remaining views are generated in four-view groups, with TCR applied during early denoising and fixed grouping used during the final denoising steps. The system inherits Wan2.2’s 121-frame generation length; the evaluation retains the first 98 frames.

Multi-GPU configurations include one 16-camera layer on four GPUs, two 16-camera layers on eight GPUs, and three 16-camera layers on eight GPUs. One 16-camera layer is sufficient for limited-pitch free-viewpoint rendering, 32 cameras cover most cases, and 48 cameras can help with complex clothing or extreme motion.

## 6. Training, 4DGS reconstruction, evaluation, and limitations

### Training data and curriculum

4DAnyone introduces MVGameHuman, generated with an in-house game engine. The dataset contains 38,000 synchronized multiview human videos, 318 actors, 24 virtual cameras per sequence, rendered resolution $2560\times1440$, and diverse clothing, motion, lighting, virtual scenes, and backgrounds.

Training also combines DNA-Rendering, SynCamVideo, Pexels, and TedTalk. DNA-Rendering provides foreground-only human multiview data; MVGameHuman supplies large-scale synthetic multiview diversity; SynCamVideo supplies multiview video generation data; and Pexels and TedTalk provide monocular in-the-wild and talking-person videos.

Training varies source-camera count, target-camera count, and frame count while approximately maintaining a fixed token budget, including configurations such as

$$
6\times41\approx4\times61\approx1\times121.
$$

The curriculum has three stages:

1. **Stage 1** uses DNA-Rendering with foreground background removal, full-body, hand, foot, and finger skeletons, and independent source-frame-range sampling probability 0.2.
2. **Stage 2** adds MVGameHuman and SynCamVideo, removes background masking, uses synchronized source and target ranges, and retains body, hand, foot, and finger skeletons.
3. **Stage 3** adds Pexels and TedTalk and drops finger keypoints because monocular finger detections are noisy.

All stages fine-tune Wan2.2-TI2V-5B at $704\times1280$, with learning rate $10^{-5}$ and LPIPS loss weight $\lambda=0.25$. The three stages take approximately 0.5, 1, and 1.5 days, respectively, on 128 H20-3E GPUs. Training is reported to converge stably with at least 32 GPUs.

For perceptual supervision, crops are sampled from full-body, face, left-hand, right-hand, and uniform image regions. The crop probabilities are

$$
p_{\mathrm{body}}=0.2,\quad
p_{\mathrm{face}}=0.2,\quad
p_{\mathrm{left}}=0.1,\quad
p_{\mathrm{right}}=0.1,\quad
p_{\mathrm{uniform}}=0.4.
$$

Each crop is $256\times256$ and is shared across all frames of a clip.

### 4DGS reconstruction

The generated multiview videos are passed to FreeTimeGS, an enhanced 4DGS system based on 4D Gaussian Splatting and LongVolcap. Its dynamic representation consists of Gaussian primitives whose positions, scales, rotations, opacities, and colors vary over time.

Initial geometry is obtained by space carving from predicted foreground masks. The Gaussian representation is then optimized with Adam using learning rate $1.6\times10^{-4}$ for 50,000 iterations on sequences of 16 cameras and 121 frames.

A conceptual rendering objective is

$$
\min_{\Theta}
\sum_{t=1}^{F}
\sum_{v=1}^{V}
\mathcal{L}_{\mathrm{render}}
\left(
\mathcal{R}(\Theta,t,v),
\hat{I}^{v}_{t}
\right),
$$

where $\Theta$ denotes the time-dependent Gaussian parameters, $\mathcal{R}$ is differentiable Gaussian rendering, and $\hat{I}^{v}_{t}$ is a generated target-view frame.

Approximate inference time is two minutes for preprocessing and GVHMR on one RTX 4090, seven minutes for four 121-frame target videos with 20 denoising steps on one H20, and 30 minutes for 4DGS optimization on one RTX 4090. Strong conditioning permits reducing diffusion from 20 to 10 denoising steps with little quality degradation.

### Evaluation

The reported evaluations use 10 DNA-Rendering scenes and three DyMVHumans scenes, with 16 approximately uniformly distributed cameras and 98 frames per sequence. Metrics include PSNR, SSIM, and LPIPS for novel-view video quality, together with downstream 4DGS reconstruction and held-out-view consistency evaluations.

The TCR sweep demonstrates an improvement from fixed grouping to the default dynamic-then-fixed schedule:

$$
\mathrm{PSNR}=22.2079,\quad
\mathrm{SSIM}=0.7880,\quad
\mathrm{LPIPS}=0.1964
$$

for fixed grouping, compared with

$$
\mathrm{PSNR}=22.6294,\quad
\mathrm{SSIM}=0.7963,\quad
\mathrm{LPIPS}=0.1906
$$

for 16 dynamic steps followed by four fixed steps.

Comparisons include MV-Performer, TrajectoryCrafter, ReCamMaster, camera-controlled diffusion approaches, and other multiview or 4D generation systems. ReCamMaster is fine-tuned using the same data and settings as 4DAnyone and equipped with the same RCP and TCR, isolating explicit skeleton-geometry conditioning from implicit camera-parameter conditioning.

### Robustness and failure modes

Skeleton conditioning improves robustness to occlusion and motion blur because GVHMR can predict a complete plausible body when image evidence is incomplete. A shifted pose estimate may still produce a coherent human, although the subject can be displaced.

The principal failure cases concern non-skeletal appearance and inaccurate human motion recovery. Loose or highly nonrigid garments can move far from the body described by the skeleton and may therefore be generated inconsistently across views. Incorrect HMR poses are propagated by the diffusion model; for example, when a dancer is en pointe but HMR predicts flat feet, the generated views inherit the incorrect foot pose.

The method is consequently constrained by monocular HMR accuracy, the representational limitations of body skeletons for clothing, props, hair, and nonhuman appendages, training coverage of actors and garments, the number and distribution of target cameras, the finite temporal length of the diffusion model, the cost of generating many target videos, and the quality of masks and generated views used by 4DGS.

A further limitation is computational. The complete pipeline requires approximately two minutes of preprocessing, seven minutes of multiview video generation, and 30 minutes of 4DGS optimization under the stated configurations. RCP prevents unbounded per-step reference-context growth, but generating more cameras still increases total denoising work.

### Single-image extension

The framework can be placed after an image-to-video or character-animation system. The demonstrated chain consists of a single input image, Wan-Animate driven by a motion video to synthesize a monocular performance video, 4DAnyone to generate multiview videos, and FreeTimeGS to reconstruct a 4D Gaussian avatar.

The resulting capability is therefore a single-image-to-4D extension mediated by a synthesized monocular performance sequence; 4DAnyone itself is defined around monocular video input.

4DAnyone’s central contribution is the integration of explicit human-geometry conditioning, multiview diffusion, fixed-length reference memory, denoising-time target-group routing, and downstream 4DGS reconstruction. RCP addresses reference-context growth, while TCR addresses target-group isolation. Their combination provides bounded per-step attention cost and broad cross-view information exchange, but does not eliminate dependence on pose estimation, diffusion quality, camera coverage, or substantial reconstruction compute.

Source: https://www.emergentmind.com/topics/4danyone