- The paper introduces HVG, a latent video diffusion model that generates multi-view, spatiotemporally coherent human videos from one image using 3D pose sequences and camera parameters.
- HVG represents pose with paired depth and normal maps rendered from volumetric ellipsoidal bones, improving occlusion handling and reducing anatomical errors and clothing-related shape leakage compared with 2D skeletons and SMPL maps.
- HVG achieves a 59.35 FID and 152.1 FVD for novel-view synthesis, outperforming LHM’s 81.60 FID and 248.4 FVD, while remaining limited by facial artifacts and evaluation on mostly horizontal camera paths.
Overview and motivation
This paper presents HVG (Human Video Generation in 4D), a latent video diffusion model that synthesizes multi-view, spatiotemporally coherent human videos from a single reference image, conditioned on 3D pose sequences and camera parameters. The work targets a specific gap in pose-guided human animation: existing methods either operate at fixed viewpoints or rely on driving signals that produce anatomically implausible motion or shape artifacts under novel views.
The authors identify two failure modes in prior conditioning representations. First, 2D skeleton–based methods such as AnimateAnyone and MimicMotion lack hierarchical limb rotations and collision constraints, producing dislocated hips, hyperextended knees, and unnatural arm twists when rendered from novel angles. Second, SMPL-based methods such as Champ oversimplify character-specific geometry, causing "shape leakage" — warped garment edges, distorted clothing, and inconsistent limb proportions. Human4DiT, the closest prior work on 360-degree human video generation, uses SMPL-derived normal maps and a computationally expensive joint viewpoint-spatial attention mechanism. HVG addresses both issues with a new pose representation, a lightweight view alignment strategy, and a progressive sampling scheme for long multi-view sequences.
Method
Dual-dimensional bone map
The core representation is a "bone map" constructed from a posed SMPL-X mesh: 23 skeletal joints are extracted, and each bone segment is modeled as a 3D ellipsoid centered at the segment midpoint, oriented along the bone direction via a rotation matrix aligning the bone vector with the canonical axis. Ellipsoid radii are set from anthropometric measurements (e.g., roughly 2.5 cm for wrists), with the major radius equal to half the bone length. These ellipsoids are rendered via perspective projection into two complementary maps: a depth map encoding z-ordering for occlusion resolution, and a normal map capturing surface orientation transformed into camera space for multi-view consistency.
The claimed advantage is that this representation retains volumetric cues absent in 2D skeletons while avoiding SMPL's dense geometric coupling to body shape, thereby decoupling shape from pose and preventing clothing distortion. This is an intermediate representation between sparse keypoints and full parametric meshes; its effectiveness rests on the assumption that ellipsoidal bone volumes suffice to convey occlusion structure without full surface geometry.
Network architecture
The DenoisingNet follows SV4D's design, initialized from pre-trained SVD-xt, with per-layer 3D convolution, spatial attention, view attention, and temporal attention blocks. Pose information enters through two guiders (one per map type) built from four convolutional layers in ControlNet style, fused by cross-attention and injected into the first convolutional block. The reference image contributes through three pathways: CLIP embeddings via cross-attention, VAE-encoded features concatenated to noise latents, and ReferenceNet appearance features integrated into down- and mid-blocks only, to limit computational cost.
A key architectural simplification concerns view attention. Human4DiT's 3D view attention operates over tokens spanning views and spatial dimensions jointly, which constrains feasible window sizes. HVG instead applies a human-centric alignment: the projected pelvis joint is used to shift and crop all subjects to a consistent image position across views, after which plain 2D attention suffices for cross-view correspondence. Camera pose and frame index are encoded with sinusoidal positional embeddings, with camera rotations expressed relative to the first view.
Progressive spatio-temporal sampling
For long multi-view generation, inference alternates between overlapping windows along the temporal dimension (e.g., 24 frames × few views) and the view dimension (e.g., 6 views × fewer frames). At each denoising timestep, temporally denoised latents and view-denoised latents are merged via weighted fusion of overlapping segments, then combined through a learned weighting strategy. A separate temporal alignment step positions the reference-image subject consistently with the first pose frame to reduce flickering. This extends progressive latent fusion techniques from long-video methods such as MimicMotion to the joint view-time setting, in contrast to Human4DiT's non-overlapping partitioning.
Experimental results
Training uses THuman2.0/2.1, CustomHuman, 2K2K (~5,000 scans rendered into 360° views) plus MVHumanNet (5,000 multi-view videos), at 576×576 resolution on 32 NVIDIA H100 GPUs, in two phases: 60,000 steps for the multi-view model, then 50,000 steps fine-tuning the multi-view multi-frame model. Evaluation covers novel-view synthesis on rendered 3D scans (50 test subjects) and joint novel-view/novel-pose synthesis on MVHumanNet (25 videos, 8 views, 200 generated videos).
On novel-view synthesis, HVG reports FID of 59.35 versus 81.60 for LHM, PSNR of 22.13 dB versus 19.82, LPIPS of 0.057, FID-VID of 13.97, and FVD of 152.1 versus 248.4 for LHM — substantial margins across every metric. On joint novel-view and novel-pose synthesis, HVG achieves FID 84.74, PSNR 20.90, LPIPS 0.092, and FVD 177.4, again outperforming all baselines including fine-tuned versions of AnimateAnyone and Champ. Notably, baselines were retrained on the authors' data where possible, so the comparison is not merely one of training-data advantage.
| Method |
FID ↓ |
SSIM ↑ |
PSNR ↑ |
LPIPS ↓ |
FID-VID ↓ |
FVD ↓ |
| MagicAnimate |
363.1 |
0.756 |
8.23 |
0.455 |
161.3 |
1323 |
| AnimateAnyone* |
106.5 |
0.892 |
18.39 |
0.093 |
30.74 |
295.2 |
| Champ* |
88.31 |
0.904 |
19.35 |
0.079 |
27.42 |
257.4 |
| MimicMotion |
223.6 |
0.905 |
19.59 |
0.082 |
24.55 |
271.1 |
| AniGS |
101.5 |
0.900 |
18.97 |
0.084 |
28.62 |
262.0 |
| LHM |
81.60 |
0.907 |
19.82 |
0.075 |
25.80 |
248.4 |
| HVG |
59.35 |
0.923 |
22.13 |
0.057 |
13.97 |
152.1 |
(fine-tuned on the authors' dataset; novel-view synthesis results.)
Ablations
The ablation on pose representations quantifies the contribution of the bone map: removing the normal map drops PSNR from 22.13 to 20.34 and raises FVD to 205.7; removing the depth map yields 20.69 / 186.2; replacing bone maps with a skeleton-only input degrades sharply to 18.52 PSNR and 277.5 FVD; using SMPL body normal maps alone gives 19.61 / 230.6. These results support the paper's central claim that the dual-map ellipsoid representation resolves occlusion and shape leakage better than either extreme of the representation spectrum. Qualitatively, skeleton-only inputs cause limb misplacement and SMPL-normal inputs cause exaggerated clothing contours, consistent with the stated failure modes.
The sampling ablation is qualitative only: with temporal-window sampling alone, adjacent views show color inconsistencies (e.g., shorts shifting white to orange); with view-window sampling alone, adjacent frames show logo appearance/disappearance. The combined strategy mitigates both, though no quantitative metric isolates the sampling contribution.
Limitations and open questions
The paper concedes a clear limitation: because it prioritizes global structural coherence across the full body, HVG introduces facial artifacts — distortions around the nose and lips — reflecting a trade-off between whole-body consistency and high-frequency local detail. The proposed remedy, a dedicated head-cropping network fused post hoc, is sketched but not implemented or evaluated. Additional open points include: the learned weighting between temporal and view latents during fusion is not analyzed in detail; the evaluation is restricted to near-horizontal 360° orbits (views captured at nearly the same height), leaving arbitrary camera trajectories untested; and the sampling-strategy benefit lacks quantitative corroboration beyond visual comparison.
Conclusion
HVG demonstrates that an intermediate pose representation — volumetric ellipsoidal bones rendered as paired depth and normal maps — combined with pelvis-based view alignment and overlapping spatio-temporal latent fusion, yields measurable gains in single-image 4D human video generation, with FVD reduced by roughly 39% relative to the strongest baseline in novel-view synthesis. The main unresolved questions concern facial fidelity under the global-coherence objective and generalization beyond horizontal orbital camera paths.
