Human4DiT: 4D Diffusion Transformer

• Human4DiT is a framework that generates 360° high-quality, spatio-temporally coherent human videos from a single image using a hierarchical 4D diffusion transformer.
• It improves consistency by factorizing self-attention across space, time, and viewpoints while integrating dedicated pathways for identity, pose, camera, and temporal controls.
• Its modality-aware training and spatio-temporally consistent sampling strategy deliver superior performance over traditional GAN and U-Net diffusion methods.

Human4DiT is a framework for generating 360-degree high-quality, spatio-temporally coherent human videos from a single image. It combines diffusion transformers for capturing global correlations across viewpoints and time with CNNs for accurate condition injection, and centers on a hierarchical 4D transformer architecture that factorizes self-attention across views, time steps, and spatial dimensions. The method is presented as the first diffusion transformer for human video generation, and it couples this architecture with rich conditioning, a multi-dimensional dataset spanning images, videos, multi-view data, and limited 4D footage, a modality-aware training strategy, and a spatio-temporally consistent sampling procedure for free-view synthesis (Shao et al., 2024).

1. Scope and problem formulation

Human4DiT addresses the task of human video generation under simultaneous motion and viewpoint control. The stated input consists of a reference image $\mathbf{y}_{r}$, a SMPL-based dynamic pose sequence $\{\mathbf{P}_{i=1}^{T}\}$, and a camera parameter sequence $\{\mathbf{c}_{i=1}^{V}\}$; the goal is to generate a video showing that person performing the given motion from arbitrary views (Shao et al., 2024).

This task is positioned against two families of earlier methods. GAN-based methods, including examples such as FOMM and G3AN, are described as struggling to maintain temporal and viewpoint consistency, especially under complex motions or large viewpoint changes, and as producing jitter and artifacts. U-Net-based diffusion methods, including AnimateAnyone, MagicAnimate, and Champ, are described as relying on local condition injection and mainly handling local framewise visual appearance, which leads to weaker global correlation reasoning across time and views, less coherent long sequences, and poor cross-view consistency (Shao et al., 2024).

A useful clarification follows from this formulation: Human4DiT is not merely a monocular animation system, and it is not a generic multi-person interaction model. Its declared regime is single-image-conditioned generation of one human subject under controlled motion and view changes. This distinguishes it from adjacent 4D human modeling tasks such as close-contact 4D instance segmentation, where the central challenge is disentangling multiple interacting bodies under severe occlusion and mesh fusion rather than synthesizing new views and motions (Yin et al., 2023).

2. Hierarchical 4D diffusion transformer

The core representation is a latent video tensor

$$\mathbf{x}_t \in \mathbb{R}^{V \times T \times H \times W \times C},$$

where $V$ denotes views, $T$ timesteps, $H \times W$ spatial dimensions, and $C$ channel depth. Human4DiT models this 4D space through a cascaded transformer framework that factorizes self-attention hierarchically across spatial, temporal, and viewpoint dimensions rather than applying full non-factorized 4D attention (Shao et al., 2024).

The hierarchy is composed of three transformer types. The 2D image transformer block performs spatial self-attention within each $H \times W$ grid for each frame at each view and time. The view transformer block then performs joint self-attention across the view and spatial dimensions for each timestep. The temporal transformer block performs self-attention across time for each view and spatial location. These three blocks are interconnected into a single 4D transformer block, and several such blocks are cascaded to form the complete 4D diffusion transformer (Shao et al., 2024).

The architecture is explicitly motivated by efficiency and global coherence. By breaking the 4D attention computation into sequential, lower-dimensional operations, computational cost is substantially reduced relative to full 4D attention, while the model still captures global, spatio-temporal correlations. The paper frames this as the architectural reason Human4DiT can scale beyond earlier GAN-based and vanilla diffusion baselines that struggle with complex motions, viewpoint changes, and generalization (Shao et al., 2024).

A common misconception is that the model is simply a DiT backbone substituted for a U-Net. The paper’s description is narrower and more specific: the innovation is a hierarchical 4D factorization over space, view, and time, with separate transformer blocks assigned to each axis and then composed into a unified diffusion backbone (Shao et al., 2024).

3. Conditioning mechanisms and controllability

Human4DiT uses dedicated injection pathways for identity, motion, camera, and temporal control. For human identity, a reference-image UNet encodes identity features $\mathbf{y}_{id}$, which are added to the transformer input tokens for pixel-level identity consistency. In parallel, a global CLIP embedding $\{\mathbf{P}_{i=1}^{T}\}$0 is extracted and injected via cross-attention in each transformer block after self-attention (Shao et al., 2024).

Camera and temporal signals are handled separately. Camera extrinsics, represented as rotation matrices for each view, are encoded as positional embeddings using sines and cosines and then mapped to the latent dimension with an MLP before being added to the view transformer output. Frame numbers are encoded as temporal positional embeddings and injected additively into the temporal transformer features (Shao et al., 2024).

Motion conditioning is based on SMPL. The SMPL mesh vertices are rendered into normal maps for each time and view, and a UNet processes these maps to obtain features $\{\mathbf{P}_{i=1}^{T}\}$1, which are added to the input tokens of the 4D transformer. In the paper’s summary, this is part of a broader conditioning design that integrates human identity, SMPL pose and motion, camera parameters, and temporal signals into different parts of the model for precise and controllable output (Shao et al., 2024).

This division of conditioning pathways matters because the paper contrasts it with earlier systems that mainly inject conditions locally. Human4DiT’s claim is not only that it conditions on more signals, but that each signal is routed to the transformer module most closely aligned with its semantics: identity at the token and cross-attention level, camera in the view transformer, and time in the temporal transformer. A plausible implication is that controllability is treated as an architectural allocation problem, not only as a conditioning-token design problem.

4. Multi-dimensional dataset and modality-aware training

Training relies on a multi-dimensional dataset with separate streams spanning 2D images, monocular videos, multi-view videos, and 3D/4D scans. The examples listed in the summary are HumanArt for 2D images, TikTok and Motion-X for monocular videos, DNA-rendering and AIST for multi-view videos, and THuman2.0 for 3D/4D scans (Shao et al., 2024).

Modality	Example datasets	Training role
2D Images	HumanArt	Train identity conditioning and 2D transformer
Monocular Videos	TikTok, Motion-X	Train spatial and temporal transformers
Multi-view Videos	DNA-rendering, AIST	Train all transformer modules
3D/4D Scans	THuman2.0	Provide dense supervision of 3D/4D structure

The training strategy is modality-aware. The data type determines which transformers are active and what conditioning is provided. The summary gives three explicit cases: 2D images only train the spatial transformer with CLIP identity conditioning; single-view videos train both spatial and temporal transformers; multi-view videos and 3D/4D data train all three transformers jointly (Shao et al., 2024).

The paper describes this design as allowing Human4DiT to fully leverage heterogeneous data for strong generalization across image, video, multi-view, and free-view tasks. This is an important part of the system definition. Human4DiT is not trained as if all inputs came from a single canonical 4D dataset; instead, it assembles supervision from heterogeneous modalities and selectively activates submodules according to what each modality can support (Shao et al., 2024).

In the broader DiT literature for human generation, later work extends this scaling logic in different directions. HumanDiT, for example, is trained on 4.5 million in-the-wild video clips totaling 14,000 hours and uses variable sequence lengths, arbitrary input resolutions, and sequence parallelism for long-form human motion video generation (Gan et al., 7 Feb 2025). This suggests that Human4DiT’s multi-stream training strategy anticipated a later emphasis on scaling data diversity and sequence length, although the two systems address different target regimes.

5. Free-view synthesis and sampling procedure

A distinct contribution of Human4DiT is its sampling strategy for free-view video. The paper states that it proposes a spatio-temporally consistent sampling strategy that synthesizes long, free-view human videos, including arbitrary trajectory, view, and motion, even with constrained input windows during inference (Shao et al., 2024).

The summary provides a combined sampling equation in which the denoising update uses two noise estimates, $\{\mathbf{P}_{i=1}^{T}\}$2 and $\{\mathbf{P}_{i=1}^{T}\}$3, obtained from different temporal and spatial windows and weighted by $\{\mathbf{P}_{i=1}^{T}\}$4 and $\{\mathbf{P}_{i=1}^{T}\}$5. The explicit interpretation attached to that equation is that sampling is designed to preserve both spatial and temporal consistency when inference cannot process the entire target sequence and view configuration in a single window (Shao et al., 2024).

This component addresses a limitation the paper assigns to previous methods: earlier systems are said to be unable to handle changes in both view and motion together. Human4DiT, by combining the 4D transformer with specialized sampling, is presented as enabling long, spatio-temporally consistent free-view videos under simultaneous viewpoint and motion change (Shao et al., 2024).

A clarification is useful here. The model’s claim to “360-degree” generation does not imply unrestricted scene reconstruction or general 3D scene synthesis. In the provided formulation, the view degree of freedom is camera-conditioned human video generation, grounded in a reference image and SMPL-based motion specification. The generated space is therefore human-centric and condition-driven rather than scene-centric.

6. Evaluation, ablations, and comparative performance

Human4DiT is evaluated quantitatively and qualitatively on four settings: Monocular Video, Multi-view Video, 3D Static Video, and Free-view Video. The reported metrics are PSNR$\{\mathbf{P}_{i=1}^{T}\}$6, SSIM$\{\mathbf{P}_{i=1}^{T}\}$7, LPIPS$\{\mathbf{P}_{i=1}^{T}\}$8, and FVD$\{\mathbf{P}_{i=1}^{T}\}$9 (Shao et al., 2024).

For the monocular-video setting, the summary reports the following values. Disco achieves PSNR 20.07, SSIM 0.661, LPIPS 0.285, and FVD 585.3. MagicAnimate achieves 21.08, 0.717, 0.256, and 550.7. AnimateAnyone achieves 22.18, 0.789, 0.195, and 479.5. Champ achieves 22.88, 0.824, 0.171, and 359.3. Human4DiT achieves 26.12, 0.888, 0.116, and 237.4. The summary adds that similarly superior results are shown in multi-view, 3D, and free-view scenarios, and states that Human4DiT outperforms all baselines by a large margin on all metrics (Shao et al., 2024).

The qualitative findings reported in the summary are also specific. Visual comparisons are said to show crisper details, better motion, and much higher temporal and view consistency, with fewer artifacts or jitters. The ablation study is said to show that removing the view transformer significantly hurts multi-view and free-view performance, and this is offered as evidence for the necessity of the 4D approach (Shao et al., 2024).

These comparisons also place Human4DiT within the transition from GAN and U-Net diffusion systems toward DiT-based human video synthesis. Later DiT-based systems pursue related but distinct objectives. HumanDiT emphasizes long-form pose-accurate generation with arbitrary resolution and video lengths up to 249 frames, while Hallo4 emphasizes direct preference optimization and temporal motion modulation for portrait animation under audio and skeletal control (Gan et al., 7 Feb 2025); (Cui et al., 29 May 2025). Human4DiT’s specific contribution within that trajectory is multi-view and free-view coherence centered on a factorized 4D attention design.

7. Position in the research landscape and downstream influence

Human4DiT occupies an early role in the use of diffusion transformers for human video generation. The paper presents it as the first diffusion transformer for this domain, contrasting it with GAN-based approaches and U-Net-based diffusion systems. Its defining claim is that global attention spanning space, time, and viewpoint, combined with precise conditioning for identity, pose, time, and viewpoint, resolves coherence failures that were persistent in earlier approaches (Shao et al., 2024).

Subsequent work has used Human4DiT directly as a component rather than only as a point of comparison. DevilSight uses Human4DiT to generate videos of a subject from alternative unseen perspectives, specifically a back-view, given only a monocular front-view input video. In that framework, the generated alternative perspective acts as pseudo-supervision for monocular avatar reconstruction, enriching previously unseen regions and regularizing the avatar representation to mitigate artifacts. DevilSight further reports physical identity fine-tuning and a patch-based denoising algorithm to improve motion consistency and resolution when Human4DiT is used in this supervisory role (Chen et al., 30 Aug 2025).

This downstream use clarifies Human4DiT’s broader significance. It is not only a generator of free-view human videos; it can also serve as a learned prior over unobserved viewpoints in reconstruction pipelines. A plausible implication is that the model’s view-consistent generative capacity is valuable wherever monocular capture leaves systematic blind spots.

At the same time, Human4DiT should not be conflated with all subsequent DiT-based human generation. HumanDiT targets long-form pose-guided video generation with a prefix-latent reference strategy rather than a hierarchical 4D view-time-space factorization, and Hallo4 targets portrait animation with direct preference optimization and temporal motion modulation rather than free-view body video generation (Gan et al., 7 Feb 2025); (Cui et al., 29 May 2025). In that taxonomy, Human4DiT is most precisely understood as a human-centric, condition-rich, multi-view diffusion-transformer framework whose primary novelty lies in efficient 4D attention factorization, modality-aware training across heterogeneous supervision sources, and spatio-temporally consistent free-view sampling (Shao et al., 2024).