MVP4D: Multi-View Portrait Video Diffusion for Animatable 4D Avatars (2510.12785v1)

Abstract: Digital human avatars aim to simulate the dynamic appearance of humans in virtual environments, enabling immersive experiences across gaming, film, virtual reality, and more. However, the conventional process for creating and animating photorealistic human avatars is expensive and time-consuming, requiring large camera capture rigs and significant manual effort from professional 3D artists. With the advent of capable image and video generation models, recent methods enable automatic rendering of realistic animated avatars from a single casually captured reference image of a target subject. While these techniques significantly lower barriers to avatar creation and offer compelling realism, they lack constraints provided by multi-view information or an explicit 3D representation. So, image quality and realism degrade when rendered from viewpoints that deviate strongly from the reference image. Here, we build a video model that generates animatable multi-view videos of digital humans based on a single reference image and target expressions. Our model, MVP4D, is based on a state-of-the-art pre-trained video diffusion model and generates hundreds of frames simultaneously from viewpoints varying by up to 360 degrees around a target subject. We show how to distill the outputs of this model into a 4D avatar that can be rendered in real-time. Our approach significantly improves the realism, temporal consistency, and 3D consistency of generated avatars compared to previous methods.

• The paper introduces MVP4D, a morphable multi-view video diffusion model generating hundreds of temporally synchronized frames for 4D avatar synthesis.
• It leverages a multi-modal training curriculum with monocular videos and static images to achieve strong spatial, temporal, and multi-view consistency.
• The framework outperforms prior methods in both self- and cross-reenactment tasks, enabling efficient real-time 4D avatar rendering.

Multi-View Portrait Video Diffusion for Animatable 4D Avatars: Technical Summary and Analysis

Introduction and Motivation

The MVP4D framework addresses the challenge of synthesizing animatable, photorealistic 4D human avatars from a single reference image. Traditional avatar creation pipelines require multi-camera rigs and extensive manual post-processing, which are prohibitive for scalable applications. Recent generative models have enabled single-image avatar synthesis, but these approaches suffer from poor multi-view and temporal consistency, especially when rendering views far from the reference. MVP4D introduces a morphable multi-view video diffusion model (MMVDM) that generates hundreds of temporally synchronized frames across 360° viewpoints, conditioned on head pose and expression, and distills these outputs into a 4D avatar suitable for real-time rendering.

Figure 1: Overview of MVP4D. The method takes as input one reference image that is encoded into the latent space of a variational autoencoder.

Model Architecture and Conditioning

MVP4D builds on the CogVideoX-2B video diffusion transformer, adapting it for multi-view generation. The architecture encodes the reference image and each view’s video into a compressed latent space via a spatio-temporal autoencoder. Conditioning signals—derived from a FLAME 3DMM and an off-the-shelf face tracker—encode camera pose, head pose, expression, and view ray information. These signals are concatenated with the latent representations and processed jointly by the transformer, which attends across spatial, temporal, and viewpoint dimensions.

Figure 2: Full visualization of the conditioning signals and the MMVDM architecture.

The model supports four generation modes, enabling flexible synthesis strategies: (1) all views/frames from the reference, (2) partial views conditioned on others, (3) temporal extension from initial frames, and (4) combined spatial-temporal conditioning. This flexibility is critical for scalable inference and efficient use of limited multi-view video data.

Figure 3: The model supports different modes to generate multiview videos based on the reference image latent $\mathbf{z}$.

Multi-Modal Training Curriculum

Due to the lack of large-scale multi-view video datasets, MVP4D employs a multi-modal training curriculum leveraging monocular videos, multi-view static images, and limited multi-view dynamic sequences. Training proceeds in three stages, progressively increasing spatial resolution, number of views, and sequence length. This curriculum enables the model to generalize to configurations (e.g., 8 views × 49 frames × 512px) not seen during training, and to synthesize synchronized multi-view videos with strong temporal and spatial consistency.

Multi-View Video Sampling and 4D Avatar Reconstruction

At inference, MVP4D uses a stepwise sampling strategy: key videos are generated for a subset of views, then additional views and frames are synthesized via clustering and conditioning on nearest key videos. This approach balances computational constraints and multi-view coverage.

Figure 4: The sampled camera view angles for 120-degree avatars (left) and 360-degree avatars (right).

Figure 5: Illustration of the step-by-step multi-view video sampling strategy.

The generated multi-view videos are distilled into a 4D avatar using deformable 3D Gaussian splatting attached to a FLAME mesh. Fine-grained details (e.g., hair, earrings) are reconstructed via structure-from-motion and keypoint triangulation, with per-Gaussian temporal deformations predicted by a U-Net conditioned on sinusoidal temporal embeddings.

Implementation Details

Classifier-free guidance (CFG) is adapted for multi-view generation by producing view-specific unconditional predictions, mitigating artifacts and improving generation quality. Attention biasing is applied to compensate for increased token entropy with more views/frames. Training requires 14 days on 8×H100 GPUs; inference for a 360° avatar (48 views × 89 frames) takes ~10.5 hours on a single RTX 6000 Ada GPU, with real-time rendering post-fitting.

Figure 6: Multi-view classifier-free guidance (CFG) improves generation quality and reduces artifacts compared to conventional CFG.

Experimental Results

Self-Reenactment

MVP4D demonstrates superior temporal consistency (JOD metric) and competitive photometric accuracy (PSNR, SSIM) compared to baselines on the Nersemble and RenderMe-360 datasets. Notably, MVP4D outperforms CAP4D-MMDM and FYE+PanoHead in 360° avatar generation, especially for views not visible in the reference.

Figure 7: Self-reenactment results on the Nersemble and RenderMe-360 datasets. MVP4D recovers fine details and structures of the face and hair that are not captured by other techniques.

Cross-Reenactment

In cross-reenactment (driving a reference image with an external video), MVP4D is preferred by human evaluators in 86% of cases over the strongest baseline (CAP4D), with high scores for visual detail, expression transfer, 3D structure, and motion quality.

Figure 8: Cross-reenactment results. MVP4D reconstructs challenging geometry and models dynamic effects such as wrinkles better than previous methods.

Figure 9: More cross-reenactment results, showing improved reconstruction of hair and realistic expressions.

Ablations

Ablation studies confirm that multi-view CFG and joint generation of more views ($V=8$) improve both quality and 3D consistency, even beyond the training configuration ($V=4$).

Extensions

MVP4D supports speech-driven avatar generation (using Hallo3 for audio-to-expression mapping) and text-to-4D avatar synthesis (by generating a reference image from a text prompt and applying MVP4D).

Figure 10: MVP4D can generate 4D avatars from audio input and a reference image, as well as from generated images.

Limitations

Artifacts in high-frequency spatio-temporal details (e.g., flickering lips) arise from latent space compression. Quality deteriorates for very long sequences due to iterative conditioning. The model is not robust to extreme lighting, as such data is absent from training.

Figure 11: Limitations of MVP4D, including artifacts in high-frequency details, quality degradation in long sequences, and failure under extreme lighting.

Implications and Future Directions

MVP4D demonstrates that video diffusion transformers, when equipped with morphable multi-view conditioning and a multi-modal training curriculum, can synthesize temporally and spatially consistent 4D avatars from a single image. This approach significantly reduces the barrier to avatar creation and animation, with direct applications in virtual reality, gaming, telepresence, and digital content creation.

Theoretically, MVP4D bridges the gap between 2D generative models (which excel at fine-grained dynamics) and 3D representations (which enforce multi-view consistency), leveraging the strengths of both. The multi-modal curriculum and flexible generation modes are critical for scaling to high spatio-temporal complexity without large multi-view video datasets.

Future work should address latent space compression artifacts, improve robustness to lighting variation, and explore direct feed-forward 3D avatar generation for further efficiency. Integration of physically-based rendering and reflectance models could enhance realism and controllability. The MVP4D pipeline may also be extended to full-body avatars and non-human subjects.

Conclusion

MVP4D introduces a scalable, flexible framework for animatable 4D avatar synthesis from a single image, leveraging a morphable multi-view video diffusion model and a multi-modal training curriculum. The method achieves state-of-the-art temporal and multi-view consistency, outperforming prior baselines in both quantitative metrics and human preference studies. While limitations remain in spatio-temporal detail and lighting robustness, MVP4D sets a new standard for single-image-driven avatar generation and opens avenues for further research in efficient, controllable, and realistic digital human synthesis.