ViSA: 3D-Aware Video Shading for Real-Time Upper-Body Avatar Creation (2512.07720v1)

Abstract: Generating high-fidelity upper-body 3D avatars from one-shot input image remains a significant challenge. Current 3D avatar generation methods, which rely on large reconstruction models, are fast and capable of producing stable body structures, but they often suffer from artifacts such as blurry textures and stiff, unnatural motion. In contrast, generative video models show promising performance by synthesizing photorealistic and dynamic results, but they frequently struggle with unstable behavior, including body structural errors and identity drift. To address these limitations, we propose a novel approach that combines the strengths of both paradigms. Our framework employs a 3D reconstruction model to provide robust structural and appearance priors, which in turn guides a real-time autoregressive video diffusion model for rendering. This process enables the model to synthesize high-frequency, photorealistic details and fluid dynamics in real time, effectively reducing texture blur and motion stiffness while preventing the structural inconsistencies common in video generation methods. By uniting the geometric stability of 3D reconstruction with the generative capabilities of video models, our method produces high-fidelity digital avatars with realistic appearance and dynamic, temporally coherent motion. Experiments demonstrate that our approach significantly reduces artifacts and achieves substantial improvements in visual quality over leading methods, providing a robust and efficient solution for real-time applications such as gaming and virtual reality. Project page: https://lhyfst.github.io/visa

• The paper introduces a two-stage paradigm combining transformer-based 3D Gaussian avatar reconstruction with a lightweight, autoregressive video diffusion renderer.
• It achieves high photorealism and stability with superior PSNR, SSIM, and LPIPS metrics, running at 15 FPS on a single A100 GPU.
• The framework’s use of 3D-aware feature conditioning and adversarial loss minimizes identity drift and structural inconsistencies for robust upper-body avatar animation.

ViSA: Real-Time 3D-Aware Video Shading for Upper-Body Avatar Creation

Introduction and Motivation

ViSA introduces a two-stage paradigm to synthesize high-fidelity, photorealistic, and temporally coherent upper-body digital avatars from a single image (2512.07720). Existing 3D avatar frameworks relying on explicit reconstruction (e.g., 3D Gaussian Splatting, NeRFs) yield structural solidity and real-time responsiveness but fall short in texture fidelity and motion dynamics, often producing blurry or unnatural results. Conversely, generative video models based on large-scale diffusion or transformer backbones achieve compelling detail and dynamism but suffer from structural drift and identity inconsistencies, rendering them inadequate for interactive and controllable avatar use-cases. ViSA bridges these divisions via an explicit-implicit hybrid: a robust, feed-forward 3D avatar reconstructor pilots a lightweight, autoregressive video diffusion shader, granting both geometric stability and photorealistic rendering.

Figure 1: Overview of the ViSA pipeline combining explicit 3D Gaussian avatar reconstruction with a real-time autoregressive video diffusion renderer, yielding photorealistic frames guided by 3D-aware features.

Methodology

Stage I: One-Shot 3D Avatar Reconstruction

ViSA's reconstruction leverages a transformer architecture to operate in canonical 3D space. Three feature streams—high-level semantics (DINOv2), low-level visual information (VAE encoder from a video diffusion backbone), and person-invariant structural priors (learnable SMPL-X vertex embeddings)—are fused, then lifted into 3D via projection using pose and shape estimations. The joint token sequence, comprising per-vertex fused features, is processed by a shallow transformer to regress Gaussian Splatting attributes that map both geometry and appearance.

This explicit 3D representation affords strong identity, shape, and appearance priors, essential for stable structural persistence during dynamic animation. The model is trained with composite objectives: pixelwise, perceptual, and silhouette (mask) loss terms measured between rendered and ground-truth images.

Stage II: Real-Time Autoregressive Video Diffusion Renderer

ViSA's video renderer is based on a few-step, autoregressive diffusion transformer (Self-forcing), optimized for causal, low-latency rollouts. Key technical adaptations are:

• 3D-Aware Feature Conditioning: Rather than deploying conventional 2D keypoint/RGB parametric guidance, the renderer incorporates dense 3D features extracted from the reconstructed avatar, concatenated as channels into the noisy latent tensors fed to the diffusion model. This mitigates identity drift and elevates context informativeness while eliminating VAE encoding bottlenecks, thus reducing computational latency.
• Reference Image Anchoring: A persistent identity reference is realized by caching the Key-Value pairs of the reference image input, continuously injected into all autoregressive generation steps via an attention cache. Spatial misalignments due to pose variation are countered with shifted Rotary Position Embedding (RoPE).

  Figure 2: The ViSA discriminator architecture employs a pretrained WAN video diffusion backbone with a conditional classification head, facilitating adversarial training guided by reference image features.

• Adversarial Distribution Preservation Loss: Naïve finetuning of distilled diffusion models typically produces blurred outputs, a result of high noise variance in the conditioning distribution. ViSA introduces a relativistic adversarial loss, employing a WAN-based frozen backbone as the critic and a conditional classification head. The strategy aligns generator outputs with authentic video distributions, improving texture sharpness and suppressing generative artifacts.

Progressive End-to-End Training

ViSA is optimized in two stages: initial end-to-end alignment of the full pipeline on short sequences (joint 3D/2D supervision and adversarial losses), then temporal extension by freezing the 3D reconstructor to reinforce longer-range consistency in generated videos.

Figure 3: Eliminating neck artifacts through targeted vertex pruning improves template quality for full-body avatars.

Dataset Curation and Annotation

The framework relies on a high-quality, large-scale dataset derived from the Speaker-5M corpus. Upper-body video clips are rigorously filtered by pose variation and keypoint confidence; SMPL-X and FLAME parameters are estimated, then jointly optimized via reprojection minimization to recover high-fidelity geometry and expression annotations. A novel facial fusion approach (FLAME+SMPL-X) eliminates neck artifacts by pruning neck vertices, ensuring more seamless head-body continuity for synthetic training.

Experimental Results

Comparative Evaluation

ViSA outperforms state-of-the-art 2D generative methods (MimicMotion, VACE, Champ) and recent 3D reconstruction-based approaches (GUAVA) on multiple fronts:

• Accuracy and Fidelity: Quantitatively, ViSA yields superior PSNR, SSIM, and LPIPS metrics, coupled with the lowest identity drift scores across both self-reenactment (same-subject) and cross-reenactment (pose transfer) benchmarks.
• Temporal and Structural Consistency: Unlike exclusively 2D approaches subject to pose instability and background hallucinations, ViSA strictly maintains identity and pose integrity across arbitrary sequences and backgrounds.
• Speed: The few-step, autoregressive diffusion renderer achieves real-time performance—15 FPS on a single A100 GPU—outpacing all previous conditional video synthesis models.

  Figure 4: On self-reenactment benchmarks, ViSA produces clearer, more consistent avatars compared to previous methods.

  

  Figure 5: Cross-reenactment experiments highlight ViSA's superior identity preservation under pose and expression variation.

Ablations

Substitution of 3D-aware feature conditioning with either keypoints or 3D-aware RGB guidance yields pronounced performance drops—keypoint conditions produce anatomical implausibilities and artifacts, while RGB-based guidance suffers from VAE encoding latency and semantic inefficacy.

Figure 6: Comparative ablation exposes the advantage of feature-based conditioning over keypoint and RGB conditions.

Inclusion of the adversarial distribution preservation loss is crucial: its removal reintroduces blurriness and artifacting, particularly in high-frequency regions.

Figure 7: Adversarial Distribution Preservation loss demonstrably sharpens results and attenuates high-frequency artifacts.

Implications and Future Directions

ViSA's hybridization of explicit 3D priors with advanced diffusion video synthesis not only advances real-time photorealistic avatar creation, but also sets a direction for scalable, controllable, and interactive digital human animation. The architectural generality and training strategy—distilled, modular, and amenable to adversarial post-training—suggests translatability to other character-driven synthesis domains, including full-body avatars, expressive hand modeling, and multi-agent animation.

Practically, this methodology scales to virtual gaming, telepresence, AR/VR embodiment, and avatar-mediated communication. Theoretically, ViSA reinforces the value of explicit geometry-guided conditioning for reliable content persistence and motivates further research into distribution-matched, adversarially regulated diffusion models.

Conclusion

ViSA (2512.07720) delivers a robust and efficient solution for single-image-driven, real-time, photorealistic upper-body avatar generation. The synergy of transformer-based 3D Gaussian reconstruction and lightweight autoregressive video diffusion—augmented with adversarially anchored finetuning—positions ViSA as a pivotal benchmark for temporally coherent, identity-consistent, and interactive avatar synthesis in AI-driven graphics and mixed-reality systems.