ViSA: 3D-Aware Real-Time Video Shading

• The paper introduces ViSA, a framework that integrates explicit geometric modeling, neural rendering, and transformer-based temporal smoothing to achieve photorealistic video relighting and avatar synthesis.
• It employs dual encoders and 3D-aware tri-plane as well as 3D Gaussian splatting techniques to ensure temporally consistent outputs and accurate view and lighting control.
• Quantitative evaluations show ViSA’s superiority in reduced lighting error, improved stability, and accelerated real-time performance, making it impactful for VR, telepresence, and gaming applications.

ViSA (Video Shading Architecture) encompasses a family of real-time, 3D-aware systems for photorealistic video relighting and avatar synthesis. These frameworks directly address traditional limitations in video-based editing and avatar generation—such as slow inference, lack of view/lighting control, texture artifacts, and motion discontinuities—by uniting explicit geometric modeling, neural rendering, and temporally consistent generative models. Recent implementations span portrait video relighting based on tri-plane NeRF variant architectures (Cai et al., 24 Oct 2024) and 3D-aware avatar creation with autoregressive video diffusion guided by 3D Gaussian splatting (Yang et al., 8 Dec 2025).

1. System Architectures and Pipelines

Two recent ViSA systems share the core requirement of complete 3D awareness, but differ in their target domain and architectural details:

A. Portrait Video Relighting Pipeline (Cai et al., 24 Oct 2024):

• Dual-Encoder Backbone: Each video frame $F_i$ is processed by an Albedo Encoder $E_A$ and a Shading Encoder $E_S$.
  • $E_A$ predicts an albedo tri-plane $T_{A_i}$ (three 32×32 feature planes, 256 channels) using DeepLabV3 ResNet-50, CNN, and ViT blocks.
  • $E_S$ predicts a shading tri-plane $T_{S_i}$ conditioned on $T_{A_i}$ and a lighting code $L$ (9D spherical harmonic coefficients) via StyleGAN2-based CNN.
• Temporal Consistency Network (TCN): Two 4-layer transformer branches (albedo and shading) receive a window of prior tri-planes, perform self/cross-attention, and output residuals $\Delta T_{A_i}$, $\Delta T_{S_i}$, leading to temporally smoothed tri-planes $\tilde{T}_{A_i}$, $\tilde{T}_{S_i}$.
• NeRF-Style Volumetric Rendering: The tri-planes condition NeRF-integration to produce photorealistic RGB outputs under arbitrary view and lighting.
• Super-Resolution: A StyleGAN2 head (from EG3D) upsamples to $512^2$.

B. Upper-Body Avatar Creation Pipeline (Yang et al., 8 Dec 2025):

• Stage 1: One-Shot 3D Gaussian Reconstruction.
  • Inputs: A single reference image $I_{ref}$.
  • Feature Extraction: Semantic features via frozen DINOv2, low-level visual features from hierarchical VAE encodings, and learnable human shape priors (per-vertex embeddings for SMPL-X).
  • 3D Lifting: Features are assembled per-vertex, passed through a 5-layer transformer to predict 3D Gaussian splat attributes (position offset, scale, quaternion, color SH coefficients, opacity, and a dense 3D feature vector).
  • Rendering: Standard 3DGS renderer to produce $I_{ren}$ and feature maps $F_{cond}(t)$ for animation.
• Stage 2: Real-Time Autoregressive Video Diffusion Shader.
  • Static Conditioning: Reference image latent embedding, precomputed and reused via attention KV caches, ensures persistent identity.
  • Dynamic Conditioning: At each frame $t$, $F_{cond}(t)$ from the animated 3DGS model is concatenated with the diffusion noise, guiding denoising.
  • Autoregressive Rollout: Each frame is generated by a distilled, causal transformer; output latents are autoregressively appended for temporal modeling.

2. Mathematical Foundations and Rendering Models

A. Tri-plane Neural Rendering (Cai et al., 24 Oct 2024):

• Classic NeRF Volume Rendering:
  • For any camera ray $r(t) = o + t d$,

  $$C(r) = \int_{t_n}^{t_f} T(t)\, \sigma(x(t))\, c(x(t), d) \; dt$$

  where $T(t)$ is accumulated transmittance, $\sigma(x)$ is volume density, and $c(x, d)$ is the view-dependent radiance. - Radiance factorization:

  $c(x, d; L) = A(x) \odot S(x; L)$

  where $A(x)$ and $S(x; L)$ are trilinear-sampled values from the smoothed albedo/shading tri-planes.

• Lighting Representation: Spherical harmonics coefficients $L \in \mathbb{R}^9$, supporting expressive low-frequency global illumination over Lambertian surfaces and enabling both cast/soft shadow effects.

B. 3D Avatar Gaussian Splatting (Yang et al., 8 Dec 2025):

• SMPL-X as Canonical Prior: Per-vertex tokens $T_i$ are constructed by combining image-sampled semantic/visual features and learned priors, projected from canonical 3D to image space.

• Gaussian Splat Attributes Prediction: Tokens are transformed to predict detailed per-vertex attributes: 3D offset, scale, quaternion, multi-band SH color, opacity, and dense feature descriptors.

• Feature-Based Conditioning for Diffusion: Dense 3D features $F_{cond}(t)$ are channel-wise concatenated with diffusion model latents, aligning generated frames tightly to the geometric and appearance priors.

3. Temporal Consistency and Conditioning Strategies

A. Transformer Temporal Smoothing (Cai et al., 24 Oct 2024):

• The TCN applies multi-head self-attention on the sequence of tri-planes, plus cross-attention between albedo and shading. It outputs corrections $\Delta T$, which are added to smooth per-frame predictions, mitigating flicker and enforcing inter-frame coherence.

• Losses combine reconstruction (RGB, albedo, shading), short-term and long-term temporal losses (using LPIPS in warped space, occlusion downweighting), and adversarial losses.

B. Identity and Temporal Conditioning (Yang et al., 8 Dec 2025):

• Static conditioning leverages precomputed KV caches for all attention layers from $I_{ref}$, with shifted Rotary PE for spatial alignment through pose deformations.

• Dynamic 3D feature conditioning directly injects per-frame 3DGS features, outperforming both sparse keypoint and rendered RGB conditioning.

• Temporal context is preserved by caching histories in the autoregressive transformer, with explicit self-rollout during training to prevent exposure bias.

4. Quantitative Performance and Comparative Evaluation

ViSA is benchmarked against both single-image relighting, optimization-driven avatar fitting, and direct video synthesis approaches:

Method	Lighting Error (LE) ↓	Instability (LI) ↓	ID ↑	LPIPS Flicker ↓	Time (s) ↓
B-DPR	0.9093	0.3041	0.5222	0.1015	200
B-SMFR	1.0929	0.3352	0.4479	0.0626	200
B-E4E	0.6384	0.1963	0.2892	0.0306	0.2
B-PTI	0.8220	0.2630	0.4728	0.1080	30
ViSA	0.7710	0.2533	0.5396	0.0159	0.03

• Real-Time Relighting: Achieves $\approx 33$ fps on RTX 4090 (Cai et al., 24 Oct 2024).

• Reconstruction Quality: LPIPS=0.240, DISTS=0.128, pose=0.036, ID=0.702 (comparable to optimization-based approaches, but at real-time speed).

• Avatar Creation: PSNR/SSIM/LPIPS/İPS-self for self-reenactment: 22.1/0.87/0.043/0.037, outperforming GUAVA (18.6/0.86/0.072/0.040) and Champ, with qualitative improvements in texture fidelity and temporal coherence (Yang et al., 8 Dec 2025).

• Autoregressive Inference: Real-time performance at 15 fps (A100), with significant latency reduction when using feature (vs. RGB) conditioning.

5. Implementation Considerations and Engineering Strategies

• Tri-plane Factorization: 3×32×32, 256 channels. Efficient trilinear sampling via custom CUDA kernels; 96 stratified points per ray in NeRF integration.

• Encoder Architectures: Albedo uses ResNet-50 (ImageNet-pretrained) and ViT (12 layers, 768 dims); Shading uses 5 conv + 4 StyleGAN2-modulated layers.

• Diffusion Model: Small, causal transformer, KD-distilled for few-step per-frame denoising.

• Super-resolution: StyleGAN2 EG3D head up to $512^2$.

• Temporal Transformers: 4 layers, 512-dim, 8 heads per branch.

• Training: Multi-stage for both modules, with 32M iterations on 8×V100 (Cai et al., 24 Oct 2024) or 32×H20 GPUs for ~5 days (Yang et al., 8 Dec 2025). Separate schedules for encoders and transformers.

• Optimization: Feature injection for diffusion and eliminating redundant VAE encodes enables a 34% faster inference (Yang et al., 8 Dec 2025).

6. Limitations and Future Directions

• Failure Modes:

  • Extreme occlusion or rare poses can lead to structure artifacts or discontinuities.
  • Incomplete training data for complex lighting (hard shadows, dramatic environment illumination) yields inconsistent or inaccurate shading.
  • Fine hair-body interactions (e.g., loose, moving hair) challenge the representation capacity of both 3DGS and tri-plane architectures.
• Scalability: Current avatar pipelines primarily target upper-body; full-body modeling, environmental integration, and end-to-end audio-driven animation are open problems.
• Performance Frontiers: Further compression of the auto-regressive model or adoption of 1-step consistency distillation are proposed to push beyond 15 fps real-time synthesis.
• Lighting Generalization: Incorporation of HDR or learnable environment-light priors is an active research direction to address pervasive relighting artifacts.

7. Broader Context and Comparative Analysis

The ViSA paradigm represents a convergence of explicit geometric priors (e.g., SMPL-X, 3DGS, tri-planes) and modern neural field/diffusion strategies, setting new state-of-the-art for real-time, high-fidelity, temporally stable video shading. By combining feed-forward encoders, spherical harmonics-based lighting, and transformer-based temporal networks (Cai et al., 24 Oct 2024), as well as integrating efficient 3D feature conditioning in autoregressive generative chains (Yang et al., 8 Dec 2025), ViSA offers a robust alternative to both computation-heavy optimization and temporally unstable direct video generation. This platform has significant implications for virtual reality, gaming, and telepresence, with future extensions toward more comprehensive avatarization—including legs, environment-aware relighting, and naturalistic, audio-driven animation (Yang et al., 8 Dec 2025, Cai et al., 24 Oct 2024).