Animated 3DGS Avatars in Diverse Scenes with Consistent Lighting and Shadows

Abstract: We present a method for consistent lighting and shadows when animated 3D Gaussian Splatting (3DGS) avatars interact with 3DGS scenes or with dynamic objects inserted into otherwise static scenes. Our key contribution is Deep Gaussian Shadow Maps (DGSM), a modern analogue of the classical shadow mapping algorithm tailored to the volumetric 3DGS representation. Building on the classic deep shadow mapping idea, we show that 3DGS admits closed form light accumulation along light rays, enabling volumetric shadow computation without meshing. For each estimated light, we tabulate transmittance over concentric radial shells and store them in octahedral atlases, which modern GPUs can sample in real time per query to attenuate affected scene Gaussians and thus cast and receive shadows consistently. To relight moving avatars, we approximate the local environment illumination with HDRI probes represented in a spherical harmonic (SH) basis and apply a fast per Gaussian radiance transfer, avoiding explicit BRDF estimation or offline optimization. We demonstrate environment consistent lighting for avatars from AvatarX and ActorsHQ, composited into ScanNet++, DL3DV, and SuperSplat scenes, and show interactions with inserted objects. Across single and multi avatar settings, DGSM and SH relighting operate fully in the volumetric 3DGS representation, yielding coherent shadows and relighting while avoiding meshing.

• The paper introduces Deep Gaussian Shadow Maps (DGSM) to compute analytic transmittance in 3D Gaussian Splatting avatars.
• It presents an efficient spherical harmonics-based relighting pipeline that modulates avatar Gaussians without explicit BRDF inversion.
• Empirical evaluations demonstrate significantly improved shadow accuracy and lighting consistency over traditional mesh-based techniques.

Consistent Relighting and Shadow Synthesis for Animated 3D Gaussian Splatting Avatars

Introduction and Scope

This paper addresses photorealistic, physically plausible illumination and shadowing in volumetrically represented scenes and avatars modeled by 3D Gaussian splatting (3DGS). Traditional shadow mapping and relighting techniques for mesh-based systems are inherently incompatible with 3DGS’s non-watertight, soft-boundary, and highly parallelizable architecture. The problem is acute in interactive or compositional workflows where dynamic avatars (humans or objects) are inserted into a static environment and fail to exhibit scene-consistent relighting and plausible self/scene shadows, producing visually disconnected results.

The principal contribution is the introduction of Deep Gaussian Shadow Maps (DGSM): a closed-form, volumetric deep-shadow mapping approach, parameterized over concentric spherical shells and octahedral atlases, rigorously adapted to the mathematical locality and translucency of 3DGS. The authors also propose an efficient spherical harmonic (SH) based relighting pipeline for avatar Gaussians, leveraging fast per-Gaussian radiance transfer without requiring explicit BRDFs or any surface meshing.

Deep Gaussian Shadow Maps (DGSM): Formulation and Implementation

The DGSM algorithm generalizes classical deep shadow mapping (DSM) to volumetric, anisotropic Gaussian primitives. Each light is surrounded by $K$ discrete spherical shells into which shadow-casting objects and avatars are inserted. The key technical insight is that the transmittance field along a light-to-point ray through a Gaussian mixture can be computed analytically due to the exponential quadratic form of the splats. This field is compactly tabulated across directions (parameterized by octahedral mapping for low distortion and efficient lookup) and radial bins from the source, forming a tensor $\mathcal{T}[u, v, k]$ of shape $K \times H \times W$.

Figure 1: Deep Gaussian Shadow Maps utilize concentric spherical shells and an octahedral direction atlas for memory-efficient, real-time shadow queries in the Gaussian domain.

At rendering, each receiver Gaussian samples the DGSM for its spatial centroid, retrieving a trilinear-interpolated transmittance corresponding to light attenuation. Efficient receiver- and occluder-space culling (inspired by 3DGS’s tiled rasterization) curbs unnecessary computations by limiting the DGSM update region and occluder checks to relevant light-space ellipsoidal footprints.

A systematic ablation demonstrates the superiority of the octahedral atlas over cubemaps for parameterization (SM-IoU: 0.830 vs 0.811; boundary F-measure: 0.796 vs 0.761) and validates the proposed opacity-to-absorption calibration (using the trace of the local precision matrix) in reducing attenuation error and improving shadow map sharpness. The necessity of both receiver- and light-space culling is highlighted by a dramatic acceleration in DGSM build times (0.13s/frame vs up to 29.1s/frame otherwise).

Spherical Harmonics-based Relighting Pipeline

For each inserted avatar, the system estimates the local incident illumination by rendering a 3DGS-based HDRI environment cubemap centered at the avatar’s position. SH coefficients are fitted via a weighted, ridge-regularized least-squares regression on the hemisphere’s radiance, producing a compressed lighting representation for subsequent analytic integration.

During rendering, the color of each avatar Gaussian is modulated by integrating the fitted SH environment with a cosine/glossy lobe aligned to the primitive’s estimated normal. This results in an exposure-robust, efficient, and plausibly relit avatar, sidestepping any explicit BRDF inversion or per-frame optimization.

Figure 2: The relighting and shadow-casting pipeline enables avatars to seamlessly reflect environment lighting and cast plausible shadows, addressing the uniform illumination artifacts of vanilla 3DGS scene compositing.

Empirical Evaluation

The methodology is evaluated across single- and multi-avatar scenes, dynamic object insertion, and highly disparate target environments (AvatarX/ActorsHQ avatars with ScanNet++, DL3DV, and SuperSplat scenes). Three bespoke illumination consistency metrics are defined: SH probe-avatar agreement in luminance (PAA–Y), avatar photometric fit (APF–Y), and chromaticity neighborhood match (NCM–ab). Across three scenes, the method improves all scores versus non-relit baselines: average $\Delta$ values are 0.253 (PAA–Y), 0.027 (APF–Y), and 2.40 (NCM–ab), signifying significantly improved lighting consistency at the color and distributional level.

Quantitative shadow accuracy is assessed against mesh-derived pseudo-ground-truth shadow maps, using attenuation error, intersection-over-union (IoU), and boundary F-measure. The full method with Monte Carlo (MC) sampling for receivers achieves a SAE of 0.058, SM-IoU 0.830, and boundary F-measure 0.796, considerably outperforming center-sample and deterministic stencil alternatives.

A user study with 12 raters returns an overall realism win rate of 75% and a lighting win rate of 87.5% (averaged over three scenes) versus baseline insertions, supporting the perceptual validity of the synthesized shadows and relighting.

Figure 3: Animated avatars and objects receive and cast temporally stable, plausible volumetric shadows, supporting both solitary and composites in complex scenes.

Figure 4: Ablation studies confirm optimal design: MC sampling, trace-based absorption mapping, and octahedral atlases yield the sharpest, most faithful shadow boundaries and smooth falloff.

Implications, Limitations, and Future Directions

The introduction of DGSM and its analytic integration with 3DGS paves the way for real-time, photorealistic human/scenario synthesis with direct compositional control and fidelity unattainable with previous mesh- or NeRF-based relighting pipelines. The avoidance of explicit meshing is especially significant for pipelines dependent on rapid editing, articulation, and dynamic scene composition, such as virtual production or robotics simulation. The use of octahedral mapping for shadow field storage is both mathematically and practically attractive for memory efficiency and hardware utilization in a differentiable rendering context.

A limitation of the approach is the reliance on single-scattering shadow approximation, potentially missing higher-order global illumination phenomena (e.g., caustics, interreflections, specular bounce), and the static-light assumption restricts dynamic illumination editing. The method assumes accurate light estimation; systematic errors in lighting could propagate to relighting/shadow artifacts. The method currently targets environments and avatars built within the same 3DGS domain—generality to mixed-representation scenes was not addressed.

Theoretically, the closed-form volumetric light transport formulation could generalize to participating media and may inspire future differentiable global illumination integrators for scene optimization. Practically, integrating data-driven or learned global illumination, supporting spatially variant and temporally dynamic lighting, and extending to anisotropic or glossy material models within 3DGS are promising directions. Joint end-to-end training strategies incorporating DGSM, illumination estimation, and per-Gaussian appearance parameters could further improve robustness, consistency, and editability in interactive systems.

Conclusion

The paper delivers a rigorous, practical solution for relighting and shadow synthesis of animated avatars and inserted objects in 3DGS-based scenes using the novel Deep Gaussian Shadow Maps paired with SH environment transfer. The method advances both the mathematical modeling and the computational engineering of compositional neural scene rendering, specifying tractable, efficient, and physically plausible light transport operators native to the volumetric, implicit structure of 3D Gaussian Splatting environments (2601.01660).