SqueezeMe: Mobile-Ready Distillation of Gaussian Full-Body Avatars (2412.15171v4)

Abstract: Gaussian-based human avatars have achieved an unprecedented level of visual fidelity. However, existing approaches based on high-capacity neural networks typically require a desktop GPU to achieve real-time performance for a single avatar, and it remains non-trivial to animate and render such avatars on mobile devices including a standalone VR headset due to substantially limited memory and computational bandwidth. In this paper, we present SqueezeMe, a simple and highly effective framework to convert high-fidelity 3D Gaussian full-body avatars into a lightweight representation that supports both animation and rendering with mobile-grade compute. Our key observation is that the decoding of pose-dependent Gaussian attributes from a neural network creates non-negligible memory and computational overhead. Inspired by blendshapes and linear pose correctives widely used in Computer Graphics, we address this by distilling the pose correctives learned with neural networks into linear layers. Moreover, we further reduce the parameters by sharing the correctives among nearby Gaussians. Combining them with a custom splatting pipeline based on Vulkan, we achieve, for the first time, simultaneous animation and rendering of 3 Gaussian avatars in real-time (72 FPS) on a Meta Quest 3 VR headset. Demo videos are available at https://forresti.github.io/squeezeme.

• The paper introduces SqueezeMe, a method for creating efficient 3D Gaussian avatars enabling real-time rendering of multiple avatars at 72 FPS on mobile VR headsets like the Meta Quest 3.
• Key innovations include generating animatable Gaussians in UV-space, simplifying the decoder via Linear Distillation, and sharing corrective computations among neighboring Gaussians.
• This research advances multi-user VR experiences on mobile hardware by significantly reducing the computational cost of high-quality avatars without substantial loss in visual fidelity.

An Evaluation of SqueezeMe: Efficient Gaussian Avatars for VR

The paper "SqueezeMe: Efficient Gaussian Avatars for VR" explores innovative approaches to developing computationally efficient avatars for virtual reality (VR) headsets. The authors address a crucial bottleneck in rendering high-quality human-like avatars on consumer-grade VR hardware, focusing primarily on the Meta Quest 3 VR headset. They explore the use of 3D Gaussian Splatting as an effective medium for creating detailed, realistic 3D avatars that can be animated in real-time.

Motivation and Background

Traditional methods of rendering avatars, such as meshes and Neural Radiance Fields (NeRF), either require significant computational resources or offer limited rendering efficiency at high levels of detail. The initial implementations of Gaussian Splatting presented a viable alternative, producing avatars with rich visual quality. However, they were constrained to operate on more powerful desktop GPUs due to computational demands. This paper sets out to bridge this gap by adapting Gaussian Splatting techniques for real-time rendering on mobile VR headsets.

Methodological Innovations

The authors introduce multiple key innovations to enhance the efficiency of Gaussian Splatted avatars:

1. UV-Space Animatable Gaussians: Unlike previous methods that generated Gaussians in pixel space, the authors propose creating Gaussians in UV-space. This change enables more coherent placement of Gaussians aligned with the avatar's mesh, reducing computational load by intelligently managing Gaussian distribution.
2. Linear Distillation: A novel approach to simplify the neural network decoder that applies corrective modifications to Gaussians by reducing a complex network to a single linear layer using PCA. This streamlining significantly speeds up the computations required to render avatars.
3. Gaussian Corrective Sharing: By recognizing similarities in required corrections across a local neighborhood of Gaussians, the authors manage to share a single corrective across multiple neighboring Gaussians, further reducing computational overhead.
4. Efficient Rendering Pipeline on Vulkan: The rendering pipeline is optimized for the mobile GPU in VR headsets using Vulkan, thus allowing multiple avatars to be rendered simultaneously at 72 FPS.

Results and Performance

Empirical evaluations of this approach demonstrate its effectiveness. The authors report achieving the rendering of three avatars in parallel on a Meta Quest 3 VR headset at a stable rate of 72 FPS. This was achieved by reducing the decoder latency from 50 ms to 0.45 ms through compression techniques and computational sharing strategies. These findings underscore a significant leap in computational efficiency without a substantial loss of visual quality, as corroborated by quantitative metrics like L1, LPIPS, PSNR, and SSIM across several user evaluations.

Implications and Future Directions

Practically, this research impacts VR development by enabling more immersive, real-time, multi-user experiences on mobile VR headsets. Theoretically, it demonstrates significant advancements in streamlining complex avatar rendering processes through UV-space mapping, linear simplification, and sharing computational tasks among model components.

Future developments could explore adaptive methods to distribute Gaussians and correctives more intelligently, thus addressing observed limitations. Enhancing adaptability could lead to further improvements in visual fidelity and computational efficiency, potentially scaling up the number of avatars that can be simultaneously rendered even further.

In conclusion, "SqueezeMe: Efficient Gaussian Avatars for VR" provides a cohesive, detailed depiction of integrating Gaussian Splatted avatars into the limited computational environments of VR headsets. It opens pathways for further refinement and adoption in VR applications, advancing toward more complex, interactive virtual worlds.