MPMAvatar: Hybrid 3D Avatar Framework

• MPMAvatar is a hybrid framework that combines physics-based MPM cloth simulation with 3D Gaussian splatting for high-fidelity avatar modeling.
• It utilizes advanced techniques like anisotropic constitutive modeling, mesh-to-grid coupling, and neural quasi-shadowing to enhance simulation and rendering accuracy.
• The system achieves robust dynamics, efficient zero-shot generalization, and significant runtime improvements over prior physics-based avatar methods.

MPMAvatar is a hybrid framework for learning and synthesizing 3D human avatars exhibiting robust, physically plausible dynamics and high-fidelity appearance. By tightly integrating a tailored Material Point Method (MPM)–based cloth simulator with photorealistic 3D Gaussian Splatting for surface representation, MPMAvatar models garment and body motion with high accuracy and supports animation from multi-view video inputs. It achieves high visual realism, generalizes seamlessly to unseen physical interactions, and demonstrates efficiency and robustness improvements over prior physics-based avatar systems (Lee et al., 2 Oct 2025).

1. Architectural Overview

MPMAvatar combines canonical mesh construction, physics-based animation, and advanced appearance modeling in a unified pipeline:

• Canonical Avatar Construction: A human mesh is reconstructed from multi-view videos, with geometry represented by a triangulated surface. Each surface triangle is augmented with physical parameters (Young’s modulus, density, rest configuration) for simulation and a local set of 3D Gaussians encoding shape, color, and opacity.
• Animation Framework: Non-garment regions of the avatar are animated via standard linear blend skinning (LBS), while garment regions—where complex dynamics are critical—are simulated using a customized MPM solver. The mesh serves as a collider for the cloth, ensuring tight contact and realistic interactions.
• Surface Representation and Rendering: The avatar’s visual appearance is rendered by projecting the per-face Gaussian splats, each characterized by position, scale, rotation (quaternion), opacity, and a color parameter modeled with spherical harmonics. To enhance realism, quasi-shadowing is included via a neural shading network that modulates per-Gaussian colors.

The simulation and rendering processes operate in parallel, so that both geometric deformations and photometric appearance are updated in lockstep during animation.

2. Physics-Based Garment Dynamics

Modeling loose garment deformation is accomplished using a modified Material Point Method (MPM) simulator with several innovations:

• Anisotropic Constitutive Model: Cloth material is treated as a codimensional continuum, where in-plane stretching is dominant. The deformation gradient $\mathbf{F}$ is decomposed ($\mathbf{F} = \mathbf{Q} \mathbf{R}$) using QR decomposition. The strain energy density is defined as

$$\hat{\psi}(\mathbf{R} | E, \nu, \kappa, \gamma) = \hat{\psi}_{\text{normal}}(\mathbf{R}_{33} | \kappa) + \hat{\psi}_{\text{shear}}(\mathbf{R}_{13}, \mathbf{R}_{23} | \gamma) + \hat{\psi}_{\text{in-plane}}(\mathbf{R}_{11}, \mathbf{R}_{12}, \mathbf{R}_{22} | E, \nu)$$

where energy is partitioned into normal, shear, and in-plane components, each governed by tunable parameters ($\kappa, \gamma, E, \nu$).

• Advanced Collision Handling: Avatars’ bodies, modeled as mesh colliders (SMPL-X or similar), are embedded into the MPM grid using B-spline weights. For each grid node $i$, the velocity and surface normal from nearby mesh faces are transferred, and collision resolution projects out velocity components normal to the collider:
  • For each affected node, update $v_i$ according to

$$\text{if } (v_i - v_i^c) \cdot n_i^c < 0,\quad v_i \leftarrow (v_i - v_i^c) - [(v_i - v_i^c) \cdot n_i^c] n_i^c + v_i^c$$

where $v_i^c$ and $n_i^c$ are the collider velocity and normal at node $i$.

• Mesh-to-Grid Coupling: Computational complexity is proportional to the number of body mesh faces, not the total grid size, yielding improved efficiency and robustness even when body mesh contains self-intersections.

This approach yields physically plausible cloth deformations, including intricate wrinkles and folds, robustly across diverse animation sequences.

3. High-Fidelity Gaussian Splatting for Appearance

Surface appearance is synthesized using 3D Gaussian splatting:

• Per-Triangle Gaussian Splat Attachment: Each mesh triangle is assigned multiple Gaussians in its local frame, each with optimized parameters (center, scale, rotation, color harmonics, opacity).
• Quasi-Shadowing Neural Network: A neural module predicts a per-Gaussian shading scalar, modulating the Gaussian’s color according to local geometry and ambient occlusion.
• Rendering Pipeline: During animation, Gaussian splats are reprojected according to simulated deformations (body via LBS, garment via MPM), composited for photorealistic rendering from arbitrary viewpoints.

Such representations enable sharp textures and seamless mapping of fine visual detail, reflecting input video data with high photometric accuracy.

4. Quantitative Evaluation and Benchmarks

Performance is assessed versus state-of-the-art physics-based avatar methods, with significant advances in three key metrics:

• Dynamics Modeling Accuracy: On geometric consistency measures (e.g., Chamfer Distance, F-Score), MPMAvatar achieves lower error (CD $\approx .42 \cdot 10^3$) and higher F-Score for garment regions, reflecting closer fidelity to real cloth behavior.
• Rendering Accuracy: On perceptual metrics (e.g., LPIPS, PSNR, SSIM), outputs are demonstrably sharper and more structurally consistent with source multi-view data.
• Simulation Robustness and Efficiency: The simulator achieves a $100\%$ success rate (without manual mesh adjustment) and average per-frame simulation time of $1.1 s$, far outperforming the $170 s$ per-frame runtime of competing methods. Rendering across free viewpoints is supported at photorealistic quality.

These results are representative across test sets that capture a range of garments and interactive pose dynamics.

5. Generalization and Zero-Shot Physical Interaction

Unlike learning-only deformation models, MPMAvatar supports zero-shot generalization to unseen scene interactions:

• Physics-Driven Generalizability: Because simulated garment dynamics are parameterized by physics, avatars respond plausibly to novel physical scenarios (e.g., interaction with a chair or sand) without additional training.
• Robustness to Unseen Inputs: The model is not limited to trajectories or interactions observed during multi-view video capture, overcoming the limitations of prior networks trained on fixed motion templates.
• Efficient Parameter Estimation: Material parameters are fit via finite-difference optimization routines for each avatar, allowing adaptation to different garment types without manual tuning.

This principled simulation approach supports scalability to new domains—including scenes featuring external object contact or nonstandard garments.

6. Future Directions and Open Challenges

Areas identified for further development include:

• Relightable Avatars: At present, static Gaussian color attributes do not support dynamic relighting. Integration of relightable neural shading or dynamic Gaussian attribute fields could further augment realism.
• Extended Physics Simulation: Extension of physics-based animation to additional avatar components (e.g., hair, non-garment body regions) may yield seamless and fully dynamic avatar models.
• Scalability of Material Parameter Fitting: Current finite-difference optimization is tractable for low-dimensional parameter spaces; investigation into differentiable and higher-resolution simulation for richly detailed garments is ongoing.

These directions may further enhance the generality and quality of simulated avatars in future applications.

7. Applications and Broader Impact

The ability to synthesize physically realistic, photorealistic avatars from multi-view video in an efficient and robust manner has direct impact in multiple domains:

• Content Creation for Virtual Reality and Film: High-fidelity avatars that respond naturally to novel interactions open new possibilities in immersive media and digital doubles.
• Telepresence and Social VR: Robust dynamics modeling supports more convincing embodiment in remote communication environments.
• Research on Cloth Dynamics and Human Motion: Data-driven but physics-aware simulation bridges the gap between pure learning systems and classical simulation, encourging adoption in physical animation, biomechanics, and graphics.

MPMAvatar thus represents a comprehensive advance, integrating tailored physics simulation with efficient appearance synthesis to achieve avatars that are accurate, realistic, robust to new scenes, and practical for real-world deployment (Lee et al., 2 Oct 2025).