MPMAvatar: Learning 3D Gaussian Avatars with Accurate and Robust Physics-Based Dynamics (2510.01619v1)

Abstract: While there has been significant progress in the field of 3D avatar creation from visual observations, modeling physically plausible dynamics of humans with loose garments remains a challenging problem. Although a few existing works address this problem by leveraging physical simulation, they suffer from limited accuracy or robustness to novel animation inputs. In this work, we present MPMAvatar, a framework for creating 3D human avatars from multi-view videos that supports highly realistic, robust animation, as well as photorealistic rendering from free viewpoints. For accurate and robust dynamics modeling, our key idea is to use a Material Point Method-based simulator, which we carefully tailor to model garments with complex deformations and contact with the underlying body by incorporating an anisotropic constitutive model and a novel collision handling algorithm. We combine this dynamics modeling scheme with our canonical avatar that can be rendered using 3D Gaussian Splatting with quasi-shadowing, enabling high-fidelity rendering for physically realistic animations. In our experiments, we demonstrate that MPMAvatar significantly outperforms the existing state-of-the-art physics-based avatar in terms of (1) dynamics modeling accuracy, (2) rendering accuracy, and (3) robustness and efficiency. Additionally, we present a novel application in which our avatar generalizes to unseen interactions in a zero-shot manner-which was not achievable with previous learning-based methods due to their limited simulation generalizability. Our project page is at: https://KAISTChangmin.github.io/MPMAvatar/

• The paper introduces a hybrid representation combining canonical meshes with 3D Gaussian splatting to accurately reconstruct dynamic avatars.
• The paper presents a tailored Material Point Method simulator with an anisotropic constitutive model that yields robust, realistic garment dynamics.
• The paper demonstrates superior efficiency and simulation accuracy over state-of-the-art methods through rigorous quantitative and qualitative evaluations.

MPMAvatar: Learning 3D Gaussian Avatars with Accurate and Robust Physics-Based Dynamics

Introduction

MPMAvatar introduces a framework for reconstructing 3D human avatars from multi-view video, integrating physically accurate garment dynamics and high-fidelity appearance modeling. The method addresses the limitations of prior approaches in animating loose garments, which often rely on piecewise linear transformations or pose-dependent correctives and thus fail to generalize to unseen motions or complex deformations. MPMAvatar leverages a tailored Material Point Method (MPM) simulator, incorporating an anisotropic constitutive model and a mesh-based collision handling algorithm, to robustly simulate garment-body interactions. The avatar is rendered using 3D Gaussian Splatting with quasi-shadowing, enabling photorealistic free-viewpoint rendering.

Hybrid Avatar Representation

MPMAvatar employs a hybrid representation, combining a canonical mesh with physical parameters for geometry and dynamics, and 3D Gaussian Splats for appearance modeling. The mesh encodes the avatar’s geometry and supports physics-based simulation, while the Gaussian Splats, attached to mesh faces and parameterized in local coordinates, enable high-fidelity rendering and natural deformation under animation.

Figure 1: Overview of dynamic avatar modeling, showing the hybrid mesh and Gaussian Splat representation, and the simulation pipeline for garment-body interaction.

Physics-Based Garment Dynamics

Material Point Method (MPM) Foundation

The MPM simulator models continuum materials using Lagrangian particles and an Eulerian grid, solving conservation of mass and momentum. The key to realistic garment simulation is the computation of the Cauchy stress tensor, which depends on the deformation gradient and a strain-energy density function $\psi$.

Anisotropic Constitutive Model

Garments exhibit codimensional structure and direction-dependent stiffness. MPMAvatar adopts the anisotropic elastoplasticity model of Jiang et al., where the strain energy is decomposed into normal, shear, and in-plane components, each penalizing different deformation modes. Material directions are tracked via the mesh, and QR decomposition ensures rotational invariance. This enables accurate modeling of garment stretching, compression, and shearing, critical for realistic cloth behavior.

Mesh-Based Collision Handling

Standard MPM collision handling assumes analytic colliders (e.g., spheres), unsuitable for mesh-based body models. MPMAvatar introduces a mesh-to-grid transfer using B-spline weights, propagating collider velocities and normals to grid nodes. Relative velocities at colliding grid nodes are projected onto the tangent space of the collider, preventing penetration and supporting efficient, robust simulation even with noisy or self-intersecting body meshes.

Learning from Multi-View Video

Mesh Tracking

Dense temporal correspondences are established via mesh tracking, using photometric and geometric losses. The canonical mesh is reconstructed from static scene methods, and subsequent frames are tracked for supervision.

Physical Parameter Estimation

Physical parameters (Young’s modulus $E$, density $\rho$, and rest geometry parameter $\alpha$) are optimized via inverse physics, minimizing vertex-wise L2 loss between simulated and tracked meshes. The rest geometry parameter $\alpha$ compensates for gravity-induced deformation in real-world captured meshes, enabling accurate simulation from non-ideal initial states.

Appearance Optimization

Gaussian Splat parameters are optimized jointly across all frames and views, leveraging tracked mesh deformations for spatial alignment. Photometric losses (L1, SSIM, LPIPS) and regularization terms ensure fidelity and stability. Quasi-shadowing is implemented via a neural network predicting ambient occlusion-based shading scalars, enhancing realism.

Experimental Results

Quantitative and Qualitative Comparisons

MPMAvatar is evaluated on ActorsHQ and 4D-DRESS datasets, outperforming state-of-the-art baselines (PhysAvatar, ARAH, TAVA, GS-Avatar) in both geometry (Chamfer Distance, F-Score) and appearance (LPIPS, PSNR, SSIM) metrics. The method achieves sharper textures, finer detail, and more accurate garment folds and wrinkles.

Figure 2: Qualitative comparisons showing superior geometry and appearance fidelity of MPMAvatar over PhysAvatar and other baselines.

Figure 3: Comparison against concurrent baselines on ActorsHQ, highlighting improved garment dynamics and rendering quality.

Robustness and Efficiency

MPMAvatar demonstrates a 100% simulation success rate and an average per-frame simulation time of 1.1 seconds, compared to 37.6% and 170 seconds for PhysAvatar. The feed-forward MPM pipeline is significantly more robust and efficient than iterative solvers (e.g., C-IPC).

Ablation Studies

Ablations confirm the necessity of the anisotropic constitutive model, physical parameter learning, and quasi-shadowing. Removing any component degrades dynamics accuracy or rendering realism, with non-anisotropic models exhibiting tearing and fixed parameters yielding suboptimal cloth deformations.

Zero-Shot Scene Interaction

The physics-based simulation generalizes to novel scene interactions, including contact with external objects (e.g., chairs, sand) unseen during training. This zero-shot capability is unattainable with learning-based simulators, which overfit to training dynamics and lack explicit physical priors.

Implementation Considerations

• Computational Requirements: The pipeline is GPU-accelerated (e.g., RTX 4090), with simulation and rendering times suitable for practical deployment.
• Scalability: Finite-difference optimization is tractable for per-garment parameters; differentiable simulation may be required for finer granularity.
• Limitations: The framework does not support relighting; non-garment regions are animated via linear blend skinning, which could be improved with strand-based simulation for hair.
• Generalization: Appearance optimization is limited to visible regions; generative priors or diffusion-based view synthesis could enhance occlusion-aware generalization.

Implications and Future Directions

MPMAvatar establishes a robust paradigm for dynamic avatar reconstruction, integrating explicit physics simulation with high-fidelity appearance modeling. The approach is extensible to relightable avatars, generative occlusion completion, and differentiable simulation for fine-grained material parameter estimation. The demonstrated zero-shot generalization to novel interactions suggests applicability in virtual reality, digital fashion, and content creation, with potential for further integration of physical commonsense priors and advanced rendering techniques.

Conclusion

MPMAvatar advances the state of the art in dynamic avatar reconstruction, achieving superior garment dynamics, rendering fidelity, and simulation robustness. The hybrid mesh-Gaussian representation, tailored MPM simulation, and mesh-based collision handling collectively enable physically plausible, photorealistic avatars from multi-view video. Future work may address relighting, fine-grained parameterization, and occlusion-aware appearance modeling, further broadening the applicability of physics-based avatars in real-world scenarios.