PIAvatar: Physically Interactive Avatars via Deformation Gradient Decoupling

Abstract: 3D human avatars have shown impressive visual fidelity driven by pose-conditioned models, yet they still lack the physical ability required for interactions with each other and environments. Although recent studies have made various attempts to incorporate physical characteristics into 3D avatars, they only exhibit limited physical deformations, often leading to constrained interaction behaviors. To resolve this issue, we present PIAvatar, a framework to simultaneously enable physically aware interactions between avatar-avatar and avatar-environment, and a non-rigid deformable human body simulation. In this work, our key insight is to decouple kinematic velocity from deformation gradient. When external forces act on avatars, the kinematic velocity induces stress which hinders the avatar's ability to achieve a desired pose. In addition, we integrate a skeletal framework within the avatar. It allows estimating its poses and real-time tracking in a closed form, even during non-rigid physical interactions. Our approach is implemented within a conventional Material Point Method framework to ensure physically consistent dynamics. We lastly evaluate the method on both human-object and human-human interaction scenarios to assess its behavior under diverse interaction settings.

• The paper presents a novel decoupling technique that separates user-driven kinematic velocities from deformation-induced stresses within MPM, enhancing physical interaction and pose tracking.
• It utilizes an embedded skeletal structure with closed-form pose extraction to maintain robust, real-time tracking during significant non-rigid deformations.
• Experimental results show order-of-magnitude improvements in pose accuracy and realistic avatar-object interactions, paving the way for physically consistent digital humans.

PIAvatar: Physically Interactive Avatars via Deformation Gradient Decoupling

Introduction

The modeling of physically plausible, non-rigid, and interactive human avatars has remained a central challenge in computer graphics and simulation. While prior work has achieved remarkable visual fidelity using pose-conditioned neural models and skeletal-driven animation, these approaches lack physical consistency necessary for realistic interactive behaviors. PIAvatar advances the field by presenting a framework that enables 3D avatars with bidirectional physical interactions and non-rigid deformation, addressing fundamental limitations of both learning-based models and previous simulation-driven methods. The core innovation is the decoupling of user-driven kinematic velocities from deformation-induced elastic stress within the Material Point Method (MPM), as well as the introduction of embedded skeletal structures for real-time pose tracking even under significant physical deformation.

Figure 1: PIAvatar supports physical 3D avatar interactions coupled with bidirectional and non-rigid deformation, driven solely by physically plausible simulation.

Background and Limitations of Prior Work

State-of-the-art avatar generation pipelines based on parametric models (e.g., SMPL-X [SMPLX_2019_Pavlakos]), Gaussians [li2024animatable], and implicit representations attain visually convincing dynamic reconstructions but remain limited to kinematic replay with no capacity for responsive physical interaction. Recent introductions of physics-aware avatars span two main lines: RL-based control (e.g., [tevet2024closd, xu2025intermimic]) and simulation-driven full or partial physical modeling (e.g., [zheng2024physavatar, lee2025mpmavatar, siyao2025half]). RL-based avatars achieve interactive behavior by policy learning but rely on simplified rigid-body models and cannot express non-rigid deformations, limiting application to stylized motion synthesis rather than physical realism.

Mesh-based FEM, C-IPC, and MPM formulations support non-rigidity but present other challenges: one-way interaction (avatar affects environment but is not affected by it), loss of articulation control under large deformation, and stress-based resistance that impedes target pose tracking. Existing two-way coupled frameworks are rare and typically require computationally expensive model-fitting or optimization for pose retrieval post-deformation.

PIAvatar Formulation

Decoupling Kinematic Velocity from Deformation Gradient

In standard MPM, each particle state is evolved by interpolating user-specified velocities onto a background grid, computing stress from deformation gradients, and transferring back to particles. For articulated avatars, user-imposed kinematic velocities intended to drive animation are inadvertently absorbed into total deformation gradients, inducing artificial internal stresses. This competes with intended motion and leads to persistent deviation from target poses, as visualized and analyzed in the supplementary material.

PIAvatar disentangles user-driven kinematic motion from physically induced deformation by explicit computation of the kinematic velocity field, which is excluded from stress accumulation. This is achieved by:

• Transferring kinematic velocities to a separate grid and updating a kinematic deformation gradient $\boldsymbol{F}_p^k$ via standard MPM operator splitting.
• After updating total deformation gradient $\boldsymbol{F}_p$, the kinematic component is factored out, and only the physically induced part is retained for stress calculation. This simple yet effective operation eliminates artificial stress accumulation from user input.
• Stresses are then computed strictly from external contacts or physically meaningful sources, preserving bidirectional interaction.

Figure 2: Motivation and comparative results for deformation gradient disentangling—basic MPM fails to reach target pose (a); removing all stress negates contact response (b); PIAvatar supports both pose control and physical interaction (c).

Skeleton-Based Pose Extraction and Real-Time Kinematic Control

PIAvatar embeds a discretized skeletal structure within the avatar mesh, using joint group particles and closed-form rigid registration (Kabsch algorithm [kabsch1976solution]) to solve for per-bone transformations directly from simulated particle states. This architecture:

• Maintains pose coherence during non-rigid deformation and contact, permitting both forward animation and robust inverse pose estimation.
• Facilitates efficient calculation of incremental per-particle kinematic velocities via LBS between canonical and posed skeletons, supporting stable, real-time animation even under dynamic external forces.
• All computations are matrix-based and solved in closed form without iterative or learning-based optimization, ensuring computational efficiency compatible with real-time frameworks.

Figure 3: Overview of the PIAvatar framework—kinematic velocity decoupling and closed-form skeletal pose tracking.

Quantitative and Qualitative Evaluation

Benchmark and Ablation

Evaluated on both Animatable Gaussians [li2024animatable] and SMPL-X body models, PIAvatar achieves an order-of-magnitude improvement over baseline MPM in terms of MSE/RMSE to ground-truth pose trajectories and pose tracking (root/joint rotation and translation errors below floating-point precision for most frames). Accuracy at 0.01m threshold remains $\geq 53\%$ after 400 frames of simulation, while standard MPM rapidly degrades to $\leq 4\%$.

Physical Interaction and Non-rigid Deformation

PIAvatar demonstrates physically consistent, two-way interactions in both avatar-object and avatar-avatar scenarios. Momentum and force transduction cause both pose deviation and visible surface deformation, which are physically interpretable and materially dependent.

Figure 4: Bidirectional non-rigid avatar-object and avatar-avatar interactions; pose and deformation reflect underlying material and interaction strength.

PIAvatar natively supports contact-propagated soft-tissue deformation (e.g., belly jiggling, hair flutter), which emerges without explicit sub-modeling, and respects regional material heterogeneity:

Figure 5: Natural soft-tissue deformation (e.g., belly wobble) driven by physical simulation, absent in conventional linear skinning.

Figure 6: Response of avatars with heterogeneous material parameters for body, hair, and garments—deformation magnitude and reaction are physically grounded.

Material and Scenario Generalization

Simulations indicate that changing Young's modulus parameterizes object and avatar stiffness; high $E$ approximates rigid-body responses while low $E$ yields compliant, squishy dynamics, faithfully reproducing soft body behavior.

Figure 7: Varying Young's modulus modulates rigidity and deformation across physical avatars.

In multi-object and variable-mass object interactions, the avatar's and objects' motion and deformation evolve consistently with physical intuition, validating the fidelity of the MPM backbone and PIAvatar’s decoupling scheme.

Figure 8: Multi-object bidirectional physical interactions supported.

Practical and Theoretical Implications

PIAvatar unifies kinematic articulation and non-rigid, physically consistent interaction, supporting applications in physically grounded AR/VR avatars, robotics, digital humans for entertainment, and embodied AI requiring veridical simulation surfaces. The framework decouples animation intent from system physics, a crucial requirement for scalable, generalizable digital actors or agents interacting in complex simulated worlds.

Theoretical significance includes a principled, computationally efficient mechanism for avoiding internal model stress from animation directives in continuum simulation, while preserving real-time closed-form inversion for articulated pose recovery. This architecture bypasses the need for computationally intractable learning or per-frame optimization for physically interactive avatars.

Limitations and Future Directions

While effective for most realistic interaction scenarios, the current approach confronts self-penetration in high-deformation regimes (armpits, garment folding) and challenges with frictionless or rapid contacts due to grid-based MPM limitations. More advanced contact and collision handling, friction modeling, and torque control would further harden the system. Joint learning-based scene understanding and physics parameter estimation, integrated with this decoupled simulation pipeline, remain open for further exploration and are critical for advancing autonomous digital humans.

Conclusion

PIAvatar advances physically plausible, non-rigid, bidirectionally interactive human avatars, leveraging kinematic deformation decoupling and embedded skeletal extraction within the MPM framework. The approach addresses key failure cases of prior simulation and learning-based methods, yielding extensive empirical gains in tracking precision and physical realism, and provides a foundation for next-generation physical digital humans in simulation, AR/VR, and embodied intelligence contexts.