PhysDiff: Physics-Guided Human Motion Diffusion Model (2212.02500v3)

Abstract: Denoising diffusion models hold great promise for generating diverse and realistic human motions. However, existing motion diffusion models largely disregard the laws of physics in the diffusion process and often generate physically-implausible motions with pronounced artifacts such as floating, foot sliding, and ground penetration. This seriously impacts the quality of generated motions and limits their real-world application. To address this issue, we present a novel physics-guided motion diffusion model (PhysDiff), which incorporates physical constraints into the diffusion process. Specifically, we propose a physics-based motion projection module that uses motion imitation in a physics simulator to project the denoised motion of a diffusion step to a physically-plausible motion. The projected motion is further used in the next diffusion step to guide the denoising diffusion process. Intuitively, the use of physics in our model iteratively pulls the motion toward a physically-plausible space, which cannot be achieved by simple post-processing. Experiments on large-scale human motion datasets show that our approach achieves state-of-the-art motion quality and improves physical plausibility drastically (>78% for all datasets).

• The paper introduces PhysDiff, a physics-guided motion diffusion model that embeds physical constraints into the diffusion process to eliminate common motion artifacts.
• It utilizes a physics-based motion projection module that employs a motion imitation policy in a simulator to ensure generated motions adhere to physical laws.
• Experiments on text-to-motion and action-to-motion tasks show significant improvements, reducing physical errors by up to 94% and enhancing motion quality.

The paper introduces PhysDiff, a physics-guided motion diffusion model designed to generate human motions that adhere to the laws of physics, addressing the common issue of physically implausible motions generated by existing motion diffusion models. These models often produce motions with artifacts like floating, foot sliding, and ground penetration, which limits their applicability in real-world scenarios. PhysDiff incorporates physical constraints into the diffusion process through a physics-based motion projection module. This module projects the denoised motion of a diffusion step onto a physically plausible space using motion imitation within a physics simulator. The projected motion then guides subsequent denoising steps, iteratively refining the motion towards physical realism.

The core idea is to embed physics constraints directly into the diffusion process, rather than applying them as a post-processing step. The authors argue that post-processing can be ineffective because the final denoised kinematic motion may deviate too significantly from physical plausibility to be corrected adequately. By iteratively applying physics and diffusion, PhysDiff maintains proximity to the data distribution while converging towards physically plausible motions.

The physics-based motion projection module at the heart of PhysDiff enforces physical constraints through motion imitation in a physics simulator. A motion imitation policy, trained on large-scale motion capture data, controls a character agent within the simulator to mimic a wide range of input motions. This process ensures that the resulting simulated motion adheres to physical laws, eliminating artifacts like floating, foot sliding, and ground penetration.

The authors evaluate PhysDiff on text-to-motion generation and action-to-motion generation tasks. The model's denoiser can be any motion-denoising network; the authors test two state-of-the-art motion diffusion models, MDM (Motion Diffusion Model) and MotionDiffuse, as denoisers within PhysDiff. On the HumanML3D benchmark for text-to-motion generation, PhysDiff demonstrates significant improvements over existing motion diffusion models, reducing physical errors by over 86% while enhancing motion quality by more than 20%, as measured by Fréchet Inception Distance (FID). For action-to-motion generation, PhysDiff achieves substantial reductions in physical error metrics on the HumanAct12 (78% improvement) and UESTC (94% improvement) datasets, along with competitive FID scores.

Further experiments explore various schedules for the physics-based projection, revealing a trade-off between physical plausibility and motion quality as the number of projection steps varies. While increasing projection steps consistently improves physical plausibility, motion quality initially improves but then declines beyond a certain number of steps. This observation suggests a need to balance the number of physics-based projection steps to achieve both high physical plausibility and motion quality. The paper also finds that incorporating the physics-based projection in later diffusion steps yields better performance than applying it in earlier steps. The authors hypothesize that motions from early diffusion steps tend towards the mean motion of the training data, and physics-based projection could inadvertently push the motion away from the data distribution.

The contributions of this work include:

• The introduction of PhysDiff, a physics-guided motion diffusion model that integrates the laws of physics into the diffusion process for generating physically plausible motions. The model is designed with a plug-and-play architecture, enabling it to be used with different kinematic diffusion models.
• The use of human motion imitation in a physics simulator as a motion projection module to enforce physical constraints.
• The demonstration of state-of-the-art performance in motion quality and a significant improvement in physical plausibility on large-scale motion datasets. The analysis provides insights into the schedules and trade-offs involved and demonstrates improvements over physics-based post-processing techniques.

The paper details related work in several key areas:

• Denoising Diffusion Models: The authors cite a range of works on score-based denoising diffusion models and their applications in image generation, text-to-speech synthesis, 3D shape generation, machine learning security, and human motion generation. They also mention techniques for conditional generation, such as classifier-free guidance, and methods for solving linear inverse problems by injecting known information into the diffusion process.
• Human Motion Generation: The paper reviews early work on deterministic human motion modeling and more recent work using deep generative models like GANs and VAEs to generate motions from various conditions, including past motions, key frames, music, text, and action labels. It also discusses the emergence of motion diffusion models and their state-of-the-art motion generation performance.
• Physics-Based Human Motion Modeling: The authors discuss the application of physics-based human motion imitation to learning locomotion skills with deep reinforcement learning (RL). They also mention the use of RL-based motion imitation for user-controllable character animation and physics-based trajectory optimization and motion imitation for 3D human pose estimation.

The PhysDiff method leverages a physics-guided motion diffusion process, incorporating a physics-based motion projection $\mathcal{P}_\pi$ to map motions to a physically plausible space. This projection uses a motion imitation policy $\pi$, trained to control a simulated character to mimic denoised motions $x^{1:H}$ within a physics simulator.

The motion diffusion process begins with a data distribution $p_0(x)$, and defines time-dependent distributions $p_t(x_t)$ through the injection of Gaussian noise. The sampling process involves solving the stochastic differential equation (SDE):

$\mathrm{d} x = - (\beta_t + \dot{\sigma}_t) \sigma_t \nabla_{x \log p_t(x) \mathrm{d} t + \sqrt{2 \beta_t} \sigma_t \mathrm{d} \omega_t$

where:

• $x$ is the motion sample.
• $\beta_t$ controls the amount of stochastic noise injected in the process.
• $\sigma_t$ defines a series of noise levels that increase over time.
• $\nabla_{x} \log p_t(x)$ is the score function.
• $\omega_t$ is the standard Wiener process.

The score function $\nabla_{x_t} \log p_t(x_t)$ recovers the minimum mean squared error (MMSE) estimator of $x$ given $x_t$:

$$x := \mathbb{E}[x | x_t] = x_t + \sigma_t^2 \nabla_{x_t} \log p_t(x_t)$$

where:

• $x$ is a denoised version of $x_t$.

The denoising autoencoder objective approximates the score function:

$$\mathbb{E}_{x \sim p_0(x), t \sim p(t), \epsilon \sim p(\epsilon)}[\lambda(t) \Vert x - D(x + \sigma_t \epsilon, t, c) \Vert_2^2]$$

where:

• $D$ is the denoiser.
• $t$ is the time step.
• $c$ is the conditional information.
• $\epsilon$ is Gaussian noise.
• $\lambda(t)$ is the loss weighting factor.

The DDIM sampling algorithm is used to perform a one-step update from time $t$ to time $s$ ($s < t$):

$$\mu_s := x + \frac{\sqrt{\sigma_s^2 - v_s}}{\sigma_t} (x_t - x)$$

where:

• $\mu_s$ is the mean.
• $v_s$ is the variance.

To incorporate physical constraints, the physics-based motion projection $\mathcal{P}_\pi$ maps the motion $x^{1:H}$ to a physically-plausible motion $x^{1:H} = \mathcal{P}_\pi(x^{1:H})$.

The authors also discuss scheduling the physics-based projection, suggesting that it should not be performed when the diffusion noise level is high, as it can push the motion away from the data distribution.

The physics-based motion projection $\mathcal{P}_\pi$ is achieved by learning a motion imitation policy $\pi$ that controls a simulated character to mimic the denoised motion $x^{1:H}$ in a physics simulator. This is formulated as a Markov decision process (MDP) $$\mathcal{M} = (\mathcal{S}, \mathcal{A}, \mathcal{T}, R, \gamma)$$, where a character agent acts according to a policy $\pi(a^h|s^h)$.

The state $s^h$ consists of the character's physical state, the input motion's next pose $x^{h+1}$, and a character attribute vector $\psi$. The agent iteratively samples an action $a^h$ from the policy $\pi$, and the simulator generates the next state $s^{h+1}$, from which the simulated pose $x^{h+1}$ is extracted.

During training, a reward $r^h$ is assigned based on the alignment between the simulated motion $x^{1:H}$ and the ground-truth motion $x^{1:H}$. Reinforcement learning (RL) is used to learn the policy $\pi$, maximizing the expected discounted return $$J(\pi) = \mathbb{E}_{\pi}\left[\sum_{h}\gamma^h r^h\right]$$.

The reward function consists of four sub-rewards:

$$r^h = w_p r^h_p + w_v r^h_v + w_j r^h_j + w_q r^h_q$$

where:

• $r^h_p$ is the pose reward.
• $r^h_v$ is the velocity reward.
• $r^h_j$ is the joint position reward.
• $r^h_q$ is the joint rotation reward.
• $w_p, w_v, w_j, w_q$ are weighting factors.

The agent state $s^h$ includes the character's joint angles, joint velocities, rigid bodies' positions, rotations, and linear and angular velocities, as well as the difference of $x^{h+1}$ with respect to the agent. The character attribute $\psi$ includes the gender and SMPL (Skinned Multi-Person Linear Model) shape parameters.

The action representation uses target joint angles of proportional derivative (PD) controllers and residual forces. A Gaussian policy $$\pi(a^h|s^h) = \mathcal{N}(\mu_\theta(s^h), \Sigma)$$ is used, where the mean action $\mu_\theta$ is output by a multi-layer perceptron (MLP) network.

The experiments involved two standard human motion generation tasks: text-to-motion and action-to-motion generation. Evaluation metrics included FID, R-Precision, Accuracy, Penetrate, Float, Skate, and Phys-Err.

The authors compared PhysDiff against state-of-the-art methods on the HumanML3D, HumanAct12, and UESTC datasets. The results demonstrated that PhysDiff achieves state-of-the-art FID and reduces Phys-Err significantly. The authors also analyzed the schedule of the physics-based projection, varying the number and placement of projection steps. They compared different schedules, including uniform, start-end, and end-spaced schedules. The results indicated that it is better to schedule the physics-based projection steps consecutively towards the end of the diffusion process. Finally, PhysDiff was compared against a post-processing baseline, demonstrating that iterative application of diffusion and physics is more effective than post-processing.