Differentiable Physics-Driven Human Representation

• DIPR is a modeling paradigm that integrates fully differentiable physics engines with human representations to yield physically plausible estimations across varied data modalities.
• It employs articulated rigid body models, soft body FEM, and Gaussian splatting for mmWave sensing to enforce biomechanical constraints and contact validity via gradient-based optimization.
• This approach improves accuracy and motion stability in human reconstructions, significantly reducing foot-skate, jitter, and unphysical artifacts in RGB, radar, and animation datasets.

A Differentiable Physics-driven Human Representation (DIPR) is a modeling paradigm that tightly couples physically grounded human models with fully differentiable simulation or analytic losses, enabling direct integration with deep learning and optimization pipelines. DIPR replaces or augments traditional kinematic or proxy-based human representation with physically consistent, differentiable models and objectives, leading to physically plausible estimations and reconstructions in vision, graphics, robotics, and sensing. Core elements include physical constraint satisfaction, gradient-based optimization through physics layers, and utility across varied data modalities, including RGB images, mmWave radar, and artist-designed animation signals.

1. Mathematical Formulation and Model Structures

DIPR frameworks encode human geometry, articulation, and/or soft-tissue deformation using representations whose parameters are optimized with respect to differentiable physics-based energy functions or dynamical equations.

Articulated Rigid Body Models employ a reduced-coordinate system with state

$q \in \mathbb{R}^D$

for joint angles, velocities $\dot q$, and body shapes $\beta$ (e.g., GHUM or SMPL). Anatomical joint-limit constraints are enforced via hinge-loss penalties:

$$L_{\mathrm{jl}} = \frac1{TK}\sum_{t,k}\Bigl\|\max(z^k_{\mathrm{lower}} - q^k_t,0) + \max(q^k_t - z^k_{\mathrm{upper}},0)\Bigr\|^2$$

Dynamics are modeled via joint-space rigid-body equations with contact:

$$M(q)\ddot q + C(q,\dot q) + g(q) + J(q)^\top \lambda = \tau$$

where $M(q)$ is the inertia matrix, $C(q,\dot q)$ the Coriolis/centrifugal vector, $g(q)$ gravity, $J(q)$ the contact Jacobian, $\lambda$ contact impulses, and $\tau$ actuator torques.

Soft Body (FEM) Models use a quasi-static energy minimum for nodal positions $u \in \mathbb{R}^{3N}$, with an actuation field $A(x)$ defined by a coordinate-based network:

$\Psi(F, A) = \min_{R\in SO(3)} \|F - R A\|_F^2$

Total energy is summed over elements and quadrature points, yielding a differentiable solver where gradients propagate through the Newton or projective dynamics steps. The actuation field $A(x)$ is parameterized continuously by an MLP $f_\theta$.

Gaussian Splatting for mmWave Sensing (Zheng et al., 28 Dec 2025) employs a mixture

$$D_i(\mathbf x) = \beta_i\;\exp\Bigl(-\tfrac{1}{2}(\mathbf x-\mathbf p_i)^T\Sigma_i^{-1}(\mathbf x-\mathbf p_i)\Bigr)\cdot \exp(j\,\boldsymbol\phi_i^T\,\mathbf v_i\,(\mathbf x-\mathbf p_i))$$

with kinematic and EM parameters forming a differentiable model of radar propagation and body structure.

2. Differentiable Physics Layers and Simulation

DIPR architectures integrate differentiable physics engines or analytic functions directly into the computational graph, allowing gradient flow from output objectives through all physical processes.

For articulated skeletons, time integration is performed using semi-implicit Euler with contact handled by differentiable unrolled linear complementarity problems (LCPs) via projected Gauss–Seidel sweeps (Gärtner et al., 2022). Dynamics are unrolled for $T$ steps as a static computation graph upon which automatic differentiation operates for end-to-end optimization.

In soft body models, the energy minimization

$u^*(\theta) = \arg\min_u E(u, \theta)$

is solved via Newton or projective dynamics, with gradients through the solver given by the solution of a linear system involving the Hessian and mixed partials (adjoint-Jacobian trick) (Yang et al., 2024).

In radar-based DIPR (Zheng et al., 28 Dec 2025), the mmWave forward model is implemented as a differentiable composition of path-loss, chirp modulation, Doppler, and antenna-geometry modulations, all supporting automatic differentiation with respect to kinematic and EM parameters.

3. Objective Functions and Optimization Strategies

Physics-based DIPR leverages loss terms reflecting physical plausibility, measurement consistency, and regularization, in conjunction with data-based losses:

• Image or Sensor Losses: Root translation loss, per-joint orientation, 2D keypoint reprojection, and/or mmWave data reconstruction
• Biomechanical Validity: Enforce joint distance constraints, velocity coherence, and joint-limit penalties
• Contact and Ground Losses: Penalize foot skate, unphysical floor penetration, and encourage base-of-support stability (CoP–CoM alignment) (Tripathi et al., 2023)
• Regularization: Control torque magnitude, actuation smoothness, parameter norms

Typical overall losses take the form

$$L = w_r L_r + w_j L_j + w_i L_i + w_l L_{\mathrm{jl}}$$

or, for radar-based DIPR,

$$L_{\text{total}}(\theta) = \lambda_{\mathrm{kine}} L_{\mathrm{kine}} + \lambda_{\mathrm{recon}} L_{\mathrm{recon}} + \lambda_{\mathrm{reg}} L_{\mathrm{reg}}$$

Optimization is performed via gradient-based methods (Adam, BFGS), often with basin-hopping for global minima avoidance and windowed parallelization for long trajectories (Gärtner et al., 2022).

4. Extensions: Modality Integration and Specializations

DIPR is adapted to multiple data sources and application domains:

• Monocular Video: Physics-constrained 3D motion estimation from single-camera RGB using articulated DIPR (DiffPhy) and contact-aware control (Gärtner et al., 2022, Li et al., 2022).
• mmWave Radar: Physics-compliant Gaussian mixtures encode both pose and reflective/signal properties to boost SNR and anatomical plausibility in radar-based HPE (Zheng et al., 28 Dec 2025).
• Soft-tissue and Facial Animation: Quasi-static FEM models with implicit neural actuation accurately track muscle, jaw, and general soft-body effects with high spatial resolution and artist-driven controllability (Yang et al., 2024).
• Pressure and Support Modeling: Intuitive physics terms—pressure heatmaps, CoP, CoM—enforce static balance and plausible ground contact within SMPL-based regression and optimization (Tripathi et al., 2023).

5. Experimental Validation and Quantitative Assessment

DIPR methodologies have been evaluated across public MoCap, in-the-wild video, radar, and facial expression datasets, using both standard accuracy and physical plausibility metrics:

Paper & Modality	Key Metrics	Main Improvements
(Gärtner et al., 2022) RGB	MPJPE-G, jitter, foot-skate	Foot skate 50%→<20%, jitter ↓40%, MPJPE-G 145→139mm
(Li et al., 2022) RGB/dyn.cam	MPJPE, ACCEL, FS, GP	MPJPE 52.5mm (Human3.6M), FS 5.8mm, GP 1.5mm
(Zheng et al., 28 Dec 2025) mmWave	MPJPE, PA-MPJPE	MPJPE improved by 7–10mm across HPE baselines
(Yang et al., 2024) mesh deformations	per-vertex error	<5mm on body, 0.4mm on face, no blendshape artifacts
(Tripathi et al., 2023) RGB	MPJPE, stability, IoU	MPJPE improved by 1.6mm, stability ↑1.1%, topple ↓14.8%

Physically plausible DIPR consistently reduces implausible sliding, jitter, floor penetration, and improves stability measures compared to kinematic-only or non-differentiable baselines.

6. Limitations and Potential Extensions

Current DIPR frameworks encounter challenges including:

• Assumption of static camera or known scene geometry in some models (Gärtner et al., 2022)
• Iterative per-sequence or per-frame optimization may be slower than feedforward nets (Zheng et al., 28 Dec 2025)
• Single-subject focus, limited multi-person or object interaction modeling
• Limited material or reflection modeling in electromagnetic regimes

Potential directions include learned DIPR initializers for efficiency, extension to moving cameras and multi-person scenes, coupling with deep deformation priors, and integration of richer material and interaction models.

7. Significance and Outlook

DIPR provides a physically consistent, differentiable foundation for human representation, bridging the gap between advances in computational human modeling, machine learning, and sensor fusion. Its modality-agnostic architecture—supporting video, radar, and simulation data—enables universal integration into modern computer vision and graphics pipelines. The capacity for gradient-based optimization through all levels of physical reasoning yields improved anatomical plausibility, motion smoothness, contact accuracy, and signal-to-noise in downstream estimation and control tasks. DIPR is positioned as a unifying representation for future research at the intersection of embodied AI, multi-modal sensing, and differentiable simulation (Gärtner et al., 2022, Zheng et al., 28 Dec 2025, Yang et al., 2024, Li et al., 2022, Tripathi et al., 2023).