MonoMSK: Monocular Kinematics & Kinetics

• MonoMSK is a framework that estimates 3D human kinematics and kinetics from monocular videos using a combination of deep learning and physically grounded simulation.
• It integrates transformer-based inverse dynamics prediction with differentiable forward simulation to enforce biomechanical causality and physical plausibility.
• The system achieves lab-grade fidelity with significant error reductions over SMPL-based methods, validated on benchmark datasets like BML-MoVi and OpenCap.

MonoMSK is a framework for estimating biomechanically realistic three-dimensional human kinematics and kinetics from monocular RGB videos. It combines data-driven deep learning with physically grounded simulation to jointly infer joint trajectories and the underlying musculoskeletal dynamics, including joint torques and ground reaction forces, producing reconstructions with anatomical accuracy and physical plausibility. Addressing critical limitations of previous monocular and SMPL-based approaches, MonoMSK is the first end-to-end system to deliver both kinematic and kinetic estimation with lab-grade fidelity using only a single video input (Koleini et al., 24 Nov 2025).

1. Motivation and Conceptual Foundations

Accurately reconstructing both kinematics (joint motions) and kinetics (forces/torques driving motion) from monocular video is a longstanding and fundamental challenge. Marker-based multiview systems coupled with musculoskeletal solvers (e.g., OpenSim) excel in biomechanical accuracy but require extensive hardware and processing. Deep learning approaches using parametric models like SMPL provide scalable visual reconstructions but adopt anatomically oversimplified skeletons and neglect the laws of physics, leading to biomechanically implausible outputs and an inability to recover kinetics. MonoMSK addresses these shortcomings by integrating learned inverse dynamics estimation with differentiable forward simulation governed by Newton–Euler and muscle dynamics, thus enforcing biomechanical causality and enabling physically valid kinetic inference (Koleini et al., 24 Nov 2025).

2. Musculoskeletal and Contact Modeling

MonoMSK is built upon a full-body musculoskeletal (MSK) model derived from the OpenSim human template. This model comprises $N_s = 24$ articulated rigid bone segments, each joint $i$ defined by a native anatomical orientation $q_i^o \in \mathbb{R}^3$, rotation coordinates $q_i^r \in \mathbb{R}^{D_i}$ (with $D_i \leq 3$), and per-segment bone scaling $s_i \in \mathbb{R}^3$. The generalized coordinate vector $q = \{T, R, q^r\}$ fully parameterizes the organism. Joint torques are given by both muscle-induced torques and idealized torque motors: $$\tau_i = \sum_{j=1}^{N_m} r_{ij} F_j(a_j, \ell_j, v_j) + \tau_i^{tm}$$ where $F_j$ follows a Hill-type muscle model. Contact with the ground is captured by six compliant spheres per foot, with interaction forces per the Hunt–Crossley normal force and Coulomb friction: $$F_{n,k} = k_n \delta_k^{1.5}(1 + c_n \dot{\delta}_k) \mathbf{n};\quad F_{t,k} = -\mu \|F_{n,k}\| \frac{v_{t,k}}{\|v_{t,k}\|+\epsilon}$$ The total ground reaction force $i$0 acts as an external input in the system’s dynamics (Koleini et al., 24 Nov 2025).

3. Physics-Regulated Inverse–Forward Loop

The core of MonoMSK is a closed loop coupling transformer-based inverse dynamics prediction with differentiable forward dynamics simulation. The Inverse Dynamics Transformer (IDT) consumes generalized coordinate and velocity sequences, encoding them with an MLP, positional encoding, and a transformer encoder: $i$1 where $i$2 are the contextualized embeddings. These predicted kinetics are propagated through a fully differentiable ODE-based forward dynamics simulator: $i$3 Simulation is carried out using explicit ODE integration (e.g., RK4). The resulting simulated trajectories are mapped to 3D landmarks using a differentiable anatomical forward kinematics (FK) layer. MonoMSK imposes a novel forward–inverse consistency loss: $i$4 to enforce that simulated dynamics and recreated kinematics remain aligned with ground truth, ensuring biomechanical plausibility (Koleini et al., 24 Nov 2025).

4. Training Protocols and Supervision

MonoMSK is trained on multi-activity datasets with both kinematic and kinetic supervision, most notably BML-MoVi (90 subjects, 21 actions), BEDLAM (271 synthetic subjects), and OpenCap. An Inverse Kinematics Transformer (IKT) estimates pose from initial mesh extractions by pretrained HMR networks (e.g., HMR2.0, TokenHMR, CameraHMR, MQ-HMR). Ground-truth kinetics are generated by solving an optimal control problem in OpenSim-Moco, which minimizes a weighted sum of muscle excitations and trajectory tracking terms subject to the full musculoskeletal system dynamics. The total loss incorporates kinetic supervision (kinetic error in both torques and ground reaction forces), forward–inverse consistency, kinematics reconstruction, and optional marker matching. Weights are $i$5, $i$6, $i$7, $i$8, with 20 epochs of supervised training followed by 5 epochs of in-loop finetuning and Adam optimization (Koleini et al., 24 Nov 2025).

5. Quantitative Evaluation and Comparative Performance

MonoMSK’s performance was evaluated using standard benchmarks:

Dataset	MAE_angle (°)	MPBLPE (mm)	MAE_λ (norm.)	MAE_τ
BML-MoVi	1.93	24.36	0.0139	0.0498
BEDLAM	[see paper]	[see paper]	[see paper]	[see paper]
OpenCap	2.84	[see paper]	[see paper]	[see paper]

MonoMSK reduced mean joint angle error (MAE_angle) on BML-MoVi by 32% (from 2.84° to 1.93°) and 11% on OpenCap over the best prior BioPose model, while yielding a 5.4% reduction in mean per-bony-landmark position error (MPBLPE). Physical plausibility metrics—acceleration and velocity errors—dropped by 30.5% and 38.8%, indicating dynamically consistent, non-jerky output. For kinetic estimation, MonoMSK is the first monocular system to report ground reaction force and joint torque errors: MAE_λ=0.0139, MAE_τ=0.0498 (normalized), surpassing alternative HMR backbones and ablation variants (Koleini et al., 24 Nov 2025).

6. Ablations, Limitations, and Prospective Directions

Ablation studies showed that accuracy improves consistently with more advanced HMR backbones, and that including kinetic and consistency loss terms drives both kinematic and kinetic error lower. A single-frame-out autoregressive temporal modeling strategy aligns best with the ODE solver supervision.

Limitations include dependence on the initial mesh/marker extraction step—the overall accuracy is bottlenecked by HMR performance. The ODE-based dynamics solver can be computationally costly and less robust to complex contact scenarios or occluding garments. Only net joint torques are predicted, with no explicit modeling of muscle excitations or fatigue. Acquiring ground-truth kinetics via optimal control is itself laborious and data-intensive. A plausible implication is that improvements in markerless pose estimation or surrogate ODE solvers could further reduce computational demands and enhance robustness. Potential future work includes direct muscle activation estimation for physiological study, adaptive contact and damping models, and self-supervised finetuning on unlabelled video (Koleini et al., 24 Nov 2025).

7. Significance and Research Context

MonoMSK demonstrates the feasibility and practical utility of combining data-driven transformers with a differentiable, anatomically grounded simulator to reconstruct full-body kinematics and kinetics from monocular video. By enforcing biomechanical causality through its physics-regulated inverse–forward loop, MonoMSK produces motion trajectories that are both accurate relative to marker-based systems and physically plausible—addressing a crucial gap for downstream applications in biomechanics, rehabilitation, sports analytics, and human–robot interaction. As the first end-to-end monocular method with validated kinetic prediction, it establishes a pathway towards lab-quality musculoskeletal analysis in unconstrained environments (Koleini et al., 24 Nov 2025).