MS-MANO: Musculoskeletal Hand Model

• MS-MANO is a musculoskeletal extension of the MANO hand model that integrates muscle-tendon actuation using 31 hand muscles and 8 wrist muscles for realistic motion tracking.
• It employs a joint-centric remapping to align muscle attachment displacements with MANO joint locations, ensuring shape adaptivity and anatomical plausibility.
• BioPR refines MANO-based estimates via simulation-in-the-loop learning, yielding improved accuracy and smoother pose trajectories, as evidenced by MPJPE reductions on DexYCB.

Searching arXiv for the primary MS-MANO paper and closely related hand-model references. MS-MANO, short for Musculoskeletal MANO, is a musculoskeletal extension of the MANO parametric hand model for visual hand dynamics analysis and tracking. It is designed to address the unnatural, robot-like artifacts that arise when hand motion is modeled as direct joint actuation by instead introducing explicit muscle- and tendon-driven actuation, physiologically constrained torque trajectories, and simulation-in-the-loop learning. The framework has two central components: the MS-MANO musculoskeletal hand model and BioPR, a biomechanical pose refinement pipeline that can refine the output of any MANO-based estimator (Xie et al., 2024).

1. Definition and motivation

MS-MANO is motivated by a specific deficiency in conventional hand tracking systems: joint-actuated formulations generate torques directly at joints without modeling how human hand motion is actually produced by muscles and tendons. In the formulation presented in "MS-MANO: Enabling Hand Pose Tracking with Biomechanical Constraints" (Xie et al., 2024), this simplification is identified as a source of unnatural or infeasible motions, especially in visually ambiguous settings such as occlusion or motion blur.

The proposed remedy is not to replace MANO, but to augment it. MANO retains its role as a learnable parametric hand representation with pose and shape adaptivity, while musculoskeletal structure is added to impose physiological realism. The paper identifies three principal contributions: muscle-/tendon-driven dynamics over 31 hand muscles plus 8 wrist muscles, physiologically constrained torque trajectories via a Hill-type muscle model and routing, and BioPR, an MLP-based refinement framework that uses musculoskeletal simulation rather than only image-based temporal features (Xie et al., 2024).

This architecture places MS-MANO in a particular position within hand modeling research. It remains compatible with standard MANO-based prediction pipelines, yet seeks to constrain those pipelines with biomechanical actuation. A plausible implication is that the framework is intended as an intermediate layer between purely image-driven hand pose estimation and more anatomically grounded simulation systems.

2. Musculoskeletal augmentation of MANO

The core technical move in MS-MANO is the integration of MANO with muscle-tendon routing, anatomical origin and insertion points, and muscle-driven torque generation. Muscles are modeled as Hill-type actuators with a contractile element (CE), parallel elastic element (PEE), and serial elastic element (SEE). The anatomical routing is transferred from MyoSuite’s MyoHand, which itself is built atop OpenSim’s MoBL and 2nd-Hand anatomical models (Xie et al., 2024).

Each muscle segment originates from an origin point $n_{\text{origin}}$ and ends at an insertion point $n_{\text{insertion}}$. Torque production from a muscle segment at a joint is defined as

$$\bm\tau_\text{m} = f(F, x)\left\lVert(\bm q - \bm j)\times \frac{\bm s_c}{\lVert\bm s_c\rVert}\right\rVert,$$

where $F$ is the contractile force, $x$ is the muscle state, $\bm q$ is the attachment point on bone, $\bm j$ is the joint location, and $\bm s_c$ is the muscle segment direction (Xie et al., 2024). The paper treats $f(F,x)$ as an abstract mapping from muscle force and state to torque magnitude and does not provide an explicit analytic form for Hill force-length or force-velocity behavior.

A central representational problem is that MyoHand routing is bone-centric, whereas MANO is joint-centric and shape-adaptive. MS-MANO therefore introduces a joint-centric remapping. With MANO joints $\mathcal{M} = \{m_i\}_{i=1}^n$ and MyoHand bone subgroups $n_{\text{insertion}}$0, the mapping

$n_{\text{insertion}}$1

associates each bone subgroup with a MANO joint, and muscle attachment displacements are remapped accordingly so that muscle routing follows changes in MANO joint locations under shape variation (Xie et al., 2024). This remapping is essential to preserving MANO’s adaptivity while making muscle geometry anatomically meaningful.

The paper also notes an anatomical mismatch between MyoHand and MANO because they originate from different human sources. For that reason, minor manual revisions to insertion points, performed with anatomical expert input, are required to avoid tendon-skin intersections and maintain plausibility (Xie et al., 2024). This is not a peripheral detail: the paper later identifies insertion-point sensitivity as a major determinant of trajectory quality and learning stability.

3. Dynamics, torque constraints, and inverse dynamics learning

The forward dynamics of MS-MANO are implemented in a physics engine using RFUniverse + Unity/Kinesis, with muscle excitation signals driving the articulated system (Xie et al., 2024). The emphasis is not on a closed-form multibody derivation. The paper explicitly does not provide a full rigid-body dynamics equation, explicit activation dynamics, tendon compliance equations, pennation parameterizations, or maximum voluntary torque curves.

Instead, the formalism centers on inverse and forward mappings between kinematic states and muscle excitation. Given a pose sequence $n_{\text{insertion}}$2 in axis-angle form and angular velocity $n_{\text{insertion}}$3, inverse dynamics is defined as

$n_{\text{insertion}}$4

and forward dynamics as

$n_{\text{insertion}}$5

Here, $n_{\text{insertion}}$6 denotes the muscle excitation inferred for the transition from frame $n_{\text{insertion}}$7 to frame $n_{\text{insertion}}$8 (Xie et al., 2024).

The inverse mapping is learned by IDNet, which predicts muscle excitation despite the absence of ground-truth excitation labels. Supervision is therefore indirect. During training, IDNet is guided by torque matching against a PD controller with inverse dynamics compensation:

$n_{\text{insertion}}$9

with reward

$$\bm\tau_\text{m} = f(F, x)\left\lVert(\bm q - \bm j)\times \frac{\bm s_c}{\lVert\bm s_c\rVert}\right\rVert,$$0

The PD controller is used only during training; at test time, motion is driven by muscle actuators alone (Xie et al., 2024).

This training strategy makes the physiological constraint operational. The simulator enforces anatomical joint ranges and collision behavior through anatomically aligned colliders with small skin clearance, but the paper does not formalize these as explicit inequality constraints. The practical effect is that torque trajectories are shaped by musculotendon routing and simulation constraints rather than by post hoc kinematic regularization alone.

4. BioPR: simulation-in-the-loop pose refinement

BioPR is the second major component of the framework. Its role is to refine an initial pose estimate from any MANO-based estimator by using the musculoskeletal simulator as a source of biomechanically plausible reference motion (Xie et al., 2024).

The pipeline proceeds in three stages. First, a base MANO estimator outputs per-frame predicted pose $$\bm\tau_\text{m} = f(F, x)\left\lVert(\bm q - \bm j)\times \frac{\bm s_c}{\lVert\bm s_c\rVert}\right\rVert,$$1 and shape parameters; angular velocities $$\bm\tau_\text{m} = f(F, x)\left\lVert(\bm q - \bm j)\times \frac{\bm s_c}{\lVert\bm s_c\rVert}\right\rVert,$$2 are then computed by finite differences. For refinement, wrist-joint velocities are ignored, so the effective refinement velocity input is $$\bm\tau_\text{m} = f(F, x)\left\lVert(\bm q - \bm j)\times \frac{\bm s_c}{\lVert\bm s_c\rVert}\right\rVert,$$3 (Xie et al., 2024).

Second, for consecutive frame pairs $$\bm\tau_\text{m} = f(F, x)\left\lVert(\bm q - \bm j)\times \frac{\bm s_c}{\lVert\bm s_c\rVert}\right\rVert,$$4, IDNet infers muscle excitation $$\bm\tau_\text{m} = f(F, x)\left\lVert(\bm q - \bm j)\times \frac{\bm s_c}{\lVert\bm s_c\rVert}\right\rVert,$$5. Starting from $$\bm\tau_\text{m} = f(F, x)\left\lVert(\bm q - \bm j)\times \frac{\bm s_c}{\lVert\bm s_c\rVert}\right\rVert,$$6, the simulator applies forward dynamics under $$\bm\tau_\text{m} = f(F, x)\left\lVert(\bm q - \bm j)\times \frac{\bm s_c}{\lVert\bm s_c\rVert}\right\rVert,$$7 and produces a biomechanically valid reference state $$\bm\tau_\text{m} = f(F, x)\left\lVert(\bm q - \bm j)\times \frac{\bm s_c}{\lVert\bm s_c\rVert}\right\rVert,$$8 (Xie et al., 2024).

Third, a lightweight MLP combines the predicted and reference poses:

$$\bm\tau_\text{m} = f(F, x)\left\lVert(\bm q - \bm j)\times \frac{\bm s_c}{\lVert\bm s_c\rVert}\right\rVert,$$9

with supervised loss

$F$0

BioPR therefore does not directly replace the base estimator’s prediction with the simulator output. Instead, it uses the simulated pose as a biomechanical prior for learned correction (Xie et al., 2024). This distinction is significant: the framework is designed to preserve the image-conditioned estimate while biasing it toward physically and anatomically plausible configurations.

5. Architecture, training regime, and implementation details

The implementation uses a relatively small model footprint. IDNet has input size 16×3×4, output size 31 for the 31 hand-muscle excitations, and two 256-dimensional hidden layers. It is trained with PPO using Stable-Baselines3 on an NVIDIA A40 GPU, with learning rate $F$1 and adaptive scheduling. Training injects Gaussian noise $F$2 degrees into joint rotations, uses 128 distributed processes × 64 agents each, a batch size of 16384 per step, approximately 10K FPS simulation throughput, and takes roughly 1 hour (Xie et al., 2024).

The simulator uses RFUniverse (Unity/Kinesis) and includes 31 muscles for the hand and 8 muscles for wrist control. Muscle maximum isometric force and contractile element length are sourced from MyoHand/MyoSuite data. The implementation also includes forearm and wrist structure via SMPLX integration to capture muscles originating in the forearm (Xie et al., 2024).

The Refine Net is an MLP with input size 93—reported as 48 for $F$3 and 45 for wrist-ignored inputs—a single 64-dimensional hidden layer, and output size 48. It is trained for 4,500 iterations with batch size 10,240 and learning rate $F$4 on an A40 GPU, taking roughly 5 minutes (Xie et al., 2024). Parameter counts remain small: approximately 0.1M for IDNet and 0.01M for Refine Net, while BioPR adds roughly 9 ms inference overhead.

The paper also compares MS-MANO against heuristic priors. Temporal smoothing is formulated as

$F$5

According to the reported ablation, temporal smoothing and TOCH yielded <0.8 mm improvement in MPJPE on DexYCB when added to gSDF, whereas PCA increased MPJPE by 7.91 mm (Xie et al., 2024). In context, this is used to argue that biomechanical simulation contributes a stronger regularizing effect than handcrafted temporal heuristics.

6. Empirical evaluation, observed behavior, and scope

The evaluation is split into two parts. First, the musculoskeletal model itself is compared with MyoSuite by stimulating specific muscles and examining joint trajectories. For ring-finger bending under FDS4_R excitation, normalized time-position plots of three ring-finger joints show similar movement patterns between MS-MANO and MyoHand, while MS-MANO exhibits a quicker response to muscle stimulation (Xie et al., 2024). The paper interprets this as suggesting fast activation behavior consistent with human physiology, but does not provide explicit numeric metrics for this comparison.

Second, BioPR is benchmarked on DexYCB and OakInk using two recent MANO-based baselines, gSDF and Deformer. On DexYCB, the reported metrics are MPJPE (mm), AUC, and AE (mm/s$F$6) (Xie et al., 2024).

Method	DexYCB results
gSDF	MPJPE 14.4, AUC 89.1, AE 30.3
gSDF + BioPR	MPJPE 12.81, AUC 89.7, AE 29.9
Deformer	MPJPE 13.64, AUC 89.6, AE 31.7
Deformer + BioPR	MPJPE 12.92, AUC 90.4, AE 30.7

These results show consistent improvements in both accuracy and temporal smoothness for the two baselines (Xie et al., 2024). OakInk is used to assess robustness in dynamic interactions, but the provided text does not include explicit numerical results.

Qualitative analysis emphasizes reduction of failure modes that are familiar in image-based hand tracking: more anatomically aligned finger tightness when grasping a mustard bottle, avoidance of odd thumb twisting, and fewer implausible penetrations under severe occlusion (Xie et al., 2024). The stated interpretation is that biomechanical constraints stabilize predictions in visually degraded conditions.

The paper also delineates the framework’s limitations. It remains dependent on the base estimator, so errors induced by occlusion or lighting can persist; it is sensitive to incorrect annotations; and extremely slow motions are difficult because weak excitations generate small torques and therefore minimal frame-to-frame offsets (Xie et al., 2024). The system likewise omits explicit joint coupling formulas, activation equations, tendon compliance equations, and object-contact force modeling. Future work is suggested in differentiable simulation, personalized muscle parameters, explicit Hill-type parameterizations, and contact-aware interaction modeling.

7. Relation to adjacent work and terminological distinctions

Within hand modeling, MS-MANO is positioned against both joint-actuated MANO-only approaches and standalone musculoskeletal simulators. Relative to MANO-only systems, its defining feature is the integration of a comprehensive musculoskeletal system adapted from MyoSuite into MANO through joint-centric mapping, while preserving MANO’s learnability and shape adaptivity (Xie et al., 2024). Relative to MyoSuite’s MyoHand, its novelty lies in embedding musculoskeletal actuation into a parametric MANO framework and coupling it to a learning-based refinement pipeline, rather than operating solely as a simulator (Xie et al., 2024).

The paper further distinguishes itself from full-body musculoskeletal integrations such as BASH and RCare World by focusing on detailed hand muscles and forearm/wrist routing for hand pose tracking tasks (Xie et al., 2024). Practical application areas identified in the paper include AR/VR avatar hands, teleoperation, human-robot interaction, dexterous manipulation, and biomechanics, where physiological realism is linked to plausibility, safety, and control (Xie et al., 2024).

The name MS-MANO also requires terminological disambiguation. In the literature provided here, MeMaHand uses MANO in a different sense, namely a token-level interaction between dense mesh prediction and MANO parameter prediction for two-hand reconstruction from a single RGB image; that work does not define MS-MANO as a musculoskeletal hand simulator (Wang et al., 2023). In a separate domain, MANO denotes Management and Orchestration in 5G/NFV systems, as in open-source MANO frameworks, benchmarking studies, and optimization-enhanced MANO for network slicing and energy efficiency (Haga et al., 2020); (Yilma et al., 2019); (Malandrino et al., 2019). Another distinct use is MaNo, standing for Matrix Norm, for unsupervised accuracy estimation under distribution shift; that paper explicitly states that the term MS-MANO does not appear there (Xie et al., 2024). In the present context, therefore, MS-MANO refers specifically to a muscle- and tendon-driven extension of MANO for physiologically constrained hand pose tracking (Xie et al., 2024).