PA-HOI: Physics-Aware Motion Capture Dataset

• PA-HOI is a physics-aware human–object interaction dataset that captures 562 motion sequences with controlled variations in object shape, size, and weight.
• It employs a hybrid optical–inertial motion capture system with SMPL-X representations and 6-DoF tracking to deliver precise kinematic data.
• The dataset supports biomechanics studies and text-to-motion generation by linking semantic captions to physically plausible motion trajectories.

Searching arXiv for the specified dataset and closely related HOI motion-capture references. The PA-HOI Motion Capture Dataset is a physics-aware human–object interaction dataset introduced to study how the physical attributes of objects—specifically shape, size, and weight—affect long-term human motion in HOI tasks. It is presented in “PA-HOI: A Physics-Aware Human and Object Interaction Dataset” (Wang et al., 8 Aug 2025). Unlike prior HOI datasets that primarily emphasize affordance, contact, or multi-object task complexity, PA-HOI makes object physics the central experimental variable. The dataset comprises 562 motion sequences of interactions between subjects of different genders and 35 3D objects varying in size, shape, and weight, with full-body and hand motion capture, 6-DoF object tracking, SMPL-X representations, and large-scale text annotations. Its stated purpose is to support the modeling and generation of physically plausible human–object motion trajectories conditioned on object attributes such as “large and heavy” or “small and light” (Wang et al., 8 Aug 2025).

1. Conceptual scope and motivation

PA-HOI, short for Physics-Aware Human-Object Interaction, was introduced to explicitly analyze how object shape, size, and weight influence human motion in HOI scenarios (Wang et al., 8 Aug 2025). The dataset is motivated by a limitation in prior resources: many established HOI or motion datasets emphasize affordance and contact—for example, where and how the body touches an object—or they focus on multi-object task complexity, but they do not systematically vary physical properties while keeping tasks comparable.

In PA-HOI, “physics-aware” means that the manipulated items are real physical objects, not merely 3D-printed replicas, and that their mass and inertia are treated as behaviorally relevant factors. For each object class, the dataset deliberately organizes size and weight into categorical groupings such as small/medium/large and light/medium/heavy, while also selecting shapes corresponding to simple geometric primitives such as spheres, cubes, and cylinders (Wang et al., 8 Aug 2025). The capture protocol is designed so that differences in these attributes produce observable kinematic differences, including increased bending for heavy loads, greater arm spread for large objects, and slower walking when transporting heavier items.

This design positions PA-HOI differently from other interaction datasets. In the comparison reported with the dataset, GRAB, OMOMO, and ARCTIC are cited as datasets oriented toward hand-object or affordance-rich interaction capture, while HIMO, ParaHome, and FORCE are associated with multi-object or more complex task settings (Wang et al., 8 Aug 2025). PA-HOI’s distinctive role is to isolate the effect of object physics on posture, velocity, motion scale, and interaction strategy.

A plausible implication is that PA-HOI is structured as an experimental resource not just for motion synthesis, but also for biomechanics-adjacent analysis of load adaptation, coordination strategies, and attribute-conditioned control policies. The paper’s own framing emphasizes questions such as how increasing weight alters body posture, bending angle, duration, and walking speed; how size changes contact patterns and coordination; and how shape affects grasp type and hand pose (Wang et al., 8 Aug 2025).

2. Dataset composition and interaction design

PA-HOI contains 562 motion sequences of human–object interaction (Wang et al., 8 Aug 2025). Each sequence involves a subject–object pair and follows a three-phase structure:

1. Pick up: the subject approaches and picks up the object.
2. Move: the subject follows a specified path.
3. Place: the subject places the object on a fixed target table and returns to a neutral stance.

The dataset includes subjects of different genders, although the exact number of participants and detailed demographic statistics are not specified. The paper states that the motion capture system is calibrated to individual skeletal dimensions, and it explicitly notes that the dataset currently involves a limited number of participants, with future work intended to expand this (Wang et al., 8 Aug 2025).

The object inventory comprises 35 objects, selected for categorical variation in size, shape, and weight. The base set contains 27 everyday objects, such as bottles, pots, boxes, and buckets. In addition, 8 objects appear in added-weight versions—for example, an empty bottle versus a filled one—and these weighted variants are treated as distinct objects, with unweighted versions suffixed “empty” (Wang et al., 8 Aug 2025). The qualitative range spans from a small bottle to a water dispenser, and from an empty plastic bottle to a barbell.

The dataset defines four basic interaction types according to contact region and manipulation mode:

Interaction type	Description
grab	Contact with the inner hand, mainly for small and medium objects
support	Holding from below, typical for lifted or carried small/medium objects
hold	Embracing or stabilizing with arms or chest, especially for large objects
push	Pushing a large object along the ground using both hands

The motion paths are also discretized into four categories: walking straight, walking left, walking right, and interacting in place (Wang et al., 8 Aug 2025). Within each recorded trial, the subject starts from a defined initial location, performs an interaction such as grab or hold, follows the assigned path, and terminates by placing the object on the target table.

The dataset paper reports that grab is the most frequent action category, particularly for small and medium, light objects, while push and support are less frequent (Wang et al., 8 Aug 2025). It also states that heavy objects require more frames, hence longer durations, than light objects for the same action, especially for hold and push.

3. Capture system, representations, and annotations

PA-HOI uses a hybrid optical–inertial motion capture system, specifically Noitom PN Hybrid VTS, with 12 infrared cameras, a full-body optical–inertial mocap suit, and Perception Neuron Studio (PNS) inertial gloves for finger articulation (Wang et al., 8 Aug 2025). For each object, the dataset attaches a K-type rigid body equipped with an inertial sensor and four optical markers, yielding real-time 6-DoF tracking of translation and rotation.

The hybrid design is described as having high robustness to occlusion, which is particularly relevant for HOI capture because large objects can obscure substantial portions of the body. The system is calibrated per subject, and the inertial gloves are used to preserve finger tracking even under occlusion (Wang et al., 8 Aug 2025). The paper does not provide explicit numerical sampling rates or absolute spatial accuracy, but it characterizes the setup as standard high-quality motion capture.

Human motion is post-processed into SMPL-X. For each frame, the representation includes global orientation $\mathbf{o} \in \mathbb{R}^3$, root translation $\mathbf{t} \in \mathbb{R}^3$, body pose $\boldsymbol{\theta}_b \in \mathbb{R}^{21 \times 3}$, and finger pose $\boldsymbol{\theta}_h \in \mathbb{R}^{30 \times 3}$ (Wang et al., 8 Aug 2025). Shape parameters are estimated using Hybrid-X and held as the subject’s body shape. The fitting objective combines joint reconstruction, temporal smoothing, and pose regularization:

$$\mathcal{L}_{joint} = \sum_{n=0}^{N} \sum_{j=0}^{J} \left\| P_n^j - \hat{P}_n^j \right\|_2^2$$

$$\mathcal{L}_{smooth} = \sum_{n=0}^{N-1} \sum_{j=0}^{J} \left\| \hat{P}_{n+1}^j - \hat{P}_n^j \right\|_2^2$$

$$\mathcal{L}_{reg} = \left\| \boldsymbol{\theta}_b \right\|_2^2 + \left\| \boldsymbol{\theta}_h \right\|_2^2$$

with overall objective

$$\mathcal{L} = \lambda_{j} \mathcal{L}_{joint} + \lambda_s \mathcal{L}_{smooth} + \lambda_{r} \mathcal{L}_{reg}$$

and weights $\lambda_j = 1$, $\lambda_s = 0.1$, and $\mathbf{t} \in \mathbb{R}^3$0 (Wang et al., 8 Aug 2025).

Each sequence additionally includes the object’s 6-DoF pose trajectory and a scaled 3D mesh generated by Tripo AI, aligned to the rigid-body tracking coordinate through a fixed offset. The dataset therefore provides frame-aligned human pose and object pose in a common global coordinate system.

A major annotation component is the text modality. Every motion sequence begins with a base description built from a unified template encoding object identity, physical attributes, action, and movement type:

$\mathbf{t} \in \mathbb{R}^3$1

These base captions are then expanded using LLMs for textual augmentation. The augmentation introduces lexical variation such as replacing “grab” with “grip,” “seize,” or “grasp,” or adding modifiers like “with the right hand” or “using both hands.” The dataset contains 562 base descriptions, one per sequence, and 2,055,060 captions after augmentation (Wang et al., 8 Aug 2025). This annotation scale is specifically intended to support text-to-motion and HOI captioning tasks.

The dataset’s physics-related metadata is categorical rather than numerical. It explicitly encodes size category, weight category, and shape class, and these attributes appear in the text annotations. However, the paper does not provide numerical masses, inertia tensors, coefficients of friction, or explicit center-of-mass coordinates (Wang et al., 8 Aug 2025).

4. Physics-aware characteristics and empirical evidence

PA-HOI was built to capture how object attributes modulate motion, and the paper presents both qualitative and quantitative evidence for this claim (Wang et al., 8 Aug 2025). In qualitative comparisons, the authors hold two attributes constant while varying the third:

• Shape variation: similar size and weight but different primitive shapes lead to different grasping patterns and different finger flexion degrees.
• Weight variation: same shape and size but different weight leads to increased body bending, more forward lean, and more careful movement.
• Size variation: same shape and weight category but different size produces a larger range of motion, wider arm spread, and more whole-body coordination.

These observations indicate that large objects induce strategies such as hugging or holding rather than one-handed grasping, while heavier objects induce more crouched or bent postures. Shape differences, by contrast, chiefly alter contact configuration and hand pose (Wang et al., 8 Aug 2025).

The paper also reports a quantitative analysis based on average frame count per sequence stratified by action type and object weight category. Heavy-weight objects require significantly more frames than light ones for the same action, with especially strong effects for hold and push (Wang et al., 8 Aug 2025). The interpretation given is that subjects move slower and more cautiously when interacting with heavy objects. The dataset therefore encodes changes in:

• Posture: bending, forward lean, arm spread, hand configuration
• Velocity and timing: longer sequence duration for heavier objects
• Motion amplitude and strategy: e.g., one-handed grasping versus whole-body holding

The paper does not compute explicit force, torque, or energy measures, but it notes that the recorded kinematics are sufficient for downstream derivation of joint velocities and related analyses (Wang et al., 8 Aug 2025). This suggests that PA-HOI can support secondary studies in motion energetics or load-sensitive control, although such analyses are not themselves part of the published benchmark.

5. Benchmark integration with text-to-motion generation

To demonstrate dataset utility, the authors integrate PA-HOI with two existing text-to-motion generation frameworks: MDM and StableMoFusion, producing PA-HOI-specific variants MDM* and StableMoFusion* (Wang et al., 8 Aug 2025). In these experiments, object physical attributes are encoded directly in the text condition, so prompts include not only the action but also phrases such as “large and heavy box.”

The evaluation uses standard text-to-motion metrics: MM-Dist, FID, Diversity, and R-Precision with top-1, top-2, and top-3 retrieval variants, all reported with 95% confidence intervals (Wang et al., 8 Aug 2025). The real-data reference row is given as:

• MM-Dist: $\mathbf{t} \in \mathbb{R}^3$2
• FID: $\mathbf{t} \in \mathbb{R}^3$3
• Diversity: $\mathbf{t} \in \mathbb{R}^3$4
• R-Precision top-1/2/3: about $\mathbf{t} \in \mathbb{R}^3$5

The reported model results are as follows:

Model	MM-Dist	FID	Diversity
MDM	$\mathbf{t} \in \mathbb{R}^3$6	$\mathbf{t} \in \mathbb{R}^3$7	$\mathbf{t} \in \mathbb{R}^3$8
MDM*	$\mathbf{t} \in \mathbb{R}^3$9	$\boldsymbol{\theta}_b \in \mathbb{R}^{21 \times 3}$0	$\boldsymbol{\theta}_b \in \mathbb{R}^{21 \times 3}$1
StableMoFusion	$\boldsymbol{\theta}_b \in \mathbb{R}^{21 \times 3}$2	$\boldsymbol{\theta}_b \in \mathbb{R}^{21 \times 3}$3	$\boldsymbol{\theta}_b \in \mathbb{R}^{21 \times 3}$4
StableMoFusion*	$\boldsymbol{\theta}_b \in \mathbb{R}^{21 \times 3}$5	$\boldsymbol{\theta}_b \in \mathbb{R}^{21 \times 3}$6	$\boldsymbol{\theta}_b \in \mathbb{R}^{21 \times 3}$7

Among the benchmarked models, StableMoFusion* achieves the best MM-Dist and FID, while MDM* also substantially improves over the original MDM (Wang et al., 8 Aug 2025). The paper interprets these improvements as evidence that training on PA-HOI allows generative models to produce motions that are closer to real motion distributions and more responsive to physics-aware textual cues.

Qualitatively, for the prompt “push a large and heavy box,” the original versions of MDM and StableMoFusion are described as generating more generic motions with less bending and weaker indications of exertion, whereas the PA-HOI-trained versions exhibit strong forward lean, more pronounced effort, and posture consistent with heavy pushing (Wang et al., 8 Aug 2025). This is presented as evidence that the dataset transfers realistic physical awareness into text-conditioned motion generation.

6. Position within the HOI dataset landscape

PA-HOI is situated by its authors relative to several prior datasets, including GRAB, BEHAVE, InterCap, OMOMO, ARCTIC, HIMO, ParaHome, and FORCE (Wang et al., 8 Aug 2025). The comparison emphasizes that PA-HOI is not the largest HOI dataset by number of sequences, but that it is unusual in simultaneously providing motion capture, multiple text descriptions per sequence, and explicit variation in shape, size, and weight.

The most direct comparison in terms of physical attributes is FORCE, described as the first dataset focusing on the weight attribute of large objects but lacking text and not covering shape and size diversity (Wang et al., 8 Aug 2025). In contrast, PA-HOI incorporates all three physical axes and links them to semantic annotations. Relative to HIMO, which contains more sequences and multiple objects, PA-HOI is narrower in scale but more targeted in experimental control over object physics.

This comparison also clarifies what PA-HOI is not. It is not primarily a markerless RGB-D interaction benchmark in the sense of “A Dataset and Evaluation for Complex 4D Markerless Human Motion Capture” (Park et al., 14 Apr 2026), which emphasizes multi-person occlusion, synchronized multi-view RGB-D, and Vicon-aligned SMPL/SMPL-X for markerless 4D evaluation rather than controlled variation of object attributes. Nor is it a narrowly constrained upper-limb manipulation corpus of the type presented in “The Effects of Selected Object Features on a Pick-and-Place Task: a Human Multimodal Dataset” (Lastrico et al., 2024), where identical cups vary mainly in weight and liquid filling. PA-HOI instead occupies an intermediate position: broader than single-task property-aware pick-and-place datasets, but more experimentally focused on object physics than general HOI corpora.

The article’s own comparison further suggests that PA-HOI complements other modern HOI resources such as “I’M HOI: Inertia-aware Monocular Capture of 3D Human-Object Interactions” (Zhao et al., 2023), which contributes a hybrid RGB-plus-object-IMU capture regime for highly dynamic interactions. PA-HOI differs by centering the physical attribute taxonomy itself rather than inertial sensing or high-speed activity diversity.

A common misconception would be to equate “physics-aware” with direct recording of physical quantities such as force, torque, friction, or inertia tensors. PA-HOI does not provide those measurements. Its notion of physics-awareness is categorical and behavioral: object size, weight, and shape are explicitly designed to induce measurable differences in motion, and those differences are captured kinematically and semantically (Wang et al., 8 Aug 2025).

7. Applications, limitations, and access

The dataset is presented as useful for several research areas: robotics, VR/AR and games, computer graphics and animation, human-computer interaction and ergonomics, and text-to-motion and HOI synthesis (Wang et al., 8 Aug 2025). In robotics, it can support learning of human handling strategies for objects with different physical properties. In animation and XR, it can support physically plausible interaction controllers that vary posture and effort with object mass or size. In language-conditioned generation, it enables models to distinguish prompts such as “heavy box” and “light box” at the motion level.

The principal limitations stated in the paper are twofold. First, facial expression is not recorded or modeled; the emphasis is strictly on body and hand motion in response to object physics. Second, the dataset contains a limited number of participants, and the authors identify subject diversity as a target for future extension (Wang et al., 8 Aug 2025). Additional limitations are implied by the reported design: the environment is a lab mocap volume with a target table and an implicit ground plane, rather than a cluttered or richly structured scene; no force plates, EMG, or tactile sensing are included; and explicit numerical object physics metadata such as mass values or inertia tensors are absent.

The dataset is publicly available via the project page:

$\boldsymbol{\theta}_b \in \mathbb{R}^{21 \times 3}$8

The paper states that the project page hosts the dataset and demos, while licensing terms are not specified in the paper itself and must be consulted on the project site (Wang et al., 8 Aug 2025).

In summary, PA-HOI is a motion-capture HOI dataset built around a controlled physical-variable design. Its defining contribution is not merely that it records human–object interaction, but that it does so in a way that makes shape, size, and weight the principal explanatory variables of motion differences. Through 562 sequences, 35 objects, SMPL-X body-and-hand motion, 6-DoF object trajectories, and 2,055,060 text captions, it establishes a benchmark for studying how object physics modulates human motion and for training generative models to reproduce those modulations from semantic descriptions (Wang et al., 8 Aug 2025).