MoviCam: Real-World Monocular Motion Benchmark

• MoviCam is a benchmark dataset for human motion estimation that provides ground-truth annotations including camera trajectories, scene geometry, 3D human pose, and contact labels.
• The dataset uses a real moving RGB camera in a controlled studio with multi-view capture to achieve accurate motion reconstruction in non-planar scenes.
• MoviCam facilitates evaluation of physics-based pose tracking methods by quantifying reconstruction accuracy and physical plausibility with metrics like MPJPE, PA-MPJPE, and foot sliding.

Searching arXiv for the MoviCam paper and related benchmark context. MoviCam is a real-world benchmark for human motion estimation from a single moving RGB camera, introduced in the context of physics-based monocular pose tracking under camera motion and non-flat scenes. It is presented as the first non-synthetic dataset that jointly provides ground-truth camera trajectories of a dynamically moving monocular RGB camera, scene geometry, 3D human motion with human-scene contact labels, and SMPL human pose and shape, with the explicit aim of evaluating world-frame human motion recovery when both the person and the camera move in a geometrically complex environment (Aytekin et al., 23 Jul 2025).

1. Origin and benchmark scope

MoviCam was introduced to address a specific benchmark gap in monocular human motion estimation: recovering human motion in global or world coordinates from a single monocular RGB camera that itself moves, while the subject interacts with a real scene that is not restricted to a flat ground plane. The motivating observation is that many monocular 3D human motion methods operate in root-relative or camera-relative coordinates, and that even methods targeting global motion under moving cameras are often evaluated either without ground-truth scene geometry or on synthetic data that does not reproduce real-world lighting, camera motion, view-dependent appearance, and geometric complexity (Aytekin et al., 23 Jul 2025).

The dataset is defined by the conjunction of five annotation types: ground-truth camera trajectories, scene geometry, 3D human motion in world coordinates, SMPL human pose and shape, and human-scene contact labels. This combination is central to its role as a benchmark. A plausible implication is that MoviCam is intended not merely to score articulated pose accuracy, but to test whether a method recovers physically coherent human motion in the world frame, including correct support relations to the scene.

The benchmark is explicitly organized around real moving-camera capture rather than simulated camera motion. This distinguishes it from settings in which camera motion is synthesized by cropping, panoramic reprojection, or virtual rendering. In the benchmark design, the relevant difficulty is the simultaneous estimation of body pose, body translation, camera motion, and scene-consistent contact under real monocular acquisition.

2. Capture setup and dataset contents

MoviCam was recorded in a controlled studio environment so that accurate ground truth could be obtained for scene mesh, human pose, and camera trajectory. The studio used 120 synchronized static multi-view cameras capturing up to 4K resolution for accurate reconstruction; in post-processing, 34 cameras at 2K resolution were used. The human motion ground truth was produced with The Captury, a markerless motion capture system, while a SONY RX10 sports camera at $1080 \times 1920$ served as the moving monocular RGB camera (Aytekin et al., 23 Jul 2025).

A checkerboard attached to the moving camera enabled the moving camera’s trajectory to be tracked by the external static cameras. Each recording involved two people: one performed the motion and interacted with the scene, and another physically operated the moving camera. This capture design makes the moving monocular camera an actual real camera moving through a real scene, rather than a simulated viewpoint.

The benchmark scene includes a table, a step stool, and various carpets scattered across the floor. The motion repertoire includes walking, jumping, stretching, squatting, and climbing or stepping on non-flat structures. These motions were chosen to create non-planar support surfaces and realistic contact patterns beyond simple floor walking.

MoviCam consists of 7 sequences, approximately 22,000 images, and 7 participants with varying heights and weights. Of the 7 sequences, 5 are on non-flat ground and 2 are on a flat surface. Each sequence features a different individual and a different moving camera trajectory.

The provided modalities and annotations include dense scene geometry as a mesh, a scene height map derived from the mesh, ground-truth 3D human pose and shape in SMPL format, human motion trajectory in the world frame, global camera poses or camera extrinsics, and contact labels for the left and right toes and heels. The human representation is described as SMPL format with 24 joints and 300 shape parameters. The scene mesh is explicitly provided for the non-flat ground setup.

3. Ground-truth generation and annotation pipeline

The ground-truth scene mesh is reconstructed from the multi-view images captured by the 120 cameras in 4K. A height map is then derived from that mesh by loading the mesh into PyBullet, generating a grid of resolution $1024 \times 1024$ over the scene, casting rays from above to below the scene for each grid cell, and recording the first intersection with the scene as local surface height. The resulting height function is queried as $h(x,z)$ for a foot location $(x,z)$ (Aytekin et al., 23 Jul 2025).

Human pose, shape, and motion are converted into SMPL-compatible annotations from The Captury output. The pipeline aligns SMPL 3D joints with the Captury skeleton, places markers at SMPL joint locations, estimates the shape parameter from the first 100 T-pose frames and then keeps it fixed, and processes the full sequence with Captury tracking using the SMPL joint markers as input. This yields per-frame pose, per-frame translation, and SMPL shape in the world frame.

Camera trajectory recovery is treated as a separate calibration problem. The moving camera carries a checkerboard, and the benchmark uses hand-eye calibration following Strobl et al. The transformations are listed as external camera to floor checkerboard $T_{EF}$, external camera to head checkerboard $T_{EH}$, and moving camera to floor checkerboard $T_{MF}$, with the hand-eye relation written as

$T_{\text{Hand-eye} = T_{EH}^{-1} T_{EF} \cdot T_{MF}^{-1}.$

The actual moving-camera trajectory is then recovered using images from 34 studio cameras, triangulating checkerboard position in each frame, averaging estimated poses from each camera, and applying the hand-eye transform.

Contact labels are generated automatically after obtaining the scene mesh and human pose. For each relevant foot joint, the distance to the closest scene vertex is computed; if that distance is less than 5 cm, the joint is labeled “in contact,” otherwise “not in contact.” The labels are provided for left toe, right toe, left heel, and right heel. This label definition follows PhysCap-style contact annotation.

4. Relationship to PhysDynPose

MoviCam is tightly coupled to PhysDynPose, the physics-based monocular pose estimation method introduced in the same work. The benchmark is used to expose failure modes of current methods, while PhysDynPose is designed specifically for the setting MoviCam captures: a person moving in a real environment with non-flat geometry while the camera also moves (Aytekin et al., 23 Jul 2025).

PhysDynPose has two stages. The kinematics module uses 4DHumans to estimate body motion and DROID-SLAM to estimate the moving-camera trajectory. Because monocular SLAM has scale ambiguity, the SLAM trajectory is aligned using the first two frames of the ground-truth camera trajectory. The method then transforms the human motion from camera coordinates into world coordinates using the estimated camera rotation and translation: $\begin{aligned} {}^w R_g &= \mathbf{R_{S}^{-1} {}^c R_g, \ {}^w t &= \mathbf{R_{S}^{-1} \left( {}^c t - \mathbf{T_{S} \right). \end{aligned}$ The provided text is typeset imperfectly, but the intended interpretation is that world-frame root orientation and translation are recovered by undoing the camera motion. The transformed pose and translation are then filtered with a One-Euro Filter using minimum cutoff frequency 0.004 and speed coefficient 0.7.

The physics module extends PIP by integrating scene geometry and root supervision. It runs in PyBullet on a floating-base humanoid and takes as input the estimated pose $q_{ref}$, estimated joint positions $1024 \times 1024$0, scene height map, and foot contact labels. The root state update is

$1024 \times 1024$1

and root supervision uses

$1024 \times 1024$2

with $1024 \times 1024$3 s.

Scene integration replaces the flat-floor condition $1024 \times 1024$4 with the scene-aware test

$1024 \times 1024$5

so foot support and penetration are evaluated against the actual scene height. The final optimization is written as

$1024 \times 1024$6

The extracted equation is visibly corrupted, but the role of scene geometry, friction, no-sliding, and no-drifting constraints is explicit.

5. Evaluation protocol and benchmark findings

MoviCam evaluates both reconstruction accuracy and physical plausibility. The baseline methods are GLAMR, WHAM, and 4DHumans; for fairness, GLAMR and WHAM are initialized using ground-truth orientation and translation from the first two frames, and a 4DHumans-based baseline is transformed into world coordinates using DROID-SLAM camera extrinsics (Aytekin et al., 23 Jul 2025).

The 3D reconstruction metrics are MPJPE, PA-MPJPE, W-MPJPE, WA-MPJPE, and RTE. The physical plausibility metrics are percentage of frames with scene penetration, average penetration depth per frame in mm, average height above the scene in mm, jitter in mm/s, and foot sliding in mm.

On non-flat scenes, the reported results are that 4DHumans has the best MPJPE at 128.01 mm, WHAM has the best PA-MPJPE at 33.88 mm, PhysDynPose has the best W-MPJPE at 779.60 mm, 4DHumans and PhysDynPose are nearly tied on WA-MPJPE at 417.83 mm and 418.65 mm respectively, and PhysDynPose has the best RTE at 1.16 versus 1.18 for 4DHumans. On flat scenes, 4DHumans again has best MPJPE at 105.70 mm, WHAM best PA-MPJPE at 33.93 mm, and PhysDynPose best W-MPJPE at 1092.72 mm, best WA-MPJPE at 489.72 mm, and best RTE at 1.87.

The physical plausibility results are also scene-dependent. On non-flat scenes, PhysDynPose achieves best foot sliding at 3.22 mm, while GLAMR, WHAM, and 4DHumans have average distance above scene values of 1370.82 mm, 1113.03 mm, and 285.54 mm respectively, compared with 377.37 mm for PhysDynPose. On flat scenes, PhysDynPose again achieves best foot sliding at 1.80 mm and the lowest average distance above the scene at 196.17 mm. The benchmark therefore shows that low penetration alone is not sufficient, since a method can avoid penetration by placing the body unrealistically high above the scene.

The paper stresses that PA-MPJPE can be misleading in this setting: a method may align well after Procrustes transformation while still having poor global placement and physically implausible scene interaction. This suggests that MoviCam’s principal contribution is methodological as much as curational: it makes scene-aware world-frame evaluation unavoidable.

6. Relation to prior datasets and current limitations

MoviCam is positioned against several prior datasets. Dynamic Human3.6M and DynaCam or TRACE-related synthetic setups simulate camera motion rather than using a real moving camera. EMDB has global human and camera trajectories but lacks scene geometry. EgoBody includes camera and motion but only incomplete scene mesh and no contact labels. SLOPER4D includes LiDAR-based scene information and global trajectories, but is described as lacking accurate foot-ground contact and not explicitly optimizing with scene geometry. Traditional motion-capture datasets such as Human3.6M, HumanEva, AMASS, and TotalCapture use static cameras and controlled studios rather than a moving monocular camera with world-grounded scene interaction (Aytekin et al., 23 Jul 2025).

Within that comparison, MoviCam’s distinctiveness lies in the joint availability of real moving-camera RGB video, ground-truth moving-camera trajectories, world-frame human motion, dense scene geometry, derived height map, and foot-scene contact labels in non-planar scenes. A plausible implication is that it serves as a benchmark not only for pose estimation, but for scene-aware physical reasoning under monocular camera motion.

The current limitations are explicit. MoviCam contains only single-person sequences, a pre-scanned static scene, and contact limited to foot-floor interaction. Correspondingly, PhysDynPose models only foot-floor contact and depends on 4DHumans for pose estimation, DROID-SLAM for camera trajectory, and ground-truth camera initialization for scale and alignment. These constraints mean that the benchmark is targeted rather than universal: it is strong for monocular moving-camera human-scene interaction in static environments, but not yet a general benchmark for arbitrary human-object interaction, multi-person scenes, or dynamic environments.