Simulating Infant First-Person Sensorimotor Experience via Motion Retargeting from Babies to Humanoids

Abstract: Motion retargeting from humans to human-like artificial agents is becoming increasingly important as humanoid robots grow more capable. However, most existing approaches focus only on reproducing kinematics and ignore the rich sensorimotor experience associated with human movement. In this work, we present a framework for simulating the multimodal sensorimotor experiences of infants using physical and virtual humanoids. From a single video, our method reconstructs the infant's body configuration by extracting its skeletal structure and estimating the full 3D pose from each frame. Then we map the reconstructed motion onto several developmental platforms: the physical iCub robot and the virtual simulators pyCub, EMFANT and MIMo. Replaying the retargeted motions on these embodiments produces simulated multisensory streams including proprioception (joints and muscles), touch, and vision. For the best-matching embodiment, the retargeting achieves sub-centimeter accuracy and enables a rich multimodal analysis of infant development as well as enhanced automated annotation of behaviors. This framework provides a unique window into the infant's sensorimotor experience, offering new tools for robotics, developmental science, and early detection of neurodevelopmental disorders. The code is available at https://github.com/ctu-vras/motion-retargeting/.

• The paper presents a novel framework for reconstructing infant 3D poses from video and retargeting them onto diverse humanoid platforms.
• It employs advanced methods like ViTPose, SMPL/SMIL models, and platform-specific algorithms to achieve sub-centimeter accuracy on simulated embodiments.
• The framework integrates multimodal sensorimotor streams, enabling automated behavioral annotation and advancing research in developmental science and embodied AI.

Simulating Infant Sensorimotor Experience via Motion Retargeting: Methods and Implications

Introduction

The paper "Simulating Infant First-Person Sensorimotor Experience via Motion Retargeting from Babies to Humanoids" (2604.27583) presents a formal framework for extracting, retargeting, and replaying infant sensorimotor behaviors in both physical and virtual humanoid robots. The primary objective is to bridge the gap between available third-person infant video datasets and the need for high-fidelity, multimodal sensorimotor data amenable to analysis in developmental science, robotics, and machine learning. The authors introduce a pipeline for reconstructing infants' body configurations from video, mapping these onto diverse humanoid platforms, and reproducing the corresponding multimodal sensory streams (vision, touch, proprioception, vestibular signals), enabling precise analysis and annotation of developmental trajectories.

Methodological Framework

The pipeline accepts single or multi-view video recordings of infants and reconstructs 3D pose and body configuration via the ViTPose network for multi-view 2D keypoint detection and subsequent triangulation. In situations lacking multi-camera setups, parametric models (e.g., SMPL, SMIL) are fit to the 2D detections to infer 3D body geometry.

The retargeted motion is mapped onto three distinct embodiments:

• iCub (physical and simulated): A robot with anthropomorphic childlike morphology, equipped with vision, touch, and proprioception sensors.
• EMFANT: An anatomically grounded musculoskeletal model, offering fine-grained biomechanical constraints and muscle-based proprioceptive feedback.
• MIMo: A scalable, multimodal infant model with simplified morphology and high-resolution touch sensing, supporting extensive kinematic flexibility.

Platform-specific retargeting methods are employed. For iCub, an analytical geometric retargeting (augmented by joint optimization and hardware constraints) is applied. MIMo uses MuJoCo-based inverse kinematics with mocap constraints, while EMFANT leverages an OpenSim-based biomechanics pipeline with PD control for physical plausibility and muscle activation.

Evaluation of Retargeting Fidelity

Retargeting accuracy is quantified by three key metrics:

• Mean Absolute Error (MAE) of 3D Keypoints: MIMo achieves the lowest error (4.6 mm), attributed to its scalable, geometry-matched design. EMFANT and iCub achieve MAE under 10 cm, with iCub performance limited by the mismatch in body proportions and hardware constraints.
• Relative End-Effector Orientation: MIMo produces the most faithful retargeting, with orientation MAE at 1.96°, outperforming EMFANT and iCub (errors consistently >5°).
• Relative Velocity of Keypoints: All platforms maintain relative velocity MAE below 1º/step (MIMo as low as 0.15°/step).

The scaling flexibility of MIMo and EMFANT provides a key advantage, supporting more accurate pose reconstruction for the infant-sized morphology, while iCub remains crucial for evaluating algorithms on real-world hardware.

Cross-Embodiment Consistency of Sensorimotor Experience

One notable analysis involves aligning and comparing the multimodal sensorimotor streams (touch, proprioception, vision) across platforms. Using PCA for dimension reduction and Generalized Procrustes Analysis for alignment, the authors report that a low-dimensional latent representation (K ≈ 20) is sufficient to capture the majority of cross-embodiment invariance. The invariance index plateaus at 0.19%, indicating moderate but significant consistency in latent sensorimotor manifolds between embodiments—even with substantial morphological differences. These findings suggest sensorimotor experiences generated from retargeted infant data remain robust to embodiment, thus supporting the validity of the pipeline for cross-platform developmental studies.

Simulation and Analysis of Multisensory Infancy

The authors demonstrate simulation of four primary sensory modalities:

• Vision: Replaying reconstructed motions with binocular cameras simulates the first-person visual stream; e.g., instances of hand regard can be directly visualized and analyzed for depth cues and field of view.
• Touch: High-resolution tactile reconstructions reveal the spatial and temporal patterns of self-touches, critical for studying body schema development.
• Proprioception: EMFANT supports biologically realistic, muscle-tendon-based proprioceptive feedback, enabling investigations into whole-body coordination at a mechanistic level.
• Vestibular Sensing: Simulated accelerometer and gyroscope data (especially in MIMo) enable monitoring of self-motion and balance during spontaneous infant movements.

This multimodal replay offers an unprecedented window into the sensorimotor contingencies faced by infants, critical for progress in developmental robotics, behavioral annotation, and understanding the emergence of embodied cognition.

Automated Annotation and Application to Developmental Science

A key practical outcome is the automated detection and annotation of developmental behaviors, particularly self-touches. Compared against expert manual coding, retargeted self-touches on simulated platforms closely reproduce empirical distributions, with iCub showing 13% of left hand touches to the left leg (cf. 9% in manual annotation), and similar correspondences for other regions and lateralization. This approach dramatically reduces labor in behavioral annotation and enables scalable analysis over large datasets.

Additionally, the pipeline supports the alignment and analysis of behavioral patterns over time and between individuals—crucial for detecting atypical development, generating normative databases, and enabling semi-supervised learning for automatic behavior coding.

Theoretical and Practical Implications

This work has several theoretical implications:

• Bridging the Embodiment Gap: The demonstrated ability to map infant movements and sensorimotor experiences onto diverse robotic platforms narrows the gap between human developmental trajectories and artificial cognitive systems.
• Multimodal Foundation Models: Access to rich, labeled, and time-synchronized multisensory streams provides invaluable training data for self-supervised multimodal learning and sensorimotor foundation models.
• Behavioral Modality Generalization: The evidence for consistent low-dimensional sensorimotor manifolds across embodiments suggests developmental robotics models can generalize across varying morphologies, increasing their scientific utility.

Practically, this approach enables:

• Large-scale, minimally invasive behavioral phenotyping from video, with potential for early diagnosis of neurodevelopmental disorders.
• Systematic benchmarking of embodied AI and imitation learning algorithms under developmentally realistic perceptual conditions.
• Use in robotics as a source of ecologically valid sensorimotor trajectories for policy learning, controller benchmarking, and resilience testing.

Future Prospects

There are promising avenues for continuing this research:

• Scaling to vast annotated video datasets, enabling population-level mappings of developmental sensorimotor experience.
• Generalization to older children and adults—including manipulation, vertical posture, and object interaction scenarios.
• Integration into end-to-end learning pipelines for robotics and cognitive modeling, further closing the loop between natural and artificial agents.

Conclusion

This paper formalizes and validates a pipeline for reconstructing, retargeting, and analyzing infant sensorimotor experience in humanoid robots. The framework attains high retargeting accuracy (sub-centimeter for MIMo), robust cross-platform sensorimotor consistency, and automatable behavioral annotation. These contributions directly support advances in developmental science, embodied AI, and the construction of sensorimotor foundation models, while providing tools for early detection of developmental anomalies and enriching the realism of robotic learning environments (2604.27583).