- The paper introduces a scalable, simulation-driven framework that replaces EMG signals by training vision-based prosthetic controllers using imitation learning.
- It generates 2,000 high-fidelity grasp demonstrations via bilevel simulation, ensuring diverse, physically consistent hand-object interactions.
- Sim-to-real evaluations show over 90% grasp success in real-world trials, highlighting superior generalization compared to real-data-trained baselines.
Simulation-Driven Imitation Learning Framework for Shared-Autonomy Prosthetic Grasping
Introduction and Motivation
This paper introduces a scalable, simulation-centric framework for training vision-based, biosignals-free shared-autonomy prosthetic hand controllers using imitation learning (2606.07389). The traditional reliance on surface EMG signals for upper-limb prosthesis control imposes cognitive and physical burdens on users, demanding continuous and explicit intention signaling. Existing approaches using partial vision for grasp selection still require user-triggered actuation. The transition towards fully autonomous, biosignals-free control paradigms has led to increased focus on vision-driven learning, but their effectiveness is fundamentally constrained by the scarcity and inconsistency of large-scale real-world human demonstration datasets, the high cost and difficulty of safe, diverse trial collection, and the limited generalization caused by narrow scenario coverage.
Scalable Simulation of Grasping Demonstrations
The framework automatically generates massive collections of reach-to-grasp episodes by integrating dexterous grasp synthesis, human-like wrist motion trajectory retargeting, and scripted reach–grasp–lift sequences in procedurally generated indoor environments. Object meshes—sampled from Google Scanned Objects—are positioned within photorealistic scenes built with Infinigen Indoors, achieving substantial domain diversity.
For each episode, grasp configurations are optimized using a bilevel simulation technique adapted from BoDex, producing physically plausible, contact-rich prosthetic hand–object interactions. Sampled wrist trajectory templates are retargeted to terminal grasp poses, preserving natural approach behavior and diversity.
(Figure 1)
Figure 1: Wide-angle corner views of ten indoor rooms were generated with Infinigen Indoors and used as procedurally diverse simulation environments.
Dataset Composition and Quality Control
The pipeline collects all pertinent sensory streams—wrist-view RGB, hand joint proprioception, and the corresponding prosthetic action commands—across 200 objects in 10 rooms, yielding 2,000 successful high-fidelity demonstrations. Each grasp is validated by stability criteria; only physically consistent, successful lift trajectories are admitted into the dataset. This enables systematic benchmarking of policy generalization to both novel objects and scenes, and provides a direct vehicle for sim-to-real policy transfer.
Imitation Learning Benchmarks and Algorithms
The authors establish a benchmark with three representative neural imitation learning architectures:
- ACT: Action-Chunking Transformer-based model with CVAE branches.
- VTM-VAE: Sequence model using a central-aware Mamba backbone for efficient image-conditioned latent representation.
- HannesImitation: Diffusion-model-based policy interface tailored for prosthetic joint trajectory forecasting.
Policy inputs are high-dimensional wrist-mounted RGB images and multi-step proprioceptive histories; outputs are high-fidelity joint action sequences. Extensive protocolization ensures fair comparison and reproducibility.
Quantitative Evaluation and Generalization
The primary evaluation is along three axes: grasp success rate (SR), open-during-reach rate (OR), and close-before-lifting rate (CR). Generalization experiments empirically demonstrate that increasing both object variety (from 10 to 100) and environmental diversity (from 2 to 10 rooms) in the training distribution consistently yields monotonic improvements in unseen-object and unseen-room policy robustness. Notably, joint scaling along both factors achieves higher absolute gains than scaling either independently.
During simulation benchmarks, ACT records the highest room-level SR (57.40% with 8 training rooms), VTM-VAE dominates object-level SR (44.90% with 100 training objects), and both maintain high intermediate rates (OR, CR). HannesImitation, the prosthetic-specific diffusion baseline, shows lower generalization, indicating that trajectory chunking and efficient visuomotor representation are critical for cross-domain success.
Sim-to-Real Transfer and Real-World Evaluation
Realistic-setting experiments feature 1,800 trials (12 participants, 3 scenes, 5 objects), with direct deployment of simulation-trained controllers to a physical PSYONIC Ability hand. The room-generalization policy achieves over 90% overall grasp success rate across all three backgrounds and all objects, while a real-data-trained baseline (VTM-VAE) collapses (0% SR) under a single domain-shifted scene despite near-perfect performance on in-domain settings. The simulation-trained policy maintains high SR, OR, and CR robustly, evidencing marked generalization to environmental changes.
Failure analysis associates the majority of errors with insufficient finger force or failure to trigger finger closure—suggesting further refinements in the contact and action policy, but not visual misclassification.
Implications and Prospects
This work makes several bold claims:
- High-fidelity, simulation-generated wrist-view datasets can fully supplant real-human demonstration for biosignals-free prosthetic grasping policy training.
- With appropriate procedural diversity, visual sim2real transfer yields greater cross-scene robustness than policies trained exclusively on real data.
- Robust generalization is predominantly driven by scene variability—object diversity contributes, but scene variation is the principal determinant of success under domain shift.
The framework has substantial implications:
- Practical: Reduces cost, risk, and time of real-world demonstration collection; democratizes data generation and evaluation for rehabilitation robotics.
- Theoretical: Provides evidence that procedural scene randomization and high-fidelity simulation are sufficient foundations for robust vision-based sim-to-real policy transfer in complex manipulation.
- AI Development: Brings forward an operational blueprint for scaling imitation learning in other domains beset by demonstration bottlenecks—suggesting similar simulation-driven regimes for other under-actuated, vision-guided control problems.
Future directions include scaling the data regime (more objects, rooms, articulated scenes), advancing contact and dynamics modeling, incorporating online policy adaptation (e.g., residual RL), and direct deployment with amputee participants.
Conclusion
The presented framework demonstrates that large-scale, procedurally diversified simulation enables end-to-end imitation learning of biosignals-free shared-autonomy prosthetic hand controllers exhibiting robust sim-to-real transfer, strong generalization to both new objects and novel scenes, and significantly superior real-world performance under domain shift compared to real-data-trained baselines. This validates simulation-driven training as a practical paradigm for autonomous prosthetic grasping, reducing reliance on user-specific biosignals or extensive real-world annotation. The research outlines a replicable path toward general-purpose, user-agnostic shared-autonomy manipulators, advancing both prosthetics and robot learning.
