ProAct: AR Testbed for Prosthetic Control

• ProAct Framework is an open-source, mixed-reality platform that integrates myoelectric input, gaze tracking, and robotic planning for full-arm prosthetic control.
• It combines advanced hardware like HoloLens 2 and EMG armbands with software including ROS, Gazebo, and MoveIt to execute and evaluate multiple control modes.
• User studies show that gaze- and context-assisted control modes significantly improve task success rates and user experience compared to direct control.

The term "ProAct Framework" encompasses several distinct frameworks, systems, and methodologies across robotics, machine learning, distributed systems, and agentic AI, each targeting different domains and problem settings. The following article focuses exclusively on the ProACT platform for intelligent prosthetic arm control as defined and presented in "ProACT: An Augmented Reality Testbed for Intelligent Prosthetic Arms" (Guptasarma et al., 2024). Selected comparative context is provided for clarity, but all technical specifics, performance metrics, and implementation details are taken verbatim from the cited work.

1. Overview and Objectives

The Prosthetic Arm Control Testbed (ProACT) is an open-source, mixed-reality platform for evaluating and advancing intelligent control strategies in upper-limb prosthetic arms. It is designed to address two primary barriers in the field: the absence of effective whole-arm, semi-autonomous prosthesis controllers, and the lack of an accessible, extensible testbed for rigorous evaluation and simulation (Guptasarma et al., 2024). Previous research had addressed isolated joints—hand, wrist, or elbow—using intent estimation (e.g., gaze, EMG), but no integrated solution existed for full 7-DoF shoulder–elbow–wrist and 20-DoF hand devices.

ProACT combines low-level myoelectric (EMG) input, real-time gaze and context inference, robotic motion planning, and augmented reality feedback to create an immersive, high-fidelity research environment for non-amputee and, prospectively, amputee subjects. The testbed enables systematic comparison of direct, shared, and semi-autonomous control paradigms and uniquely supports sequential task modeling, essential for real-world prosthesis usability.

2. System Architecture and Key Components

Hardware

• Microsoft HoloLens 2 HMD: Provides real-time inside-out SLAM tracking and gaze vector estimation.
• OyMotion Myo-style EMG armband: Eight surface electrodes stream raw EMG via Bluetooth LE.
• Optical motion-capture markers on a shoulder harness: Ensure precise base-frame localization for virtual–physical alignment.
• Desktop workstation: Ubuntu 20.04, Intel i9 CPU, running Gazebo physics at 1 kHz.

Software Stack

• ROS Noetic: Core for all inter-process messaging.
• Gazebo + ODE: Rigid-body dynamics simulation of the Modular Prosthetic Limb (MPL)—7 DoF shoulder-elbow-wrist, 20 DoF hand.
• Unity 2020.3: AR rendering, linked via ROS# (ROS–Unity bridge).
• MoveIt: Trajectory execution for prosthetic kinematics.
• OMPL (Open Motion Planning Library): Sample-based path planning.
• KDL (Kinematics and Dynamics Library): Jacobian-based inverse kinematics.
• LibEMG: Real-time LDA classification of five discrete EMG gestures.

AR Integration

• Gazebo disseminates joint/pose data as ROS topics; ROS# relays these into Unity for AR rendering.
• Virtual MPL, task objects, and gaze cursor are superimposed with accurately tracked shoulder/table reference.
• HoloLens gaze rays, fused with head pose and mocap reference, enable robust gaze-in-scene computation.

3. Intelligent Control Pipeline

ProACT supports four progressively intelligent control modes (A–D):

Mode	Description	Core Mechanisms
A	Direct joint control	EMG gesture cycles/selects/actuates joints
B	Direct end-effector control	EMG cycles through Cartesian axes; Jacobian inv.
C	Gaze-assisted shared control	2D gaze → object probability; OMPL path plan
D	Context-assisted shared control	Gaze + Bayesian fusion with task-state prior

Mode A: Direct Joint Control

• Joint selection: "hand-open" (HO) EMG gesture cycles across joints.
• Movement: "wrist-flex" (WF) or "wrist-extend" (WE) increments/decrements current joint.
• Grasp: "hand-close" (HC) toggles open/close.

Mode B: Direct End-Effector Control

• Axis selection via HO.
• Cartesian twist $(\dot{x}, \omega \in \mathbb{R}^6)$ command via WF/WE, mapped to joints through Jacobian pseudoinverse:

$$q_{t+1} = q_t + J(q_t)^\# \begin{bmatrix} \dot{x} \ \omega \end{bmatrix} \Delta t$$

• Grasp toggled with HC.

Mode C: Gaze-Assisted Shared Control

• For each block $i$ (pixel coordinate $p_i$), estimate

$$p(g \mid a_i) \propto \exp\bigl(-\frac{1}{2\sigma^2}\|g-p_i\|^2\bigr)$$

where $g$ is user gaze in Unity's image plane.

• Highest-probability object is automatically highlighted; selection is locked and OMPL plans executed on user command.
• "No-motion" (NM) gesture aborts planning.

Mode D: Context-Assisted Shared Control

• Mode C plus Bayesian update with task state $s$:

$$p(a_i \mid g, s) \propto p(g \mid a_i) \, p(a_i \mid s)$$

• Priors $p(a_i \mid s)$ are set based on recent pick color, e.g., Table 1 in (Guptasarma et al., 2024), biasing intent inference toward the correct sequence.

4. Sequential Task Modeling

A minimal two-state variable $s \in \{\text{last‐Red},\text{last‐Blue}\}$ encodes sequential context, updated on each successful block drop and biasing the intent prior as above. This approximates a first-order Markov model over block-color transitions. The approach allows ProACT to test not just isolated movements but also performance on temporally structured tasks, representing a critical advance for prosthesis control research.

5. Experimental Design, Metrics, and Results

User Study Protocol

• Subjects: Eight able-bodied, right-handed adults (5 male, 3 female, ages 22–30).
• Design: Within-subjects; each completed three 5-min Box-and-Blocks Task trials in each mode (A, B, C, D).
• Task: Move four colored, numbered cylinders (R1→B1→R2→B2) in a fixed sequence over a 0.75 m vertical partition.
• Trial randomization counteracted order effects.

Evaluation Metrics

• Success Rate: Number of correctly delivered blocks per trial (max 4).
• Failure Types: Dropped on floor, dropped wrong side, crossed but not placed, wrong color order.
• Pick/Place Durations: Timing of grasp–release and release–next grasp.
• Placement Accuracy: Proximity to target center.
• Subjective Ratings (5-point Likert): Ease, agency, perceived speed.

Results Summary

Mode	Mean Successes/trial ($\pm$ SD)	User-preferred modes	Notable Findings
A	2.21 $\pm$ 1.50	–	High variance, slow/erratic picks
B	1.96 $\pm$ 1.60	–	Comparable or lower to joint control
C	3.29 $\pm$ 0.95	High (top-ranked)	Fewer failures, improved consistency
D	3.67 $\pm$ 0.56	6/8 participants	Best uniformity, top for ease, agency, speed

• Modes C and D (gaze/context shared autonomy) reduced all error types, improved user ratings, and standardized performance across participants.
• Pick durations were faster and more consistent with gaze/context modes; place and precision metrics were near parity once blocks were in grasp.
• Six of eight users ranked Mode D as preferred and top-ranked for all subjective measures.

6. System Availability, Extensibility, and Impact

• Resource: Open-source at https://armlabstanford.github.io/proact
• Interoperability: Modular; supports any URDF-compatible manipulator, new intent prediction algorithms (e.g., EMG regression, vision-based classifiers), and flexible augmentation for new tasks or entire amputee trials.
• Research Utility: By leveraging standard robotics (ROS, Gazebo, MoveIt, OMPL, KDL) and AR/VR (Unity, HoloLens), ProACT reduces barriers for heterogeneous research teams and fosters replicable, extensible experimentation.

ProACT establishes the first platform for end-to-end, mixed-reality, intelligent whole-arm prosthesis research. Empirical evidence demonstrates that even simple gaze- and context-based intent inference can yield substantial improvements in both performance and user experience over direct myoelectric control. The testbed enables future studies across a range of intelligent prosthetic algorithms, user populations, and task complexities (Guptasarma et al., 2024).