Fingers as Legs: Learning Self-Supported Locomotion and Manipulation with an Anthropomorphic Hand

Abstract: A walking robotic hand must use the same fingers to move its body, support its weight, and interact with the environment. We show how an anthropomorphic hand can learn these skills while retaining its finger design and position controller. Onboard power and computation make the platform self-contained. Our reinforcement learning approach accounts for the hand's unequal fingers, with training in a simulator calibrated from hardware measurements. In simulation, the hand moves faster with our reward formulation than with tuned rewards originally designed for quadrupeds. On hardware, task-specific policies enable untethered crawling, steering, and fall recovery. While supporting its own weight, the hand also executes successive keyboard commands without vision and pushes an object to targets using overhead visual feedback. These results demonstrate a compact mobile manipulator that reuses its fingers for locomotion and interaction, without a separate locomotion mechanism.

• The paper demonstrates that an unmodified anthropomorphic hand can autonomously locomote and manipulate objects by reusing fingers for both tasks.
• A hardware-calibrated structured reward system launches locomotion based on the finger's settling stance, enabling mobility over diverse surfaces and enhancing non-nail participation
• Specific tasks included 29 successful keyboard presses, 15 visually guided object deliveries, and 28 of 32 simulated recovery tasks.
• follow_up_questions”: [“What specific tasks did the anthropomorphic hand successfully perform?”, “How does the policy inference system work in this context?”, “What are the performance metrics for each experiment and specific tasks discussed?”, “How does the hardware-calibrated reward system improve maneuverability?”, “Find recent papers about self-supported locomotion and manipulation studies.” ] } 'eightpoint_frows'nguagfe' ] 'te see'ts'ou knowdledgKfe} KurtzcOwsoreafaantv-aastian-tangentractions' fi'fingers' movingvpositionco--m'ders---prmove!inegerboostelea'-oradeakquated-maciantiit!Somethingouastic-meanwithout'-responsible'fievrreeatr;t!linguaus}-.....falsiacitationNavigationgrabthdflift-lastlfutour-

System concept and research objective

“Fingers as Legs: Learning Self-Supported Locomotion and Manipulation with an Anthropomorphic Hand” investigates whether a conventional anthropomorphic hand can function as an untethered mobile manipulator without altering its finger morphology or factory position controller (2609.17172). The central constraint is that the same five fingers must alternately or simultaneously provide propulsion, support the $818$-gram body, and execute manipulation actions. This differs from designs that add dedicated legs, use symmetric modular digits, or reconfigure a hand into a different locomotion morphology.

The platform is an off-the-shelf WUJI right hand with 20 actuated joints, four per finger. The original hand weighs $738$ grams; a dorsal module adds $80$ grams and integrates a Raspberry Pi Zero 2 W, an IMU, a battery, communication electronics, and safety supervision. Policy inference runs at $50$ Hz, while the serial driver operates at $500$ Hz. The hand retains its non-backdrivable actuators, factory controller gains, finger kinematics, and position-command interface. Consequently, the learning system does not directly command torques: it outputs incremental joint-position targets that are filtered and tracked by the embedded controller.

The morphology creates a substantially different learning problem from standard quadrupedal locomotion. The thumb is opposed to the other digits, the fingers have unequal reach and kinematic structure, and the palm rests at a non-horizontal angle. These properties prevent straightforward use of symmetry-based policy architectures and make generic quadruped footfall assumptions inappropriate. The paper therefore treats the hand’s settled fingertip configuration as the primary geometric reference and learns locomotion around that morphology.

Hardware-calibrated learning framework

All task policies are feedforward multilayer perceptrons trained with PPO in Isaac Lab and deployed as ONNX models on the onboard computer. The actor receives proprioceptive observations consisting of joint positions, gravity direction, angular velocity, previous policy output, and task-specific variables. Each observation term is represented by an eight-sample temporal history. The actor outputs bounded incremental joint-position commands, with task-dependent scales of $0.060$ radians for crawling and recovery, $0.028$ radians for keyboard pressing, and $0.040$ radians for pushing.

A significant methodological component is the calibration of simulation to the behavior of the hand’s existing position controller. Hardware measurements identify a closed-loop delay of approximately $19$ ms, a $3$ Hz command-filter cutoff, filtered joint speeds of approximately $738$0 and $738$1 rad/s for flexion and abduction, respectively, and a kinetic fingertip friction coefficient of $738$2, with measured values ranging from $738$3 to $738$4. The measured joint stiffness is $738$5 times the nominal uncalibrated model stiffness. These parameters are incorporated into simulation together with command delay, actuator variation, friction randomization, inertial variation, center-of-mass perturbations, and IMU bias.

This calibration is important because the deployed policy acts through a filtered position-control loop rather than an idealized torque interface. The simulation-to-real strategy consequently models not only rigid-body dynamics and contact, but also the controller-induced dynamics that determine how rapidly a fingertip can move and how much force it can sustain. The remaining sim-to-real gap is addressed through domain randomization around the hardware estimates, including fingertip friction in the interval $738$6, actuator-gain scaling in $738$7, and payload center-of-mass offsets.

Morphology-adapted locomotion reward

The main technical contribution is a locomotion reward based on stance geometry rather than prescribed gait timing. The method first obtains a nominal crawl stance by settling the hand with an $738$8-gram payload. A stance-calibrated frame $738$9 removes the nominal palm tilt while remaining attached to the root orientation. Commands, fingertip positions, and velocities are expressed in this frame, allowing the policy to reason about motion relative to a level reference without artificially constraining whole-body orientation.

For each fingertip, the method stores an individual nominal position. The footprint objective penalizes deviations from these five positions, with lateral and vertical errors weighted four times more strongly than fore-aft errors. The resulting objective behaves as a set of anisotropic virtual springs: it maintains a stable support arrangement while leaving the fingers freedom to move along the travel direction. Crucially, the objective does not specify ground contact locations or a fixed sequence of footfalls. The policy must learn which finger steps, when it steps, and how the remaining contacts stabilize the body.

Two auxiliary terms shape the learned behavior. A lift objective encourages command-dependent stepping frequency, increasing from approximately $80$0 to $80$1 Hz over the relevant command-speed range. A direction objective discourages airborne fingertips from moving opposite to the commanded direction. The reward also includes planar and yaw-velocity tracking, undesired-contact penalties, vertical-motion and angular-rate penalties, action-rate regularization, and effort-related terms. Yaw tracking is introduced only after a forward gait has developed, avoiding simultaneous optimization of stepping and turning.

The reward design explicitly rejects the assumption that periodic timing should be imposed externally. Standard legged-robot rewards often encode swing intervals, phase relations, or predefined foot-position targets. In contrast, the present formulation anchors each digit to its own morphological stance and leaves temporal coordination to the policy. This is particularly appropriate for an asymmetric hand, but it also makes the learned behavior dependent on the chosen reference stance and on the contact dynamics represented during training.

Simulation ablations and quantitative evidence

The reward study uses 12 training seeds, 4,096 parallel environments, 8,000 PPO iterations, and 256 randomized evaluation episodes per configuration. The authors compare their formulation against raw and tuned versions of an ANYmal-D quadruped reward adapted to the hand, as well as variants that remove the footprint, lift, direction, or combined lift-and-direction objectives.

The proposed formulation moves faster than the tuned stock reward by $80$2 cm/s, with a 95% confidence interval of $80$3 cm/s, and is faster in 10 of 12 seeds. This result is notable because the comparison is not merely between a generic baseline and an uncalibrated method: the tuned stock reward was selected through an additional reward-weight search. The implication is that quadrupedal reward structure is not directly transferable to this asymmetric anthropomorphic morphology, even after manual tuning.

The footprint objective accounts for the clearest supported performance gain. Removing it decreases task speed by $80$4 cm/s, with a confidence interval of $80$5 cm/s, and reduces five-finger participation. The proposed formulation also improves non-nail participation by $80$6, with a confidence interval of $80$7. This distinction matters because total contact frequency can conceal whether the hand is relying on anatomically unfavorable nail-side contacts. The tuned stock reward achieves higher mean five-finger participation in some comparisons, but it produces more nail-side planting in every seed.

The ablations do not establish an independent speed benefit for the lift or direction objectives. Removing the direction term increases five-finger participation by $80$8, with an interval of $80$9, while its posture effects remain uncertain. The authors retain the direction term because it was part of the hardware-trained policy, but the simulation evidence does not justify attributing the principal locomotion improvement to it. This is an important qualification: the paper supports the footprint objective as the central contribution, whereas the auxiliary terms are primarily shaping mechanisms whose causal value remains unresolved.

Reward-weight sweeps further show that footprint and lift terms affect contact behavior differently. Doubling the footprint weight yields a $50$0 higher worst-tip planting fraction than doubling the lift weight, with a confidence interval of $50$1 across 10 of 12 seed-matched comparisons. Increasing the footprint weight beyond its deployed value continues to improve worst-tip planting, but speed changes are uncertain. Mean five-finger participation peaks at twice the deployed footprint weight and decreases at $50$2 times the deployed value. Thus, the reward exhibits a non-monotonic trade-off between support participation and locomotion performance.

Untethered locomotion and recovery

On hardware, the hand crawls without external power or computation across 14 surfaces, including rubber, carpet, hardwood, tile, metal grating, asphalt, concrete, stone, artificial turf, grass, and gravel. These demonstrations establish environmental breadth but are qualitative rather than controlled measurements of speed, energy, or success probability across surfaces.

The hand’s steering behavior is asymmetric. Without a steering command, it drifts right at approximately $50$3/s. Open-loop commands produce command-scaled right turns, whereas left-turn rates plateau. Across 21 trajectories, the mean path speed is $50$4 m/s. A proportional controller using onboard IMU heading compensates for the drift and closes the heading loop. The hand reaches both $50$5 heading steps and a $50$6 right turn, but the $50$7 left turns fall short because the drift-canceling offset pushes the effective command beyond the trained range. The result demonstrates heading regulation in both directions, while also exposing the limited symmetry and command envelope of the learned controller.

Fall recovery is handled by a separate PPO policy trained from randomized side-lying configurations. In simulation, the complete recovery procedure rights the hand in 28 of 32 trials, with a median recovery time of $50$8 s and mean absolute joint error of $50$9 radians relative to the crawl stance. On hardware, recovery succeeds in 21 of 25 trials, corresponding to $500$0, including 11 of 14 thumb-side falls and 10 of 11 wrist-side falls. The four failures occur when fingers catch on one another and stall. Once the hand is upright, an attitude and angular-velocity detector initiates a $500$1-second joint-target ramp into the crawl stance. The recovery policy therefore extends the system beyond locomotion from a prepared initial condition, although it remains vulnerable to self-entanglement during contact-rich righting motions.

Self-supported manipulation

The keyboard task evaluates whether the hand can remove individual fingers from support, press keys, and preserve its body stance without visual feedback. The task policy receives a requested-key indicator and the nominal pressing fingertip position in the stance-calibrated frame. The keyboard is manually aligned before each evaluation block, and encoder-based forward kinematics provides the pressing fingertip location. Translation of the hand relative to the keyboard is not estimated online.

Over 32 consecutive commands across $500$2 seconds, the hand executes 29 correct presses, with a median command-to-keystroke latency of $500$3 s. The maximum body tilt is $500$4. All three failures involve the Up key and activate the adjacent Right Shift key, indicating a systematic press-location offset rather than stochastic loss of support. The same physical-keyboard interface successfully executes optimal nine-move and twelve-move Sokoban solutions for one- and two-box levels. These results demonstrate sequential self-supported interaction, but the absence of vision and the need for manual initial registration substantially constrain the task.

The pushing experiment adds overhead visual feedback. A camera tracks a dorsal hand marker and a $500$5 mm PLA cube weighing $500$6 grams. The policy receives the object position relative to the hand, world-frame object velocity, and object-to-target displacement in the stance-calibrated frame. Training includes camera latency of one to three control steps, 3% observation dropout, and 2 mm position noise.

One policy learns approach, contact, and transport jointly. Across 15 deliveries with target distances from 10 to 40 cm, the final target error averages 17 mm, with a range of 5 to 37 mm. A delivery is counted only if the cube enters a 2 cm target radius and remains within 5 cm for one second. The result shows that locomotion and manipulation can be coupled without allocating a dedicated support limb. It does not, however, establish robustness to clutter, occlusion, varying object friction, or target motion.

Limitations and open questions

The evaluation relies on task-specific policies rather than a unified locomotion-manipulation policy. Crawling, recovery, keyboard pressing, and pushing each have separate actors, and the transitions between them are externally structured. The hand therefore demonstrates a collection of compatible capabilities rather than autonomous task-level sequencing.

The manipulation experiments also depend on external setup. Keyboard operation requires manual alignment and assumes that alignment remains valid throughout the sequence. Object pushing uses an overhead camera, so the perception system is not onboard and the task is restricted to the camera’s workspace. The conclusion that the hand is self-contained applies to power and computation for policy execution, not to all sensing used in the manipulation demonstrations.

Locomotion remains asymmetric and only partially characterized. The hand exhibits a persistent rightward drift, unequal turning rates, and an effective command limitation that prevents some left-turn trials from reaching their targets. The 14-surface evaluation is qualitative, and the paper does not report systematic success rates, energy consumption, slip statistics, or speed distributions by surface type.

The reward ablations also leave a causal question open. They provide evidence for the footprint objective, but not for an independent benefit of the direction objective, and they do not fully disentangle the lift term from curriculum effects and other reward components. Hardware validation is performed only with the complete deployed reward formulation, not with matched hardware policies trained under the ablated alternatives. Finally, recovery fails in $500$7 of hardware trials, primarily because of finger interference, indicating that the contact topology of self-righting remains insufficiently controlled.

Conclusion

The paper demonstrates that an unmodified anthropomorphic hand can serve as an untethered mobile manipulator by reusing its fingers for propulsion, support, and interaction. Its principal methodological result is a stance-calibrated footprint reward that accommodates unequal finger geometry without imposing mirrored morphology or a fixed gait schedule. Hardware-calibrated simulation enables transfer through the hand’s existing filtered position controller, while onboard power and computation support untethered deployment.

The strongest quantitative evidence concerns the footprint objective: it improves simulated task speed relative to tuned quadrupedal rewards, increases non-nail participation, and produces the clearest ablation gains. Hardware experiments further establish crawling across diverse surfaces, $500$8 fall-recovery success, 29 correct presses in 32 keyboard commands, and 15 visually guided object deliveries with 17 mm mean final error. The remaining questions concern unified policy composition, autonomous registration and perception, robustness to contact interference, and systematic characterization of locomotion across surfaces and disturbances.