GOTT: Framework for Dexterous Manipulation

• The framework GOTT is a dexterous-manipulation system that integrates robot-agnostic object motion and reach specifications, converting them into precise multi-fingered robot execution with a robust, closed-loop contact acquisition system known for its stability, despite various errors and imperfections during execution.
• GOTT decomposes manipulation into three stages: reach (approach and initialization), acquire (stable contact), and move (object trajectory tracking), enhancing the success of dynamic manipulation tasks across varied robot arm hand configurations and motion blindspots.
• The GOTT policies show high success on lifting, pouring, and erasing in laboratory conditions using a variety of robots and hand types. The best average success was 96% in MPEG lifting while in freezing of object motion.
• Wherever possible, Falls are predicted and object motion predicted with realtime feedback control switching
• Use of several visual and contact sensors to assess the performance and contact entry condition.
• Flexible to integrate execution policy adjustments from future aware prediction and planning data from human actions, thus optimizing object specific reach parameters.

GOTT (“General Object Trajectory Tracking”) is a dexterous-manipulation framework that converts robot-agnostic object motion and an approximate reach specification into multi-fingered robot execution. Its central mechanism is a reusable, closed-loop contact-acquisition primitive that separates manipulation into reach, acquire, and move. The framework is designed to establish stable opposing contact despite imperfect initialization, object-pose error, object displacement, geometry mismatch, and differences among robot arms and hand morphologies (Liu et al., 2 Oct 2026).

1. Problem formulation and design principles

GOTT represents manipulation intent primarily through a desired object-pose trajectory,

$\tau_o=\{T_{O,t}\}_{t=0}^{T},$

where $T_{O,t}\in SE(3)$ is the desired pose of the object frame at time $t$. This representation is robot-agnostic: tasks such as sweeping, drilling, pouring, or erasing can be specified by object motion rather than by the joint trajectories of a particular robot.

An object trajectory alone does not specify the hand–object contact region, finger assignments, contact-establishment strategy, collision avoidance, or compensation for object-pose and geometry errors. A stable geometric grasp may also be functionally inappropriate. For example, grasping a drill by its chuck can prevent tool use, while a loose brush-handle grasp may fail during sweeping.

GOTT addresses this interface problem by requiring the upstream system to provide only an approximately reachable, task-relevant initialization. The local contact-rich interaction is delegated to a shared closed-loop policy. Its core interface is

$$(\tau_o,\rho)\longrightarrow \text{robot execution},$$

where $\rho$ is a reach specification describing how the hand should approach an appropriate contact region.

This decomposition differs from open-loop grasp execution. Open-loop methods assume that the object pose, approach trajectory, final hand configuration, and resulting contacts are sufficiently accurate before execution begins. GOTT instead allows the initial configuration to be a pre-grasp or approximate contact configuration and uses feedback to refine it.

2. Reach–acquire–move execution

GOTT executes manipulation in three sequential stages.

2.1 Reach

The reach specification is

$\rho=(\xi_{\mathrm{app}},g_{\mathrm{init}}),$

where $\xi_{\mathrm{app}}$ is an approach trajectory and $g_{\mathrm{init}}$ is an approximate hand configuration at the end of that approach. The latter is explicitly not required to be a final grasp.

The system switches from reach to acquisition when the minimum distance between the hand links and the object surface satisfies

$$d_{\min}(\text{hand},\partial\mathcal M)<\epsilon_{\mathrm{approach}},$$

where $\mathcal M$ is the object mesh. The upstream planner remains responsible for functional contact selection, collision-free approach, and downstream reachability, but it need not synthesize a perfectly executable multi-finger grasp.

2.2 Acquire

The contact-acquisition policy receives local hand–object geometry and proprioceptive history and outputs incremental wrist and joint commands,

$T_{O,t}\in SE(3)$0

The policy operates closed loop and can respond to object-pose estimation error, object displacement, imperfect hand initialization, mesh-to-object mismatch, and uncertain or delayed contact.

Acquisition terminates through a separate proprioceptive contact predictor. Stable contact requires the thumb and at least one non-thumb finger to be predicted in contact for five consecutive control steps. This criterion encodes opposing contact rather than merely proximity or enclosure.

2.3 Move

After stable contact is detected, the robot performs a short lift and tracks $T_{O,t}\in SE(3)$1 with a pose-conditioned controller. If $T_{O,t}\in SE(3)$2 denotes the estimated hand–object transform, a target hand pose is obtained from the desired object pose and the estimated grasp transform. The target is additionally corrected using observed object-pose error.

Conceptually, the target can be expressed as

$T_{O,t}\in SE(3)$3

with an additional correction based on

$T_{O,t}\in SE(3)$4

The move stage assumes that the hand–object transform remains approximately fixed after acquisition. This makes the framework suitable for grasp-and-move tasks, but limits deliberate in-hand manipulation, finger gaiting, unscrewing, and regrasping.

3. Reach-specification generation

GOTT keeps the contact-acquisition primitive and tracking backend fixed while allowing reach specifications to originate from different sources.

3.1 Future-aware object-centric planning

The default planner receives an object mesh, an object trajectory, and a task description:

$T_{O,t}\in SE(3)$5

A vision-LLM processes four rendered object views and identifies functional parts and task-relevant grasp points. The selected two-dimensional points are back-projected onto the object mesh and aggregated into a task-relevant surface region

$T_{O,t}\in SE(3)$6

The implementation forms the final region from the union of $T_{O,t}\in SE(3)$7 cm surface neighborhoods around consistent back-projected points. For drill use, functional reasoning should select the pistol grip rather than the chuck.

BODex’s lightweight initialization procedure samples $T_{O,t}\in SE(3)$8 candidate hand poses and joint configurations:

$T_{O,t}\in SE(3)$9

Each candidate is approximate rather than a final grasp. GOTT then:

1. plans a collision-free approach trajectory using cuRobo;
2. assumes a temporary rigid hand–object attachment;
3. rolls out the hand motion implied by $t$0;
4. solves arm inverse kinematics over the entire trajectory;
5. rejects candidates with infeasible approaches, joint-limit violations, collisions, or failed inverse kinematics.

Among valid candidates, it selects the one minimizing cumulative arm motion,

$t$1

This is future-aware because initial candidates that become infeasible during subsequent object motion are rejected.

3.2 Human and external specifications

Reach specifications can also be supplied by human demonstrations or external models. Such sources may provide a task-relevant contact region, an initial hand configuration, an approach trajectory, or a reference hand–object trajectory.

For DexYCB and OakInk demonstrations, the human approach segment is retargeted to the robot hand to obtain $t$2 and $t$3. The endpoint need not be an exact grasp because the acquisition policy refines it.

Object trajectories can originate from tracked human demonstrations using FoundationPose, generated videos with depth estimation and pose tracking, or manually specified object-pose keyframes interpolated into trajectories. The contact-acquisition and tracking components remain unchanged across these sources.

4. Contact-acquisition primitive and cross-embodiment control

4.1 Training and control setup

The primitive is trained with PPO in IsaacLab. Training uses:

• 16 YCB objects;
• multiple stable resting poses per object;
• 4,000 initial hand configurations per object–resting-pose pair;
• XHand, Sharpa Wave, and Allegro hands;
• 4,096 parallel environments;
• 2,000 PPO epochs;
• 60 control steps per episode;
• a 10 Hz control rate;
• approximately $t$4 million environment steps;
• approximately 24 hours on one NVIDIA L40S GPU.

The policy must establish contact, lift the object at least $t$5 cm, and maintain stability. Domain randomization varies object scale, mass, friction, restitution, robot-link friction and restitution, actuator stiffness and damping, and random push forces.

The observation contains per-link hand–object displacement, canonical hand-link positions, canonical joint positions, previous joint targets, and joint and link masks. Hand–object geometry is represented using $t$6 sampled object-surface points. For canonical link $t$7 at position $t$8, the nearest-object displacement is

$t$9

where $$(\tau_o,\rho)\longrightarrow \text{robot execution},$$0 is the object point cloud transformed into world coordinates.

The policy uses five frames of proprioceptive history, $$(\tau_o,\rho)\longrightarrow \text{robot execution},$$1, and produces a 32-dimensional action:

$$(\tau_o,\rho)\longrightarrow \text{robot execution},$$2

with

$$(\tau_o,\rho)\longrightarrow \text{robot execution},$$3

The first six components specify wrist translation and rotation deltas; the remaining 26 specify canonical hand-joint deltas. Actions are clipped to $$(\tau_o,\rho)\longrightarrow \text{robot execution},$$4 and converted to residual target commands. Wrist translation and rotation are scaled to $$(\tau_o,\rho)\longrightarrow \text{robot execution},$$5 m and $$(\tau_o,\rho)\longrightarrow \text{robot execution},$$6 rad per control step, respectively. The policy is an MLP with hidden layers $$(\tau_o,\rho)\longrightarrow \text{robot execution},$$7.

4.2 Reward structure

The reward combines distance, contact, lift, and object-stability terms:

$$(\tau_o,\rho)\longrightarrow \text{robot execution},$$8

The weights are

$$(\tau_o,\rho)\longrightarrow \text{robot execution},$$9

with object-stability weights annealed from $\rho$0 to $\rho$1 over $\rho$2 steps.

The distance term prioritizes fingertips and the thumb:

$\rho$3

The contact reward is

$\rho$4

with force-based contact threshold $\rho$5. Grasping requires thumb contact and contact by at least one non-thumb finger.

The lift reward activates only after opposing contact and rewards a lift of up to $\rho$6 cm:

$\rho$7

where

$\rho$8

Planar displacement and orientation stability are encouraged through

$\rho$9

and

$\rho=(\xi_{\mathrm{app}},g_{\mathrm{init}}),$0

where

$\rho=(\xi_{\mathrm{app}},g_{\mathrm{init}}),$1

4.3 Canonical representation across hands

GOTT maps different hands into a shared representation of 21 canonical links and 26 canonical joints. The links comprise one palm keypoint and four keypoints for each of five fingers. Missing links are filled using the nearest available links and indicated by a binary link mask.

The joint representation contains six thumb slots and five slots for each of the four other fingers. Native hand joints are mapped through an index map $\rho=(\xi_{\mathrm{app}},g_{\mathrm{init}}),$2 and sign vector $\rho=(\xi_{\mathrm{app}},g_{\mathrm{init}}),$3:

$\rho=(\xi_{\mathrm{app}},g_{\mathrm{init}}),$4

Missing slots are zero-padded and masked. The active degrees of freedom are 12 for XHand, 16 for Allegro, and 22 for Sharpa Wave. No hand-specific encoder or decoder is used.

Canonicalization alone is not sufficient for morphology transfer. The experiments indicate that successful zero-shot transfer depends strongly on joint training across multiple hand morphologies.

4.4 Contact switching

A separate two-layer MLP contact predictor receives five-frame histories of joint positions, joint targets, and link positions. It predicts five finger-contact probabilities:

$\rho=(\xi_{\mathrm{app}},g_{\mathrm{init}}),$5

At inference, probabilities are thresholded at $\rho=(\xi_{\mathrm{app}},g_{\mathrm{init}}),$6, and the system switches to the move stage when the thumb and at least one other finger are predicted in contact for five consecutive steps. The predictor achieved per-step false-positive rates of $\rho=(\xi_{\mathrm{app}},g_{\mathrm{init}}),$7–$\rho=(\xi_{\mathrm{app}},g_{\mathrm{init}}),$8, false-negative rates of $\rho=(\xi_{\mathrm{app}},g_{\mathrm{init}}),$9–$\xi_{\mathrm{app}}$0, per-episode switch accuracy of $\xi_{\mathrm{app}}$1–$\xi_{\mathrm{app}}$2, and median switching lag of at most one step.

5. Evaluation and empirical results

5.1 Simulation lifting

The simulation evaluation uses 9,588 DexGraspNet object assets, 54,149 object-scale-pose instances, four object scales, and XHand, Sharpa, and Allegro hands. Success requires stable contact, a lift of at least $\xi_{\mathrm{app}}$3 cm, and stability over the evaluation horizon.

Method	Setting	Average success
BODex	Open loop	38.7%
DRO	Open loop	40.5%
CrossDex	Specialist	46.6%
CrossDex	Unified	51.7%
GOTT	Specialist	75.4%
GOTT	Unified	78.7%

The unified GOTT policy achieves $\xi_{\mathrm{app}}$4 average success. It was trained on only 16 YCB objects but evaluated on 9,588 DexGraspNet assets.

5.2 Real-world acquisition

Real-world experiments use Franka Research 3 and Tianji arms with XHand and Sharpa Wave hands. Object meshes are captured with an iPhone, SAM2 performs segmentation, and FoundationPose estimates and tracks six-dimensional object pose at $\xi_{\mathrm{app}}$5 Hz. GOTT runs at $\xi_{\mathrm{app}}$6 Hz.

Each platform performs 50 trials over ten objects, with success requiring a $\xi_{\mathrm{app}}$7 cm lift and stable holding for at least three seconds:

• Franka–Sharpa: $\xi_{\mathrm{app}}$8;
• Franka–XHand: $\xi_{\mathrm{app}}$9;
• Tianji–XHand: $g_{\mathrm{init}}$0.

The tested objects include five seen and five unseen objects.

5.3 Unseen hand morphologies

The unified policy is evaluated zero-shot on Allegro variants with removed finger links. Its average success is $g_{\mathrm{init}}$1, compared with $g_{\mathrm{init}}$2 for per-variant oracle policies. An Allegro-only policy achieves only $g_{\mathrm{init}}$3 average success on these variants. The result indicates that cross-morphology training, together with canonicalization, is important for transfer.

5.4 End-to-end object-centric tasks

Four real-world tasks are evaluated on Franka–Sharpa: trash cleanup with a brush, mustard pouring, whiteboard erasing, and drill use. Each is repeated ten times.

Method	Average grasp success	Average task success
DRO	57.5%	35.0%
BODex	57.5%	35.0%
GOTT	87.5%	80.0%

GOTT therefore improves average end-to-end task success from $g_{\mathrm{init}}$4 to $g_{\mathrm{init}}$5. Baseline failures include unsuitable contact regions, object displacement during approach, cage-like grasps without firm opposing contact, and slipping during object motion. DRO achieves $g_{\mathrm{init}}$6 on trash cleanup because its loose brush-handle grasp does not survive sweeping.

5.5 Human demonstrations and workspace robustness

All 499 left-hand DexYCB sequences are tested after retargeting the human approach to XHand. GOTT achieves $g_{\mathrm{init}}$7–$g_{\mathrm{init}}$8 success across IsaacLab, ManiSkill, and MuJoCo, compared with $g_{\mathrm{init}}$9 for kinematic replay in MuJoCo. SPIDER achieves $$d_{\min}(\text{hand},\partial\mathcal M)<\epsilon_{\mathrm{approach}},$$0, but requires physics-informed optimization for each demonstration and reoptimization when object states or trajectories change.

In 600 simulated SimToolReal eraser episodes, SimToolReal achieves $$d_{\min}(\text{hand},\partial\mathcal M)<\epsilon_{\mathrm{approach}},$$1 near its training configuration but drops to $$d_{\min}(\text{hand},\partial\mathcal M)<\epsilon_{\mathrm{approach}},$$2 below table offset $$d_{\min}(\text{hand},\partial\mathcal M)<\epsilon_{\mathrm{approach}},$$3 m. GOTT achieves $$d_{\min}(\text{hand},\partial\mathcal M)<\epsilon_{\mathrm{approach}},$$4 near the training configuration and $$d_{\min}(\text{hand},\partial\mathcal M)<\epsilon_{\mathrm{approach}},$$5 below that offset. GOTT’s failures are mainly associated with arm inverse-kinematics singularities under constrained end-effector orientations.

5.6 Ablation findings

The full unified policy achieves $$d_{\min}(\text{hand},\partial\mathcal M)<\epsilon_{\mathrm{approach}},$$6 average success. Removing link-position observations reduces this to $$d_{\min}(\text{hand},\partial\mathcal M)<\epsilon_{\mathrm{approach}},$$7, while using only one history frame reduces it to $$d_{\min}(\text{hand},\partial\mathcal M)<\epsilon_{\mathrm{approach}},$$8. Removing the contact reward or lift reward reduces average success to $$d_{\min}(\text{hand},\partial\mathcal M)<\epsilon_{\mathrm{approach}},$$9 and $\mathcal M$0, respectively. Removing the distance reward gives $\mathcal M$1 average success on training hands but reduces unseen-Allegro performance from $\mathcal M$2 to $\mathcal M$3.

These results associate performance with temporal proprioceptive history, explicit per-link geometry, contact reward, and lift reward. The distance term has a smaller effect on training-hand success but helps avoid finger inversion and distorted bending and improves morphology transfer.

6. Scope, limitations, and technical significance

GOTT’s sequential decomposition does not eliminate high-level planning failures. If the reach specification is poor, the acquisition policy may fail rather than request a revised plan. The approach also depends on SAM2 segmentation, FoundationPose tracking, calibrated hand kinematics, object meshes, and the mapping into canonical link and joint representations. Severe hand–object occlusion can corrupt pose estimates, and the current system does not use tactile or visuotactile sensing.

The move controller assumes an approximately fixed hand–object transform,

$\mathcal M$4

which limits deliberate in-hand manipulation. Arm-specific reachability and inverse-kinematics feasibility remain necessary; the workspace experiments demonstrate that GOTT can still fail because of arm singularities or constrained orientations. The method is primarily evaluated on rigid-object grasp acquisition followed by object motion and does not establish universal performance for deformable objects, highly dynamic manipulation, tactile-only manipulation, or arbitrary changes in hand topology.

The central technical contribution is an interface between object-centric intent and embodiment-specific control:

$\mathcal M$5

Its empirical significance lies in treating contact acquisition, rather than only grasp synthesis or trajectory tracking, as a reusable dexterous-control primitive. The reported results include $\mathcal M$6 average simulated lifting success for the unified policy, $\mathcal M$7, $\mathcal M$8, and $\mathcal M$9 real-world lifting success across three arm–hand platforms, $T_{O,t}\in SE(3)$00 average end-to-end task success, and $T_{O,t}\in SE(3)$01 zero-shot success on unseen Allegro variants. GOTT consequently provides a modular mechanism for connecting foundation-model, human-demonstration, video-derived, or manually specified object-centric intent to reliable multi-fingered robot execution.