Geometry-aware Dexterous Pushing & Pulling

• The paper introduces a framework that synthesizes pre-contact hand poses conditioned on object geometry, enabling effective nonprehensile pushing and pulling.
• It leverages compact shape descriptors and a conditional diffusion model to generate and validate candidate poses through collision checking and motion planning.
• The approach couples geometry-aware contact synthesis with downstream ranking, achieving higher success rates than grasp-transfer baselines on diverse objects.

Geometry-aware Dexterous Pushing and Pulling (GD2P) is a nonprehensile manipulation framework for multi-fingered robotic hands that treats pushing and pulling as a problem of synthesizing pre-contact hand poses conditioned on object geometry and pushing/pulling direction, then executing the resulting interactions through standard motion planning rather than direct end-to-end contact-dynamics control (Li et al., 22 Sep 2025). In this formulation, the central learned object is not a low-level torque policy but a distribution over hand configurations that are likely to produce effective pushes or pulls once the arm-hand system is translated along a prescribed direction. The framework is explicitly geometry-aware because it conditions pose generation on compact object-shape descriptors and evaluates candidates using collision checking, direction alignment, and task-space outcome measures.

1. Conceptual scope and problem setting

GD2P addresses tabletop manipulation of a single object on a flat surface, with the goal of moving, aligning, or repositioning that object without grasping it firmly. The task is motivated by settings in which direct grasping is difficult because of object geometry, size, or the relation between the object and the environment. In GD2P, dexterous hands are used as nonprehensile manipulators: they establish supportive, geometry-matched contacts with the object and then induce object motion through a straight-line arm-hand translation along a desired planar direction (Li et al., 22 Sep 2025).

A hand pose is defined as

$H = (\theta, T),$

where $\theta \in \mathbb{R}^d$ are the hand joint angles and $T \in SE(3)$ is the wrist pose. For the Allegro and LEAP setups discussed in the paper, $d=16$. Each trial is specified by a planar direction vector

$$u_{\rm dir} \in \mathbb{R}^3,\quad (u_{\rm dir})_z = 0,$$

and a target object position

$u_{\rm targ} \in \mathbb{R}^3.$

The hand is moved to a pre-contact pose and then translated along $u_{\rm dir}$ over a fixed distance of $0.2$ m. In simulation, a trial is successful if, after execution, the final object center satisfies

$$\| \mathbf{c}_{\rm final} - u_{\rm targ} \|_2 \le 0.03 \text{ m},$$

and the object orientation change is at most $45^\circ$ (Li et al., 22 Sep 2025).

This setup makes a precise methodological claim. GD2P does not attempt to learn the full nonprehensile dynamics directly. Instead, it reframes pushing and pulling as a pre-contact pose synthesis problem: if the hand approaches the object in a geometry-consistent pose, then a comparatively simple translational motion can produce useful manipulation outcomes. A common misunderstanding is to read GD2P as a generic dexterous control policy; more precisely, it is a hybrid pipeline that couples geometry-conditioned pose generation with downstream motion planning and execution.

2. Geometric representation and formal objective

The object is represented by a surface point cloud

$\theta \in \mathbb{R}^d$0

with configuration

$\theta \in \mathbb{R}^d$1

GD2P encodes geometry using a Basis Point Set (BPS) representation

$\theta \in \mathbb{R}^d$2

defined by a canonical set of basis points $\theta \in \mathbb{R}^d$3 and distances

$\theta \in \mathbb{R}^d$4

This yields a fixed-length representation that captures overall shape and scale (Li et al., 22 Sep 2025).

The objective is to find a pose $\theta \in \mathbb{R}^d$5 such that translating the arm-hand system along $\theta \in \mathbb{R}^d$6 moves the object close to $\theta \in \mathbb{R}^d$7 without undesirable events such as toppling, loss of contact, or collisions. Geometry-awareness enters at several levels. First, the BPS descriptor conditions the generative model, so the predicted pose distribution depends on object shape. Second, the optimization used to construct the training data explicitly biases the palm normal toward the pushing or pulling direction. Third, candidate poses are ranked partly by the alignment between palm orientation and desired manipulation direction.

The directional alignment term in the pose-synthesis energy is

$\theta \in \mathbb{R}^d$8

where $\theta \in \mathbb{R}^d$9 is the palm normal in world coordinates. Minimizing this term encourages

$T \in SE(3)$0

An additional arm-feasibility term

$T \in SE(3)$1

penalizes upward-pointing palm normals, which would suggest the arm is likely underneath and colliding with the table (Li et al., 22 Sep 2025).

The resulting interpretation is geometric rather than purely kinematic: the hand is asked to approach with a pose that is not merely reachable, but also shape-matched and direction-matched. This is why GD2P often uses palm support for flat objects, partial wraps for volumetric objects, and higher or lower contact placement depending on toppling risk.

3. Dataset construction and diffusion-based pose synthesis

GD2P relies on a large offline corpus of successful pushing and pulling poses. Candidate poses are first generated by contact-guided optimization, then filtered by physics-based validation in IsaacGym, and finally used to train a conditional diffusion model (Li et al., 22 Sep 2025).

For each hand, candidate contact points are predefined over the palm surface, finger links, and fingertips. In the Allegro example, fingertips use 96 candidates each, several finger links use 16 candidates each, and the palm uses 128 candidates. Given an object and manipulation direction, GD2P minimizes the energy

$T \in SE(3)$2

where $T \in SE(3)$3 is a force-closure estimator from DFC, $T \in SE(3)$4 penalizes hand-object distance, $T \in SE(3)$5 penalizes joint-limit violations, $T \in SE(3)$6 penalizes penetration and self-collision, and the last two terms impose directional alignment and arm feasibility. Optimization is performed with RMSProp and step-size decay, with Simulated Annealing added to escape local minima; diversity is increased by randomly resampling subsets of contact candidate indices per run (Li et al., 22 Sep 2025).

Each candidate is then validated by attaching the hand to the arm model, moving to the optimized pre-contact pose, executing a straight-line translation of length $T \in SE(3)$7 m along $T \in SE(3)$8, and checking the success criteria. Because optimization without dynamics modeling has low raw success rate, successful poses are augmented by perturbations generated via Halton sequences: rotation noise up to $T \in SE(3)$9, translation noise up to $d=16$0 mm, and joint-angle noise up to $d=16$1 rad. This yields the final training set of approximately 2.3k object meshes and approximately 1.3 million successful pushing and pulling hand poses (Li et al., 22 Sep 2025).

The generative model is a conditional diffusion network adapted from ConditionalUnet1D. For the Allegro setup, the pose vector has dimension $d=16$2, reported as pushingpose_dim = 25; the global conditioning input is the BPS descriptor with global_cond_dim = 4096. The DDPM scheduler uses num_diffusion_timesteps = 100, beta_schedule = squaredcos_cap_v2, and prediction_type = epsilon. Training uses batch size $d=16$3, $d=16$4 epochs, Adam with learning rate $d=16$5 and weight decay $d=16$6, gradient clipping at $d=16$7, and a cosine learning-rate scheduler with warmup (Li et al., 22 Sep 2025).

Methodologically, this stage situates GD2P between analytic grasp synthesis and policy learning. The offline optimizer and simulator define what successful contact geometries look like; the diffusion model then learns a conditional distribution over those geometries.

4. Test-time planning and execution

At test time, GD2P first constructs object geometry, then samples candidate poses from the diffusion model, filters them through arm-hand motion planning, ranks the feasible ones, and executes the best-scoring option (Li et al., 22 Sep 2025).

If the object mesh is known, points are sampled directly. For unseen daily objects, the paper uses a NeRF-based reconstruction pipeline consisting of multi-view RGB capture, COLMAP plus Nerfstudio for reconstruction, Stable Normal for normal maps, and 2D Gaussian Splatting to obtain a dense point cloud. BPS is then computed from the reconstructed point cloud. Candidate hand poses are sampled in the object frame and transformed to the world frame using the estimated object pose. In some simulation experiments, the system samples $d=16$8 candidates.

Full arm-hand trajectories are planned with cuRobo. For each candidate pose $d=16$9, cuRobo attempts to find a collision-free arm trajectory from a start posture to the desired wrist and hand configuration. If no such path exists, the pose is marked infeasible and discarded. The remaining poses are evaluated by

$$u_{\rm dir} \in \mathbb{R}^3,\quad (u_{\rm dir})_z = 0,$$0

where

$$u_{\rm dir} \in \mathbb{R}^3,\quad (u_{\rm dir})_z = 0,$$1

$$u_{\rm dir} \in \mathbb{R}^3,\quad (u_{\rm dir})_z = 0,$$2 is a binary collision indicator during simulated execution, and $$u_{\rm dir} \in \mathbb{R}^3,\quad (u_{\rm dir})_z = 0,$$3 reuses the palm-direction alignment term. The selected pose is

$$u_{\rm dir} \in \mathbb{R}^3,\quad (u_{\rm dir})_z = 0,$$4

This ranking stage is integral rather than incidental: the paper includes an ablation called GD2P w/o Ranking, and the reported performance drop shows that sampling alone is not sufficient (Li et al., 22 Sep 2025).

The framework also supports multi-step manipulation. A global planner such as RRT* computes a collision-free 2D path for the object center around obstacles; waypoints are transformed into the robot frame; for each path segment, GD2P is called again with a new direction $$u_{\rm dir} \in \mathbb{R}^3,\quad (u_{\rm dir})_z = 0,$$5. The hand pose is therefore re-synthesized at each segment rather than reused, which is consistent with the central premise that effective nonprehensile manipulation depends on local geometry and local direction, not just object identity.

5. Experimental evidence and empirical profile

The main real-world platform uses a Franka Emika Panda arm with an Allegro Hand in a $$u_{\rm dir} \in \mathbb{R}^3,\quad (u_{\rm dir})_z = 0,$$6 workspace. The object set comprises 8 3D-printed objects and 6 real everyday objects, including a vase, blender, bowl, cake box, cow, camera, lamp, bottle, black box, toy avocado, ranch bottle, spray bottle, coconut water, and cookie box. Each object is tested in 3 directions around a circle, with each direction attempted 5 times, except where some directions are infeasible because of kinematics. The paper also reports 840 real-world experiments with an Allegro Hand in comparison to baselines (Li et al., 22 Sep 2025).

Across the 14 evaluated objects and 3 directions, GD2P reports average success rates of approximately 61.4%, 47.1%, and 57.1% for the three directions. The ablation GD2P w/o Ranking reports approximately 50%, 40%, and 51.4%. The Nearest Neighbor baseline reports approximately 32.9%, 28.6%, and 32.9%, while Pre-Trained Grasp Pose reports approximately 27.1%, 31.4%, and 37.1%. These results are accompanied by qualitative observations: grasp-based baselines often produce contacts that are poorly aligned with the pushing direction, and nearest-neighbor retrieval can transfer hand poses from geometrically similar but contact-incompatible objects (Li et al., 22 Sep 2025).

A separate case study on tall objects compares GD2P to a manually defined flat “spatula-like” hand pose. Across 60 trials on 6 tall objects, the fixed pose succeeds 18/60 times, or 30%, with failures often attributed to toppling or sliding off the flat fingers. This comparison is especially significant because it isolates the effect of geometry-conditioned hand configuration rather than overall motion planning.

Cross-morphology transfer is evaluated on an xArm7 with a LEAP Hand. Using the same 14 objects and 3 directions, the paper reports 210 trials and an overall success rate of approximately 68.1%. The result does not show hand-independence in a strong formal sense, but it does indicate that the pipeline is not specific to a single hand morphology (Li et al., 22 Sep 2025).

These findings support a narrower conclusion than “dexterous hands solve nonprehensile manipulation.” What they show, more precisely, is that a large-scale prior over geometry-matched pre-contact poses, combined with downstream planning and ranking, can outperform grasp-transfer and retrieval baselines on diverse real objects.

6. Position within geometry-aware dexterous manipulation research

GD2P belongs to a broader shift from object-agnostic contact heuristics toward explicitly geometric representations, hierarchical contact reasoning, and environment-aware dexterity. Earlier work on geometry-aware dexterous manipulation had already shown that conditioning policies on object point clouds and normals improves generalization across diverse shapes; a single geometry-aware multi-task policy was shown to perform in-hand manipulation across over 100 real-world objects and generalize to held-out ones (Huang et al., 2021). In a complementary direction, DexRepNet++ proposed a hand-object interaction representation built from hand-centric occupancy, nearest-point distances and normals, and local geometric descriptors, and showed that such representations substantially improve dexterous manipulation generalization even though pushing and pulling were not its explicit focus (Liu et al., 25 Feb 2026).

Within nonprehensile manipulation specifically, GD2P follows and departs from several lines of work. ExDex used reinforcement learning and extrinsic dexterity to relocate ungraspable objects to table edges or walls, where environmental geometry makes grasping possible, thereby making wall and edge geometry part of the manipulation strategy itself (Wang et al., 29 Mar 2025). GAPG, in clutter, coupled a geometry-aware grasp evaluation module with a push evaluation module, predicting whether pushes would convert non-graspable states into graspable ones from point cloud geometry (Xiao et al., 22 Mar 2026). “Where to Touch, How to Contact” introduced a hierarchical RL–MPC contact-intention interface that predicts object-surface contact locations and post-contact object-level subgoals, achieving near-100% success on pushing and 3D reorientation with substantially reduced data (Xie et al., 16 Jan 2026).

A distinct but related strand concerns detailed contact kinematics. Mesh-based roll-slide integration via geodesic tracing extends Montana-style contact reasoning to manifold meshes and is directly relevant when pushing or pulling must follow high-fidelity surface geometry rather than primitive approximations (Wang et al., 17 Aug 2025). Contact-state graph approaches such as Dexterous Manipulation Graphs and their dual-arm extension formulate in-hand manipulation as graph search over reachable fingertip contact states, with edges corresponding to sliding and pivoting primitives grounded in surface geometry (Cruciani et al., 2018, Cruciani et al., 2019). Planning for dexterous ungrasping emphasizes secure rolling and sliding contact transitions and shows how digit asymmetry can expand the space of feasible geometry-driven motions against the environment (Kim et al., 2021). For articulated objects, DragMesh-2 and its PICA training mechanism address a more contact-driven pulling regime in which the target joint cannot be directly actuated and motion must arise through sustained hand-handle contact under varying damping (Zhang et al., 13 Jun 2026). At the control-theoretic end, SafePBDS provides pullback CBF constructions and task-manifold action interfaces that compute safe configuration-manifold accelerations from objectives and constraints defined on heterogeneous geometric spaces, which is pertinent when pushing and pulling must be both dexterous and certifiably safe (Wu et al., 20 May 2026).

Taken together, these works locate GD2P within a research program that treats dexterous nonprehensile manipulation as a problem of contact geometry, contact sequencing, and contact feasibility, rather than only one of end-effector trajectory design.

7. Assumptions, limitations, and prospective extensions

GD2P is developed under several explicit assumptions. The paper assumes quasi-static pushing, a flat workspace with sufficient friction, access to object geometry either through an exact mesh or an approximate reconstruction, and a single object, single-hand setting. Execution is open-loop once a hand pose and translational trajectory are selected. The study covers pushing and pulling only; other nonprehensile behaviors such as tilting, rolling, and pivoting are identified as outside the presented scope (Li et al., 22 Sep 2025).

These limitations clarify both the contribution and the boundary of the method. Because the learned model predicts pre-contact poses rather than closed-loop contact adjustments, failures caused by friction mismatch, contact stiffness mismatch, or object-motion drift are mitigated mainly by large-scale data and by the ranking stage, not by online feedback correction. The use of NeRF-based reconstruction for daily objects also means that geometric conditioning is only as accurate as the reconstructed point cloud. The paper explicitly identifies several potential improvements: broader nonprehensile repertoires, closed-loop control with vision or tactile feedback, better modeling of friction and mass, richer task-level criteria, and stronger sim-to-real adaptation (Li et al., 22 Sep 2025).

A plausible implication is that GD2P is best understood as a scalable pose-prior module for dexterous nonprehensile manipulation rather than a full theory of pushing and pulling. Its most durable contribution is the reframing of the task: instead of learning full contact-rich dynamics end to end, it learns where and how a dexterous hand should arrive at the object so that subsequent pushing or pulling becomes effective. That perspective is likely to remain central as future systems incorporate tactile feedback, articulated objects, cluttered scenes, and hybrid analytic-learning contact models.