---
title: Geometry-aware Dexterous Pushing & Pulling
url: https://www.emergentmind.com/topics/geometry-aware-dexterous-pushing-and-pulling-gd2p
type: topic
---

# Geometry-aware Dexterous Pushing & Pulling

Geometry-aware Dexterous Pushing and Pulling (GD2P) is a nonprehensile manipulation framework for multi-fingered robotic hands that treats pushing and pulling as a problem of synthesizing **pre-contact hand poses** conditioned on **object geometry** and **pushing/pulling direction**, then executing the resulting interactions through standard motion planning rather than direct end-to-end contact-dynamics control [2509.18455]. In this formulation, the central learned object is not a low-level torque policy but a distribution over hand configurations that are likely to produce effective pushes or pulls once the arm-hand system is translated along a prescribed direction. The framework is explicitly geometry-aware because it conditions pose generation on compact object-shape descriptors and evaluates candidates using collision checking, direction alignment, and task-space outcome measures.

## 1. Conceptual scope and problem setting

GD2P addresses tabletop manipulation of a **single object** on a **flat surface**, with the goal of moving, aligning, or repositioning that object **without grasping** it firmly. The task is motivated by settings in which direct grasping is difficult because of object geometry, size, or the relation between the object and the environment. In GD2P, dexterous hands are used as nonprehensile manipulators: they establish supportive, geometry-matched contacts with the object and then induce object motion through a straight-line arm-hand translation along a desired planar direction [2509.18455].

A hand pose is defined as
$$
H = (\theta, T),
$$
where $\theta \in \mathbb{R}^d$ are the hand joint angles and $T \in SE(3)$ is the wrist pose. For the Allegro and LEAP setups discussed in the paper, $d=16$. Each trial is specified by a planar direction vector
$$
u_{\rm dir} \in \mathbb{R}^3,\quad (u_{\rm dir})_z = 0,
$$
and a target object position
$$
u_{\rm targ} \in \mathbb{R}^3.
$$
The hand is moved to a pre-contact pose and then translated along $u_{\rm dir}$ over a fixed distance of $0.2$ m. In simulation, a trial is successful if, after execution, the final object center satisfies
$$
\| \mathbf{c}_{\rm final} - u_{\rm targ} \|_2 \le 0.03 \text{ m},
$$
and the object orientation change is at most $45^\circ$ [2509.18455].

This setup makes a precise methodological claim. GD2P does not attempt to learn the full nonprehensile dynamics directly. Instead, it reframes pushing and pulling as a **pre-contact pose synthesis** problem: if the hand approaches the object in a geometry-consistent pose, then a comparatively simple translational motion can produce useful manipulation outcomes. A common misunderstanding is to read GD2P as a generic dexterous control policy; more precisely, it is a hybrid pipeline that couples geometry-conditioned pose generation with downstream motion planning and execution.

## 2. Geometric representation and formal objective

The object is represented by a surface point cloud
$$
P = \{ \mathbf{p}_i \in \mathbb{R}^3 \}_{i=1}^{N},
$$
with configuration
$$
S_{\rm obj} \in SE(3).
$$
GD2P encodes geometry using a **Basis Point Set (BPS)** representation
$$
B \in \mathbb{R}^{4096},
$$
defined by a canonical set of basis points $\{\mathbf{b}_k\}_{k=1}^{4096}$ and distances
$$
B_k = \min_{\mathbf{p}\in P} \|\mathbf{b}_k - \mathbf{p}\|_2.
$$
This yields a fixed-length representation that captures overall shape and scale [2509.18455].

The objective is to find a pose $H^* = (\theta^*, T^*)$ such that translating the arm-hand system along $u_{\rm dir}$ moves the object close to $u_{\rm targ}$ without undesirable events such as toppling, loss of contact, or collisions. Geometry-awareness enters at several levels. First, the BPS descriptor conditions the generative model, so the predicted pose distribution depends on object shape. Second, the optimization used to construct the training data explicitly biases the palm normal toward the pushing or pulling direction. Third, candidate poses are ranked partly by the alignment between palm orientation and desired manipulation direction.

The directional alignment term in the pose-synthesis energy is
$$
E_{\rm dir} = - \frac{u_{\rm dir}^{\mathsf T} v_{\rm palm}}{\|u_{\rm dir}\|_2 \,\|v_{\rm palm}\|_2},
$$
where $v_{\rm palm}$ is the palm normal in world coordinates. Minimizing this term encourages
$$
v_{\rm palm} \parallel u_{\rm dir}.
$$
An additional arm-feasibility term
$$
E_{\rm arm} = \max \bigl( 0, (v_{\rm palm})_z \bigr)
$$
penalizes upward-pointing palm normals, which would suggest the arm is likely underneath and colliding with the table [2509.18455].

The resulting interpretation is geometric rather than purely kinematic: the hand is asked to approach with a pose that is not merely reachable, but also shape-matched and direction-matched. This is why GD2P often uses palm support for flat objects, partial wraps for volumetric objects, and higher or lower contact placement depending on toppling risk.

## 3. Dataset construction and diffusion-based pose synthesis

GD2P relies on a large offline corpus of successful pushing and pulling poses. Candidate poses are first generated by **contact-guided optimization**, then filtered by **physics-based validation** in IsaacGym, and finally used to train a conditional diffusion model [2509.18455].

For each hand, candidate contact points are predefined over the palm surface, finger links, and fingertips. In the Allegro example, fingertips use 96 candidates each, several finger links use 16 candidates each, and the palm uses 128 candidates. Given an object and manipulation direction, GD2P minimizes the energy
$$
E = E_{\rm fc} + w_{\rm dis} E_{\rm dis} + w_{\rm j} E_{\rm j} + w_{\rm pen} E_{\rm pen} + w_{\rm dir} E_{\rm dir} + w_{\rm arm} E_{\rm arm},
$$
where $E_{\rm fc}$ is a force-closure estimator from DFC, $E_{\rm dis}$ penalizes hand-object distance, $E_{\rm j}$ penalizes joint-limit violations, $E_{\rm pen}$ penalizes penetration and self-collision, and the last two terms impose directional alignment and arm feasibility. Optimization is performed with RMSProp and step-size decay, with Simulated Annealing added to escape local minima; diversity is increased by randomly resampling subsets of contact candidate indices per run [2509.18455].

Each candidate is then validated by attaching the hand to the arm model, moving to the optimized pre-contact pose, executing a straight-line translation of length $0.2$ m along $u_{\rm dir}$, and checking the success criteria. Because optimization without dynamics modeling has low raw success rate, successful poses are augmented by perturbations generated via Halton sequences: rotation noise up to $\pm 2.5^\circ$, translation noise up to $\pm 5$ mm, and joint-angle noise up to $\pm 0.05$ rad. This yields the final training set of approximately **2.3k** object meshes and approximately **1.3 million** successful pushing and pulling hand poses [2509.18455].

The generative model is a conditional diffusion network adapted from **ConditionalUnet1D**. For the Allegro setup, the pose vector has dimension $25$, reported as `pushingpose_dim = 25`; the global conditioning input is the BPS descriptor with `global_cond_dim = 4096`. The DDPM scheduler uses `num_diffusion_timesteps = 100`, `beta_schedule = squaredcos_cap_v2`, and `prediction_type = epsilon`. Training uses batch size $16$, $200$ epochs, Adam with learning rate $10^{-4}$ and weight decay $10^{-6}$, gradient clipping at $1.0$, and a cosine learning-rate scheduler with warmup [2509.18455].

Methodologically, this stage situates GD2P between analytic grasp synthesis and policy learning. The offline optimizer and simulator define what successful contact geometries look like; the diffusion model then learns a conditional distribution over those geometries.

## 4. Test-time planning and execution

At test time, GD2P first constructs object geometry, then samples candidate poses from the diffusion model, filters them through arm-hand motion planning, ranks the feasible ones, and executes the best-scoring option [2509.18455].

If the object mesh is known, points are sampled directly. For unseen daily objects, the paper uses a NeRF-based reconstruction pipeline consisting of multi-view RGB capture, COLMAP plus Nerfstudio for reconstruction, Stable Normal for normal maps, and 2D Gaussian Splatting to obtain a dense point cloud. BPS is then computed from the reconstructed point cloud. Candidate hand poses are sampled in the object frame and transformed to the world frame using the estimated object pose. In some simulation experiments, the system samples $N=200$ candidates.

Full arm-hand trajectories are planned with **cuRobo**. For each candidate pose $H_i = (\theta_i, T_i)$, cuRobo attempts to find a collision-free arm trajectory from a start posture to the desired wrist and hand configuration. If no such path exists, the pose is marked infeasible and discarded. The remaining poses are evaluated by
$$
V(H) = \alpha L_{\rm goal} + \beta L_{\rm coll} + \gamma L_{\rm dir},
$$
where
$$
L_{\rm goal} = \|\mathbf{c}_{\rm final}(H) - u_{\rm targ}\|_2,
$$
$L_{\rm coll}$ is a binary collision indicator during simulated execution, and $L_{\rm dir}$ reuses the palm-direction alignment term. The selected pose is
$$
H^* = \arg\min_H V(H).
$$
This ranking stage is integral rather than incidental: the paper includes an ablation called **GD2P w/o Ranking**, and the reported performance drop shows that sampling alone is not sufficient [2509.18455].

The framework also supports multi-step manipulation. A global planner such as RRT* computes a collision-free 2D path for the object center around obstacles; waypoints are transformed into the robot frame; for each path segment, GD2P is called again with a new direction $u_{\rm dir}$. The hand pose is therefore re-synthesized at each segment rather than reused, which is consistent with the central premise that effective nonprehensile manipulation depends on local geometry and local direction, not just object identity.

## 5. Experimental evidence and empirical profile

The main real-world platform uses a **Franka Emika Panda** arm with an **Allegro Hand** in a $60\text{ cm} \times 60\text{ cm}$ workspace. The object set comprises **8 3D-printed objects** and **6 real everyday objects**, including a vase, blender, bowl, cake box, cow, camera, lamp, bottle, black box, toy avocado, ranch bottle, spray bottle, coconut water, and cookie box. Each object is tested in **3 directions** around a circle, with each direction attempted **5 times**, except where some directions are infeasible because of kinematics. The paper also reports **840 real-world experiments with an Allegro Hand** in comparison to baselines [2509.18455].

Across the 14 evaluated objects and 3 directions, GD2P reports average success rates of approximately **61.4%**, **47.1%**, and **57.1%** for the three directions. The ablation **GD2P w/o Ranking** reports approximately **50%**, **40%**, and **51.4%**. The **Nearest Neighbor** baseline reports approximately **32.9%**, **28.6%**, and **32.9%**, while **Pre-Trained Grasp Pose** reports approximately **27.1%**, **31.4%**, and **37.1%**. These results are accompanied by qualitative observations: grasp-based baselines often produce contacts that are poorly aligned with the pushing direction, and nearest-neighbor retrieval can transfer hand poses from geometrically similar but contact-incompatible objects [2509.18455].

A separate case study on tall objects compares GD2P to a manually defined flat “spatula-like” hand pose. Across **60 trials** on **6 tall objects**, the fixed pose succeeds **18/60** times, or **30%**, with failures often attributed to toppling or sliding off the flat fingers. This comparison is especially significant because it isolates the effect of geometry-conditioned hand configuration rather than overall motion planning.

Cross-morphology transfer is evaluated on an **xArm7** with a **LEAP Hand**. Using the same **14 objects** and **3 directions**, the paper reports **210 trials** and an overall success rate of approximately **68.1%**. The result does not show hand-independence in a strong formal sense, but it does indicate that the pipeline is not specific to a single hand morphology [2509.18455].

These findings support a narrower conclusion than “dexterous hands solve nonprehensile manipulation.” What they show, more precisely, is that a large-scale prior over **geometry-matched pre-contact poses**, combined with downstream planning and ranking, can outperform grasp-transfer and retrieval baselines on diverse real objects.

## 6. Position within geometry-aware dexterous manipulation research

GD2P belongs to a broader shift from object-agnostic contact heuristics toward explicitly geometric representations, hierarchical contact reasoning, and environment-aware dexterity. Earlier work on geometry-aware dexterous manipulation had already shown that conditioning policies on object point clouds and normals improves generalization across diverse shapes; a single geometry-aware multi-task policy was shown to perform in-hand manipulation across over 100 real-world objects and generalize to held-out ones [2111.03062]. In a complementary direction, DexRepNet++ proposed a hand-object interaction representation built from hand-centric occupancy, nearest-point distances and normals, and local geometric descriptors, and showed that such representations substantially improve dexterous manipulation generalization even though pushing and pulling were not its explicit focus [2602.21811].

Within nonprehensile manipulation specifically, GD2P follows and departs from several lines of work. ExDex used reinforcement learning and extrinsic dexterity to relocate ungraspable objects to **table edges** or **walls**, where environmental geometry makes grasping possible, thereby making wall and edge geometry part of the manipulation strategy itself [2503.23120]. GAPG, in clutter, coupled a geometry-aware grasp evaluation module with a push evaluation module, predicting whether pushes would convert non-graspable states into graspable ones from point cloud geometry [2603.21195]. “Where to Touch, How to Contact” introduced a hierarchical RL–MPC contact-intention interface that predicts object-surface contact locations and post-contact object-level subgoals, achieving near-100% success on pushing and 3D reorientation with substantially reduced data [2601.10930].

A distinct but related strand concerns detailed contact kinematics. Mesh-based roll-slide integration via geodesic tracing extends Montana-style contact reasoning to manifold meshes and is directly relevant when pushing or pulling must follow high-fidelity surface geometry rather than primitive approximations [2508.12439]. Contact-state graph approaches such as Dexterous Manipulation Graphs and their dual-arm extension formulate in-hand manipulation as graph search over reachable fingertip contact states, with edges corresponding to sliding and pivoting primitives grounded in surface geometry [1803.00346; 1904.11382]. Planning for dexterous ungrasping emphasizes secure rolling and sliding contact transitions and shows how digit asymmetry can expand the space of feasible geometry-driven motions against the environment [2108.13580]. For articulated objects, DragMesh-2 and its PICA training mechanism address a more contact-driven pulling regime in which the target joint cannot be directly actuated and motion must arise through sustained hand-handle contact under varying damping [2606.15133]. At the control-theoretic end, SafePBDS provides pullback CBF constructions and task-manifold action interfaces that compute safe configuration-manifold accelerations from objectives and constraints defined on heterogeneous geometric spaces, which is pertinent when pushing and pulling must be both dexterous and certifiably safe [2605.21811].

Taken together, these works locate GD2P within a research program that treats dexterous nonprehensile manipulation as a problem of **contact geometry**, **contact sequencing**, and **contact feasibility**, rather than only one of end-effector trajectory design.

## 7. Assumptions, limitations, and prospective extensions

GD2P is developed under several explicit assumptions. The paper assumes **quasi-static pushing**, a **flat workspace with sufficient friction**, access to object geometry either through an exact mesh or an approximate reconstruction, and a **single object, single-hand setting**. Execution is **open-loop** once a hand pose and translational trajectory are selected. The study covers **pushing and pulling** only; other nonprehensile behaviors such as tilting, rolling, and pivoting are identified as outside the presented scope [2509.18455].

These limitations clarify both the contribution and the boundary of the method. Because the learned model predicts pre-contact poses rather than closed-loop contact adjustments, failures caused by friction mismatch, contact stiffness mismatch, or object-motion drift are mitigated mainly by large-scale data and by the ranking stage, not by online feedback correction. The use of NeRF-based reconstruction for daily objects also means that geometric conditioning is only as accurate as the reconstructed point cloud. The paper explicitly identifies several potential improvements: broader nonprehensile repertoires, closed-loop control with vision or tactile feedback, better modeling of friction and mass, richer task-level criteria, and stronger sim-to-real adaptation [2509.18455].

A plausible implication is that GD2P is best understood as a scalable **pose-prior module** for dexterous nonprehensile manipulation rather than a full theory of pushing and pulling. Its most durable contribution is the reframing of the task: instead of learning full contact-rich dynamics end to end, it learns where and how a dexterous hand should arrive at the object so that subsequent pushing or pulling becomes effective. That perspective is likely to remain central as future systems incorporate tactile feedback, articulated objects, cluttered scenes, and hybrid analytic-learning contact models.

Source: https://www.emergentmind.com/topics/geometry-aware-dexterous-pushing-and-pulling-gd2p