Pose-Agnostic Robotic Functional Grasping via Observation-Action Canonicalization

Abstract: Functional robotic grasping requires a policy that generalizes across diverse object geometries and poses while maintaining task-specific contact precision. We study this challenge through mug-handle grasping, where thin handles, instance variation, and upright or inverted placements make both perception and control sensitive to object configuration. Grasp pose detection methods operate open-loop and are sensitive to estimation errors on thin handle structures. Learned visuomotor policies must implicitly learn to handle the coupled variation in visual appearance and action direction induced by different object placements, limiting generalization. We propose AnyMug, a canonicalized visuomotor reinforcement learning framework for functional grasping that trains a single closed-loop policy entirely in simulation and deploys it zero-shot on a real robot. AnyMug introduces observation-action canonicalization, which transforms both the depth observation and the predicted end-effector action into a shared object-centric frame. The policy therefore sees a consistent mug-centered view and emits actions in a canonical direction regardless of mug placement, allowing the same grasping behavior to be reused across configurations. A handle-aware reward further encourages precise approach, gripper alignment, and opposing-finger placement, while a pose curriculum and domain randomization improve training stability and sim-to-real transfer. In simulation, AnyMug achieves over 93% success rate on both unseen upright and inverted mugs and transfers zero-shot to a real Franka Panda, reaching 80% success rate on 5 held-out physical mugs across both pose categories.

• The paper introduces observation-action canonicalization to align depth images and control outputs for consistent mug-handle grasping.
• It employs a handle-aware reward structure and dense RL training with pose-category curriculum, achieving over 93% success in simulation and 80% in real-world tests.
• The method demonstrates robust sim-to-real transfer and highlights the critical role of explicit finger opposition for functional grasps.

Pose-Agnostic Mug-Handle Grasping with Canonicalized Observation-Action RL

Problem Definition and Motivation

Functional robotic grasping, particularly for objects with category-specific affordances like mugs, necessitates policies that generalize not only across diverse instance geometries but also across arbitrary object poses and spatial placements. Mug-handle grasping is uniquely constrained: handles are thin, curved, and highly variable across instances, with task-relevant contacts often only achievable in specific spatial configurations. Traditional open-loop grasp pose detection is brittle under estimation error, especially for thin handles, while closed-loop visuomotor approaches typically require implicitly learning a multitude of placement-dependent approach strategies, limiting generalization and robustness.

Methodological Innovations

The presented framework, AnyMug, addresses these limitations by introducing observation-action canonicalization within a reinforcement learning (RL) paradigm. The central innovation is the transformation of both depth observations and end-effector actions into a shared mug-centric reference frame, explicitly decoupling the underlying task from object placement or handle orientation. Observational canonicalization warps depth images such that the mug is always centered and the handle aligned to a canonical direction. Action canonicalization expresses policy outputs—translational, rotational, and gripper commands—in the same mug-centric frame, after which they are de-canonicalized for execution in the world frame.

The RL policy is trained in simulation using PPO, leveraging a dense handle-aware reward structure that incorporates reachability, gripper orientation alignment, finger opposition, and action smoothness. The finger opposition term is essential, encouraging opposing finger placement around the handle prior to closure—a condition critical for functional grasps and experimentally shown to be indispensable (removal yields 0% success rate). Training is augmented with pose-category curriculum learning (progressive balancing over upright/inverted poses) and domain randomization (mug geometry, placement, pose-estimation noise) to enhance stability and sim-to-real transfer.

Experimental Results

AnyMug is evaluated both in high-fidelity simulation and zero-shot on a real Franka Panda robotic arm. In simulation, AnyMug achieves 93.18% success rate on upright mugs and 94.19% on inverted mugs, outperforming analytical motion-planning and advanced learning-based baselines. Motion-planning degrades rapidly as pose estimation noise increases (dropping to 62.68%/59.03% SR with 0.03 m noise), while Diffusion Policy and equivariant alternatives remain stably below 55% SR, unable to learn robust finger alignment and closure timing. The domain-agnostic canonicalization—unique to AnyMug—proves vital: ablations show that removing canonicalization reduces SR to 31.53% and 40.99% for upright/inverted mugs, with failures dominated by empty grasps.

Zero-shot real-world deployment (5 unseen physical mugs, 2 trials per pose) yields 80% SR per pose category, outperforming motion planning (70%/60%) and Diffusion Policy (20%/10%). AnyMug's failures are split between tip/fall and empty grasps, indicating remaining sensitivity to handle alignment errors and perceptual artifacts in physical environments. These results underscore the importance of canonicalized closed-loop feedback for functional grasping, particularly under real-world uncertainty.

Sim-to-sim backend transfer (PhysX-to-Newton) without retraining demonstrates that the canonicalized observation-action interface generalizes across physical simulators, maintaining 85.2-90.0% SR under pose perturbations.

Theoretical and Practical Implications

Canonicalization of both perception and action fundamentally reduces the complexity of pose-induced variation, allowing instance-level adaptation to focus exclusively on handle geometry and contact timing. This approach is not merely equivariant in the network architecture, but couples visual normalization directly with control, enabling a single closed-loop policy to generalize functionally across placements and orientations. The stringent ablation analysis confirms the necessity of explicit finger opposition and canonicalization for high-fidelity handle grasping.

Practically, the paradigm supports robust zero-shot sim-to-real transfer, a major hurdle for RL-based manipulation. The explicit canonicalization enables domain-reduced sim training to produce policies that remain effective on physical platforms, even with real-world pose estimation noise and contact variation.

Future Directions

The current framework is limited to mug-handle grasping with upright/inverted pose categories. Extension to broader object classes, additional pose modes (e.g., tilted, side-lying), and downstream tasks (placing, pouring) remain open. Robustness to severe occlusion and reflective surfaces is not yet addressed; multi-view or active perception strategies are promising augmentations. Furthermore, integrating canonicalized representations in policy architectures for more complex tasks may yield similar generalization and transfer benefits.

Conclusion

AnyMug demonstrates a comprehensive approach to pose-agnostic, functional mug-handle grasping through simultaneous canonicalization of observations and actions in a mug-centric frame. Explicit handle-aware closed-loop RL with robust training protocols attains superior generalization and real-world performance relative to conventional and equivariant learning baselines. The results substantiate the criticality of canonicalization and task-centric rewards for category-level manipulation, offering a transferable methodology for future robotic functional grasping research (2606.21148).