KGN-Pro: Keypoint-Based 6-DoF Grasp Synthesis

• KGN-Pro is a keypoint-based 6-DoF grasp synthesis network that leverages probabilistic 2D-3D correspondence learning for end-to-end optimization.
• The method replaces the non-differentiable PnP stage with a probabilistic layer that uses weighted re-projection errors and confidence maps from paired RGB-D inputs.
• Empirical evaluations demonstrate improved grasp cover and success rates over previous baselines, especially in precision-demanding and cluttered multi-object scenarios.

Searching arXiv for the KGN-Pro paper and closely related context.

Search query: "KGN-Pro Keypoint-Based Grasp Prediction through Probabilistic 2D-3D Correspondence Learning"

arXiv search: KGN-Pro

KGN-Pro is a keypoint-based 6-DoF grasp synthesis network introduced for robotic manipulation as “Keypoint-Based Grasp Prediction through Probabilistic 2D-3D Correspondence Learning” (Chen et al., 20 Jul 2025). It retains the efficiency and fine-grained object grasping of earlier KGN variants while replacing the conventional non-differentiable Perspective-n-Point (PnP) stage with a probabilistic, differentiable formulation. The method operates on paired RGB-D images, predicts 2D grasp keypoints and confidence weights, and uses weighted re-projection errors to transmit 3D supervision back into the 2D prediction heads. In the reported experiments, this design improves grasp cover rate and success rate relative to prior point-cloud and keypoint-based baselines, particularly under stringent precision requirements and in cluttered multi-object scenes (Chen et al., 20 Jul 2025).

1. Problem setting and design objective

Flexible 6-DoF grasp estimation is presented as a basic function for high-level robotic manipulation tasks such as rearrangement and vision-language manipulation (Chen et al., 20 Jul 2025). The motivating contrast is between two established families of methods. Point-cloud-only approaches directly generate grasps from depth-derived 3D geometry, but they are reported to struggle with small objects, few and noisy depth samples, sensor noise, dense scenes, and heavy preprocessing. By contrast, 2D keypoint approaches retain an image-space representation and can exploit fine-grained visual cues, but earlier KGN-style pipelines rely on a non-differentiable PnP block and therefore cannot propagate 3D supervision back into the image encoder (Chen et al., 20 Jul 2025).

KGN-Pro is defined by the attempt to preserve the advantages of the 2D keypoint formulation while enabling direct 3D optimization and end-to-end learning (Chen et al., 20 Jul 2025). The central technical claim is that the network turns PnP into a probabilistic, differentiable layer by modeling weighted re-projection errors and introducing a 2D confidence map that reflects the reliability of each 2D-3D correspondence. This allows gradients to flow from pose likelihoods to the 2D keypoint predictions and correspondence confidences, rather than confining supervision to 2D heatmap or offset targets alone (Chen et al., 20 Jul 2025).

This design also targets a specific failure mode of sparse-correspondence grasp estimation. With only four gripper keypoints, PnP can be sensitive to localization noise. KGN-Pro addresses this by learning per-keypoint confidence weights and by optimizing a pose-distribution objective rather than a hard geometric estimate (Chen et al., 20 Jul 2025).

2. Network architecture and 2D-3D correspondence construction

The system takes paired RGB-D images as input (Chen et al., 20 Jul 2025). The use of RGB-D is motivated as a compromise between modalities: RGB contributes texture cues useful for small-object detection, while depth preserves spatial reasoning needed for grasp geometry. The backbone is DLA-34 with deformable convolutions, producing multi-scale features at $1/4$ of the input resolution (Chen et al., 20 Jul 2025).

The network contains two principal prediction components. The keypoint extractor produces a Keypoint Map consisting of a center heatmap $H$, a sub-pixel center offset $S$, and a center-to-keypoint offset map $O$. The confidence extractor outputs a 2D confidence map whose values $w_i^{2D}$ are used to weight the contribution of each predicted 2D point during re-projection error minimization (Chen et al., 20 Jul 2025).

The 2D-3D correspondence structure is fixed by the gripper model. KGN-Pro uses four predefined 3D corner points on the gripper and associates them with four predicted 2D keypoints. The paper gives the 2D keypoint construction as (Chen et al., 20 Jul 2025):

$$p_{i,k}^{2D} = \left( \left(u_i,v_i\right) + S_i{\left(\Delta u,\Delta v \right)}\right) + O_{i}^{k}, \quad k=1,\ldots,4$$

where $(u_i, v_i)$ is the $i$-th selected center from $H$, $S_i$ refines it to sub-pixel precision, and $H$0 offsets from the center to keypoint $H$1 (Chen et al., 20 Jul 2025).

A grasp pose is represented as $H$2, where $H$3 and $H$4, with projection computed under the standard pinhole camera model (Chen et al., 20 Jul 2025). For a gripper-frame 3D point $H$5, camera intrinsics $H$6, and pose $H$7, image coordinates are obtained by

$H$8

with $H$9 (Chen et al., 20 Jul 2025).

The output is a set of diverse 6-DoF grasp candidates, ranked by confidence and re-projection error for downstream execution (Chen et al., 20 Jul 2025).

3. Probabilistic PnP formulation

The defining methodological contribution of KGN-Pro is the replacement of a hard, non-differentiable PnP solver with a probabilistic PnP layer (Chen et al., 20 Jul 2025). Instead of producing a pose estimate solely by geometric optimization external to the learning graph, the method defines a likelihood over poses through weighted re-projection errors and then optimizes a distributional objective.

The per-correspondence likelihood is reported as (Chen et al., 20 Jul 2025):

$S$0

where $S$1 is the re-projection error and $S$2 encodes uncertainty. Combining all correspondences with confidence weights yields the likelihood

$S$3

In this construction, $S$4 acts like a learned per-keypoint precision: higher confidence increases the penalty assigned to its re-projection error, while low-confidence correspondences are down-weighted (Chen et al., 20 Jul 2025).

The posterior $S$5 is aligned with a target distribution $S$6 by minimizing a KL divergence, with $S$7 defined by nearest-neighbor matching to a ground-truth grasp (Chen et al., 20 Jul 2025). Because grasping admits multiple valid poses for the same scene, the paper does not impose a single canonical target pose; instead, each prediction is associated with at most one physically feasible ground-truth grasp through nearest-neighbor matching (Chen et al., 20 Jul 2025). This is an important distinction from standard rigid pose estimation.

The predicted log-likelihood term is approximated by Adaptive Multiple Importance Sampling (AMIS). The paper reports the approximation in the following form (Chen et al., 20 Jul 2025):

$S$8

The backward signal is expressed through the expected gradient of the re-projection error under the predicted pose distribution (Chen et al., 20 Jul 2025):

$S$9

The paper explicitly notes that KGN-Pro does not unroll Gauss-Newton or Levenberg-Marquardt steps. Instead, differentiability is obtained through the probabilistic objective and sampling-based gradients, described as being in the spirit of EPro-PnP (Chen et al., 20 Jul 2025).

4. Losses, supervision, and training protocol

KGN-Pro combines conventional keypoint supervision with the probabilistic PnP objective (Chen et al., 20 Jul 2025). The keypoint losses are a binary focal loss $O$0 for the center heatmap $O$1, an $O$2 loss $O$3 for the sub-pixel offset $O$4, and an $O$5 loss $O$6 for the center-to-keypoint offsets $O$7. These are combined with the KL-divergence-based probabilistic PnP loss $O$8 (Chen et al., 20 Jul 2025).

The reported total loss is

$O$9

The implementation details state that “the weights for each branch of the model are empirically set as follows: $w_i^{2D}$0” (Chen et al., 20 Jul 2025). The paper also gives the multi-grasp objective over $w_i^{2D}$1 predictions as

$w_i^{2D}$2

The 3D supervision source is a synthetic dataset spanning six categories—Cylinder, Ring, Stick, Sphere, Semi-sphere, and Cuboid—in both single-target and multi-object scenes (Chen et al., 20 Jul 2025). The dataset is split 80% for training and 20% for testing. Training runs for 300 epochs with an initial learning rate of $w_i^{2D}$3, decayed by a factor of 10 at epochs 200 and 250 (Chen et al., 20 Jul 2025).

Several implementation constraints are explicit. The method requires camera intrinsics $w_i^{2D}$4, known gripper corner points, and paired RGB-D frames (Chen et al., 20 Jul 2025). No robust loss variants such as Huber or Tukey are used; robustness is instead attributed to the confidence-weighted likelihoods and the distributional pose modeling (Chen et al., 20 Jul 2025).

5. Empirical evaluation

The reported baselines are CenterGrasp, Contact-GraspNet, KGN, KGNv2, and a direct 3D regression baseline labeled Direct3D (Chen et al., 20 Jul 2025). Evaluation uses Grasp Success Rate (GSR), defined as the number of successful grasps divided by the number of executed grasp attempts, and Scene Completion Rate (SCR), defined as the fraction of objects eventually grasped and removed after multiple attempts (Chen et al., 20 Jul 2025). Controlled evaluations vary positional tolerance, such as $w_i^{2D}$5 cm, and angular offset tolerance, such as $w_i^{2D}$6, to test precision (Chen et al., 20 Jul 2025).

Under the $w_i^{2D}$7 criterion, KGN-Pro reports the following object success rates. In single-object scenes: CenterGrasp $w_i^{2D}$8, KGN $w_i^{2D}$9, KGNv2 $$p_{i,k}^{2D} = \left( \left(u_i,v_i\right) + S_i{\left(\Delta u,\Delta v \right)}\right) + O_{i}^{k}, \quad k=1,\ldots,4$$0, and KGN-Pro $$p_{i,k}^{2D} = \left( \left(u_i,v_i\right) + S_i{\left(\Delta u,\Delta v \right)}\right) + O_{i}^{k}, \quad k=1,\ldots,4$$1. In multi-object scenes: CenterGrasp $$p_{i,k}^{2D} = \left( \left(u_i,v_i\right) + S_i{\left(\Delta u,\Delta v \right)}\right) + O_{i}^{k}, \quad k=1,\ldots,4$$2, KGN $$p_{i,k}^{2D} = \left( \left(u_i,v_i\right) + S_i{\left(\Delta u,\Delta v \right)}\right) + O_{i}^{k}, \quad k=1,\ldots,4$$3, KGNv2 $$p_{i,k}^{2D} = \left( \left(u_i,v_i\right) + S_i{\left(\Delta u,\Delta v \right)}\right) + O_{i}^{k}, \quad k=1,\ldots,4$$4, and KGN-Pro $$p_{i,k}^{2D} = \left( \left(u_i,v_i\right) + S_i{\left(\Delta u,\Delta v \right)}\right) + O_{i}^{k}, \quad k=1,\ldots,4$$5 (Chen et al., 20 Jul 2025).

Under the $$p_{i,k}^{2D} = \left( \left(u_i,v_i\right) + S_i{\left(\Delta u,\Delta v \right)}\right) + O_{i}^{k}, \quad k=1,\ldots,4$$6 criterion, the reported single-object rates are CenterGrasp $$p_{i,k}^{2D} = \left( \left(u_i,v_i\right) + S_i{\left(\Delta u,\Delta v \right)}\right) + O_{i}^{k}, \quad k=1,\ldots,4$$7, KGN $$p_{i,k}^{2D} = \left( \left(u_i,v_i\right) + S_i{\left(\Delta u,\Delta v \right)}\right) + O_{i}^{k}, \quad k=1,\ldots,4$$8, KGNv2 $$p_{i,k}^{2D} = \left( \left(u_i,v_i\right) + S_i{\left(\Delta u,\Delta v \right)}\right) + O_{i}^{k}, \quad k=1,\ldots,4$$9, and KGN-Pro $(u_i, v_i)$0. In multi-object scenes the reported values are CenterGrasp $(u_i, v_i)$1, KGN $(u_i, v_i)$2, KGNv2 $(u_i, v_i)$3, and KGN-Pro $(u_i, v_i)$4 (Chen et al., 20 Jul 2025).

The paper further states that generalization to 50 novel scenes confirms strong performance under varied objects and conditions. In single-object tests, KGN-Pro achieves approximately $(u_i, v_i)$5 GSR under lenient thresholds and approximately $(u_i, v_i)$6 GSR under the tightest setting of $(u_i, v_i)$7 cm and $(u_i, v_i)$8. In multi-object scenes it consistently exceeds KGN and slightly improves over KGNv2, while the point-cloud baselines lag notably under high precision because of reconstruction artifacts and small-object sensitivity (Chen et al., 20 Jul 2025).

Physical experiments use household objects similar to, and more complex than, those in the synthetic dataset (Chen et al., 20 Jul 2025). Candidate grasps are ranked by 2D confidence and re-projection error, then executed by a robot arm, with success defined as lifting to $(u_i, v_i)$9 m and stable placement. The reported single-object GSR values are Cylinder $i$0 $i$1, Sphere $i$2 $i$3, Stick $i$4 $i$5, Cuboid $i$6 $i$7, and Other $i$8 $i$9 (Chen et al., 20 Jul 2025). In multi-object scenes across camera angles $H$0, $H$1, $H$2, $H$3, and $H$4, GSR decreases with increased occlusion, but SCR remains high at $H$5 to $H$6 (Chen et al., 20 Jul 2025).

The ablation findings emphasize three points. First, Direct3D underperforms relative to keypoint-based methods. Second, the confidence map $H$7 improves robustness by down-weighting unreliable correspondences and filtering low-quality keypoints during inference. Third, the differentiable probabilistic PnP transmits 3D supervision into $H$8, $H$9, and $S_i$0, directly addressing the non-differentiability bottleneck of earlier KGN methods (Chen et al., 20 Jul 2025).

6. Limitations, reproducibility, and terminological scope

The reported limitations are those typical of keypoint-based PnP systems, but they are stated in specifically operational terms (Chen et al., 20 Jul 2025). Performance depends on accurate 2D keypoint localization and camera calibration. Severe occlusion or reflective surfaces may degrade keypoint reliability and depth quality. The Monte Carlo approximation introduces variance, and poor proposal distributions $S_i$1 can slow training or reduce stability. The system also requires a known gripper model and camera intrinsics; category-level shape variation beyond the gripper frame is not directly modeled (Chen et al., 20 Jul 2025).

The reproducibility details are partial but concrete. The project website is listed as https://waitderek.github.io/kgnpro/. The paper reports 300 training epochs, an initial learning rate of $S_i$2, decay by $S_i$3 at epochs 200 and 250, multi-scale features at $S_i$4 resolution, and a DLA-34 backbone with deformable convolutions (Chen et al., 20 Jul 2025). Hardware is not specified.

The stated future directions include learning better proposal distributions, integrating differentiable optimization steps for faster convergence, incorporating robust loss variants or uncertainty modeling for outliers, extending to task-oriented grasps with semantic constraints, exploiting large-scale multimodal datasets, and integrating language-conditioned grasp generation (Chen et al., 20 Jul 2025). These directions suggest that the present method is positioned as a bridge between efficient image-space grasp synthesis and richer multimodal or semantics-aware robotic manipulation.

The term “KGN-Pro” also has a broader interpretive use outside robotic grasping. In the protein-science context, a “KGN-Pro” style approach is described as a knowledge-graph-based protein network that integrates Gene Ontology semantics with protein entities and multimodal alignments (Cheng et al., 2022). In medical LLM evaluation, “KGN-Pro” is used interpretively as a generic knowledge-graph probing paradigm for eliciting and assessing causal reasoning pathways (Rosenbaum et al., 2024). Those usages are conceptually unrelated to the robotic method formally introduced as KGN-Pro in 2025; they indicate terminological convergence around knowledge-graph probing rather than a shared technical lineage (Cheng et al., 2022, Rosenbaum et al., 2024).