---
title: 'KGN-Pro: Keypoint-Based 6-DoF Grasp Synthesis'
url: https://www.emergentmind.com/topics/kgn-pro
type: topic
---

# KGN-Pro: Keypoint-Based 6-DoF Grasp Synthesis

Searching arXiv for the KGN-Pro paper and closely related context.

Search query: "KGN-Pro Keypoint-Based Grasp Prediction through Probabilistic 2D-3D Correspondence Learning"

arXiv search: KGN-Pro

KGN-Pro is a keypoint-based 6-DoF grasp synthesis network introduced for robotic manipulation as “Keypoint-Based Grasp Prediction through Probabilistic 2D-3D Correspondence Learning” [2507.14820]. It retains the efficiency and fine-grained object grasping of earlier KGN variants while replacing the conventional non-differentiable Perspective-n-Point (PnP) stage with a probabilistic, differentiable formulation. The method operates on paired RGB-D images, predicts 2D grasp keypoints and confidence weights, and uses weighted re-projection errors to transmit 3D supervision back into the 2D prediction heads. In the reported experiments, this design improves grasp cover rate and success rate relative to prior point-cloud and keypoint-based baselines, particularly under stringent precision requirements and in cluttered multi-object scenes [2507.14820].

## 1. Problem setting and design objective

Flexible 6-DoF grasp estimation is presented as a basic function for high-level robotic manipulation tasks such as rearrangement and vision-language manipulation [2507.14820]. The motivating contrast is between two established families of methods. Point-cloud-only approaches directly generate grasps from depth-derived 3D geometry, but they are reported to struggle with small objects, few and noisy depth samples, sensor noise, dense scenes, and heavy preprocessing. By contrast, 2D keypoint approaches retain an image-space representation and can exploit fine-grained visual cues, but earlier KGN-style pipelines rely on a non-differentiable PnP block and therefore cannot propagate 3D supervision back into the image encoder [2507.14820].

KGN-Pro is defined by the attempt to preserve the advantages of the 2D keypoint formulation while enabling direct 3D optimization and end-to-end learning [2507.14820]. The central technical claim is that the network turns PnP into a probabilistic, differentiable layer by modeling weighted re-projection errors and introducing a 2D confidence map that reflects the reliability of each 2D-3D correspondence. This allows gradients to flow from pose likelihoods to the 2D keypoint predictions and correspondence confidences, rather than confining supervision to 2D heatmap or offset targets alone [2507.14820].

This design also targets a specific failure mode of sparse-correspondence grasp estimation. With only four gripper keypoints, PnP can be sensitive to localization noise. KGN-Pro addresses this by learning per-keypoint confidence weights and by optimizing a pose-distribution objective rather than a hard geometric estimate [2507.14820].

## 2. Network architecture and 2D-3D correspondence construction

The system takes paired RGB-D images as input [2507.14820]. The use of RGB-D is motivated as a compromise between modalities: RGB contributes texture cues useful for small-object detection, while depth preserves spatial reasoning needed for grasp geometry. The backbone is DLA-34 with deformable convolutions, producing multi-scale features at \(1/4\) of the input resolution [2507.14820].

The network contains two principal prediction components. The keypoint extractor produces a Keypoint Map consisting of a center heatmap \(H\), a sub-pixel center offset \(S\), and a center-to-keypoint offset map \(O\). The confidence extractor outputs a 2D confidence map whose values \(w_i^{2D}\) are used to weight the contribution of each predicted 2D point during re-projection error minimization [2507.14820].

The 2D-3D correspondence structure is fixed by the gripper model. KGN-Pro uses four predefined 3D corner points on the gripper and associates them with four predicted 2D keypoints. The paper gives the 2D keypoint construction as [2507.14820]:

$$
p_{i,k}^{2D} = \left( \left(u_i,v_i\right) + S_i{\left(\Delta u,\Delta v \right)}\right) + O_{i}^{k}, \quad k=1,\ldots,4
$$

where \((u_i, v_i)\) is the \(i\)-th selected center from \(H\), \(S_i\) refines it to sub-pixel precision, and \(O_i^k\) offsets from the center to keypoint \(k\) [2507.14820].

A grasp pose is represented as \(y=(R,t)\), where \(R \in SO(3)\) and \(t \in \mathbb{R}^3\), with projection computed under the standard pinhole camera model [2507.14820]. For a gripper-frame 3D point \(X\), camera intrinsics \(K\), and pose \(y=(R,t)\), image coordinates are obtained by

$$
u = \pi(K(RX + t)),
$$

with \(\pi([x,y,z]^T) = [x/z, y/z]^T\) [2507.14820].

The output is a set of diverse 6-DoF grasp candidates, ranked by confidence and re-projection error for downstream execution [2507.14820].

## 3. Probabilistic PnP formulation

The defining methodological contribution of KGN-Pro is the replacement of a hard, non-differentiable PnP solver with a probabilistic PnP layer [2507.14820]. Instead of producing a pose estimate solely by geometric optimization external to the learning graph, the method defines a likelihood over poses through weighted re-projection errors and then optimizes a distributional objective.

The per-correspondence likelihood is reported as [2507.14820]:

$$
p(x_i | y) = \frac{1}{\sqrt{2\pi\sigma_i^2} \exp\left( -\frac{\| E_i(y) \|^2}{2\sigma_i^2} \right),
$$

where \(E_i(y)\) is the re-projection error and \(\sigma_i\) encodes uncertainty. Combining all correspondences with confidence weights yields the likelihood

$$
p(X|y) \propto \exp -\frac{1}{2} \sum_{i=1}^N \| w_i^{2D} E_i(y)\|^2.
$$

In this construction, \(w_i^{2D}\) acts like a learned per-keypoint precision: higher confidence increases the penalty assigned to its re-projection error, while low-confidence correspondences are down-weighted [2507.14820].

The posterior \(p(y|X)\) is aligned with a target distribution \(t(y)\) by minimizing a KL divergence, with \(t(y)\) defined by nearest-neighbor matching to a ground-truth grasp [2507.14820]. Because grasping admits multiple valid poses for the same scene, the paper does not impose a single canonical target pose; instead, each prediction is associated with at most one physically feasible ground-truth grasp through nearest-neighbor matching [2507.14820]. This is an important distinction from standard rigid pose estimation.

The predicted log-likelihood term is approximated by Adaptive Multiple Importance Sampling (AMIS). The paper reports the approximation in the following form [2507.14820]:

$$
L_{\mathrm{pred} \approx \log \frac{1}{K} \sum_{j=1}^K \underbrace{\frac{\exp -\frac{1}{2} \sum_{i=1}^N\left\|E_i\left(y_j\right)\right\|^2}{q\left(y_j\right)}}_{v_j \text { (importance weight) }}.
$$

The backward signal is expressed through the expected gradient of the re-projection error under the predicted pose distribution [2507.14820]:

$$
\frac{\partial L_{\mathrm{pred}}}{\partial(\cdot)}=
\underset{y \sim p(y \mid X)}{\mathbb{E}}
\frac{\partial}{\partial(\cdot)}
\frac{1}{2} \sum_{i=1}^N\left\|E_i(y)\right\|^2 .
$$

The paper explicitly notes that KGN-Pro does not unroll Gauss-Newton or Levenberg-Marquardt steps. Instead, differentiability is obtained through the probabilistic objective and sampling-based gradients, described as being in the spirit of EPro-PnP [2507.14820].

## 4. Losses, supervision, and training protocol

KGN-Pro combines conventional keypoint supervision with the probabilistic PnP objective [2507.14820]. The keypoint losses are a binary focal loss \(L_H\) for the center heatmap \(H\), an \(L_1\) loss \(L_S\) for the sub-pixel offset \(S\), and an \(L_1\) loss \(L_O\) for the center-to-keypoint offsets \(O\). These are combined with the KL-divergence-based probabilistic PnP loss \(L_{\mathrm{KL}}\) [2507.14820].

The reported total loss is

$$
L=\lambda_H L_H+\lambda_S L_S+\lambda_O L_O + \lambda_{\mathrm{KL}} L_{\mathrm{KL}}.
$$

The implementation details state that “the weights for each branch of the model are empirically set as follows: \(\lambda_Y = 1, \lambda_O = 1, \lambda_J = 1, \lambda_{\mathrm{KL}} = 0.1\)” [2507.14820]. The paper also gives the multi-grasp objective over \(M\) predictions as

$$
L_{\text{KL}}=\sum_{j=1}^M \left[ L_{\text{pred}}^j +
\left( \frac{1}{2} \sum_{i=1}^N \big\| w_i^{2D} E_i(\hat{y}_j) \big\|^2 \right)
\right].
$$

The 3D supervision source is a synthetic dataset spanning six categories—Cylinder, Ring, Stick, Sphere, Semi-sphere, and Cuboid—in both single-target and multi-object scenes [2507.14820]. The dataset is split 80% for training and 20% for testing. Training runs for 300 epochs with an initial learning rate of \(10^{-4}\), decayed by a factor of 10 at epochs 200 and 250 [2507.14820].

Several implementation constraints are explicit. The method requires camera intrinsics \(K\), known gripper corner points, and paired RGB-D frames [2507.14820]. No robust loss variants such as Huber or Tukey are used; robustness is instead attributed to the confidence-weighted likelihoods and the distributional pose modeling [2507.14820].

## 5. Empirical evaluation

The reported baselines are CenterGrasp, Contact-GraspNet, KGN, KGNv2, and a direct 3D regression baseline labeled Direct3D [2507.14820]. Evaluation uses Grasp Success Rate (GSR), defined as the number of successful grasps divided by the number of executed grasp attempts, and Scene Completion Rate (SCR), defined as the fraction of objects eventually grasped and removed after multiple attempts [2507.14820]. Controlled evaluations vary positional tolerance, such as \(1.0/1.5/2.0\) cm, and angular offset tolerance, such as \(10^\circ/20^\circ\), to test precision [2507.14820].

Under the \(10^\circ\) criterion, KGN-Pro reports the following object success rates. In single-object scenes: CenterGrasp \(20.5/44.5/56.8\), KGN \(60.0/71.7/78.7\), KGNv2 \(91.4/97.1/97.7\), and KGN-Pro \(93.4/96.5/98.5\). In multi-object scenes: CenterGrasp \(13.9/33.3/50.4\), KGN \(69.3/73.6/83.2\), KGNv2 \(84.9/87.9/89.9\), and KGN-Pro \(90.9/93.9/96.9\) [2507.14820].

Under the \(20^\circ\) criterion, the reported single-object rates are CenterGrasp \(19.5/54.5/66.8\), KGN \(82.2/84.0/91.5\), KGNv2 \(99.4/99.5/100.0\), and KGN-Pro \(99.1/99.5/100.0\). In multi-object scenes the reported values are CenterGrasp \(33.5/54.5/71.8\), KGN \(86.0/86.7/94.3\), KGNv2 \(90.9/93.9/96.1\), and KGN-Pro \(95.2/97.5/98.1\) [2507.14820].

The paper further states that generalization to 50 novel scenes confirms strong performance under varied objects and conditions. In single-object tests, KGN-Pro achieves approximately \(96\%\) GSR under lenient thresholds and approximately \(80\%\) GSR under the tightest setting of \(1\) cm and \(10^\circ\). In multi-object scenes it consistently exceeds KGN and slightly improves over KGNv2, while the point-cloud baselines lag notably under high precision because of reconstruction artifacts and small-object sensitivity [2507.14820].

Physical experiments use household objects similar to, and more complex than, those in the synthetic dataset [2507.14820]. Candidate grasps are ranked by 2D confidence and re-projection error, then executed by a robot arm, with success defined as lifting to \(0.1\) m and stable placement. The reported single-object GSR values are Cylinder \(96\%\) \((24/25)\), Sphere \(100\%\) \((25/25)\), Stick \(92\%\) \((23/25)\), Cuboid \(92\%\) \((23/25)\), and Other \(80\%\) \((20/25)\) [2507.14820]. In multi-object scenes across camera angles \(90^\circ\), \(75^\circ\), \(60^\circ\), \(45^\circ\), and \(30^\circ\), GSR decreases with increased occlusion, but SCR remains high at \(80\%\) to \(100\%\) [2507.14820].

The ablation findings emphasize three points. First, Direct3D underperforms relative to keypoint-based methods. Second, the confidence map \(w_i^{2D}\) improves robustness by down-weighting unreliable correspondences and filtering low-quality keypoints during inference. Third, the differentiable probabilistic PnP transmits 3D supervision into \(H\), \(S\), and \(O\), directly addressing the non-differentiability bottleneck of earlier KGN methods [2507.14820].

## 6. Limitations, reproducibility, and terminological scope

The reported limitations are those typical of keypoint-based PnP systems, but they are stated in specifically operational terms [2507.14820]. Performance depends on accurate 2D keypoint localization and camera calibration. Severe occlusion or reflective surfaces may degrade keypoint reliability and depth quality. The Monte Carlo approximation introduces variance, and poor proposal distributions \(q(y)\) can slow training or reduce stability. The system also requires a known gripper model and camera intrinsics; category-level shape variation beyond the gripper frame is not directly modeled [2507.14820].

The reproducibility details are partial but concrete. The project website is listed as https://waitderek.github.io/kgnpro/. The paper reports 300 training epochs, an initial learning rate of \(10^{-4}\), decay by \(0.1\) at epochs 200 and 250, multi-scale features at \(1/4\) resolution, and a DLA-34 backbone with deformable convolutions [2507.14820]. Hardware is not specified.

The stated future directions include learning better proposal distributions, integrating differentiable optimization steps for faster convergence, incorporating robust loss variants or uncertainty modeling for outliers, extending to task-oriented grasps with semantic constraints, exploiting large-scale multimodal datasets, and integrating language-conditioned grasp generation [2507.14820]. These directions suggest that the present method is positioned as a bridge between efficient image-space grasp synthesis and richer multimodal or semantics-aware robotic manipulation.

The term “KGN-Pro” also has a broader interpretive use outside robotic grasping. In the protein-science context, a “KGN-Pro” style approach is described as a knowledge-graph-based protein network that integrates Gene Ontology semantics with protein entities and multimodal alignments [2207.10080]. In medical LLM evaluation, “KGN-Pro” is used interpretively as a generic knowledge-graph probing paradigm for eliciting and assessing causal reasoning pathways [2412.10982]. Those usages are conceptually unrelated to the robotic method formally introduced as KGN-Pro in 2025; they indicate terminological convergence around knowledge-graph probing rather than a shared technical lineage [2207.10080; 2412.10982].

Source: https://www.emergentmind.com/topics/kgn-pro