EPro-PnP: Probabilistic 6-DoF Pose Estimation

• EPro-PnP is a probabilistic framework that reformulates the classical PnP problem as a continuous, differentiable SE(3) layer for robust 6-DoF object pose estimation.
• It learns 2D–3D correspondences and confidence weights via a KL-divergence loss, integrating continuous Softmax attention into deep vision architectures.
• Benchmark evaluations show that EPro-PnP achieves state-of-the-art performance in pose accuracy and monocular 3D detection with extensible design for further geometric estimation.

EPro-PnP is a generalized, end-to-end probabilistic Perspective-n-Points (PnP) framework enabling robust 6-DoF object pose estimation and monocular 3D object detection from single RGB images. It reconceptualizes classical PnP as a differentiable probabilistic layer, outputting a distribution of poses on the SE(3) manifold and learning all 2D-3D correspondences and confidence weights as intermediate network variables under a KL-divergence loss. This approach unifies prior methods and makes PnP a continuous analogue of Softmax, facilitating principled training and integration within deep vision architectures (Chen et al., 2022, Chen et al., 2023).

1. Probabilistic PnP Formulation on SE(3)

Classical PnP solvers estimate the optimal pose $y = (R, t) \in \mathrm{SE}(3)$ by minimizing a weighted reprojection error given $N$ correspondences $$X = \{(x_i^{\rm 3D}, x_i^{\rm 2D}, w_i^{\rm 2D})\}_{i=1}^N$$, where $x_i^{\rm 3D} \in \mathbb{R}^3$, $x_i^{\rm 2D} \in \mathbb{R}^2$, and $w_i^{\rm 2D} \in \mathbb{R}_+^2$:

$$y^* = \arg\min_y \frac{1}{2} \sum_{i=1}^N \| w_i^{\rm 2D} \circ (\pi(Rx_i^{\rm 3D} + t) - x_i^{\rm 2D}) \|^2$$

EPro-PnP interprets the negative reprojection error as an un-normalized likelihood:

$$p(X|y) = \exp\left( -\frac{1}{2} \sum_i \|f_i(y)\|^2 \right)$$

With a uniform prior over SE(3), the posterior is a continuous density:

$$p(y|X) = \frac{p(X|y)}{\int_{\mathrm{SE}(3)} p(X|y) \, dy}$$

This framework resembles the categorical Softmax, but is defined in the continuous domain of SE(3), supporting multi-modal pose distributions (Chen et al., 2022, Chen et al., 2023).

2. Network Outputs and Continuous Softmax Attention

The network does not regress pose directly. Instead, it outputs:

• 3D object coordinates: $\{x_i^{\rm 3D}\}$
• 2D image correspondences: $\{x_i^{\rm 2D}\}$
• Per-point weights: $\{w_i^{\rm 2D}\}$

Weights are normalized by a spatial Softmax with a global scale:

$$\sum_i w_{i,u}^{\rm 2D} = \sum_i w_{i,v}^{\rm 2D} = 1$$

This mechanism generalizes attention to the continuous pose setting, allowing the framework to learn both correspondences and their significance via end-to-end backpropagation. The posterior $p(y|X)$ then acts as a Softmax denominator, yielding densities over pose space rather than discrete probabilities (Chen et al., 2022).

3. KL-Divergence Loss and Backpropagation

Training minimizes KL divergence between a narrow Dirac-like target distribution $t(y) \approx \delta(y - y_{gt})$ (centered at ground truth) and the predicted pose posterior:

$L_{\rm KL} = D_{\mathrm{KL}}(t(y) \,\|\, p(y|X))$

Expanding (omitting constants), the loss decomposes as:

$$L_{\rm KL} = \underbrace{\frac{1}{2}\sum_i \|f_i(y_{gt})\|^2}_{L_{\rm tgt}} \;+\; \underbrace{\log \int_{\mathrm{SE}(3)} \exp\left(-\frac{1}{2}\sum_i\|f_i(y)\|^2\right) \, dy}_{L_{\rm pred}}$$

Here $L_{\rm tgt}$ is standard reprojection error at ground truth; $L_{\rm pred}$ (the log-partition function) penalizes unnormalized likelihood mass of incorrect poses, yielding a discriminative, stable training objective.

Gradient backpropagation is implemented via:

$$\nabla_\theta L_{\rm KL} = \nabla_\theta\left(\frac{1}{2}\sum_i\|f_i(y_{gt})\|^2\right) - \mathbb{E}_{y \sim p(y|X)} \left[ \nabla_\theta \left( \frac{1}{2} \sum_i\|f_i(y)\|^2 \right)\right]$$

The expectation is approximated using Adaptive Multiple Importance Sampling (AMIS), with importance weights

$$v_j = \frac{\exp(-\frac{1}{2} \sum_i \|f_i(y_j)\|^2)}{q(y_j)}$$

This ensures that gradients backpropagate through all network-produced correspondence variables. Optionally, derivative regularization is applied using a local Gauss-Newton step:

$$\Delta y = - (J^T J + \varepsilon I)^{-1} J^T F(y^*)$$

$L_{\rm reg} = l(y^* + \Delta y, y_{gt})$

This encourages the LM iterate at inference to move toward ground truth (Chen et al., 2022, Chen et al., 2023).

4. Network Architectures and Integration

Dense Correspondence Network

Built on CDPN [Li et al. ICCV'19] with a ResNet-34 backbone. Outputs a $64 \times 64$ map of dense 3D coordinates and two-channel weights, normalized via spatial Softmax and global scale. The EPro-PnP layer extracts sampled correspondences (typically 512 per object) and computes the KL-divergence plus derivative regularization losses.

Deformable Correspondence Network

Based on FCOS3D [Wang et al. ICCVW'21] with ResNet-101-DCN. Employs object queries and Deformable DETR-style attention, sampling $N = n_{head} \times n_{hpts}$ 2D locations and features. Each feature is decoded to a 3D coordinate and weight; the EPro-PnP layer then probabilistically fuses these for pose estimation. Object-level features predict localization confidence, global weight scale, box parameters, and additional attributes. Auxiliary coordinate-regression and GMM-based reprojection losses regularize training. The same AMIS-based KL objective applies (Chen et al., 2022, Chen et al., 2023).

5. Comparative Evaluation

LineMOD 6DoF Pose

• Baseline (CDPN-Full): Mean [email protected] = 63.21%
• PnP (EPnP+LM only): 45.75%
• EPro-PnP (basic KL): 65.88%
• • Derivative regularization: 67.76%
• • Pretrained rotation head: 73.22%
• • Extended schedule: 74.19%

State-of-the-art performance achieved: EPro-PnP ADD-S 0.1d = 95.80%, 5°5cm = 98.54% (matching/exceeding DPOD, HybridPose, PVNet+RePOSE).

nuScenes Monocular 3D Detection

• CenterNet: NDS=0.328, mAP=0.306
• FCOS3D: NDS=0.372, mAP=0.295
• PGD [Wang et al. CoRL’21]: NDS=0.422, mAP=0.361
• Basic EPro-PnP: NDS=0.425, mAP=0.349, mAOE=0.363, mATE=0.676
• • sparse regression: NDS=0.430, mAP=0.352, mAOE=0.337
• • test-time flip: NDS=0.439, mAP=0.361

On nuScenes test set, EPro-PnP+flip obtains NDS=0.453, mAP=0.373 (PGD: 0.448/0.386, FCOS3D: 0.428/0.358) (Chen et al., 2022, Chen et al., 2023).

6. Strengths, Limitations, and Extensibility

Strengths

• Fully differentiable probabilistic PnP layer: continuous Softmax on SE(3)
• End-to-end learning of 2D–3D correspondences and weights from scratch (no surrogate mask/point supervision required)
• Handles pose ambiguity and multi-modal distributions inherently
• Unifies previous differentiable PnP works (DSAC, BlindPnP, BPnP) within this probabilistic framework
• Motivates novel network architectures utilizing attention-like mechanisms

Limitations

• Training cost: Monte Carlo KL loss increases training time by approximately 70% relative to a reprojection-only baseline
• Requires tuning sample count, AMIS iterations, proposal distributions
• Current deformable architecture possesses redundant FLOPs and memory; not exhaustively optimized

Prospective Extensions

• Generalization to other geometric layers (e.g., bundle adjustment, ICP, structured SVM)
• Exploration of learned proposal distributions, normalizing flows for sampling efficiency
• Cost reduction via quasi-Monte Carlo or learned quadrature over SE(3)
• Joint modeling of aleatoric/epistemic uncertainty by combining EPro-PnP with Bayesian neural networks

A plausible implication is that EPro-PnP's framework may extend beyond PnP, serving as a template for probabilistic, differentiable geometric estimators in varied vision problems (Chen et al., 2022, Chen et al., 2023).

7. Relationship to Prior Differentiable PnP Approaches

Prior frameworks, such as BPnP, DSAC++, or BlindPnP, utilize implicit differentiation or Laplace approximation centered on the local PnP optimum—these fail under multi-modal distributions or ambiguity, only capturing a narrow subset of the posterior. Some alternatives backpropagate solely the reprojection loss, risking degenerate minima unless heavily regularized. EPro-PnP instead applies the full KL-divergence between target and predicted distributions, stably learning all correspondences and weights from scratch, and outperforming prior methods on key benchmarks (Chen et al., 2023).