Uncertainty Quantification for Visual Object Pose Estimation

Abstract: Quantifying the uncertainty of an object's pose estimate is essential for robust control and planning. Although pose estimation is a well-studied robotics problem, attaching statistically rigorous uncertainty is not well understood without strict distributional assumptions. We develop distribution-free pose uncertainty bounds about a given pose estimate in the monocular setting. Our pose uncertainty only requires high probability noise bounds on pixel detections of 2D semantic keypoints on a known object. This noise model induces an implicit, non-convex set of pose uncertainty constraints. Our key contribution is SLUE (S-Lemma Uncertainty Estimation), a convex program to reduce this set to a single ellipsoidal uncertainty bound that is guaranteed to contain the true object pose with high probability. SLUE solves a relaxation of the minimum volume bounding ellipsoid problem inspired by the celebrated S-lemma. It requires no initial guess of the bound's shape or size and is guaranteed to contain the true object pose with high probability. For tighter uncertainty bounds at the same confidence, we extend SLUE to a sum-of-squares relaxation hierarchy which is guaranteed to converge to the minimum volume ellipsoidal uncertainty bound for a given set of keypoint constraints. We show this pose uncertainty bound can easily be projected to independent translation and axis-angle orientation bounds. We evaluate SLUE on two pose estimation datasets and a real-world drone tracking scenario. Compared to prior work, SLUE generates substantially smaller translation bounds and competitive orientation bounds. We release code at https://github.com/MIT-SPARK/PoseUncertaintySets.

• The paper introduces SLUE, a statistically principled method quantifying 6-DoF pose uncertainty via conformal prediction and convex relaxations based on the generalized S-Lemma.
• It efficiently computes minimum-volume ellipsoidal bounds for joint translation and rotation uncertainty, outperforming baselines in tightness and speed.
• Empirical evaluations on LineMOD-Occlusion, YCB-Video, and drone-car tracking validate SLUE's real-time applicability and robustness for downstream planning in robotics.

Statistically Principled Uncertainty Quantification for Monocular Visual Object Pose Estimation

Overview

The paper "Uncertainty Quantification for Visual Object Pose Estimation" (2511.21666) proposes SLUE (S-Lemma Uncertainty Estimation), a statistically rigorous methodology for quantifying uncertainty in 6-DoF object pose estimation from monocular RGB images. SLUE leverages distribution-free conformal prediction for keypoint localization and propagates these pixel uncertainty sets through a hierarchy of convex relaxations—rooted in a generalization of the S-Lemma—to derive explicit ellipsoidal bounds on object pose. These bounds are guaranteed, under minimal distributional assumptions, to include the true object pose with high probability.

Unlike prior work reliant on strong parametric noise models, spherical uncertainty bounds, or model-dependent heuristics, the SLUE framework solves explicitly for an interpretable, minimum-volume outer ellipsoid enclosing the implicit, non-convex pose constraint set arising from semantic keypoint detections. Joint and marginal uncertainty bounds (in translation and rotation) can be efficiently extracted, and empirical results demonstrate substantial gains in tightness and speed compared to contemporary set-membership approaches.

Measurement Model and Pose Uncertainty Constraints

SLUE builds upon the paradigm of keypoint-based pose estimation. A neural front-end provides pixel positions for a sparse set of annotated 3D object keypoints, each accompanied by a high-confidence uncertainty set—derived via split conformal prediction and modeled as infinity-norm (axis-aligned) pixel error bounds.

The object pose uncertainty set is implicitly defined as the set of all translations and orientations consistent with backprojection constraints (each keypoint must reproject into its high-confidence uncertainty set) and chirality (positive depth). The intersection of these $N$ sets yields a generally non-convex region in pose space. The paper explicitly demonstrates that, even with correct conformal calibration, the worst-case coverage guarantee for the joint pose set can degrade rapidly with $N$ due to possible error correlation among keypoints; empirical calibration suggests that coverage is generally intermediate between best- and worst-case bounds.

Figure 1: Keypoint detection provides 2D pixel coordinates and conformal uncertainty sets by calibrating detection error, visualized as blue boxes, for each semantically labeled keypoint.

Figure 2: SLUE projects high-probability keypoint uncertainty sets from 2D image space to non-convex quadratic pose constraints, reduces them to explicit ellipsoidal bounds via the S-Lemma hierarchy, and visualizes the resulting pose uncertainty set centered on the estimator.

S-Lemma-Based Bounding Ellipsoid Formulation

To obtain an explicit, interpretable bound for practical downstream use, SLUE seeks a minimum-volume ellipsoid centered at a pose estimate, guaranteed to contain the true pose whenever the implicit pose constraints are satisfied.

The key technical contribution is recasting the minimum-volume bounding ellipsoid problem over quadratic (and polynomial) constraints into a semidefinite program using a generalized S-Lemma. The classical S-Lemma dualizes containment over one constraint; here, a relaxation hierarchy leverages sum-of-squares programming to guarantee convergence of the ellipsoidal outer bound as the relaxation order increases.

Figure 3: Visualization of the bounding ellipsoid hierarchy at increasing relaxation orders $\kappa$. As $\kappa$ rises, SLUE converges to the tightest volume ellipsoidal outer bound.

Joint bounds are formulated in the $13$-dimensional space of homogeneous pose variables (translation vector and vectorized rotation matrix), with $15$ equality constraints to encode $\mathrm{SO}(3)$ and $5N$ inequalities from backprojection and chirality. Marginalization is performed via projection to translation and to axis-angle parameterizations for interpretation; the latter employs a geometric mapping from matrix deviations to skew-symmetric axis-angle vectors using the Rodrigues formula.

Empirical Evaluation and Comparative Analysis

SLUE is evaluated on the LineMOD-Occlusion (LM-O), YCB-Video (YCB-V), and a real-world drone-car tracking sequence (CAST) across $>10k$ images, comparing against baselines (RANSAG, GRCC) that employ fixed-shape ellipsoidal outer bounds.

Figure 4: CDFs of translational and angular uncertainty volumes; SLUE yields significantly tighter translation bounds across all datasets, and competitive angular bounds, particularly at first-order relaxation.

Figure 5: Image-plane projections of second-order joint ellipsoidal pose uncertainty, revealing typical uncertainty geometry concentrated along the optical axis due to scale ambiguity.

Strong numerical results are reported. First- and second-order SLUE ellipsoids are often several orders of magnitude smaller in volume than competing relaxations for translation; angular bounds are competitive, with some conservatism attributable to the choice of infinity-norm. Importantly, SLUE simultaneously optimizes the joint uncertainty ellipsoid (not two independently), preserving correlation between translation and orientation uncertainty—a critical property for downstream planning and control algorithms.

Figure 6: Projections of first and second-order uncertainty ellipsoids for translation and rotation space, showing expressiveness of SLUE in capturing scale ambiguity and uncertainty geometry beyond spherical baselines.

SLUE's convex formulation and efficient quaternion-based implementation yield runtimes that are frequently lower than keypoint detection, and much lower than prior relaxations—making it suitable for real-time robotics.

Implications, Limitations, and Future Directions

SLUE introduces a robust, distribution-free framework for certifying uncertainty in monocular object pose estimation. It is agnostic to the pose estimator, requires only calibrated keypoint uncertainty sets, and produces explicit bounds that are interpretable and practical for robust planning under uncertainty.

The general methodology can be leveraged to initialize downstream active perception, robust control, and multi-object tracking pipelines with statistically rigorous uncertainty primitives. The approach’s independence from estimator-specific noise models enables broad transferability.

Potential directions for future work include:

• Robust multiple-testing integration: Incorporating combinatorial reasoning when some keypoints are outside their uncertainty sets, mirroring outlier-robust estimation literature.
• Adaptive ellipsoid centering: Jointly optimizing estimator center and ellipsoid geometry within the S-lemma hierarchy for improved tightness.
• Symmetry-aware uncertainty: Extending the framework to account for orientation ambiguity due to object symmetries and multimodal pose hypotheses.
• Temporal fusion: Deploying SLUE in sequential frame contexts to perform Bayesian fusion or multi-view set intersection for further reduction of uncertainty over time.

Conclusion

SLUE establishes an efficient, statistically guaranteed procedure for quantifying high-probability uncertainty bounds in monocular object pose estimation using convex optimization and sum-of-squares relaxations. The method yields tighter, more expressive uncertainty sets than previous approaches, is computationally tractable for real-time deployment, and prioritizes correct coverage guarantees without strong modeling assumptions. SLUE paves the way for principled uncertainty integration across robotics perception and control systems.