Globally Optimal Pose from Orthographic Silhouettes

Abstract: We solve the problem of determining the pose of known shapes in $\mathbb{R}^3$ from their unoccluded silhouettes. The pose is determined up to global optimality using a simple yet under-explored property of the area-of-silhouette: its continuity w.r.t trajectories in the rotation space. The proposed method utilises pre-computed silhouette-signatures, modelled as a response surface of the area-of-silhouettes. Querying this silhouette-signature response surface for pose estimation leads to a strong branching of the rotation search space, making resolution-guided candidate search feasible. Additionally, we utilise the aspect ratio of 2D ellipses fitted to projected silhouettes as an auxiliary global shape signature to accelerate the pose search. This combined strategy forms the first method to efficiently estimate globally optimal pose from just the silhouettes, without being guided by correspondences, for any shape, irrespective of its convexity and genus. We validate our method on synthetic and real examples, demonstrating significantly improved accuracy against comparable approaches. Code, data, and supplementary in: https://agnivsen.github.io/pose-from-silhouette/

• The paper introduces a method that leverages silhouette area and elliptical aspect ratio signatures to achieve correspondence-free, globally optimal pose estimation.
• It separates orientation and translation recovery by initially optimizing rotation through efficient branch-and-bound techniques and refining via local non-linear optimization.
• Empirical evaluations show an 85–89% reduction in mean orientation error compared to existing methods, proving robust performance under noise.

Globally Optimal Pose Estimation from Orthographic Silhouettes

Introduction

The problem of estimating the pose of a 3D rigid object given only the observed 2D silhouette—without reliance on image textures, point correspondences, or other cues—has been a longstanding challenge. "Globally Optimal Pose from Orthographic Silhouettes" (2604.09199) establishes, for the first time, a practical and globally optimal solution to the correspondence-free pose-from-silhouette (PFS) problem for general object topology and geometry under orthographic (and approximately, perspective) projection. This approach leverages the continuity properties of geometric silhouette signatures, specifically the area of the silhouette and the aspect ratio of a best-fit ellipse, to efficiently branch and search the rotation group $\mathrm{SO}(3)$, allowing exactly optimal (up to discretization limits) 3-DoF orientation and 2-DoF translation recovery.

Formalization of the PFS Problem

The PFS task is formulated as minimizing the Hausdorff distance between the observed (possibly noisy) input silhouette and the reprojected silhouette of a known 3D template, under the action of $\mathrm{SE}(3)$ transformations. The problem is ill-posed for symmetries or degenerate projections, but is treated without restrictions on shape convexity, genus, or distinctiveness. The authors precisely formalize the constraint set and show under reasonable assumptions that at least one solution exists with vanishing Hausdorff distance.

Methodological Innovations

Silhouette Area Signature and Rotation Space Branching

The central theoretical insight is that the area of the unoccluded orthographic silhouette is a continuous function with respect to (almost all) continuous rotation trajectories in $\mathbb{SO}(3)$ when the object is specified by a mesh or point cloud. This enables a geometric “shape signature” mapping from a substantial portion of the $\mathrm{SO}(3)$ group to $\mathbb{R}$. By discretizing the rotation group (via the Postel ball and disc parameterizations) and pre-computing a dense response (signature) surface via projection and area computation, the search for feasible orientations is strongly pruned.

The translation component is handled by exploiting the invariance of the silhouette area and its centroid under in-plane ($XY$) translation. Thus, the pose search is separated: orientation is recovered first, then the translation that aligns silhouette centroids.

Elliptical Aspect Ratio as Auxiliary Signature

While the area signature constrains the orientation except for in-plane ($Z$-axis) rotations, this residual ambiguity is tackled by the auxiliary use of the aspect ratio of algebraically fitted ellipses to the silhouettes. Aspect ratio varies with orientation (except for certain symmetric or degenerate cases), and is empirically shown to be monotonic or locally constant almost everywhere. Pre-computation of this second signature allows further branching and elimination of infeasible orientations.

Multi-Stage Candidate Pose Generation and Refinement

Each observed silhouette yields, by intersection with the area and aspect ratio signature surfaces, a set of candidate orientations in $\mathrm{SO}(3)$. These are exhaustively but efficiently checked by direct silhouette projection and Hausdorff evaluation. Final selection is performed via a local non-linear refinement on the $\mathrm{SE}(3)$ manifold, initialized at the best-scoring global candidate.

For perspective cases, the approach can be extended by incorporating a depth prior, as exact global optimality is no longer tractable due to coupling of depth and shape signatures.

Empirical Evaluation

The proposed algorithm is evaluated on synthetic and real datasets, including the rigid, high-genus, and highly symmetric objects, and real segmented data from benchmark datasets. With additive noise, its mean orientation error ($\mathrm{OE}$) for all shapes remains below $\mathrm{SE}(3)$0, and the worst-case errors are bounded ($\mathrm{SE}(3)$1), which constitutes a substantial improvement (85–89% reduction in mean $\mathrm{SE}(3)$2) over contemporary baselines and local/non-convex optimization strategies.

Robustness to silhouette noise is systematically analyzed, with the probability of successful estimation as a function of candidate pose depth and silhouette deformation illustrated below.

(Figure 1)

Figure 1: Success rate of pose estimation across the seven best solution candidates, denoted C$\mathrm{SE}(3)$3 ($\mathrm{SE}(3)$4), highlighting graceful degradation under increasing silhouette noise.

Parameter ablation demonstrates that the method maintains accuracy over wide discretization intervals, with greater sampling density trading off against runtime but not numerical stability. The candidate set size $\mathrm{SE}(3)$5 is shown to scale intuitively with object symmetry: spheres produce many ambiguous solutions, while typical real-world geometries yield unique or nearly unique optima.

Qualitative Results

To validate the method’s performance on non-trivial, real images featuring complex, asymmetric shapes and cluttered backgrounds, the approach is applied to the “complex_movable_handheld” sequences of the BCOT dataset. The estimated and ground-truth silhouettes demonstrate visually precise and consistent alignment.

(Figure 2)

Figure 2: Qualitative results on the asymmetric shapes from complex_movable_handheld sequence—ground-truth (red) versus estimated (blue) silhouettes—demonstrating alignment even in challenging configurations.

Discussion and Implications

The evidence supports the assertion that, in the correspondence-free regime, the combination of area and aspect-ratio shape signatures is sufficient for practical global pose recovery under orthographic projection, except for objects and viewpoints that induce geometric ambiguity. The framework is efficient relative to general branch-and-bound alternatives by virtue of response-surface pre-computation and declarative feature intersection.

Practical implications include advances in settings lacking texture or feature correspondences, e.g., in robotics, industrial inspection, or medical imaging where cross-modal alignment relies exclusively on object outlines. The generalization to perspective, while not strictly globally optimal, is analytically and empirically justified when a depth prior is available, and the pipeline accommodates extensions for enhanced robustness or shape uncertainty quantification.

Future research may focus on integrating higher-order or learned global shape signatures, characterizing the identifiability limits in the presence of symmetry and occlusion, and further parallelizing the search and refinement for real-time deployment.

Conclusion

This work provides the first practical, globally optimal solution (up to discretization and noise) to the rigid pose-from-silhouette problem in the absence of correspondences or intensity cues, substantiated by rigorous mathematical analysis and strong numerical results. The approach constitutes a methodological and practical benchmark for silhouette-based 3D object localization and will likely serve as a foundational technique for future research in geometric perception, model-based vision, and robotics (2604.09199).