DirectionNet: Discrete Deep Pose Estimation

• DirectionNet is a deep learning framework for wide-baseline camera pose estimation that predicts 5D relative poses through discrete spherical probability distributions.
• It factorizes the pose into independent directional components, leveraging a two-stage encoder-decoder architecture to mitigate challenges from occlusion and low image overlap.
• Experimental results show up to 50% lower rotation and translation errors than direct regression methods, demonstrating its robust performance on synthetic and real-world datasets.

DirectionNet is a deep learning framework for wide-baseline relative camera pose estimation, specifically designed to predict discrete probability distributions over the 5D space of relative camera poses between image pairs. Traditional regression-based and keypoint-driven methods for pose estimation face significant challenges in scenarios with large viewpoint changes, severe occlusion, or low image overlap. DirectionNet circumvents these limitations by factorizing the relative pose and leveraging dense, spherical discretization, resulting in substantial gains over direct regression methods in challenging synthetic and real-world datasets (Chen et al., 2021).

1. Factorized Pose Parameterization and Model Architecture

DirectionNet represents the 5D relative pose $(R, t)$, where $R \in SO(3)$ is the relative rotation and $t \in S^2$ is the unit translation direction, via four independent 3D unit vectors: three orthonormal axes for rotation ($x, y, z \in S^2$), and one for translation ($t \in S^2$). The alignment of $[x \; y \; z]$ approximates $R$, and, together with $t$, fully parameterizes the 5D pose.

The model is a fully convolutional, two-stage encoder-decoder neural network. The decoder predicts, for each image pair, four separate $H \times W$ equirectangular grids that correspond to soft probability maps over $S^2$ for each of the direction vectors ($R \in SO(3)$0). Each map is converted to a distribution via softplus activation and spherical normalization: $R \in SO(3)$1 where $R \in SO(3)$2 discretize the sphere.

Spherical expectation yields predicted directions: $R \in SO(3)$3 with $R \in SO(3)$4, and the unit direction $R \in SO(3)$5.

For rotation, the three unit vectors $R \in SO(3)$6 are projected onto $R \in SO(3)$7 via the orthogonal Procrustes solution using SVD: $R \in SO(3)$8 An alternative Gram–Schmidt variant uses only $R \in SO(3)$9 and $t \in S^2$0, setting $t \in S^2$1.

2. Discrete Spherical Probability Modeling

Each predicted direction is modeled as a discrete probability mass function over the sphere, parameterized as an $t \in S^2$2 equirectangular grid. This discretization allows the network to represent multimodal and uncertain pose hypotheses efficiently, using only $t \in S^2$3 bins versus $t \in S^2$4 for naive quaternion binning in $t \in S^2$5.

The spherical expectation aggregates the predicted probability mass into a single direction hypothesis, and this procedure applies independently to each of the four direction components. This factorization transforms the intrinsically high-dimensional pose regression into the estimation of multiple structured 2D distributions on the sphere.

3. Loss Supervision and Training Strategy

DirectionNet employs a composite supervision strategy for each direction:

• Directional Alignment (negative cosine similarity):

$t \in S^2$6

for unit vectors $t \in S^2$7.

• Distribution Map Regression (spherical MSE):

$t \in S^2$8

where $t \in S^2$9 is a von Mises–Fisher heatmap centered at the ground truth direction.

• Concentration Loss:

$x, y, z \in S^2$0

encouraging unimodality of the predicted distribution.

The final loss combines these terms per direction, with fixed weights $x, y, z \in S^2$1 $x, y, z \in S^2$2: $x, y, z \in S^2$3

Data augmentation uses random perturbations of up to $x, y, z \in S^2$4 in rotation before derotation, in addition to standard leaky-ReLU and dropout. The architecture features no fully connected regression layers.

4. Experimental Protocol and Quantitative Evaluation

Experiments utilize both synthetic (InteriorNet A/B) and real (Matterport3D A/B) datasets, operating on $x, y, z \in S^2$5 image crops (90° FoV) with controlled rotations. Each dataset comprises 1M training and approximately 1k test pairs with non-overlapping scenes.

Key evaluation metrics include:

• Rotation error:

$x, y, z \in S^2$6

• Translation direction error:

$x, y, z \in S^2$7

Performance on Matterport-B ($x, y, z \in S^2$8 rotation baseline) is summarized below:

Method	mean(ΔR)	med(ΔR)	mean(Δt)	med(Δt)
6D direct-regression	18.23°	7.69°	39.06°	25.07°
Quaternion reg.	28.38°	19.23°	48.99°	34.94°
DirectionNet-6D	14.85°	3.69°	23.60°	9.42°
DirectionNet-9D (ours)	13.60°	3.54°	21.26°	8.90°

DirectionNet-9D shows a ≈25% reduction in rotation error and ≈46% in translation error versus best direct-regression baseline.

Classic feature-based (SIFT+LMedS) approaches attain low errors only with high image overlap, with significant performance degradation below 30% overlap. DirectionNet, by contrast, demonstrates stable performance as overlap decreases.

5. Ablation Analysis and Model Behavior

• SVD vs Gram–Schmidt projection: The SVD-based (DirectionNet-9D) method outperforms the Gram–Schmidt (DirectionNet-6D) variant for rotation recovery.
• Single-stage vs two-stage derotation: Omitting image derotation results in ≈15–30% higher error rates, underscoring the benefit of the two-stage approach.
• Loss component importance: Removing the distribution map regression term ($x, y, z \in S^2$9) increases mean rotation error from 3.96° to ≈14.7°, while ablating either the cosine similarity or concentration losses yields milder degradations (to ≈4.8°). This finding highlights the central role of dense distributional supervision.
• Discretization scheme: Spherical direction binning significantly outperforms naive $t \in S^2$0 quaternion binning, supporting the design choice of independent $t \in S^2$1 parameterizations for scalability.

Qualitative results show DirectionNet robustly predicts valid epipolar geometry even under heavy occlusion and minimal overlap. Failure modes tend to involve ambiguous, repetitive, or textureless scenes, or extreme viewpoint changes.

6. Limitations, Implications, and Future Directions

DirectionNet establishes that discrete, structured output spaces—when suitably factorized—enable viable learning for high-dimensional pose estimation problems. The method is currently limited in handling absolute translation scale, scenarios with negligibly small translation baselines, and scenes with symmetry-induced pose ambiguity.

Potential areas for future work include the integration of spherical CNN decoders to exploit rotation equivariance, extension to essential matrix estimation (with unknown scale), and modeling dependencies between the predicted direction vectors. Joint distributions could improve robustness by capturing inter-vector correlations, addressing independence assumptions in the current factorization.

By recasting 5D pose estimation as four discrete spherical estimation problems, and aggregating via expectation and projection, DirectionNet achieves up to ~50% lower errors than prevailing direct-regression techniques on wide-baseline benchmarks (Chen et al., 2021).