Pairwise-Relative Camera Pose Encodings (CaPE)

• The paper introduces a deep Siamese CNN architecture with spatial pyramid pooling that efficiently encodes relative camera poses without relying on explicit intrinsic calibration.
• It leverages transfer learning and multi-scale feature extraction to achieve precise 3D reconstruction and robust visual localization in varied real-world conditions.
• Experimental results show that CaPE outperforms traditional SURF/ORB pipelines in low-texture and wide-baseline scenarios by reducing orientation and translation errors.

Pairwise-Relative Camera Pose Encodings (CaPE) are algorithmic and architectural strategies for representing, learning, or leveraging the geometric relationship between pairs of camera views, typically in the context of deep learning for visual localization, 3D reconstruction, or novel view synthesis. Central to these methods is the encoding of relative rather than absolute camera poses, which enables invariance to global coordinate frames, improved inductive bias for geometric consistency, and greater flexibility across a range of tasks and data regimes. The following sections provide a technical and comprehensive overview of CaPE as originally formulated and deployed in "Relative Camera Pose Estimation Using Convolutional Neural Networks" (Melekhov et al., 2017).

1. Network Architecture and Pose Encoding

The foundational CaPE approach utilizes a Siamese convolutional neural network (CNN) comprising two identical branches for feature extraction from paired RGB images. Each branch follows an AlexNet-inspired architecture initialized via transfer learning from a Hybrid-CNN model pretrained on large-scale datasets (e.g., ImageNet, Places). Structural details of one branch include the sequence:

• ConvB1₍₉₆,11,4,0₎ → ReLU → pool₍₃,2₎
• ConvB2₍₂₅₆,5,1,2₎ → ReLU → pool₍₃,2₎
• ConvB3₍₃₈₄,3,1,1₎ → ReLU
• ConvB4₍₃₈₄,3,1,1₎ → ReLU
• ConvB5₍₂₅₆,3,1,1₎ → ReLU → [optional final pooling]

Variants include:

• cnnA: last pooling, output feature map 6×6
• cnnB: no final pooling, output feature map 13×13

The extracted features from both input images are concatenated and passed through two fully connected regression layers outputting a seven-dimensional pose vector:

$\Delta p = [\Delta q,\, \Delta t]$

where $\Delta q$ is a 4D quaternion (relative orientation), and $\Delta t$ is a 3D translation direction.

Spatial Pyramid Pooling (SPP) is optionally appended to each branch, enabling fixed-length representations for arbitrary input image sizes by pooling over multiple spatial bins at multiple scales, thus preserving hierarchical spatial features relevant for robust relative geometry estimation.

2. Input/Output and Post-processing

The network consumes a pair of raw RGB images—each corresponding to a different camera viewpoint of a static scene. Each branch processes one image, and their deep features are merged for regression. The direct output is the 7D $\Delta p$, where:

• $\Delta q$ (quaternion) is normalized at inference to ensure it represents a valid 3D rotation (unit-norm).
• $\Delta t$ is normalized to encode translation direction only (up-to-scale), as the network lacks access to absolute scale.

This representation captures the pairwise geometric transformation between the input camera views, abstracting away from any global coordinate frame.

3. Training Methodology and Loss Formulation

Training is performed end-to-end with a loss function explicitly balancing rotation and translation prediction errors:

$$\mathcal{L} = \| \widehat{\Delta t} - \Delta t \|_2 + \beta \, \| \widehat{\Delta q} - \Delta q \|_2$$

where $\beta$ (set to 10 experimentally) scales the orientation component to match its typical range and impact.

Weight initialization is performed with Hybrid-CNN (combining ImageNet and Places pretraining), allowing effective transfer of low-level representations and accelerating convergence. The training dataset comprises approximately $581\,\text{K}$ image pairs from the 1DSfM project (landmarks/crowdsourced photos), subsequently refined on the DTU dataset for high-precision pose accuracy. The network is optimized using Adam, a learning rate of $10^{-4}$, batches of size 128, and weight decay $10^{-5}$.

4. Comparative Performance Against Traditional Pipelines

The CaPE CNN approach is evaluated against classical local feature-based systems (e.g., SURF, ORB), which require feature extraction, matching, and robust essential matrix estimation—typically requiring precise camera intrinsics and often failing in scenes with repetitive or textureless content.

Quantitatively, on the DTU dataset (robot-mounted, ground-truth-calibrated), the CNN method—particularly with SPP (cnn-spp)—demonstrates lower median errors in relative orientation and translation direction, especially in challenging cases:

• Handles large viewpoint baselines and low-texture areas more robustly than SURF/ORB.
• Does not require explicit knowledge of intrinsic calibration, simplifying deployment across heterogeneous datasets and cameras.

5. The Role and Impact of Spatial Pyramid Pooling (SPP)

SPP layers enable the network to accommodate variable input image sizes (by design, necessary for images originating from uncontrolled datasets) while maintaining fixed-size representations for the regression head. Multiple pyramid levels (e.g., 1×1, 2×2, 3×3, 6×6 for cnnAspp; additional 13×13 for cnnBspp) capture local and global context, critical for resolving ambiguities in wide-baseline matching. Empirical evaluation indicates that SPP-enhanced architectures consistently outperform non-SPP variants, with improved accuracy and generalization stemming from the preservation of multiscale geometric cues.

6. Practical Applications and Broader Implications

The CaPE paradigm is directly applicable to several computer vision domains:

• Structure from Motion (SfM): Direct estimation of pairwise camera poses for incremental or global scene reconstruction.
• SLAM and Visual Odometry: Robust tracking even under adverse conditions (repetitive/textureless scenes, variable intrinsics).
• Visual Navigation/Relocalization: Efficient pose inference without reliance on global/scene-specific coordinate frames or feature engineering.

The system’s independence from intrinsic calibration, scalability enabled by end-to-end learning, and robustness across scene variabilities point toward future extensions where multi-stage networks provide hierarchical pose refinement, and dense feature representations fuse with explicit geometric constraints to further enhance performance and reliability.

7. Summary Table: CaPE vs. Classical Pipelines (as in (Melekhov et al., 2017))

Approach	Input	Outputs	Robustness to Intrinsics	Performance in Challenging Scenes
CaPE Siamese CNN (SPP)	Pair RGB images	Relative pose	Not required	High: handles repetitive/textureless/large baseline scenes
SURF/ORB + Essential	Pair RGB images, K	Relative pose	Required for E-matrix	Degraded in low-texture, repetitive, or poorly calibrated data

In conclusion, Pairwise-Relative Camera Pose Encodings (CaPE) as implemented in the Siamese CNN/SPP pipeline offer a methodologically robust, scale- and coordinate-invariant foundation for geometric inference in 3D vision, surpassing traditional local feature pipelines under complex real-world conditions and enabling a new class of efficient, learning-based visual localization systems.