IUP-Pose: Decoupled Relative Pose Regression

• IUP-Pose is a relative pose regression method that decouples rotation and translation estimation through a geometry-driven, homography-based pipeline.
• It employs an implicit dense alignment module and iterative uncertainty-aware rotation refinement before estimating translation from de-rotated features.
• The method achieves real-time performance with 70 FPS and balanced AUC metrics, making it effective for calibrated two-view camera motion estimation.

Searching arXiv for the IUP-Pose paper and directly related relative pose regression work. IUP-Pose is a relative pose regression method for estimating calibrated two-view camera motion from image pairs by combining end-to-end differentiability, explicit projective-geometric structure, and real-time efficiency. It is formulated for the standard task of predicting relative rotation $\mathbf{R} \in SO(3)$ and translation direction $\mathbf{t} \in \mathbb{S}^2$ from two overlapping images and their intrinsics, but departs from monolithic regression by treating rotation and translation as distinct subproblems. Its defining design is a geometry-driven decoupled iterative pipeline: cross-view features are first aligned implicitly, rotation is estimated and refined with uncertainty, and the resulting rotational homography $\mathbf{H}_\infty$ is then used to realign features before translation prediction (Wang et al., 20 Mar 2026).

1. Scope and problem formulation

IUP-Pose addresses relative pose regression (RPR), the problem of inferring the relative camera motion between two calibrated views $\mathbf{I}_0, \mathbf{I}_1$ with intrinsics $\mathbf{K}_0, \mathbf{K}_1$. The method predicts an axis-angle rotation vector and an unnormalized translation vector as

$$(\hat{\boldsymbol{\omega}}, \tilde{\mathbf{t}})=f_{\boldsymbol{\theta}}(\mathbf{I}_0,\mathbf{I}_1,\mathbf{K}_0,\mathbf{K}_1),$$

with the final pose recovered through

$$\hat{\mathbf{R}}=\mathrm{Exp}(\hat{\boldsymbol{\omega}}^\wedge), \qquad \hat{\mathbf{t}}=\frac{\tilde{\mathbf{t}}}{\|\tilde{\mathbf{t}}\|_2}.$$

This choice reflects the two-view calibrated setting, where translation is identifiable only up to scale (Wang et al., 20 Mar 2026).

The method is motivated by a trade-off identified in prior relative pose pipelines. Feature-matching systems can be highly accurate, but their use of non-differentiable matching and RANSAC blocks end-to-end gradient flow. By contrast, recent ViT-based regressors remain differentiable but are described as too expensive for real-time edge deployment. IUP-Pose explicitly frames its contribution around these bottlenecks, identifying two architectural weaknesses in existing RPR methods: the coupling of rotation and translation estimation, and insufficient cross-view feature alignment (Wang et al., 20 Mar 2026).

A central implication is that IUP-Pose should be understood not merely as a lightweight regressor, but as a regression architecture organized around projective geometry. Rotation is treated as the global alignment component, while translation is treated as the residual parallax component after de-rotation. This suggests a design closer in spirit to geometry-aware deep estimation than to direct black-box pose regression.

2. Geometric decomposition and decoupled estimation

The conceptual foundation of IUP-Pose is a homography decomposition. For a local plane $\pi=[\mathbf{n}^\top,d]^\top$, the induced homography is

$$\mathbf{H}=\mathbf{K}_2\left(\mathbf{R}+\frac{\mathbf{t}\mathbf{n}^\top}{d}\right)\mathbf{K}_1^{-1}.$$

For general scenes, the paper uses a dense local-plane approximation,

$$\mathbf{H}_{ij}=\mathbf{K}_2\left(\mathbf{R}+\frac{\mathbf{t}\mathbf{n}_{ij}^\top}{d_{ij}}\right)\mathbf{K}_1^{-1},$$

and factorizes it into a rotation-only component and a translation-dependent residual: $\mathbf{t} \in \mathbb{S}^2$0 Here $\mathbf{t} \in \mathbb{S}^2$1 is the infinite homography induced purely by rotation, while $\mathbf{t} \in \mathbb{S}^2$2 captures local translation and scene-geometry effects (Wang et al., 20 Mar 2026).

This decomposition motivates the paper’s decoupled pipeline. Rotation is estimated first because it corresponds to the global alignment term $\mathbf{t} \in \mathbb{S}^2$3. Once a rotation estimate is available, features from one view are rewarped using the associated rotational homography, reducing rotational disparity and isolating the translation-sensitive residual structure required by the translation decoder. The paper argues that joint regression of rotation and translation introduces redundancy and prevents a globally optimal solution, because the two quantities depend on different visual cues: rotation on broad global alignment, translation on local parallax and scene structure (Wang et al., 20 Mar 2026).

This suggests that the novelty of IUP-Pose is not simply the presence of separate heads, but the use of a specific geometric factorization to justify a rotate-then-translate estimation order. The architecture operationalizes that factorization through explicit feature warping.

3. Input representation and implicit dense alignment

The network takes each RGB image and augments it with normalized image-plane coordinates derived from the intrinsics. For pixel coordinate $\mathbf{t} \in \mathbb{S}^2$4, the normalized coordinate is

$\mathbf{t} \in \mathbb{S}^2$5

and the final encoder inputs are

$\mathbf{t} \in \mathbb{S}^2$6

Thus each input is a 5-channel tensor: RGB plus $\mathbf{t} \in \mathbb{S}^2$7 (Wang et al., 20 Mar 2026). This directly injects camera-geometry information into the backbone rather than leaving intrinsics to be inferred implicitly from appearance.

The backbone is a ResNet-based encoder whose first convolution is adapted from 3 channels to 5. Only the $\mathbf{t} \in \mathbb{S}^2$8-resolution feature map is passed to the alignment module, a design choice explicitly made for speed. Training and evaluation use resized images with width $\mathbf{t} \in \mathbb{S}^2$9 and height $\mathbf{H}_\infty$0, so the stride-32 map is approximately $\mathbf{H}_\infty$1 spatially (Wang et al., 20 Mar 2026).

The alignment mechanism, termed Implicit Dense Alignment (IDA), has two components: Shared Spatial Pyramid Pooling-Fast (SPPF) and Multi-Head Bi-Cross Attention (MHBC). SPPF first computes a bottleneck feature $\mathbf{H}_\infty$2, then recursively applies max pooling: $\mathbf{H}_\infty$3 and fuses the resulting multi-scale context through

$\mathbf{H}_\infty$4

The paper interprets SPPF as enlarging the receptive field while also behaving like an implicit keypoint detector through local maxima (Wang et al., 20 Mar 2026).

After SPPF, the model applies a single bidirectional cross-attention layer. The basic alignment block is

$\mathbf{H}_\infty$5

where

$\mathbf{H}_\infty$6

In this formulation, $\mathbf{H}_\infty$7 supplies content aggregation, $\mathbf{H}_\infty$8 injects global positional structure through compressed attention patterns, and $\mathbf{H}_\infty$9 adds local positional enhancement using dilated depthwise convolutions with dilation rates $\mathbf{I}_0, \mathbf{I}_1$0 and $\mathbf{I}_0, \mathbf{I}_1$1. The query and key dimension is compressed such that $\mathbf{I}_0, \mathbf{I}_1$2, reducing attention cost (Wang et al., 20 Mar 2026).

The resulting aligned cross-view features are not explicit correspondences in the classical sense, but the paper presents them as correspondence-like internal representations learned purely from pose supervision, without explicit matching supervision. A plausible implication is that IUP-Pose occupies an intermediate position between matcher-based geometry pipelines and direct regressors: it retains differentiable cross-view interaction while avoiding hard correspondence extraction.

4. Iterative rotation estimation and uncertainty propagation

The pose decoder is organized into three stages: coarse rotation, refined rotation, and translation. The two rotation stages share parameters and each predicts both an axis-angle rotation and a diagonal uncertainty estimate (Wang et al., 20 Mar 2026). The coarse stage outputs $\mathbf{I}_0, \mathbf{I}_1$3, from which the rotational homography is formed as

$\mathbf{I}_0, \mathbf{I}_1$4

This homography is applied in feature space: $\mathbf{I}_0, \mathbf{I}_1$5 The refined rotation stage then operates on these re-aligned features while also receiving the previous pose estimate and its uncertainty (Wang et al., 20 Mar 2026).

The final rotation is composed multiplicatively: $\mathbf{I}_0, \mathbf{I}_1$6 Each stage predicts a diagonal covariance, denoted $\mathbf{I}_0, \mathbf{I}_1$7 and $\mathbf{I}_0, \mathbf{I}_1$8, and the final propagated covariance is

$\mathbf{I}_0, \mathbf{I}_1$9

This uncertainty is used in two ways: as a heteroscedastic loss term during training and as an explicit conditioning signal passed to later stages (Wang et al., 20 Mar 2026). The paper states that uncertainty is positively correlated with prediction error, so propagation provides the refinement stage with information about the reliability of the initial estimate.

The decoder architecture used by the rotation and translation branches is lightweight and shared in form. It incorporates FiLM conditioning on camera intrinsics, input pose, and input uncertainty, and includes a MoE adapter with 2 experts and a View Fusion block. The View Fusion block contains 4 iterations of Robust Bottleneck Blocks, V-aware gates, progressively increasing kernel size $\mathbf{K}_0, \mathbf{K}_1$0 from 1 to 3 for cross-view mixing, depthwise separable refinement, residual addition, and a final view-wise softmax attention fusion (Wang et al., 20 Mar 2026). The paper does not provide a full layer-by-layer decoder specification in the supplied text, which matters for exact reimplementation.

This staged design is central to the method’s identity. Rather than repeatedly refining a single entangled pose vector, IUP-Pose performs a shallow but structured sequence of transformations in which rotation estimates directly modify the features consumed by subsequent stages.

5. Translation estimation after de-rotation

After the final rotation estimate is obtained, features are realigned again using the final rotational homography $\mathbf{K}_0, \mathbf{K}_1$1. The translation decoder then predicts the translation direction from the de-rotated feature pair (Wang et al., 20 Mar 2026). In the geometric interpretation established earlier, this stage estimates the residual structure associated with $\mathbf{K}_0, \mathbf{K}_1$2, not the full camera motion jointly.

This ordering is a defining operational property of IUP-Pose. Translation is not estimated from the original pair of features, but from features after rotational disparity has been explicitly reduced. The method thus imposes a separation between global rotational consistency and local parallax reasoning. The paper describes this as “explicit dense information flow” because the inter-stage connection is not only symbolic pose conditioning but also geometrically transformed feature maps (Wang et al., 20 Mar 2026).

A plausible implication is that the translation stage is easier to optimize than in a standard direct regressor, because it does not need to simultaneously explain orientation mismatch and translation-induced displacement. The ablations support this interpretation, especially through the large gains obtained when homography warping is enabled.

6. Training objective, datasets, and empirical behavior

The overall training loss is

$\mathbf{K}_0, \mathbf{K}_1$3

For each rotation stage, the prediction is converted to a rotation matrix and compared to the ground truth through the relative rotation

$\mathbf{K}_0, \mathbf{K}_1$4

The geodesic angle loss is

$\mathbf{K}_0, \mathbf{K}_1$5

and the uncertainty loss is

$\mathbf{K}_0, \mathbf{K}_1$6

The stage loss becomes

$\mathbf{K}_0, \mathbf{K}_1$7

with $\mathbf{K}_0, \mathbf{K}_1$8. Translation is trained with an angular Huber loss,

$\mathbf{K}_0, \mathbf{K}_1$9

using $$(\hat{\boldsymbol{\omega}}, \tilde{\mathbf{t}})=f_{\boldsymbol{\theta}}(\mathbf{I}_0,\mathbf{I}_1,\mathbf{K}_0,\mathbf{K}_1),$$0 rad and $$(\hat{\boldsymbol{\omega}}, \tilde{\mathbf{t}})=f_{\boldsymbol{\theta}}(\mathbf{I}_0,\mathbf{I}_1,\mathbf{K}_0,\mathbf{K}_1),$$1 (Wang et al., 20 Mar 2026).

Training is performed on MegaDepth following the LoFTR protocol with 368 training scenes and about 1.5 million image pairs, after pretraining on ScanNet and then fine-tuning on MegaDepth. Images are resized to $$(\hat{\boldsymbol{\omega}}, \tilde{\mathbf{t}})=f_{\boldsymbol{\theta}}(\mathbf{I}_0,\mathbf{I}_1,\mathbf{K}_0,\mathbf{K}_1),$$2, training pairs are filtered by overlap ratio $$(\hat{\boldsymbol{\omega}}, \tilde{\mathbf{t}})=f_{\boldsymbol{\theta}}(\mathbf{I}_0,\mathbf{I}_1,\mathbf{K}_0,\mathbf{K}_1),$$3, and evaluation is conducted on the unfiltered MegaDepth1500 benchmark (Wang et al., 20 Mar 2026). Optimization uses AdamW with learning rate $$(\hat{\boldsymbol{\omega}}, \tilde{\mathbf{t}})=f_{\boldsymbol{\theta}}(\mathbf{I}_0,\mathbf{I}_1,\mathbf{K}_0,\mathbf{K}_1),$$4, weight decay $$(\hat{\boldsymbol{\omega}}, \tilde{\mathbf{t}})=f_{\boldsymbol{\theta}}(\mathbf{I}_0,\mathbf{I}_1,\mathbf{K}_0,\mathbf{K}_1),$$5, gradient clipping $$(\hat{\boldsymbol{\omega}}, \tilde{\mathbf{t}})=f_{\boldsymbol{\theta}}(\mathbf{I}_0,\mathbf{I}_1,\mathbf{K}_0,\mathbf{K}_1),$$6, a one-cycle schedule, 4,000 warm-up steps, 320,000 total training steps, batch size 20 image pairs, 10 data-loader workers, and 4 NVIDIA A800 GPUs (Wang et al., 20 Mar 2026).

The main benchmark result reported for IUP-Pose is AUC@5 = 27.9, AUC@10 = 52.6, and AUC@20 = 73.3 on MegaDepth1500, with 70 FPS, 14.3 ms latency per image pair, and 37M parameters on a single NVIDIA RTX 4090 (Wang et al., 20 Mar 2026). Compared with other RPR methods in the paper’s table, this is not the highest-accuracy result: Reloc3r-512 reaches AUC@20 = 81.2 at 40 FPS, and non-RPR systems such as ROMA and Efficient LoFTR are more accurate still, though slower or non-differentiable as complete pipelines. The paper’s explicit claim is therefore an accuracy-efficiency trade-off, not absolute state of the art (Wang et al., 20 Mar 2026).

The method also reports AUC@10 across overlap regimes:

• $$(\hat{\boldsymbol{\omega}}, \tilde{\mathbf{t}})=f_{\boldsymbol{\theta}}(\mathbf{I}_0,\mathbf{I}_1,\mathbf{K}_0,\mathbf{K}_1),$$7: 24.8
• $$(\hat{\boldsymbol{\omega}}, \tilde{\mathbf{t}})=f_{\boldsymbol{\theta}}(\mathbf{I}_0,\mathbf{I}_1,\mathbf{K}_0,\mathbf{K}_1),$$8: 50.5
• $$(\hat{\boldsymbol{\omega}}, \tilde{\mathbf{t}})=f_{\boldsymbol{\theta}}(\mathbf{I}_0,\mathbf{I}_1,\mathbf{K}_0,\mathbf{K}_1),$$9: 57.5
• $$\hat{\mathbf{R}}=\mathrm{Exp}(\hat{\boldsymbol{\omega}}^\wedge), \qquad \hat{\mathbf{t}}=\frac{\tilde{\mathbf{t}}}{\|\tilde{\mathbf{t}}\|_2}.$$0: 58.3
• overall: 52.6 (Wang et al., 20 Mar 2026)

This suggests that IUP-Pose remains functional even at low overlap, though performance still improves with stronger overlap.

7. Ablations, significance, and limitations

The ablation study is especially important for interpreting IUP-Pose as a coherent method rather than a collection of modules. Starting from a weak baseline with AUC@5 = 4.1, AUC@10 = 14.2, and AUC@20 = 31.3, the paper adds successive components: RT-Dec, Iter, IDA, Uncert, Homo, and Pre. The final configuration reaches 27.9 / 52.6 / 73.3 (Wang et al., 20 Mar 2026).

The decoupled rotation-translation design raises performance from 4.1 / 14.2 / 31.3 to 8.4 / 19.0 / 37.9. Adding iterative refinement yields 10.6 / 23.3 / 42.9. IDA increases this to 14.8 / 28.5 / 45.2. Uncertainty propagation further improves to 16.8 / 31.3 / 48.8. The largest single architectural gain comes from homography warping, which lifts results to 22.5 / 40.9 / 57.6. Finally, ScanNet pretraining produces the full 27.9 / 52.6 / 73.3 result (Wang et al., 20 Mar 2026).

These ablations support three conclusions. First, the method’s core claim about decoupling is empirically consequential rather than cosmetic. Second, implicit dense alignment contributes particularly at strict thresholds, which is consistent with the notion that better alignment sharpens precise relative-pose estimation. Third, rotational homography warping is the most important geometry-specific ingredient beyond generic deep refinement. This strongly supports the paper’s framing of $$\hat{\mathbf{R}}=\mathrm{Exp}(\hat{\boldsymbol{\omega}}^\wedge), \qquad \hat{\mathbf{t}}=\frac{\tilde{\mathbf{t}}}{\|\tilde{\mathbf{t}}\|_2}.$$1-based realignment as the key bridge between projective geometry and learned regression (Wang et al., 20 Mar 2026).

The paper also visualizes MHBC attention maps and reports high attention scores such as 0.863 and 0.872 on correct corresponding structures, and low scores such as 0.146 on textureless or occluded regions (Wang et al., 20 Mar 2026). This is presented as qualitative evidence that meaningful correspondences emerge without direct matching supervision.

The main limitations are also explicit. Performance degrades on indoor datasets such as ScanNet when rotations are large, translations are dominant, and homography warping moves corresponding pixels outside image bounds. The authors state that IDA partially mitigates this but does not solve it fully, and suggest adaptive warping strategies and multi-scale feature pyramids as future directions (Wang et al., 20 Mar 2026). This suggests that the current feature-space de-rotation is most reliable when the infinite-homography approximation remains meaningful within the feature-map support.

More broadly, IUP-Pose assumes calibrated cameras, overlapping fields of view, and that stride-32 features retain sufficient information for regression. It estimates only translation direction rather than metric translation. It is also not a correspondence-estimation method in the classical sense, so it does not provide explicit matches or epipolar-consistency diagnostics.

In summary, IUP-Pose is best characterized as a geometry-driven, fully differentiable, real-time relative pose regressor that organizes estimation around a homography-based decoupling of rotation and translation. Its main contribution is not maximum benchmark accuracy, but a principled architecture that combines implicit cross-view alignment, iterative uncertainty-aware refinement, feature-space de-rotation, and lightweight deployment characteristics in a single end-to-end system (Wang et al., 20 Mar 2026).