RBE-Flow: Recurrent Bayesian Estimation on Feature Manifolds for Cross-Modal Registration

Abstract: Cross-modal image registration is essential for multi-sensor perception but remains fundamentally challenging due to severe non-linear radiometric discrepancies and geometric distortions. Existing deterministic matching methods lack uncertainty awareness, struggling to navigate the resulting highly non-convex optimization landscape and frequently accumulating errors in ambiguous regions. In this paper, we propose RBE-Flow, a novel framework that reformulates dense cross-modal flow estimation as a closed-loop recurrent Bayesian estimation problem on learned feature manifolds. Diverging from standard feed-forward regression, RBE-Flow establishes a robust self-correcting mechanism by deeply coupling feature-metric non-linear optimization with probabilistic state updates. Specifically, a Recurrent Manifold Optimization (RMO) block iteratively generates flow observations and their associated uncertainties, which are then optimally assimilated into the prior state via an Uncertainty-Adaptive Probabilistic Update (UAPU) using deterministic sigma-point projection. Crucially, the resulting calibrated posterior covariance is fed back to adaptively regularize the damping of subsequent optimization steps, allowing the system to modulate its convergence based on predictive confidence. To ensure stable probabilistic training, we introduce a hybrid supervision scheme featuring a geometry-aware rectified NLL loss that structurally prevents variance collapse. Extensive experiments on challenging OSdataset, WHU-OPT-SAR, and RoadScene benchmarks demonstrate that RBE-Flow consistently achieves state-of-the-art performance, outperforming existing methods by a significant margin, particularly under strict sub-pixel criteria. Project page: https://github.com/NEU-Liuxuecong/RBE-Flow

• The paper introduces a recurrent Bayesian estimation framework integrated with feature manifold optimization for robust cross-modal dense correspondence.
• It leverages multi-scale CNN feature extraction and uncertainty-adaptive probabilistic updates to achieve state-of-the-art AEPE and sub-pixel alignment.
• Empirical results demonstrate significant improvements in registration accuracy across benchmarks while maintaining low computational and memory overhead.

RBE-Flow: Recurrent Bayesian Estimation on Feature Manifolds for Cross-Modal Registration

Introduction

Cross-modal image registration aims to establish dense geometric correspondences across images acquired from distinct sensing modalities, such as optical-SAR and RGB-infrared. This problem presents acute challenges stemming from severe non-linear radiometric shifts, intense geometric distortions, and modality-specific artifacts which undermine the reliability of conventional photometric alignment or local feature matching paradigms. Deterministic matching frameworks fail to account for the high intrinsic uncertainty and ambiguity present in cross-modal scenarios, leading to instability and prevalence of erroneous local minima. The RBE-Flow architecture addresses these difficulties by reformulating cross-modal dense flow estimation as a closed-loop recurrent Bayesian estimation problem on learned feature manifolds, fundamentally integrating non-linear optimization with probabilistic state updates.

Methodological Overview

RBE-Flow is constructed around a cohesive pipeline that utilizes deep neural representations as feature manifolds, leveraging them with explicit, probabilistically grounded, recurrent optimization for dense correspondence estimation.

Figure 1: Overview of the proposed RBE-Flow architecture. RBE-Flow formulates the alignment task as a closed-loop recurrent Bayesian estimation on feature manifolds.

Multi-Scale Feature Extraction

A weight-shared CNN encoder extracts hierarchical features at 1/2, 1/4, and 1/8 spatial resolutions, capturing both global semantic context and local structural cues. The fusion of these features through self-attention and cross-attention mechanisms ensures robust, detail-preserving representations for subsequent probabilistic inference.

Global Flow Initialization

Initial correspondences are established by computing a dense 4D correlation volume at 1/8 scale, which produces pixel-level soft correspondences via expectation over the probability-normalized correlation scores. Unlike approaches that regress global affine parameters, this dense formulation enables the model to absorb local discrepancies and curtail error propagation.

Recurrent Manifold Optimization (RMO) Block

Flow refinement proceeds within a recurrent loop, where each iteration solves a feature-metric non-linear least-squares problem on the feature manifold. Covariance-adaptive damping, guided by the posterior uncertainty from the previous step, adaptively regularizes the optimization. The computed analytic Jacobian, gradient, and LM-style damping factor enable the system to interpolate between Gauss-Newton and conservative gradient descent updates, contingent on the predicted confidence.

Uncertainty-Adaptive Probabilistic Update (UAPU)

Each RMO-produced flow increment and its uncertainty are assimilated with the prior state using a deterministic sigma-point projection framework, yielding a minimum mean square error (MMSE)-optimal posterior over the flow field. The posterior covariance update propagates through the recurrent loop, gating future optimization steps and enforcing uncertainty-driven damping. This mechanism realizes a closed feedback system where estimation confidence directly modulates the course of manifold optimization.

Training Objective

A hybrid loss function supervises both initialization and recurrent probabilistic refinement. The latter employs a geometry-aware rectified negative log-likelihood (NLL) loss, which sets a dynamically scaled lower bound on predicted variance, preventing variance collapse and enforcing uncertainty calibration proportionally to geometric error.

Empirical Evaluation

Extensive experiments demonstrate the superiority of RBE-Flow on three demanding benchmarks: OSdataset (Optical-SAR, diverse terrains), WHU-OPT-SAR (Optical-SAR, urban scenes), and RoadScene (RGB-IR, variable traffic). The method is benchmarked against both classical hand-crafted and recent deep learning-based approaches, including sparse, semi-dense, and dense methods.

Figure 2: Inlier correspondence visualization across multiple benchmarks. RBE-Flow maintains geometrically stable, dense correspondences even under severe modality gaps and structural distortions.

The core findings are summarized as follows:

• State-of-the-art AEPE: RBE-Flow achieves the lowest average end-point error (AEPE) across all datasets. On OSdataset, AEPE drops to 0.77px (a 45.4% improvement over the previous best); on WHU-OPT-SAR, AEPE falls to 0.53px (nearly a threefold improvement).
• Superior Sub-Pixel CMR: RBE-Flow yields consistently higher correct match rates (CMR) under strict thresholds, demonstrating sub-pixel alignment robustness. For instance, on WHU-OPT-SAR, it retains 80.0% CMR at 0.7px, outperforming all baselines.
• Stable Generalization Across Modalities: Competing approaches exhibit sharp performance declines under strict matching, but RBE-Flow maintains high accuracy even as error thresholds decrease.

Ablation and Analysis

Ablation studies confirm the critical importance of each architectural component:

• The global flow initialization is essential for robust coarse alignment.
• The recurrent manifold optimization block provides fine-scale refinement sensitive to uncertainty-aware damping.
• The uncertainty-adaptive probabilistic update loop substantially improves both average and strict-threshold performance, especially boosting sub-pixel match ratios.
• The proposed geometry-aware NLL prevents overconfident, degenerate solutions and leads to well-calibrated uncertainty maps.

Efficiency analysis shows that RBE-Flow attains this performance with only marginal computational and memory overhead compared to deterministic baselines, supporting real-time deployment.

Implications and Future Perspectives

RBE-Flow introduces an explicit probabilistic optimization paradigm to cross-modal registration, circumventing the pitfalls of overconfidence and instability that plague deterministic and implicitly optimized approaches. By closing the loop between uncertainty estimation and optimization trajectory, the method achieves not only stronger empirical accuracy but also more reliable, uncertainty-calibrated outputs.

For practical applications, this architecture is directly relevant for robust perception in remote sensing analytics, multi-sensor fusion in autonomous vehicles, and medical imaging scenarios that demand reliable spatial alignment under modality-induced ambiguities. Theoretically, the framework paves the way for integrating recursive Bayesian estimation with learned non-linear feature manifolds, suggesting avenues to extend beyond 2D flow estimation toward full 3D multi-modal scene registration, spatial tracking, and probabilistic geometric reasoning in large-scale vision systems.

Conclusion

RBE-Flow establishes a new benchmark for cross-modal dense correspondence estimation by tightly integrating recurrent Bayesian state estimation with explicit manifold optimization. Its mechanism for dynamically coupling uncertainty with optimization feedback sets a new trajectory for uncertainty-aware alignment, with demonstrated state-of-the-art results under stringent evaluation regimes. The modular nature and low computational cost of RBE-Flow suggest its applicability as a backbone for future multi-modal perceptual systems requiring robust, interpretable, and reliable spatial reasoning.