SFMambaNet: Spectral-Frequency Enhanced Selective State Space Model for Correspondence Pruning

Abstract: Correspondence pruning aims to identify inliers from an initial set of correspondences. Most existing Graph Neural Network (GNN)-based methods rely on geometric features mapped from coarse Euclidean coordinates, which struggle to capture the subtle geometric consistencies presented by inliers. While Mamba-based methods possess global receptive fields and long sequence modeling capabilities, they tend to accumulate substantial inconsistent features within the hidden state space, making it difficult to distinguish inliers from outliers. In this paper, we integrate frequency domain perception into this task for the first time and propose SFMambaNet, a novel Spectral-Frequency enhanced Mamba-based two-view correspondence pruning network. Our method is collaboratively composed of two components: First, we design a Local Spectral-Geometric Attention (LSGA) block. LSGA incorporates spectral positional encoding into local graph interactions and introduces multi-scale Mamba processing to enhance the capture of subtle geometric consistencies and improve local feature discriminability. Building upon this, we design a Spectral-Integrated Global Mamba (SIGM) block. SIGM embeds a frequency gating mechanism within the state space, utilizing the frequency information provided by LSGA to explicitly suppress high-frequency noise accumulation within hidden states and mitigate the propagation of inconsistent features. This enhances inlier-outlier separability and achieves robust global context modeling capabilities with nearly linear complexity. Extensive experiments demonstrate that SFMambaNet outperforms current state-of-the-art methods on several challenging tasks. The code is available at https://github.com/Kirito14IT/SFMambaNet.

• The paper introduces a spectral-enhanced selective state space model that robustly separates inlier and outlier correspondences to improve geometric vision tasks.
• It employs a two-stage architecture with Local Spectral-Geometric Attention and Spectral-Integrated Global Mamba modules to refine noisy matches using frequency-domain cues.
• Experimental results demonstrate significant gains in pose estimation and homography accuracy while maintaining competitive computational efficiency.

SFMambaNet: Spectral-Frequency Enhanced Selective State Space Model for Correspondence Pruning

Introduction and Motivation

Reliable two-view correspondence filtering is pivotal for a range of geometric vision applications, including SfM, SLAM, image registration, and camera localization. The persistence of outliers in initial matches—particularly in environments with repetitive structures or strong photometric changes—significantly degrades downstream performance, motivating discriminative, scalable pruning approaches. Existing GNN-based pruning networks predominantly rely on geometric features derived from point coordinates, which has limited efficacy in capturing the nuanced geometric consistencies that distinguish inliers. Similarly, Mamba-based State Space Models (SSMs) provide global context and long sequence modeling but accumulate inlier–outlier ambiguity within hidden states due to insufficient frequency domain discrimination.

The central contribution of "SFMambaNet: Spectral-Frequency Enhanced Selective State Space Model for Correspondence Pruning" (2606.04493) is the principled integration of frequency-domain perception into the selective state space modeling pipeline for correspondence pruning. SFMambaNet demonstrates that introducing spectral context at both local and global scales yields superior separability between inlier and outlier matches, explicit suppression of high-frequency feature noise, and improved robustness on challenging geometric estimation tasks.

Architectural Overview

SFMambaNet instantiates an iterative two-stage pruning and verification framework, leveraging frequency-informed modules for context aggregation and outlier suppression.

The overall architecture processes an initial $N \times 4$ correspondence matrix, passing it through two sequential pruning blocks (each composed of a Local Spectral-Geometric Feature Extractor (LSGFE) and a Spectral-Global Context Aggregator (SGCA)), followed by parametric model (e.g., essential matrix) estimation and full-size verification. The core innovation lies in the design and interplay between the LSGFE (which realizes Local Spectral-Geometric Attention, LSGA) and the SGCA (which deploys Spectral-Integrated Global Mamba, SIGM).

Figure 1: SFMambaNet network architecture: iterative pruning and verification, with explicit spectral modeling in both local and global context modules.

Spectral-Frequency Integration

Unlike prior models, SFMambaNet introduces frequency domain representation and gating at two critical levels:

• Local: Spectral expansion of relative neighborhood coordinates injects high-frequency priors to enrich local graph representations, improving discrimination of subtle geometric stabilities.
• Global: Frequency-domain gating is imposed within the Mamba SSM to filter unstable high-frequency artifacts during long-range sequence propagation, thereby fortifying global consensus estimation.

This dual spectral modeling yields improved correspondence discriminability and reduces the propagation of inconsistent features across the network.

Local Spectral-Geometric Attention (LSGA)

The LSGA block first maps raw correspondences into a high-dimension via MLP/ResNet layers. An explicit local graph is constructed for each point, encoding position and residual features among $k$-nearest neighbors. Conventional MaxPooling-based aggregation within the local graph is insufficient for capturing structural high-frequency perturbations relevant to separating inliers from outliers.

LSGA augments local aggregation by injecting Fourier-based spectral position encoding into local neighborhoods. Relative coordinates are projected via a learned Gaussian matrix to generate sinusoidal features that expose high-frequency variations. These are fused as additional input to a learned attention mechanism, yielding query-key-value projections for neighborhood-wise aggregation weighted by both raw feature affinity and spectral-geometric compatibility.

Figure 2: Internal schematic of the LSGA block, detailing the spectral-geometric encoding, local attention, cluster pooling, and multi-scale propagation paths.

After attention-based accumulation, LSGA implements dual-branch aggregation: a DiffPool clusters features for macro context, while a bidirectional Mamba module aggregates spatial dependencies. Cluster-level features are further processed by a custom Bidirectional Cluster Spatial Mamba (BiCSM) to reinforce regional consensus. Final local features emerge from a multi-scale interaction mechanism that fuses point-wise and cluster-aware representations.

Figure 3: Local Geometric Attention pipeline: spectral encoding is integrated into the query/key/value paradigm for neighbor-aware local aggregation.

Ablation studies confirm that spectral encoding and the hierarchical LSGA structure yield notable improvements over GNN-style or vanilla Mamba models, driving accuracy gains of up to 8–10 mAP points on challenging splits.

Spectral-Integrated Global Mamba (SIGM)

The SIGM block operates on the cluster-ordered, locally enhanced sequence outputted by LSGA. Its core is a two-stage (forward and backward) bidirectional scan using Mamba modules, interleaved with spectral filtering.

• Forward Pass: The features are propagated through Mamba, then converted to the frequency domain via FFT. A learned frequency-domain gating operation (elementwise complex modulation) suppresses high-frequency components indicative of noisy or outlier matches. An IFFT transforms the filtered sequence back to the spatial domain.
• Backward Pass: The frequency-purified sequence is reversed and passed through a second Mamba + frequency gating stage, further consolidating robust long-range consensus.

This process ensures that the final state representations are dominated by low-frequency, consensus-driven signal components, which are characteristic of true inlier correspondences.

Figure 4: SIGM block schematic: dual Mamba/frequency gating enables sequential suppression of high-frequency noise propagation.

Frequency Diagnostics: FFT-based analysis demonstrates that SIGM explicitly attenuates high-frequency spectral energy in the outlier population while preserving low-frequency consensus inliers, confirming its role as a learned spectral regularizer.

Figure 5: Spectral energy distributions before and after SIGM: high-frequency suppression for outliers, preservation of consensus cues for inliers.

Experimental Evaluation

SFMambaNet is evaluated on standard vision benchmarks (outdoor YFCC100M, indoor SUN3D, HPatches), in both correspondence extraction and downstream pose estimation tasks, under both SIFT and SuperPoint descriptors.

Camera Pose Estimation: On YFCC100M (unknown split), SFMambaNet achieves mAP5$^\circ$ of 73.83%, outperforming prior art including MatchMamba (67.60%) and BCLNet (66.08%). On indoor SUN3D, a corresponding ~1% improvement is observed.

Outlier Removal: SFMambaNet yields best-in-class F-scores and precision, slightly sacrificing recall for significantly more reliable inlier sets—a byproduct of aggressive but accurate pruning justified by the subsequent verification stage.

Figure 6: Outlier removal qualitative results: SFMambaNet suppresses more false matches and preserves superior inlier density in hard cases.

Homography Estimation: On HPatches, SFMambaNet attains top accuracy across all error thresholds, confirming its robustness to challenging planar transformations.

Figure 7: Homography estimation curves on HPatches: SFMambaNet dominates across error regimes.

Ablations indicate both LSGA's spectral attention and SIGM's gating are indispensable, each yielding unique accuracy gains versus GNN or SSM-only alternatives. Computational efficiency analysis reveals that SFMambaNet achieves state-of-the-art accuracy with a competitive parameter budget and run-time profile, notably surpassing heavy transformer-based networks in speed and size.

Figure 8: Trade-off between accuracy (mAP5$^\circ$) and parameter count: SFMambaNet attains highest accuracy per model size.

Implications and Outlook

SFMambaNet's explicit spectral integration at multiple network levels reveals the utility of frequency-domain representations for fine-grained geometric reasoning—beyond conventional coordinate- or attention-driven spatial models. Unlike classical hand-crafted filters, learnable spectral gating mechanisms can robustly regularize state transitions in SSMs, offering a new axis for model design in vision tasks plagued by subtle or high-frequency outlier noise.

Practically, SFMambaNet sets a new benchmark for feature correspondence filtering where both high recall and high-precision outlier removal are required for robust pose or geometry estimation, with competitive compute overhead. Theoretically, the demonstrated separation of inliers and outliers in the frequency domain motivates further investigation into spectral priors and implicit geometric cues in other graph, point cloud, and sequential modeling tasks.

Conclusion

SFMambaNet delivers a principled, empirically validated framework for correspondence pruning, integrating spectral-geometric encoding and global frequency gating via the Mamba SSM. The resulting network achieves superior inlier/outlier separation, geometric robustness, and computational efficiency, and advances the state of the art across canonical geometric vision benchmarks. Future directions include extending spectral-enhanced SSMs to dense matching, multi-view geometry, and downstream tasks such as 3D reconstruction, and further exploring implicit spectral priors in vision models.