UniCorrn: Unified 2D/3D Correspondence Model

• UniCorrn is a unified correspondence model that handles 2D-2D, 2D-3D, and 3D-3D matching via shared weights and modality-specific backbones.
• It employs a dual-stream Transformer decoder that separately refines appearance and positional features while using a query-based interface.
• The model leverages both dot-product and Gaussian attention along with auxiliary supervision to achieve state-of-the-art registration recall on benchmark datasets.

UniCorrn is a correspondence model for visual geometric matching across image-to-image (2D-2D), image-to-point cloud (2D-3D), and point cloud-to-point cloud (3D-3D) settings. It is presented as the first correspondence model with shared weights that unifies geometric matching across all three tasks, replacing task-specific designs with a single architecture that combines modality-specific backbones, shared encoder and decoder components, and a dual-stream decoder that maintains separate appearance and positional feature streams. Its central premise is that Transformer attention naturally captures cross-modal feature similarity, enabling end-to-end learning through stack-able layers and flexible query-based correspondence estimation across heterogeneous modalities (Goswami et al., 5 May 2026).

1. Problem setting and unified scope

Visual correspondence across image-to-image (2D-2D), image-to-point cloud (2D-3D), and point cloud-to-point cloud (3D-3D) geometric matching forms the foundation for numerous 3D vision tasks. The paper characterizes the prevailing landscape as one in which current methods use task-specific designs with separate models for each modality combination. UniCorrn instead uses shared weights across all three tasks, with a single correspondence framework spanning 2D and 3D inputs (Goswami et al., 5 May 2026).

The model is organized around a query-based interface. Source keypoints are mapped directly to target coordinates, and this interface is intended to remain flexible across 2D and 3D. In the formulation given in the paper, the decoder extracts features at source keypoints, produces query features $F$ and initial point coordinates $K$, and predicts target coordinates $K_{\rm pred} \in \mathbb{R}^{N\times \{2|3\}}$ together with confidence scores $C \in (0,1)^N$.

A common assumption in prior practice is that each modality pair requires a separate matcher. UniCorrn is explicitly designed against that assumption. This suggests that the work is less a specialized matcher for one benchmark than a unification attempt at the level of correspondence representation, attention, and supervision.

2. Architectural organization

The architecture begins with modality-specific backbones. For 2D images, the encoder is a ViT encoder, exemplified by ViT-B or ViT-L. For 3D point clouds, it uses a Point Transformer v3 encoder (PTv3). When source and target share a modality, as in 2D-2D matching, the same backbone is applied Siamese-style. Rotary position embeddings (RoPE) encode relative token positions in both modalities (Goswami et al., 5 May 2026).

After backbone encoding, UniCorrn uses a shared feature-fusion encoder. This component takes the two streams of tokens—image and/or point tokens—and alternates self-attention within each stream with cross-attention between streams. Weight-shared Transformer blocks let features from one modality attend to the other. The use of shared blocks is central to the claim that one set of parameters can learn cross-modal geometric priors.

The decoder upsamples fused features back to fine spatial or 3D resolution. The 2D path uses PixelShuffle, while the 3D path uses the learned upsampler in PTv3. Features are then extracted at source keypoints via bilinear interpolation in 2D or Gaussian-weighted kNN interpolation in 3D, producing the query features $F$ and initial coordinates $K$. These are processed by $L$ stacked dual-stream Transformer layers.

The final prediction heads are modality-specific only where the output geometry requires it. The positional stream output $P$ is passed through a linear head to regress target coordinates, whereas the appearance stream output $F$ is passed through a shared MLP to produce confidence estimates. This division preserves a shared correspondence mechanism while allowing the output space to remain either 2D or 3D.

3. Attention mechanisms and the dual-stream decoder

The paper defines both vanilla self- and cross-attention and a Gaussian attention variant. Given token sets $X=[x_i]$ and $K$0, each of dimension $K$1,

$K$2

The row-normalized attention matrix $K$3 and output $K$4 are

$K$5

In self-attention, $K$6; in cross-attention, $K$7 and $K$8 come from different modalities (Goswami et al., 5 May 2026).

UniCorrn also introduces a Gaussian attention variant based on an RBF-style kernel over $K$9 distances:

$K_{\rm pred} \in \mathbb{R}^{N\times \{2|3\}}$0

The paper states that dot-product attention can be sensitive to feature scale, and that the Gaussian form better captures non-linear similarity and is akin to classic descriptor matching with Gaussian kernels.

The core innovation is the dual-stream Transformer decoder. It keeps an appearance stream $K_{\rm pred} \in \mathbb{R}^{N\times \{2|3\}}$1 and a positional stream $K_{\rm pred} \in \mathbb{R}^{N\times \{2|3\}}$2 in parallel, while sharing the same attention weights $K_{\rm pred} \in \mathbb{R}^{N\times \{2|3\}}$3 for both. Query features and their current coordinate guesses are first augmented by RoPE:

$K_{\rm pred} \in \mathbb{R}^{N\times \{2|3\}}$4

where $K_{\rm pred} \in \mathbb{R}^{N\times \{2|3\}}$5 is the target feature set and $K_{\rm pred} \in \mathbb{R}^{N\times \{2|3\}}$6 their coordinates. Attention is then computed using either the vanilla or Gaussian form on $K_{\rm pred} \in \mathbb{R}^{N\times \{2|3\}}$7.

The appearance stream is updated as

$K_{\rm pred} \in \mathbb{R}^{N\times \{2|3\}}$8

preserving a residual connection. The positional stream uses a separate learned absolute positional encoding $K_{\rm pred} \in \mathbb{R}^{N\times \{2|3\}}$9 and is updated as

$C \in (0,1)^N$0

The paper specifies that $C \in (0,1)^N$1 is initialized to zero. After the final layer, coordinates are recovered through

$C \in (0,1)^N$2

where $C \in (0,1)^N$3 is the Moore–Penrose pseudoinverse of $C \in (0,1)^N$4.

Because $C \in (0,1)^N$5 and $C \in (0,1)^N$6 remain disentangled but both see the same attention matrix, the decoder can stack $C \in (0,1)^N$7 layers and iteratively refine appearance representations and coordinate estimates. This suggests that UniCorrn treats attention not only as a feature-fusion primitive but also as the mechanism that jointly drives descriptor refinement and explicit coordinate regression.

4. Training objective, supervision, and data composition

UniCorrn is trained end-to-end on three tasks simultaneously: 2D-2D, 2D-3D, and 3D-3D. The total loss is

$C \in (0,1)^N$8

with task-level loss

$C \in (0,1)^N$9

The confidence-aware $F$0 term is

$F$1

where $F$2 is the predicted confidence and $F$3 is a weight for the entropy term (Goswami et al., 5 May 2026).

Descriptor learning is enforced with an InfoNCE-based contrastive loss applied both to upsampled fused descriptors and to final appearance features:

$F$4

In addition, each decoder layer receives auxiliary layer-wise supervision: the intermediate $F$5 is trained with $F$6 to the ground truth, weighted by factor $F$7. The paper states that this strongly guides early-layer attention.

The training data mix is explicitly heterogeneous. For 2D-2D, the model uses seven large image datasets, including MegaDepth, ScanNet++, and CO3D. For 2D-3D and 3D-3D, it combines real correspondences from 7Scenes and 3DMatch with pseudo point clouds derived from dense depth maps from ScanNet++ and ARKitScenes, oversampling scarce 2D-3D and 3D-3D pairs to match 2D-2D volume. This pseudo-point-cloud augmentation is presented as a mechanism for alleviating scarcity in cross-modal and 3D correspondence supervision.

Optimization uses AdamW with weight decay $F$8, cosine learning-rate decay, and gradient clipping. The schedule has two stages: Stage 1 trains 2D-2D and 3D-3D, initializing from CroCo v2, for 40 epochs; Stage 2 adds 2D-3D and trains all three tasks for 30 epochs.

5. Quantitative results across tasks

The reported evaluation covers all three correspondence regimes and emphasizes registration recall for 2D-3D and 3D-3D, with competitive pose-estimation AUC for 2D-2D (Goswami et al., 5 May 2026).

Task	Benchmark / metric	Reported result
2D-3D	7Scenes, registration recall RR@5cm/5°	2D3D-MATR: 75.8%; Diff-Reg: 83.8%; UniCorrn: 91.0%
3D-3D	3DLoMatch, low-overlap registration recall	GeoTransformer / RoITr / PEAL-3D: $F$9–$K$0; UniCorrn: 83.2%
2D-2D	MegaDepth, ScanNet-1500, AUC@5°–20°	Competitive compared to RoMa and DKM

On 7Scenes, UniCorrn is reported to surpass the best prior method by $K$1 absolute in registration recall, improving from Diff-Reg at $K$2 RR to $K$3 RR. On 3DLoMatch, UniCorrn achieves $K$4 RR, described as more than an $K$5 absolute gain over prior state of the art. In the abstract, the gain on 3DLoMatch is summarized as $K$6 in registration recall.

For 2D-2D, the model is described as competitive with specialist matchers such as RoMa and DKM on MegaDepth and ScanNet-1500, while retaining the advantage of also supporting 2D-3D and 3D-3D in one model. The paper’s emphasis is therefore not only on peak benchmark accuracy in a single regime, but on joint performance under a shared-weight design.

6. Architectural interpretation and significance

The paper enumerates several key architectural and algorithmic insights. Transformer attention itself is framed as a natural learnable cost volume. The dual-stream design decouples appearance versus geometric reasoning, allowing residual stacking while still producing explicit coordinates. Gaussian attention, defined as an RBF kernel on $K$7, is reported to outperform dot-product attention and to better capture non-linear similarity. The query-based interface allows user-specified source keypoints to map directly to target coordinates across both 2D and 3D. Joint weight sharing across all modalities is intended to enable cross-modal geometric priors to be learned, while pseudo-point-cloud augmentation addresses limited 2D-3D and 3D-3D data, and auxiliary supervision at each layer enforces correct attention early, speeding convergence and improving final accuracy (Goswami et al., 5 May 2026).

Taken together, these points situate UniCorrn as a unification of correspondence estimation rather than a simple aggregation of three separate heads. The design preserves modality-specific encoders where raw signal structure differs, but shares the feature-fusion and correspondence machinery where the paper argues the underlying problem structure is common. This suggests that the main contribution lies in identifying which parts of geometric matching can be shared across modalities without collapsing 2D appearance information and 3D positional reasoning into a single undifferentiated representation.

A likely misconception is that unification across 2D-2D, 2D-3D, and 3D-3D must necessarily reduce performance relative to specialist systems. The reported results are presented precisely against that concern: UniCorrn remains competitive on pure 2D-2D tasks while surpassing specialized, task-specific matchers by $K$8–$K$9 percentage points in registration recall on challenging 2D-3D and 3D-3D benchmarks.