Cross-View Mask Matching

• Cross-view mask matching is a framework for establishing consistent object correspondences via segmented masks across diverse camera viewpoints.
• It utilizes multiple formulations—from binary segmentation to global clustering—to tackle challenges such as occlusion, fragmented masks, and viewpoint shifts.
• Applications include 3D reconstruction and video analysis, leveraging geometry cues, cyclic consistency, and self-supervised learning for enhanced performance.

Cross-view mask matching denotes the problem of establishing consistent correspondences between segmented regions observed from different viewpoints. In the recent literature, the term covers several closely related formulations: matching an object mask from a source image to the corresponding mask in a target image in egocentric–exocentric video (Liao et al., 6 Aug 2025), assigning globally consistent identities to per-view masks before lifting them into 3D Gaussian Splatting (Park et al., 2 Jul 2026), using coarse 3D segments as a common reference for multi-view mask consolidation in zero-shot 3D instance segmentation (Zhao et al., 10 Apr 2026), predicting co-visible masks as structured priors for feature matching (Pan et al., 2 Jun 2026), and enforcing cross-view consistency among segment tokens in text-supervised segmentation (Ren et al., 2023). Across these settings, the explicitly identified difficulties are drastic viewpoint and appearance variation, fragmented masks, inconsistent identities across views, partial overlap, occlusion, and projection drift (Liao et al., 6 Aug 2025, Park et al., 2 Jul 2026, Taggenbrock et al., 10 Jan 2025, Zhao et al., 10 Apr 2026, Gao et al., 15 Apr 2026).

1. Problem formulations and representational units

A common formalization treats cross-view mask matching as conditional binary segmentation. In this setting, the inputs are a source image $I_s$, a binary query mask $M_s\in\{0,1\}^{H\times W}$, and a target image $I_t$; the output is a binary mask $\hat M_t$ that segments the same object in the target view (Yan et al., 22 Feb 2026). A closely related formulation appears in challenge systems such as ObjectRelator, which writes the Ego$\to$Exo task as $M^x_{t,\mathrm{pred}} = F(I^x_t; M^e_t)$ and symmetrically supports Exo$\to$Ego prediction (Fu et al., 6 Jun 2025).

A second formulation operates over proposal sets rather than directly over pixels. DOMR begins from $N$ bounding-box proposals and corresponding candidate masks in each view, produced by YOLO-UniOW and SAM 2, and builds proposal-level embeddings from SAM 2 prompt tokens, positional encodings, and CLIP label embeddings. After stacking the $2N$ proposal embeddings and processing them with multi-head self-attention and cross-attention to image features, it computes cosine similarities and two directed softmax scores,

$s_{ij}^{src},\qquad s_{ij}^{tgt},$

then chooses a one-to-one correspondence through the “mix” rule

$M_s\in\{0,1\}^{H\times W}$0

This formulation makes the matching problem explicitly object-centric and one-to-one (Liao et al., 6 Aug 2025).

A third formulation uses global clustering across many views. In the 3D Gaussian Splatting pipeline of “Consistent Scene Understanding in 3D Gaussian Splatting via Multi-Cue Mask Refinement,” each refined 2D mask $M_s\in\{0,1\}^{H\times W}$1 is associated with a descriptor $M_s\in\{0,1\}^{H\times W}$2. Two masks from different views are linked when

$M_s\in\{0,1\}^{H\times W}$3

and connected components of the resulting graph define global instance clusters $M_s\in\{0,1\}^{H\times W}$4. The outputs then extend beyond 2D masks to per-Gaussian lifted features $M_s\in\{0,1\}^{H\times W}$5, discrete instance IDs $M_s\in\{0,1\}^{H\times W}$6, and reliability flags obtained from majority voting and variance filtering (Park et al., 2 Jul 2026).

These formulations differ in output space—target mask prediction, proposal correspondence, or global clustering—but all treat segmentation masks as the primary carrier of object identity across views. This suggests that “mask matching” is not a single algorithmic primitive but a family of correspondence problems coupled to segmentation, clustering, or 3D lifting.

2. Object-centric ego–exo correspondence architectures

In ego–exo correspondence, cross-view mask matching has largely been framed as an object-level reasoning problem over source masks and target images. ObjectRelator augments the query mask with a textual description generated by LLaVA from the masked ego region and the full frame, embeds the mask and text into a common $M_s\in\{0,1\}^{H\times W}$7-dimensional space, fuses them by

$M_s\in\{0,1\}^{H\times W}$8

and applies a cross-view object alignment loss

$M_s\in\{0,1\}^{H\times W}$9

during training. On the Ego-Exo4D challenge test set it reports IoU $I_t$0 for Ego$I_t$1Exo and $I_t$2 for Exo$I_t$3Ego, compared with $I_t$4 and $I_t$5 for XSegTx (Fu et al., 6 Jun 2025).

O-MaMa explicitly redefines cross-image segmentation as mask matching. Its Mask-Context Encoder pools dense DINOv2 features over each FastSAM candidate mask to obtain a local descriptor $I_t$6 and over an extended box to obtain a context descriptor $I_t$7. An Ego$I_t$8Exo cross-attention module enriches the mask descriptor with global cues from the opposite view, and a shallow MLP maps the concatenated representation into a shared embedding space optimized with an InfoNCE loss. Hard negatives are selected by Delaunay-triangulation adjacency and second-order neighbors. Reported test v2 IoU is $I_t$9 for Ego$\hat M_t$0Exo and $\hat M_t$1 for Exo$\hat M_t$2Ego (Mur-Labadia et al., 6 Jun 2025).

DOMR extends this object-centric line by jointly modeling multiple objects rather than directly matching individual object masks to image features. Its Dense Object Matcher constructs proposal embeddings

$\hat M_t$3

applies $\hat M_t$4 attention blocks over all source and target proposals, and uses a complete-graph self-attention mechanism to encode inter-object relationships. A subsequent Mask Refinement head initializes query tokens from matched embeddings and positional encodings, passes them through $\hat M_t$5 “two-way” transformer blocks, and predicts dynamic convolution kernels for refined masks. On Ego-Exo4D, DOMR reports $\hat M_t$6 mIoU on Ego$\hat M_t$7Exo and $\hat M_t$8 on Exo$\hat M_t$9Ego, outperforming the prior best ObjectRelator by $\to$0 and $\to$1 (Liao et al., 6 Aug 2025).

V$\to$2-SAM adapts SAM2 to cross-view correspondence through two prompt generators. V$\to$3-Anchor uses DINOv3 patch features, a dense cosine-similarity heatmap, foreground-constrained matching, and coordinate transformation to produce a geometry-based coordinate prompt. V$\to$4-Visual uses a Visual Prompt Matcher with transformer cross-attention and a CNN-based structural branch to produce an appearance-based feature prompt. Three experts—Anchor, Visual, and Fusion—produce candidate masks, and a Post-hoc Cyclic Consistency Selector back-projects each mask to the ego view and picks the most consistent expert. On Ego-Exo4D, the multi-experts+PCCS model reports $\to$5 IoU for Ego2Exo / Exo2Ego (Pan et al., 25 Nov 2025).

Cycle-consistent mask prediction provides another object-centric formulation. CCMP encodes the source mask by normalized mask pooling of dense source features into a single condition token, prepends that token to target-view visual tokens in a ViT encoder, and predicts $\to$6 with a conditional binary segmentation head. It optimizes a forward segmentation loss plus a backward reconstruction loss that re-encodes the predicted target mask and reconstructs the original source mask,

$\to$7

The same backward loss enables test-time training. On Ego-Exo4D, the full model with TTT reports $\to$8 mIoU; on HANDAL-X zero-shot evaluation it reports $\to$9 IoU (Yan et al., 22 Feb 2026).

VGGT-Segmentor replaces direct appearance matching with geometry-enhanced feature alignment. A frozen VGGT backbone predicts depth, camera parameters, and point maps, reprojects source points into the target view, and feeds geometry-aware source and target features into a three-stage Union Segmentation Head consisting of mask prompt fusion, point-guided prediction, and iterative mask refinement. On Ego-Exo4D, VGGT-S reports $M^x_{t,\mathrm{pred}} = F(I^x_t; M^e_t)$0 average IoU in zero-shot mode and $M^x_{t,\mathrm{pred}} = F(I^x_t; M^e_t)$1 in fully supervised mode, surpassing DOMR by $M^x_{t,\mathrm{pred}} = F(I^x_t; M^e_t)$2 points (Gao et al., 15 Apr 2026).

3. Global identity assignment and 3D lifting

When cross-view mask matching is embedded in 3D reconstruction, the central issue becomes consistency across many views rather than a single source–target pair. In the multi-cue 3DGS framework, per-view refined masks are matched by a simple undirected graph whose edges are thresholded cosine similarities between mask descriptors from different views. Global mask IDs are the connected components of that graph. For each 3D Gaussian primitive $M^x_{t,\mathrm{pred}} = F(I^x_t; M^e_t)$3, the framework then computes a weighted average descriptor

$M^x_{t,\mathrm{pred}} = F(I^x_t; M^e_t)$4

assigns a discrete ID by majority vote over projected masks, and removes unreliable Gaussians when the across-view feature variance

$M^x_{t,\mathrm{pred}} = F(I^x_t; M^e_t)$5

exceeds $M^x_{t,\mathrm{pred}} = F(I^x_t; M^e_t)$6. The paper’s qualitative ablation isolates the effect of cross-view matching: without global matching, instance IDs flicker across viewpoints; with the full pipeline, colors and IDs remain consistent across wide-baseline views (Park et al., 2 Jul 2026).

MV3DIS uses a different mechanism built around coarse 3D segments rather than descriptor graphs alone. Its Stage 1 forms coarse 3D segments $M^x_{t,\mathrm{pred}} = F(I^x_t; M^e_t)$7 from superpoints and coarse per-frame label maps. Stage 2 projects each $M^x_{t,\mathrm{pred}} = F(I^x_t; M^e_t)$8 into all frames, computes a visibility indicator

$M^x_{t,\mathrm{pred}} = F(I^x_t; M^e_t)$9

and retains candidate masks only when frame visibility $\to$0 and mask visibility $\to$1 exceed thresholds. Each candidate mask receives a 3D coverage vector $\to$2, and its consistency score is the average cosine similarity of that coverage vector with the others in the same candidate set,

$\to$3

NMS over these scores yields refined 2D maps that are then fed back into superpoint refinement. On ScanNetV2, the ablation reports mAP $\to$4 for coarse segments, $\to$5 with region refinement only, $\to$6 with 3D-guided mask matching but no depth weighting, and $\to$7 for the full method (Zhao et al., 10 Apr 2026).

These two systems instantiate distinct but compatible notions of global consistency. The 3DGS method treats mask matching as graph clustering in descriptor space, whereas MV3DIS uses 3D coverage distributions anchored by coarse geometry. A plausible implication is that multi-view mask matching becomes more robust when the identity decision is coupled to a 3D consistency test rather than left entirely to pairwise appearance similarity.

4. Geometry, co-visibility, and structural priors

A persistent theme in recent work is that appearance-only mask matching is often insufficient under large viewpoint shifts. Geometry can enter the pipeline as coordinate prompts, explicit reprojection, co-visibility supervision, or hard attention constraints.

V$\to$8-SAM introduces geometry-aware prompting through V$\to$9-Anchor. Given DINOv3 patch features $N$0 and $N$1, it computes a dense cosine-similarity heatmap

$N$2

restricts source patches to the query foreground, recovers matched coordinates $N$3, stratifies those points by a distance threshold, and transforms them into the SAM2 coordinate frame before prompt encoding. This explicitly reuses geometric correspondences as prompts for mask prediction rather than treating geometry as a post-processing step (Pan et al., 25 Nov 2025).

VGGT-S uses an even stronger geometric prior. Its backbone predicts source depth $N$4, intrinsics $N$5, and relative pose $N$6, enabling reprojection of source points by

$N$7

The paper explicitly notes that naïvely projecting source points via VGGT and directly splatting a mask leads to systematic pixel drift, whereas the Union Segmentation Head pulls the mask back onto the true object silhouette in the target view (Gao et al., 15 Apr 2026).

SAMatcher relocates the problem from object identity to co-visible region estimation. Rather than directly matching local features, it first predicts co-visible masks $N$8 and boxes $N$9 using a frozen SAM-HQ encoder, symmetric cross-view fusion via interleaving and “DoubleBlocks,” and joint supervision from point-sampled BCE, Dice, box regression, and mask–box consistency. On MegaDepth, augmenting LoFTR with SAMatcher raises AUC@20° from $2N$0 to $2N$1 and mAA@20° from $2N$2 to $2N$3 (Pan et al., 2 Jun 2026).

EpiMask uses an explicit epipolar-distance mask to restrict cross-attention in satellite image matching. From a patch-wise affine approximation to the RPC camera model, it computes a local fundamental matrix $2N$4, evaluates a symmetric epipolar distance $2N$5, and defines a binary mask $2N$6 by thresholding that distance. All cross-attention logits outside the epipolar band are set to $2N$7 before softmax. The method warms up without masking for the first $2N$8 epochs and then narrows the band-width. On SatDepth it reports up to $2N$9 improvement in matching accuracy compared to re-trained ground-based models (Deshmukh et al., 23 Mar 2026).

These results clarify that geometry does not merely regularize correspondence; in several systems it defines the admissible match set itself. This suggests that high-variance cross-view settings benefit when the mask matcher is constrained by camera or scene structure before appearance similarity is applied.

5. Supervision regimes and optimization objectives

Cross-view mask matching is trained under markedly different supervision regimes, ranging from fully supervised pairwise labels to correspondence-free pretraining.

DOMR trains its Dense Object Matcher with a contrastive loss over matched proposals only,

$s_{ij}^{src},\qquad s_{ij}^{tgt},$0

while the refinement head is trained separately, with DOM frozen, using focal and Dice losses combined as

$s_{ij}^{src},\qquad s_{ij}^{tgt},$1

This separation assigns matching and boundary completion to different optimization stages (Liao et al., 6 Aug 2025).

O-MaMa remains fully supervised but emphasizes discriminative metric learning. For a source embedding $s_{ij}^{src},\qquad s_{ij}^{tgt},$2, one positive mask embedding $s_{ij}^{src},\qquad s_{ij}^{tgt},$3, and hard-mined negatives, it minimizes an InfoNCE objective

$s_{ij}^{src},\qquad s_{ij}^{tgt},$4

Its hard negative adjacent mining strategy forces the model to differentiate nearby objects rather than arbitrary negatives (Mur-Labadia et al., 6 Jun 2025).

ViewCo addresses cross-view ambiguity in text-supervised segmentation with a Siamese student–teacher GroupViT setup. It defines positive pairs as diagonal matches between segment tokens from two augmented views of the same image and optimizes a bi-directional cross-view segmentation consistency loss $s_{ij}^{src},\qquad s_{ij}^{tgt},$5 together with a text-to-views contrastive loss. Trained on large-scale image–text data, it reports average gains of up to $s_{ij}^{src},\qquad s_{ij}^{tgt},$6, $s_{ij}^{src},\qquad s_{ij}^{tgt},$7, and $s_{ij}^{src},\qquad s_{ij}^{tgt},$8 mIoU on PASCAL VOC2012, PASCAL Context, and COCO, respectively (Ren et al., 2023).

Self-supervised partial cycle-consistency handles partial overlap explicitly. It replaces unknown hard match matrices $s_{ij}^{src},\qquad s_{ij}^{tgt},$9 with soft operators $M_s\in\{0,1\}^{H\times W}$00, constructs several pairwise and triplewise cycle matrices, and builds a pseudo-mask $M_s\in\{0,1\}^{H\times W}$01 that indicates where cycles can actually exist. A masked margin loss then applies a larger margin $M_s\in\{0,1\}^{H\times W}$02 on pseudo-cycle diagonals and a smaller margin $M_s\in\{0,1\}^{H\times W}$03 elsewhere. On DIVOTrack, cycle variations alone raise F1 from $M_s\in\{0,1\}^{H\times W}$04 to $M_s\in\{0,1\}^{H\times W}$05, partial masking to $M_s\in\{0,1\}^{H\times W}$06, and time-divergent sampling to $M_s\in\{0,1\}^{H\times W}$07 (Taggenbrock et al., 10 Jan 2025).

CCMP uses cycle-consistency in a supervised correspondence setting but exploits the backward loss for test-time training. During inference it freezes most weights, fine-tunes the last $M_s\in\{0,1\}^{H\times W}$08 transformer layers and mask head for $M_s\in\{0,1\}^{H\times W}$09 steps—reported examples are $M_s\in\{0,1\}^{H\times W}$10, $M_s\in\{0,1\}^{H\times W}$11 for Ego$M_s\in\{0,1\}^{H\times W}$12Exo—at $M_s\in\{0,1\}^{H\times W}$13, using only the source-mask reconstruction loss (Yan et al., 22 Feb 2026).

VGGT-S introduces a different self-supervised route. It pretrains on unpaired single images by generating a pseudo mask $M_s\in\{0,1\}^{H\times W}$14, applying VGGT-adaptive or VGGT-nonadaptive augmentations, and optimizing the same segmentation loss used in supervised training,

$M_s\in\{0,1\}^{H\times W}$15

The paper characterizes the resulting model as “correspondence-free pretrained” (Gao et al., 15 Apr 2026).

6. Empirical behavior, limitations, and open directions

Several recurring empirical findings cut across the literature. First, successful systems rarely rely on direct isolated-mask appearance matching. DOMR explicitly contrasts itself with methods that directly match individual object masks to image features and instead leverages positional and semantic relationships among objects (Liao et al., 6 Aug 2025). V$M_s\in\{0,1\}^{H\times W}$16-SAM combines geometry-aware and appearance-aware prompts and then delegates final selection to a cyclic-consistency selector (Pan et al., 25 Nov 2025). VGGT-S separates geometric point transfer from final segmentation refinement because direct projection causes pixel drift (Gao et al., 15 Apr 2026).

Second, refinement and consistency filtering are repeatedly shown to matter. DOMR reports that its Mask Refinement head corrects boundary errors and fills holes in SAM proposals (Liao et al., 6 Aug 2025). The 3DGS pipeline reports that without cross-view matching, refined masks still flicker in ID across viewpoints, whereas the full pipeline yields consistent colors and IDs (Park et al., 2 Jul 2026). MV3DIS shows a further mAP increase when 3D-guided mask matching is augmented with depth consistency weighting (Zhao et al., 10 Apr 2026). SAMatcher’s ablations show that removing symmetric cross-view interaction drops mask IoU by $M_s\in\{0,1\}^{H\times W}$17 and box IoU by $M_s\in\{0,1\}^{H\times W}$18, while removing mask–box consistency drops mask IoU by $M_s\in\{0,1\}^{H\times W}$19 and box IoU by $M_s\in\{0,1\}^{H\times W}$20 (Pan et al., 2 Jun 2026).

The reported limitations are likewise consistent. The 3DGS matching stage is reliant on the discriminability of per-mask descriptors, and fixed $M_s\in\{0,1\}^{H\times W}$21 can under-link or over-link masks; occlusions and drastic lighting changes can also cause high variance and removal of valid Gaussians (Park et al., 2 Jul 2026). MV3DIS identifies threshold sensitivity in $M_s\in\{0,1\}^{H\times W}$22, $M_s\in\{0,1\}^{H\times W}$23, and $M_s\in\{0,1\}^{H\times W}$24, noisy consistency scores when few views see an object clearly, and a static-scene assumption that breaks for dynamic or deformable objects (Zhao et al., 10 Apr 2026). V$M_s\in\{0,1\}^{H\times W}$25-SAM notes reliance on pretrained DINO features, no explicit use of camera extrinsics, and heavier compute than single-expert baselines (Pan et al., 25 Nov 2025). VGGT-S remains frame-based and ignores temporal consistency, while higher-resolution bottleneck fusion improves IoU only marginally at high compute cost (Gao et al., 15 Apr 2026).

A common misconception is that cross-view mask matching is synonymous with bipartite assignment between two images. The literature contains one-to-one matching rules such as DOMR’s $M_s\in\{0,1\}^{H\times W}$26 (Liao et al., 6 Aug 2025), but also connected-component clustering across all masks (Park et al., 2 Jul 2026), coverage-vector consistency scoring relative to coarse 3D segments (Zhao et al., 10 Apr 2026), cyclic expert selection (Pan et al., 25 Nov 2025), and co-visibility mask prediction before feature correspondence (Pan et al., 2 Jun 2026). Another misconception is that the task is necessarily fully supervised. ViewCo, partial cycle-consistency, CCMP test-time training, and VGGT-S single-image pretraining all show that cross-view consistency itself can serve as supervision (Ren et al., 2023, Taggenbrock et al., 10 Jan 2025, Yan et al., 22 Feb 2026, Gao et al., 15 Apr 2026).

Taken together, these results suggest that current high-performing formulations treat cross-view mask matching as a composite problem involving object representation, structural context, geometric admissibility, and post-hoc consistency enforcement rather than a single similarity computation. The most explicit future directions in the surveyed work are to incorporate camera pose or 3D reasoning (Pan et al., 25 Nov 2025), learn thresholds or affinity metrics end-to-end (Park et al., 2 Jul 2026), integrate temporal modules (Gao et al., 15 Apr 2026), and replace hard assignments with soft or probabilistic identity updates (Park et al., 2 Jul 2026).