Dense Object Matching and Refinement

• DOMR is a two-stage framework for cross-view object matching and segmentation in ego–exo videos, leveraging dense contextual cues for robust correspondence.
• It employs a Dense Object Matcher using transformer-based self and cross-attention to fuse visual, spatial, and semantic features from multiple proposals.
• Its mask refinement stage improves annotation quality and segmentation accuracy, achieving significant IoU gains on the Ego-Exo4D benchmark.

Searching arXiv for the DOMR paper and closely related dense matching work to ground the article in current literature. arXiv search: "Dense Object Matching and Refinement cross-view segmentation" Dense Object Matching and Refinement (DOMR) is a two-stage framework for cross-view object correspondence and segmentation between synchronized egocentric (first-person) and exocentric (third-person) videos. It is centered on a Dense Object Matcher (DOM) that jointly models multiple objects across both views, rather than treating correspondence as an isolated single-object search problem, and a Mask Refinement (MR) head that improves the completeness and accuracy of the matched masks. On the Ego-Exo4D benchmark, DOMR reports a mean IoU of 49.7% on Ego$\to$Exo and 55.2% on Exo$\to$Ego, outperforming previous methods by 5.8% and 4.3%, respectively (Liao et al., 6 Aug 2025).

1. Task formulation and operating assumptions

DOMR addresses cross-view object correspondence under a synchronized source-target video setting. Given a source video $V^{\text{src}}$, a target video $V^{\text{tgt}}$, and a set of source masks $M^{\text{src}}$ describing objects of interest in one view, the goal is to identify the corresponding target masks $M^{\text{tgt}}$ in the other view. The framework is defined for both Ego $\rightarrow$ Exo and Exo $\rightarrow$ Ego transfer (Liao et al., 6 Aug 2025).

A defining constraint is that the method operates without camera intrinsics/extrinsics, IMU, or sensor metadata, and without relying on semantic object names as supervision at inference. The task is therefore posed under large viewpoint changes, frequent scale variation, object ambiguity, and severe asymmetry between views. The same object may be tiny in one view, heavily occluded, or confounded with several visually similar instances.

Within this formulation, DOMR is explicitly motivated as a dense contextual alternative to prior approaches that search for one object at a time. Its central premise is that surrounding objects provide disambiguating evidence. In ego-exo scenes, where multiple instances of the same category may co-occur, contextual object layout and cross-view relational structure become part of the correspondence signal rather than incidental background information.

2. Proposal generation and multi-cue object encoding

DOMR begins from an object proposal pipeline composed of YOLO-UniOW-L as the open-vocabulary detector and SAM 2.1 Hiera-B+ to convert bounding-box proposals into masks. For an input frame $\mathcal{I}$, the detector produces $N$ box proposals,

$\to$0

which are then used as prompts for SAM 2 to obtain mask proposals,

$\to$1

The reported proposal coverage on Ego-Exo4D indicates that this stage is strong but imperfect, which is one of the reasons DOMR includes a separate refinement module.

View	Box IoU	Mask IoU
Ego-view	78.5	77.4
Exo-view	71.5	67.1

For each proposal, DOM constructs a composite embedding from three components. The first is a token embedding $\to$2 produced by SAM and associated with the prompt used to generate the mask. The second is a positional embedding,

$\to$3

which encodes the box corners and thereby the object’s location and extent. The third is a label embedding $\to$4, obtained by embedding the YOLO-UniOW category label through CLIP text encoding. These components are fused as

$\to$5

This representation explicitly combines visual, spatial, and semantic cues. The design matters because the framework is intended to resolve ambiguity not only from appearance, but also from object layout and category-level distinctions across views (Liao et al., 6 Aug 2025).

3. Dense Object Matcher and explicit relational modeling

The Dense Object Matcher is the core correspondence module. Instead of matching a single source object independently against target candidates, it concatenates all source and target proposal embeddings into

$\to$6

and processes them jointly through transformer blocks (Liao et al., 6 Aug 2025).

The first stage is multi-head self-attention over all proposals: $\to$7 where $\to$8 is a learnable embedding indicating whether a token belongs to the source or target view. This operation allows representations to be influenced by surrounding proposals, rather than remaining independent object descriptors.

The second stage is cross-attention from proposal tokens to SAM 2 image features: $\to$9 followed by

$V^{\text{src}}$0

Here $V^{\text{src}}$1 denotes image features and $V^{\text{src}}$2 is image positional encoding. The resulting stack models within-view interactions, cross-view interactions, and object-to-image feature grounding.

After transformer processing, the source and target proposal embeddings are denoted $V^{\text{src}}$3 and $V^{\text{src}}$4. Their similarity is computed by cosine similarity,

$V^{\text{src}}$5

This induces a full source-target similarity matrix over proposals. The framework’s stated rationale is that dense inter-object reasoning improves correspondence under ambiguity, especially when several objects share similar local appearance.

4. Matching strategies and training objective

DOMR discusses three matching formulations. The simplest is a top-1 rule,

$V^{\text{src}}$6

which selects the target proposal with maximum similarity for each source object. The paper notes that this can be unstable when many similar objects appear or when the two views contain very different proposal sets (Liao et al., 6 Aug 2025).

DOM therefore defines bidirectional softmax scores. The source-normalized score is

$V^{\text{src}}$7

and the target-normalized score is

$V^{\text{src}}$8

with $V^{\text{src}}$9 a learnable temperature.

The default mix strategy combines both directions: $V^{\text{tgt}}$0 This enforces a form of symmetric consistency: a favored source-to-target match should also be favored in the reverse direction. In ablation, DOM without MIX achieves 47.2 / 53.3 IoU on Ego$V^{\text{tgt}}$1Exo / Exo$V^{\text{tgt}}$2Ego, whereas DOM with MIX reaches 49.0 / 53.4. The improvement is therefore concentrated on Ego$V^{\text{tgt}}$3Exo. The paper also states that Hungarian matching performs poorly in this setting because not every object necessarily has a one-to-one counterpart across views.

Training uses a contrastive cross-entropy loss on proposal embeddings aligned to ground truth by highest IoU. For each ground-truth object $V^{\text{tgt}}$4, the source and target proposals with highest IoU are selected, and source-direction and target-direction cross-entropy terms are computed against all other proposals as negatives. Only samples with proposal IoU greater than 0.5 in both views are used, so poor proposals do not inject supervision noise. The final loss averages the source and target terms and masks them by this proposal-quality indicator.

5. Mask refinement and stage separation

The second stage, Mask Refinement, is introduced because correct proposal selection does not guarantee annotation-quality segmentation. Proposal masks can miss parts, exhibit boundary bias, or be over-segmented or under-segmented. MR is therefore responsible for improving completeness, boundary alignment, and annotation consistency (Liao et al., 6 Aug 2025).

Architecturally, MR uses a SAM 2-like two-way transformer block with query-to-image attention, image-to-query attention, self-attention on query tokens, and MLP updates. Its query is initialized from a learnable output token, the source object embedding $V^{\text{tgt}}$5, the matched target embedding $V^{\text{tgt}}$6, and positional embeddings of the target box. The output token is then converted by an MLP into convolution kernels that are applied to upsampled image features to predict refined masks, mirroring the mask decoder logic in SAM 2.

A notable training decision is that DOM is frozen during MR training. The paper states that joint training hurts DOM’s coarse matching ability, whereas freezing DOM lets MR specialize in refinement. In the reported ablation, DOMR with frozen DOM reaches the best final result, namely 49.7 on Ego$V^{\text{tgt}}$7Exo and 55.2 on Exo$V^{\text{tgt}}$8Ego. This makes DOMR a clear example of a matching-then-refinement architecture in which the stages are optimized for distinct roles rather than jointly collapsed into a single end-to-end objective.

6. Experimental setting and reported performance

DOMR is evaluated on the Ego-Exo4D ego-exo correspondence benchmark, which contains 1,335 takes, 5,566 target objects, 742K egocentric masks, 1.1M exocentric masks, and about 1.8 million masks total, with annotations sampled at 1 FPS (Liao et al., 6 Aug 2025). The annotated objects include wearer-interacted objects, activity-relevant environmental objects, and body parts.

Training is conducted without data augmentation. The proposal generator is YOLO-UniOW-L, the mask encoder/backbone is SAM 2.1 Hiera-B+, and DOM uses 6 attention blocks. Optimization uses AdamW with DOM learning rate $V^{\text{tgt}}$9, DOMR learning rate $M^{\text{src}}$0, weight decay $M^{\text{src}}$1, and 800,000 training iterations. The learning rate decays after 533,336 and 733,337 iterations. The loss weights are $M^{\text{src}}$2 and $M^{\text{src}}$3. Training is reported on 4× NVIDIA A800 GPUs with batch size 16. The main metric is mean IoU between predicted and ground-truth masks.

The headline benchmark numbers are as follows:

Method	Ego$M^{\text{src}}$4Exo IoU	Exo$M^{\text{src}}$5Ego IoU
ObjectRelator	43.9	50.9
XView-XMem + XSegTx	36.9	36.1
DOMR	49.7	55.2

Compared to ObjectRelator, DOMR improves by +5.8 on Ego$M^{\text{src}}$6Exo and +4.3 on Exo$M^{\text{src}}$7Ego. The paper further emphasizes that DOMR is a spatial-only method, yet it outperforms methods that use spatio-temporal modeling. This suggests that dense relational object reasoning itself is a strong signal for ego-exo cross-view correspondence.

7. Relation to dense matching-and-refinement pipelines, strengths, and limitations

DOMR is object-centric, but it is closely related in structure to broader dense matching-and-refinement pipelines. RoMa v2 is described as following the same broad “coarse correspondence estimation + fine refinement” philosophy that Dense Object Matching and Refinement pipelines usually adopt, although it is designed for generic dense two-view matching rather than object-specific correspondence (Edstedt et al., 19 Nov 2025). RoMa v2 first estimates a dense approximate warp and confidence, then refines that warp with local high-resolution reasoning to recover subpixel precision and uncertainty. In that sense, “matching-then-refinement,” the prediction of confidence or uncertainty alongside correspondence, and the separation between coarse reasoning and local correction are directly relevant to DOMR design.

The difference is equally explicit. RoMa v2 operates on full dense two-view matching of scenes, whereas DOMR is framed around object-centric dense matching between ego and exo views and relies on a proposal set constructed from detector and mask-generator outputs. A plausible implication is that DOMR inherits both the benefits and the liabilities of proposal-based pipelines: it can reason richly over a compact set of objects, but missed detections or weak masks can bound the quality of downstream correspondence and refinement.

The strengths reported for DOMR are strong empirical performance on a difficult benchmark, explicit context modeling among multiple objects, practical use of off-the-shelf components, robustness to visually ambiguous scenes, and measurable gains from refinement. The stated limitations are that performance depends on proposal quality, the approach is built around synchronized ego-exo pairs rather than unsynchronized or weakly aligned settings, broader generalization beyond Ego-Exo4D is not yet fully established, and sparse or poor proposals degrade correspondence quality (Liao et al., 6 Aug 2025). These limitations are intrinsic to the framework’s operating assumptions rather than incidental implementation details.

Taken together, DOMR establishes dense object matching among multiple proposals as the organizing principle for cross-view segmentation in ego-exo video. Its distinctive contribution is not merely adding refinement after matching, but making correspondence itself relational: visual similarity is conditioned by positional and semantic cues, and objects are interpreted in the context of other objects in both views.