Deploy DINO with Many-to-Many Association

Abstract: Motivated by the limited generalization of supervised image matching models to unseen image domains, we explore the zero-shot deployment of DINO features for this task. The generalist visual representation extracted from DINO has inherent ambiguity when used to match feature points among semantically similar instances, prompting us to adopt a many-to-many (m-to-m) matching paradigm. However, the existing robust mechanism under m-to-m data association is computationally heavy, which requires finding a maximum-cardinality matching in the inlier association graph for each parameter evaluation. To address this inefficiency, we introduce a novel likelihood perspective, which interprets the existing method as a zeroth-order approximation of otherwise intractable likelihood calculation,and inspires us to propose a faster and finer-grained robust mechanism, termed as Harmonic Consensus Maximization (HCM). Take camera pose estimation as an exemplifying downstream task, we demonstrate that general-purpose visual features, used out of the box without any adaptation, can compete with specialized matching models on out-of-distribution datasets when mated with m-to-m association and the HCM mechanism.

• The paper introduces a novel many-to-many association paradigm using HCM to robustly match DINO features for geometric correspondence, improving performance on out-of-distribution datasets.
• It employs a probabilistic framework where harmonic consensus maximization replaces exhaustive matching, reducing computational complexity while enhancing pose estimation accuracy.
• Empirical results on camera pose estimation demonstrate that the proposed pipeline outperforms traditional one-to-one matching methods under challenging OOD scenarios.

Deploying DINO Features for Geometric Matching via Many-to-Many Association

Introduction

This paper addresses the challenge of deploying generalist visual features—specifically from DINOv2/v3 models—for geometric correspondence in images, focusing on settings where supervised models exhibit limited domain generalization. The principal observation is that, despite providing strong semantic descriptors, DINO features produce inherent ambiguities in geometric matching, particularly among semantically similar but distinct image regions. The authors propose embracing these ambiguities by shifting from the classical one-to-one (1-to-1) association paradigm to a many-to-many (m-to-m) association framework. To effectively utilize this framework, the paper introduces Harmonic Consensus Maximization (HCM), a robust, efficient alternative to the commonly used Matching Cardinality Maximization (MCM). The empirical evaluation demonstrates the competitiveness of DINO+m-to-m+HCM pipelines on out-of-distribution (OOD) datasets for downstream tasks like camera pose estimation.

Inherent Ambiguity in DINO Features for Geometric Correspondence

DINOv3 features capture rich semantic information, but their representation is not fine-grained enough to guarantee correct geometric matches. This semantic richness results in feature similarity patterns where the most similar regions are not necessarily the correct geometric correspondences. The ambiguity arises from both intra-instance variation (e.g., repetitive local patterns) and inter-instance ambiguities (e.g., semantically similar parts in different objects or backgrounds).

Figure 1: DINOv3 feature visualizations reveal persistent ambiguity in geometric correspondence, with high feature similarity observed among incorrect associations in both intra- and inter-instance scenarios.

This finding invalidates the suitability of 1-to-1 association for geometric matching with DINO features, motivating a paradigm shift.

Many-to-Many Association: Formulation and Graph-Theoretic Insights

The many-to-many paradigm accepts multiple candidate correspondences for each feature point. This is formalized using bipartite graphs, where nodes in each image correspond to candidate keypoints and edges represent potential associations found via mutual K-Nearest Neighbors (MKNN) in the embedding space. The m-to-m setting increases recall at the expense of reduced precision, reflecting the dense, ambiguous associations found in DINO feature spaces.

Probabilistic Formulation and Harmonic Consensus Maximization

The classical robust estimation (e.g., RANSAC) relies on maximizing the consensus under 1-to-1 association, using residuals and a threshold to delineate inliers and outliers. Under m-to-m association, the computational challenge increases: for each hypothesis, identifying the largest mutually compatible inlier set corresponds to finding a maximum matching in a bipartite graph, an operation with superlinear complexity.

To address this, the authors develop a unified probabilistic framework. They interpret the existing MCM approach as a zeroth-order approximation to the true—but intractable—likelihood under the model. The likelihood under the m-to-m setting involves marginalizing over all permissible matching configurations, a combinatorial sum.

Figure 2: A correct inlier under a geometric model may correspond to a spurious association, highlighting the need for probabilistic modeling of association ambiguity.

Figure 3: A toy example illustrating likelihood calculation in the presence of ambiguous associations.

Building on this, the authors propose HCM, which discards the computationally expensive matching step and instead employs marginal probability assignments to associations, leading to a robust likelihood-based score per hypothesis. HCM’s evaluation complexity is linear in the number of associations, and its continuous, symmetry-enforcing scoring provides finer discrimination among ambiguous candidates.

Practical Pipeline: DINO, Unsupervised Keypoint Detection, and MKNN Association

The authors deploy the DINO features, interpolated at SuperPoint-detected keypoints (achieving sub-patch pixel localization), and use MKNN to admit top-K mutual matches per keypoint. Marginal probabilities are assigned not by feature similarity but by solving a convex optimization problem that enforces soft row/column sum constraints over the association graph, leading to uniform and structure-regularized probability distributions, which are crucial given DINO’s homogenized similarity patterns.

The empirical evaluation of association group precision demonstrates that increasing K improves recall (group precision), especially in challenging OOD scenarios characterized by large viewpoint or scene variation.

Figure 4: Left image group precision $p_x$ vs. MKNN hyperparameter $K$ and viewpoint difficulty, establishing the trade-off between recall and ambiguity.

Camera Pose Estimation: Robust Zero-Shot Deployment

Camera pose estimation is used as the primary downstream task. The authors evaluate pipelines on ScanNet (in-domain for most supervised methods) and NAVI-Multi/Wild (OOD, object-centric datasets with significant viewpoint and background variation). Their evaluation considers both correspondence quality (Precision, Dense Recall, Success Ratio) and pose accuracy (AUC at various thresholds).

Key observations include:

• DINO+m-to-m+HCM pipelines achieve superior recall and matchability on OOD data compared to supervised matchers.
• HCM consistently improves both speed and pose estimation accuracy relative to MCM, due to its finer likelihood-based scoring.
• The pipeline demonstrates minimal sensitivity to marginal probability hyperparameters, indicating robustness across OOD and varying overlap regimes.

  Figure 5: Sensitivity analysis of HCM to group precision hyperparameter $q$ on the Wild-Average dataset, illustrating stability across a broad operational range.

Implications and Future Directions

The demonstrated efficacy of m-to-m association and likelihood-based robust estimation with DINO features highlights the need to rethink canonical 1-to-1 matching paradigms when deploying semantic foundation models in geometric tasks. The results challenge the notion that increasingly complex supervised pipelines are universally necessary for robust correspondence, especially in OOD regimes.

Practically, this work suggests that:

• Foundation model features should be coupled with loose association schemes and probabilistic robust estimation for reliable deployment in new domains.
• Specialized matching heads trained on in-domain data do not guarantee generalization; robust association and estimation pipelines are necessary.
• There exists potential for integrating HCM into supervised matchers’ assignment matrices and for tuning foundation model backbones with likelihood-derived objectives.

Theoretically, the paper advances the probabilistic understanding of matching under relaxed association constraints and demonstrates that maximizing global model fit need not require exhaustive combinatorial search.

Conclusion

This work systematically analyzes and addresses the ambiguity intrinsic to generalist DINO features for geometric matching, proposing and validating a many-to-many association paradigm and the Harmonic Consensus Maximization mechanism. Empirical results establish that, when equipped with principled association and likelihood-based robust estimation, zero-shot pipelines leveraging DINO features are not only competitive but can surpass state-of-the-art supervised matchers on out-of-distribution image matching and pose estimation tasks. The framework sets the stage for further integration of probabilistic optimization into geometric vision, particularly as semantic foundation models proliferate.