PanMatch: Unified Dense 2D Matching

• PanMatch is defined as a foundation model that reformulates stereo, optical flow, feature matching, and depth estimation into a single dense 2D displacement estimation task.
• It leverages a frozen DINOv2 encoder with a FlowFormer backbone and a specialized feature transformation pipeline to generate multi-scale, matching-oriented representations.
• PanMatch demonstrates robust cross-task performance with measurable gains, including improved zero-shot optical flow and stereo metrics, showcasing its unified approach.

Searching arXiv for PanMatch and related correspondence-matching papers. PanMatch is a foundation model for two-frame correspondence matching that treats stereo matching, optical flow, feature matching, and two-view depth estimation as instances of a single problem: dense 2D displacement estimation. The model’s defining premise is that task unification should occur at the level of representation and supervision rather than through task-specific heads or ensembles. In this formulation, the same model weights predict pixel-wise displacement fields across domains and tasks, with stereo disparity, feature correspondences, and depth recovered as downstream interpretations of the same displacement output. PanMatch combines a frozen Large Vision Model feature extractor, a feature transformation pipeline tailored to dense correspondence, and a displacement-estimation backbone, and is pretrained on a cross-domain corpus of near 1.8 million samples assembled from optical flow, stereo, and depth-related data (Zhang et al., 11 Jul 2025).

1. Conceptual formulation

PanMatch is defined by the claim that any two-frame correspondence matching task can be expressed as prior-free all-pairs pixel matching under a common 2D displacement field. The paper writes the correspondence relation as

$$\mathbf{I}_{ref}(u_1, v_1) = \mathbf{I}_{tar}(u_2 - \Delta u, v_2 - \Delta v),$$

where $(\Delta u, \Delta v)$ is the displacement vector between reference and target coordinates (Zhang et al., 11 Jul 2025).

Within this formulation, optical flow is the direct prediction target. Stereo matching becomes a constrained special case in rectified geometry, where vertical displacement is zero and disparity is recovered from horizontal displacement via

$\Delta D = - \Delta u, \Delta v = 0.$

Feature matching is obtained by estimating dense displacement first and then extracting reliable correspondences by post-processing, specifically with forward-backward cyclic consistency. Two-view depth estimation is likewise reduced to correspondence estimation followed by geometric conversion under known camera pose and intrinsics (Zhang et al., 11 Jul 2025).

This unification distinguishes PanMatch from earlier “unified” matchers that still retained task-specific architectural structure or fine-tuning. A plausible implication is that PanMatch’s contribution is less a new geometric prior than a reformulation of task boundaries: stereo, flow, and wide-baseline matching are treated as different observation regimes of the same displacement field rather than separate prediction problems.

2. Architecture and feature transformation pipeline

PanMatch uses the same matching backbone across tasks, with the final system instantiating a frozen DINOv2-reg-giant encoder and FlowFormer as the displacement-estimation baseline (Zhang et al., 11 Jul 2025). The architecture is organized around three components: multi-scale all-purpose feature generation, feature transformation, and cost aggregation/regression.

The input is an image pair

$$\mathcal{I}=\{\mathbf{I}_{ref}, \mathbf{I}_{tar}\},$$

and the output is a dense displacement field $(\Delta u, \Delta v)$. Task-specific outputs are not produced by separate heads; instead, stereo disparity, sparse feature matches, and depth are derived from the same displacement predictions (Zhang et al., 11 Jul 2025).

A central claim of the paper is that standard task-trained encoders are insufficiently general for cross-domain matching, while features from Large Vision Models such as DINOv2, SAM, and DAMv2 are substantially more transferable but not directly optimized for dense matching. PanMatch therefore introduces a feature transformation pipeline that converts frozen LVM features into matching-oriented multi-scale representations (Zhang et al., 11 Jul 2025).

Three modules define this pipeline:

1. Position-aware guided feature upsampling: low-resolution LVM features are upsampled using adaptive, guidance-conditioned attention rather than fixed interpolation.
2. Hierarchical adaptation network: multi-layer LVM features are fused with CNN guidance features using a U-shaped FPN-like decoder with ConvNeXt blocks.
3. Cross-view matching constraint: an auxiliary contrastive objective encourages transformed features from corresponding locations to become discriminative for matching (Zhang et al., 11 Jul 2025).

This design suggests that PanMatch does not replace dense matching pipelines wholesale. Rather, it keeps an optical-flow-style regression mechanism and moves most of the innovation into representation quality and supervision unification.

3. Multi-task representation and geometry

The paper explicitly details how multiple tasks map into the common displacement representation. For stereo matching, disparity is recovered from horizontal displacement. For feature matching, dense predictions are filtered using forward-backward cyclic consistency. For two-view depth, PanMatch uses standard projective geometry to convert correspondences into depth under known camera parameters (Zhang et al., 11 Jul 2025).

The paper defines

$$\mathbf{H}=\mathbf{K}_2 \mathbf{R}_2 \mathbf{R}_1^{-1} \mathbf{K}_1^{-1}, \qquad \mathbf{B}=-\mathbf{K}_2 \mathbf{R}_2 \mathbf{R}_1^{-1} \mathbf{T}_1 + \mathbf{K}_2 \mathbf{T}_2,$$

and derives two depth expressions, $Z_u(u_1,v_1)$ and $Z_v(u_1,v_1)$, from the recovered pixel correspondences. To address numerical inconsistency, depth is estimated by least squares as

$$Z_{lsm}(u_1, v_1)=(\mathbf{A}^\mathrm{T} \mathbf{A})^{-1} \mathbf{A}^\mathrm{T} \mathbf{b}.$$

The paper notes that depth recovery can become only semi-dense under degenerate motion such as zero translation or pure optical-axis motion (Zhang et al., 11 Jul 2025).

The significance of this section is methodological rather than purely geometric. PanMatch does not attempt to learn separate stereo- or depth-specific reasoning modules. Instead, it treats geometry as a deterministic readout stage operating on a universally predicted correspondence field. This suggests a shift from task-specific modeling toward correspondence-first modeling.

4. Feature adaptation and training objectives

PanMatch’s feature adaptation pipeline is mathematically explicit. For position-aware guided upsampling, the high-resolution feature at location $\mathbf{p}$ is reconstructed as

$(\Delta u, \Delta v)$0

with adaptive attention weights

$(\Delta u, \Delta v)$1

The neighborhood $(\Delta u, \Delta v)$2 is defined as the 3×3 nearest neighbors in low-resolution feature space (Zhang et al., 11 Jul 2025).

To fuse scales, PanMatch uses a multi-scale patch embedding scheme that aligns 1/2, 1/4, 1/8, and 1/16 features to a common 1/8-scale grid with patch sizes 4, 2, 1, and 1/2 respectively, followed by projection, concatenation, and MLP fusion (Zhang et al., 11 Jul 2025).

The cross-view matching constraint computes a 3D score volume

$(\Delta u, \Delta v)$3

and applies a pixel-wise InfoNCE objective with temperature $(\Delta u, \Delta v)$4: $(\Delta u, \Delta v)$5 The total loss is

$(\Delta u, \Delta v)$6

where $(\Delta u, \Delta v)$7 is the displacement loss from the underlying FlowFormer-style baseline (Zhang et al., 11 Jul 2025).

The paper explicitly states that occluded regions are not excluded from the contrastive loss. This suggests an emphasis on globally meaningful representation learning rather than supervision restricted only to visible matches.

5. Data construction and training regime

A major component of PanMatch is its multi-task pretraining corpus, described as a cross-domain dataset with near 1.8 million samples from stereo matching, optical flow, and feature-matching-related domains (Zhang et al., 11 Jul 2025). The training set includes optical flow datasets, disparity datasets, and depth datasets converted into dense displacement supervision.

The paper lists the following sources and approximate sizes: FlyingChairs (~23k), FlyingThings (~81k), Monkaa (~35k), AutoFlow (~27k), Dynamic Replica (~144k), TartanAir (~306k), Kubric (~132k), CVO (~125k), VirtualKITTI2 (~42k), Hypersim (~367k), MegaDepth (~285k), CREStereo (~200k), and FallingThings (~62k) (Zhang et al., 11 Jul 2025).

The unification of labels is itself a methodological contribution. Optical flow datasets already provide displacement; stereo labels are converted using $(\Delta u, \Delta v)$8; and depth datasets are converted to displacement using camera geometry. For posed rigid-scene data such as Hypersim and MegaDepth, adjacent views with similar perspectives are selected and overlapping regions are used to derive correspondence supervision (Zhang et al., 11 Jul 2025).

Training uses a three-stage curriculum. Stage 1 trains a common optical-flow baseline following FlowFormer protocol. Stage 2 trains PanMatch for 300k iterations with batch size 8 and crop size $(\Delta u, \Delta v)$9 on the full mixed dataset. Stage 3 fine-tunes for 60k iterations at crop size $\Delta D = - \Delta u, \Delta v = 0.$0 on a selected subset of datasets. Optimization uses AdamW with a one-cycle schedule and maximum learning rate $\Delta D = - \Delta u, \Delta v = 0.$1. Augmentations include color jitter, asymmetric occlusion, image flipping, random vertical shift for stereo crops, and random rotation of stereo pairs to balance horizontal and vertical displacement learning (Zhang et al., 11 Jul 2025).

This suggests that PanMatch’s cross-task behavior depends not only on architectural unification but also on systematic label homogenization across heterogeneous sources.

6. Empirical performance, robustness, and limitations

PanMatch is evaluated across stereo matching, optical flow, feature matching, and depth estimation using the same weights (Zhang et al., 11 Jul 2025). The paper reports that PanMatch outperforms prior unified models such as UniMatch and Flow-Anything on cross-task evaluations and remains comparable to many task-specific methods on task-oriented benchmarks.

For optical flow cross-task zero-shot evaluation, PanMatch achieves 0.32 EPE on Infinigen and 0.31 EPE on Spring, reported as 16% and 23% better than the second-best methods respectively (Zhang et al., 11 Jul 2025). For stereo cross-task evaluation, the paper reports improvements over the second best by 30% on Middlebury, 16% on ETH3D, and 10% on KITTI12 (Zhang et al., 11 Jul 2025).

Zero-shot stereo results include 3.39 on Middlebury Half, 3.64 on Middlebury Quarter, 3.27 on KITTI 2015, 2.77 on KITTI 2012, and 1.79 on ETH3D (Zhang et al., 11 Jul 2025). Zero-shot optical flow results include 0.85 on Sintel Clean, 1.67 on Sintel Final, 1.94 KITTI F1-epe, and 5.54 KITTI F1-all (Zhang et al., 11 Jul 2025).

On feature matching, PanMatch remains below the strongest dedicated matchers on ScanNet and MegaDepth but performs competitively, and on WxBS achieves mAA@10 = 64.2, exceeding LoFTR and DKM while remaining below RoMa (Zhang et al., 11 Jul 2025). For two-view depth, it achieves competitive metrics and is often best in Sq Rel or RMSE on several datasets, despite not being a dedicated depth-regression model (Zhang et al., 11 Jul 2025).

The paper places special emphasis on abnormal-domain robustness. On Booster, a difficult dataset with specular and transparent surfaces, PanMatch attains EPE 1.95, outperforming several robust benchmark-tuned baselines. Qualitative results are also reported on rainy DrivingStereo scenes, night Oxford RobotCar scenes, transparent objects, and satellite imagery from UrbanSemantic3D, where PanMatch is said to continue producing meaningful outputs while many prior models fail (Zhang et al., 11 Jul 2025).

The ablation study attributes gains to all major modules. Guided upsampling outperforms bilinear, deconvolution, and pixel shuffle variants. The combination of FPN, multi-scale embedding, and InfoNCE yields the best overall results. Frozen LVM choice also matters: the strongest reported encoder variant is DINOv2-G, which outperforms smaller or more conventional backbones in the reported metrics (Zhang et al., 11 Jul 2025).

Several limitations are explicit. Two-view depth recovery can be numerically unstable under degenerate motion. PanMatch is not always best on benchmark-tuned seen datasets where specialized models benefit from task-specific fine-tuning. It also trails the strongest specialist matchers on some feature-matching benchmarks. Finally, the method’s performance depends materially on heavy frozen LVM backbones such as DINOv2-G (Zhang et al., 11 Jul 2025).

A common misconception would be to describe PanMatch as merely another unified architecture with shared weights. The paper’s actual claim is narrower and more specific: unification comes from re-expressing tasks as dense displacement estimation and then endowing an optical-flow-style matcher with sufficiently general features and sufficiently diverse supervision. This suggests that PanMatch is best understood not as a generic multi-head matcher, but as a displacement-centric correspondence foundation model (Zhang et al., 11 Jul 2025).