FlowIt: Global Matching for Optical Flow with Confidence-Guided Refinement

Abstract: We present FlowIt, a novel architecture for optical flow estimation designed to robustly handle large pixel displacements. At its core, FlowIt leverages a hierarchical transformer architecture that captures extensive global context, enabling the model to effectively model long-range correspondences. To overcome the limitations of localized matching, we formulate the flow initialization as an optimal transport problem. This formulation yields a highly robust initial flow field, alongside explicitly derived occlusion and confidence maps. These cues are then seamlessly integrated into a guided refinement stage, where the network actively propagates reliable motion estimates from high-confidence regions into ambiguous, low-confidence areas. Extensive experiments across the Sintel, KITTI, Spring, and LayeredFlow datasets validate the efficacy of our approach. FlowIt achieves state-of-the-art results on the competitive Sintel and KITTI benchmarks, while simultaneously establishing new state-of-the-art cross-dataset zero-shot generalization performance on Sintel, Spring, and LayeredFlow.

• The paper introduces a global matching approach for optical flow that leverages optimal transport for robust initialization.
• The methodology combines a hierarchical transformer backbone with confidence-guided refinement to effectively handle occlusions and large displacements.
• Results on Sintel, KITTI, and other benchmarks demonstrate state-of-the-art accuracy and strong zero-shot generalization.

FlowIt: Global Matching for Optical Flow with Confidence-Guided Refinement

Introduction and Motivation

Optical flow estimation is fundamental for a wide range of vision tasks requiring temporally coherent dense correspondence, with applications from action recognition to autonomous navigation and video inpainting. Traditional deep learning-based pipelines predominantly employ local refinement architectures that rely on iterative, local matching, leading to difficulties in modeling large displacements and scenarios with textureless or occluded regions. Global matching strategies provide stronger modeling capacity for long-range correspondences but suffer from scalability issues due to the computational overhead of dense cost volumes, typically necessitating low-resolution processing and introducing errors in high-displacement regions.

FlowIt introduces a global matching-based architecture that leverages a hierarchical vision transformer backbone, an optimal transport-based flow field initialization, and a confidence-guided refinement regime. The approach aims to robustly estimate optical flow with state-of-the-art accuracy while retaining generalization capability across domains, particularly when faced with occlusion, ambiguity, or adverse scene properties such as transparency and reflection.

Architecture Overview

FlowIt is organized into distinct stages:

1. Efficient Feature Extraction: FlowIt utilizes a CNN backbone with a Feature Pyramid Network (FPN) to extract multi-scale feature representations from input image pairs. These features are subsequently refined using a Multi-Resolution Transformer (MRT) that supports horizontal and vertical information flow across spatial resolutions. This design provides both local detail capture and global context sharing.
2. Global Optimal Transport-Based Flow Initialization: At the core of FlowIt is the modeling of optical flow initialization as an optimal transport problem. Using $1/4$-resolution feature maps, the all-pairs 4D correlation volume is regularized via entropy and solved with the Sinkhorn algorithm. The result is a soft, globally consistent correspondence probability tensor that enables robust estimation even in ambiguous scenarios. Occlusion and confidence maps are simultaneously derived, with occlusion handled via a dustbin strategy in the transport plan and confidence directly computed as the summed matching probability around the expected correspondence.
3. Confidence-Guided Refinement: The initial predictions are refined with a two-tier regime:
   • A single global refinement step propagates reliable motion cues from high-confidence regions, determined by the initial confidence map, into uncertain or occluded regions via a UNet-based aggregation module.
   • This is followed by three local refinement iterations (axis-wise, not coupled) leveraging local cost volumes and prior predictions, iteratively sharpening the flow while maintaining numerical stability via logit space accumulation for both occlusion and confidence maps.

Predictions are upsampled using convex combinations of coarse flow vectors, guided by learnable weights, for full-resolution results. The complete architecture supports multiple model sizes, scaling from lightweight to large configurations for various computational demands.

Qualitative Performance on Benchmark Datasets

FlowIt’s performance is systematically validated across leading optical flow benchmarks, including Sintel, KITTI, Spring, and LayeredFlow. On the challenging and diverse KITTI benchmark, FlowIt achieves notable improvements in both quantitative accuracy and qualitative detail retention compared to established baselines.

Figure 1: Qualitative Results on the KITTI test set demonstrate superior detail recovery by FlowIt (XL) compared to leading methods such as WAFT, DPFlow, and AnyFlow.

On Sintel, FlowIt’s global matching and confidence propagation contribute to fine detail preservation and reliable handling of complex nonrigid motion.

Figure 2: Results on Sintel (training set) illustrate FlowIt’s ability to reconstruct subtle motions and maintain high accuracy, even in the presence of spatial ambiguities.

Zero-shot generalization—evaluation under strong domain shift—is a primary focus. FlowIt models, particularly the (L) and (XL) variants, establish new state-of-the-art results or match the top-tier models in terms of endpoint error and high-precision matching across unseen datasets.

Figure 3: Zero-shot generalization on Sintel (clean) shows FlowIt surpassing or matching the accuracy of models specifically trained or adapted to this domain.

Figure 4: FlowIt maintains robustness in the more challenging Sintel (final), supporting strong claims regarding its generalization capability.

On less conventional benchmarks such as LayeredFlow, which emphasizes non-Lambertian effects like reflections and transparency, FlowIt retains competitive performance.

Figure 5: FlowIt (XL) demonstrates improved correspondence estimation for transparent and reflective surfaces, outperforming competing approaches, as seen in qualitative results on LayeredFlow.

Technical Contributions and Analysis

Several design choices underpin FlowIt's capabilities:

• Global Optimal Transport Initialization: By replacing local matching maxima with a globally regularized probabilistic transport plan, FlowIt obtains robust initial matches not susceptible to local minima.
• Confidence/occlusion Supervision and Refinement: Joint supervision and propagation of confidence/occlusion maps not only help in explicit error correction during refinement but also provide a measure of reliability for downstream consumers.
• Transfer Learning from Stereo Depth: Initialization with stereo foundation models further enhances training efficiency and final accuracy, despite task divergence, evidencing the synergy between correspondence estimation tasks.
• Ablation on Refinement Strategies: Decoupling axis-wise updates for the flow field improves optimization stability and final accuracy over coupled updates.

Numerical Results and Claims

FlowIt (XL) achieves leading performance on Sintel (Clean), competitive results on Sintel (Final), and a new best score on KITTI’s non-occluded regions. For zero-shot transfer, FlowIt’s (L) and (XL) variants outperform or match all published baselines across Sintel (train Final), Spring, and LayeredFlow. These claims are strongly substantiated by both quantitative benchmarks and qualitative figures throughout the study. Notably, the architecture does not require excessive computational resources, and ablation confirms robust trade-offs for lighter-weight variants.

Implications and Future Directions

The practical implication of FlowIt is clear: global matching with confidence-guided refinement achieves robust, generalizable optical flow estimation in the presence of scene complexity, occlusions, and distribution shift. The explicit reliability estimation is directly applicable to safety-critical pipelines such as autonomous navigation and video post-processing, enabling downstream modules to reason about uncertainty.

Theoretically, the results encourage a paradigm where global matching establishes strong priors, while uncertainty reasoning and confidence propagation are integrated into the refinement process, rather than treated as post-hoc or auxiliary tasks.

Future research could extend these principles to multi-frame flow, integrate the architecture with large-scale pretraining from foundation models, or adapt global matching approaches for other dense prediction problems in computer vision. Efficient scaling to ultra-high resolutions or temporal consistency across longer horizons are also compelling directions given the architecture’s modularity and demonstrated robustness.

Conclusion

FlowIt presents an effective synthesis of hierarchical transformer-based feature extraction, optimal transport-based matching, and confidence-aware refinement for optical flow. Extensive benchmarks and generalization tests validate its strong performance, especially for large displacements, occlusions, and unfavorable visual phenomena. Its design advances the state-of-the-art in both operational accuracy and reliability estimation, and informs future directions for generalized dense correspondence modeling.