Taming Flow Matching with Unbalanced Optimal Transport into Fast Pansharpening (2503.14975v1)

Abstract: Pansharpening, a pivotal task in remote sensing for fusing high-resolution panchromatic and multispectral imagery, has garnered significant research interest. Recent advancements employing diffusion models based on stochastic differential equations (SDEs) have demonstrated state-of-the-art performance. However, the inherent multi-step sampling process of SDEs imposes substantial computational overhead, hindering practical deployment. While existing methods adopt efficient samplers, knowledge distillation, or retraining to reduce sampling steps (e.g., from 1,000 to fewer steps), such approaches often compromise fusion quality. In this work, we propose the Optimal Transport Flow Matching (OTFM) framework, which integrates the dual formulation of unbalanced optimal transport (UOT) to achieve one-step, high-quality pansharpening. Unlike conventional OT formulations that enforce rigid distribution alignment, UOT relaxes marginal constraints to enhance modeling flexibility, accommodating the intrinsic spectral and spatial disparities in remote sensing data. Furthermore, we incorporate task-specific regularization into the UOT objective, enhancing the robustness of the flow model. The OTFM framework enables simulation-free training and single-step inference while maintaining strict adherence to pansharpening constraints. Experimental evaluations across multiple datasets demonstrate that OTFM matches or exceeds the performance of previous regression-based models and leading diffusion-based methods while only needing one sampling step. Codes are available at https://github.com/294coder/PAN-OTFM.

• The paper introduces the Optimal Transport Flow Matching (OTFM) framework that replaces multi-step diffusion with a one-step high-quality pansharpening approach.
• It integrates unbalanced optimal transport by employing f-divergence to relax strict marginal constraints and improve stability in remote sensing data.
• Experimental results on WV3, GaoFen-2, and QuickBird datasets demonstrate that OTFM achieves competitive performance while significantly reducing computational overhead.

Summary of "Taming Flow Matching with Unbalanced Optimal Transport into Fast Pansharpening"

This essay provides an authoritative analysis of the paper titled "Taming Flow Matching with Unbalanced Optimal Transport into Fast Pansharpening". The paper introduces the Optimal Transport Flow Matching (OTFM) framework which integrates unbalanced optimal transport (UOT) principles into flow matching for enhancing the efficiency of pansharpening, a crucial task in remote sensing. The key innovation lies in utilizing UOT to facilitate one-step, high-quality pansharpening, rather than relying on the extensive and computationally demanding multi-step sampling processes typical of traditional diffusion models.

Diffusion Models and Flow Matching

In recent approaches, diffusion models have shown significant promise for image processing tasks including pansharpening. They typically use stochastic differential equations (SDEs) for image synthesis, requiring multiple sampling steps to achieve high-quality results. The OTFM framework diverges from this by leveraging flow matching, which simplifies training and sampling by approximating the transport between data distributions through direct interpolation from initial noise to the target distribution. This approach streamlines the process into single-step generation without compromising quality.

Unbalanced Optimal Transport

The integration of unbalanced optimal transport represents a fundamental advancement in the OTFM framework. Traditional optimal transport (OT) methods enforce strict marginal restrictions which can lead to stability issues and reduced network flexibility. UOT relaxes these constraints by introducing f-divergence, allowing for more adaptable mass transportation solutions that are particularly advantageous when dealing with the intrinsic disparities inherent in remote sensing data, such as spatial and spectral inconsistencies.

Figure 1: Key distinction from previous diffusion models. Traditional diffusion models typically sample from a Gaussian distribution, requiring numerous iterative steps (e.g., 1000) to achieve results. In contrast, our OTFM harnesses the power of unbalanced optimal transport, enabling high-quality pansharpening with just one-step sampling step.

Application and Performance

Experimental results illustrate the practicality and efficiency of the OTFM framework across multiple datasets, such as WV3, GaoFen-2, and QuickBird. OTFM consistently achieved or surpassed performance benchmarks set by regression-based and leading diffusion-based methods. The adoption of a pansharpening-specific regularization term within the UOT framework further enhances the model's robustness, ensuring adherence to spatial and spectral consistency while mitigating computational overhead.

Figure 2: Training and one-step sampling diagrams of the proposed OTFM. Due to the flow matching velocity construction, the UOT-mapped HRMS can be obtained in a single step.

Network Architecture

The paper details a bespoke U-net architecture for the mapping network within the OTFM framework, utilizing neighborhood attention mechanisms to efficiently capture global feature responses. This architecture is crucial for the pansharpening task, providing robust feature encoding under the conditions set by PAN and LRMS inputs. The potential network is designed to operate as a conditioned discriminator, optimizing the UOT dual formulation to predict high-quality pansharpened images directly from the input data.

Conclusion

The OTFM framework represents a significant step forward in pansharpening techniques by addressing the computational challenges of existing methods. By leveraging UOT within a flow matching context, it enables efficient single-step generation of high-quality images. This advancement aligns well with the ongoing trend in AI research towards creating more efficient, scalable, and adaptable models. Future research could explore extending the methodology to other remote sensing tasks and further optimizations in network architecture to enhance efficacy under varying conditions.

Figure 3: Visual comparisons on WV3 (1-2 rows) and GaoFen-2 (3-4 rows) cases. The second and fourth rows are error maps.