Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Content-Adaptive Non-Local Convolution for Remote Sensing Pansharpening (2404.07543v1)

Published 11 Apr 2024 in cs.CV and eess.IV

Abstract: Currently, machine learning-based methods for remote sensing pansharpening have progressed rapidly. However, existing pansharpening methods often do not fully exploit differentiating regional information in non-local spaces, thereby limiting the effectiveness of the methods and resulting in redundant learning parameters. In this paper, we introduce a so-called content-adaptive non-local convolution (CANConv), a novel method tailored for remote sensing image pansharpening. Specifically, CANConv employs adaptive convolution, ensuring spatial adaptability, and incorporates non-local self-similarity through the similarity relationship partition (SRP) and the partition-wise adaptive convolution (PWAC) sub-modules. Furthermore, we also propose a corresponding network architecture, called CANNet, which mainly utilizes the multi-scale self-similarity. Extensive experiments demonstrate the superior performance of CANConv, compared with recent promising fusion methods. Besides, we substantiate the method's effectiveness through visualization, ablation experiments, and comparison with existing methods on multiple test sets. The source code is publicly available at https://github.com/duanyll/CANConv.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (47)
  1. Context-driven fusion of high spatial and spectral resolution images based on oversampled multiresolution analysis. IEEE Transactions on Geoscience and Remote Sensing, 40(10):2300–2312, 2002.
  2. Full-resolution quality assessment of pansharpening: Theoretical and hands-on approaches. IEEE Geoscience and Remote Sensing Magazine, 10(3):168–201, 2022.
  3. k-means++: the advantages of careful seeding. In Proceedings of the 18th Annual ACM-SIAM Symposium on Discrete Algorithms. SIAM, 2007.
  4. Joseph W. Boardman. Automating spectral unmixing of AVIRIS data using convex geometry concepts. 1993.
  5. A non-local algorithm for image denoising. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), pages 60–65 vol. 2, 2005.
  6. Dynamic region-aware convolution. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8060–8069, 2021.
  7. Dynamic convolution: Attention over convolution kernels. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11027–11036, 2020.
  8. A New Adaptive Component-Substitution-Based Satellite Image Fusion by Using Partial Replacement. IEEE Transactions on Geoscience and Remote Sensing, 49(1):295–309, 2011.
  9. Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering. IEEE Transactions on Image Processing, 16(8):2080–2095, 2007.
  10. Deformable convolutional networks. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 764–773, 2017.
  11. Detail Injection-Based Deep Convolutional Neural Networks for Pansharpening. IEEE Transactions on Geoscience and Remote Sensing, 59(8):6995–7010, 2021.
  12. Machine Learning in Pansharpening: A benchmark, from shallow to deep networks. IEEE Geoscience and Remote Sensing Magazine, 10(3):279–315, 2022.
  13. Yinyang K-means: A drop-in replacement of the classic K-means with consistent speedup. In Proceedings of the 32nd ICML - Volume 37, Lille, France, 2015. JMLR.org.
  14. A Variational Pan-Sharpening With Local Gradient Constraints. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10257–10266. IEEE, 2019.
  15. Hypercomplex Quality Assessment of Multi/Hyperspectral Images. IEEE Geoscience and Remote Sensing Letters, 6(4):662–665, 2009.
  16. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, Las Vegas, NV, USA, 2016. IEEE.
  17. Pansharpening via Detail Injection Based Convolutional Neural Networks. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(4):1188–1204, 2019.
  18. Dynamic Filter Networks. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2016.
  19. LAGConv: Local-Context Adaptive Convolution Kernels with Global Harmonic Bias for Pansharpening. Proceedings of the AAAI Conference on Artificial Intelligence, 36(1):1113–1121, 2022.
  20. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
  21. Fully Trainable and Interpretable Non-local Sparse Models for Image Restoration. In Computer Vision – ECCV 2020, pages 238–254. 2020.
  22. Involution: Inverting the Inherence of Convolution for Visual Recognition. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12316–12325, Nashville, TN, USA, 2021a. IEEE.
  23. Cross-Patch Graph Convolutional Network for Image Denoising. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 4631–4640, Montreal, QC, Canada, 2021b. IEEE.
  24. Non-local sparse models for image restoration. In 2009 IEEE 12th International Conference on Computer Vision, pages 2272–2279, Kyoto, 2009. IEEE.
  25. Pansharpening by Convolutional Neural Networks. Remote Sensing, 8(7):594, 2016.
  26. Review of the pansharpening methods for remote sensing images based on the idea of meta-analysis: Practical discussion and challenges. Information Fusion, 46:102–113, 2019.
  27. A New Pansharpening Algorithm Based on Total Variation. IEEE Geoscience and Remote Sensing Letters, 11(1):318–322, 2014.
  28. Source-Adaptive Discriminative Kernels based Network for Remote Sensing Pansharpening. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, pages 1283–1289, 2022.
  29. Neural Nearest Neighbors Networks. In Advances in Neural Information Processing Systems, 2018.
  30. U-Net: Convolutional Networks for Biomedical Image Segmentation. In Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, pages 234–241, Cham, 2015. Springer International Publishing.
  31. Pixel-adaptive convolutional neural networks. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11158–11167, 2019.
  32. Variational Pansharpening by Exploiting Cartoon-Texture Similarities. IEEE Transactions on Geoscience and Remote Sensing, 60:1–16, 2022.
  33. Interpretable model-driven deep network for hyperspectral, multispectral, and panchromatic image fusion. IEEE Trans. Neural Netw. Learn. Syst., pages 1–14, 2023.
  34. Gemine Vivone. Robust Band-Dependent Spatial-Detail Approaches for Panchromatic Sharpening. IEEE Transactions on Geoscience and Remote Sensing, 57(9):6421–6433, 2019.
  35. Contrast and Error-Based Fusion Schemes for Multispectral Image Pansharpening. IEEE Geoscience and Remote Sensing Letters, 11(5):930–934, 2014.
  36. Full Scale Regression-Based Injection Coefficients for Panchromatic Sharpening. IEEE Transactions on Image Processing, 27(7):3418–3431, 2018.
  37. A new benchmark based on recent advances in multispectral pansharpening: Revisiting pansharpening with classical and emerging pansharpening methods. IEEE Geoscience and Remote Sensing Magazine, 9(1):53–81, 2021.
  38. Lucien Wald. Data Fusion. Definitions and Architectures - Fusion of Images of Different Spatial Resolutions. 2002.
  39. Fusion of satellite images of different spatial resolutions: Assessing the quality of resulting images. Photogrammetric engineering and remote sensing, 63(6):691–699, 1997.
  40. SSconv: Explicit Spectral-to-Spatial Convolution for Pansharpening. In Proceedings of the 29th ACM International Conference on Multimedia, pages 4472–4480, 2021.
  41. Dynamic Cross Feature Fusion for Remote Sensing Pansharpening. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 14667–14676. IEEE, 2021.
  42. LRTCFPan: Low-rank tensor completion based framework for pansharpening. IEEE Trans. on Image Process., 32:1640–1655, 2023.
  43. Memory-augmented model-driven network for pansharpening. In ECCV, pages 306–322, 2022.
  44. PanNet: A Deep Network Architecture for Pan-Sharpening. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 1753–1761, 2017.
  45. Decoupled Dynamic Filter Networks. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6643–6652, 2021.
  46. Cross-Scale Internal Graph Neural Network for Image Super-Resolution, 2020.
  47. Deformable convnets v2: More deformable, better results. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9300–9308, 2019.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - Duanyll/CANConv: Content-Adaptive Non-Local Convolution for Remote Sensing Pansharpening (58 stars)