Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
158 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Hyperspectral and multispectral image fusion with arbitrary resolution through self-supervised representations (2405.17818v2)

Published 28 May 2024 in cs.CV and eess.IV

Abstract: The fusion of a low-resolution hyperspectral image (LR-HSI) with a high-resolution multispectral image (HR-MSI) has emerged as an effective technique for achieving HSI super-resolution (SR). Previous studies have mainly concentrated on estimating the posterior distribution of the latent high-resolution hyperspectral image (HR-HSI), leveraging an appropriate image prior and likelihood computed from the discrepancy between the latent HSI and observed images. Low rankness stands out for preserving latent HSI characteristics through matrix factorization among the various priors. However, the primary limitation in previous studies lies in the generalization of a fusion model with fixed resolution scales, which necessitates retraining whenever output resolutions are changed. To overcome this limitation, we propose a novel continuous low-rank factorization (CLoRF) by integrating two neural representations into the matrix factorization, capturing spatial and spectral information, respectively. This approach enables us to harness both the low rankness from the matrix factorization and the continuity from neural representation in a self-supervised manner.Theoretically, we prove the low-rank property and Lipschitz continuity in the proposed continuous low-rank factorization. Experimentally, our method significantly surpasses existing techniques and achieves user-desired resolutions without the need for neural network retraining. Code is available at https://github.com/wangting1907/CLoRF-Fusion.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (48)
  1. (2006). MTF-tailored multiscale fusion of high-resolution MS and Pan imagery. Photogrammetric Engineering & Remote Sensing, 72(5), 591–596. doi: 10.14358/PERS.72.5.591
  2. (2007). Improving component substitution pansharpening through multivariate regression of MS +++ pan data. IEEE Transactions on Geoscience and Remote Sensing, 45(10), 3230–3239. doi: 10.1109/TGRS.2007.901007
  3. (2021). Image generators with conditionally-independent pixel synthesis. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 14278–14287). doi: 10.1109/CVPR46437.2021.01405
  4. (2015). 220 band aviris hyperspectral image data set: June 12, 1992 indian pine test site 3. Purdue University Research Repository, 10(7), 991.
  5. (2021). Learning continuous image representation with local implicit image function. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 8628–8638). doi: 10.1109/CVPR46437.2021.00852
  6. (2017). Nonnegative matrix factorization using nonnegative polynomial approximations. IEEE Signal Processing Letters, 24(7), 948–952. doi: 10.1109/LSP.2017.2697680
  7. (2004). Exploiting spectral and spatial information in hyperspectral urban data with high resolution. IEEE Geoscience and Remote Sensing Letters, 1(4), 322–326. doi: 10.1109/LGRS.2004.837009
  8. (2023). Implicit neural feature fusion function for multispectral and hyperspectral image fusion. arXiv preprint arXiv:2307.07288.
  9. (2017). Hyperspectral image super-resolution via non-local sparse tensor factorization. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 5344–5353). doi: 10.1109/CVPR.2017.411
  10. (2021). Regularizing hyperspectral and multispectral image fusion by CNN denoiser. IEEE Transactions on Neural Networks and Learning Systems, 32(3), 1124–1135. doi: 10.1109/TNNLS.2020.2980398
  11. (2016). Hyperspectral image super-resolution via non-negative structured sparse representation. IEEE Transactions on Image Processing, 25(5), 2337–2352. doi: 10.1109/TIP.2016.2542360
  12. (2021). Generative models as distributions of functions. In International Conference on Artificial Intelligence and Statistics (AISTATS) (pp. 2989–3015).
  13. (2014). Subspace-based support vector machines for hyperspectral image classification. IEEE Geoscience and Remote Sensing Letters, 12(2), 349–353. doi: 10.1109/LGRS.2014.2341044
  14. (2024). Band-wise hyperspectral image pansharpening using CNN model propagation. IEEE Transactions on Geoscience and Remote Sensing, 62, 1-18. doi: 10.1109/TGRS.2023.3339337
  15. (2014). Weighted-rxd and linear filter-based rxd: Improving background statistics estimation for anomaly detection in hyperspectral imagery. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 7(6), 2351–2366. doi: 10.1109/JSTARS.2014.2302446
  16. (2024). Two spectral–spatial implicit neural representations for arbitrary-resolution hyperspectral pansharpening. IEEE Transactions on Geoscience and Remote Sensing, 62, 1-21. doi: 10.1109/TGRS.2024.3380067
  17. (2023). Multiscale spatial-spectral transformer network for hyperspectral and multispectral image fusion. Information Fusion, 96, 117–129. doi: 10.1016/j.inffus.2023.03.011
  18. (2023). Model-guided deep unfolded fusion network with nonlocal spatial-spectral priors for hyperspectral image super-resolution. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 16, 4607-4625. doi: 10.1109/JSTARS.2023.3272370
  19. (2000, January 4). Process for enhancing the spatial resolution of multispectral imagery using pansharpening. Google Patents. (US Patent 6,011,875)
  20. (2018). 2018 ieee GRSS data fusion contest: Multimodal land use classification [technical committees]. IEEE Geoscience and Remote Sensing Magazine, 6(1), 52–54. doi: 10.1109/MGRS.2018.2798161
  21. (2015). Hyperspectral pansharpening: A review. IEEE Geoscience and Remote Sensing Magazine, 3(3), 27–46. doi: 10.1109/MGRS.2015.2440094
  22. (2024). Low-rank tensor function representation for multi-dimensional data recovery. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(5), 3351-3369. doi: 10.1109/TPAMI.2023.3341688
  23. (2021). NeRF: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1), 99–106. doi: 10.1145/3503250
  24. (2022). Deep SURE for unsupervised remote sensing image fusion. IEEE Transactions on Geoscience and Remote Sensing, 60, 1–13. doi: 10.1109/TGRS.2022.3215902
  25. (1999). Full rank factorization of matrices. Mathematics Magazine, 72(3), 193–201. doi: 10.2307/2690882
  26. (2022). Beyond periodicity: Towards a unifying framework for activations in coordinate-mlps. In European Conference on Computer Vision (ECCV) (Vol. 13693, pp. 142–158). doi: 10.1007/978-3-031-19827-4_9
  27. (2024). SPDER: Semiperiodic damping-enabled object representation. In International Conference on Learning Representations (ICLR).
  28. (2014). A convex formulation for hyperspectral image super-resolution via subspace-based regularization. IEEE Transactions on Geoscience and Remote Sensing, 53(6), 3373–3388. doi: 10.1109/TGRS.2014.2375320
  29. (2020). Implicit neural representations with periodic activation functions. In Advances in Neural Information Processing Systems (NeurIPS) (Vol. 33, pp. 7462–7473).
  30. (2021). Deep maximum a posterior estimator for video denoising. International Journal of Computer Vision, 129(10), 2827–2845. doi: 10.1007/S11263-021-01510-7
  31. (2021). Neural geometric level of detail: Real-time rendering with implicit 3D shapes. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 11358–11367). doi: 10.1109/CVPR46437.2021.01120
  32. (2014). A critical comparison among pansharpening algorithms. IEEE Transactions on Geoscience and Remote Sensing, 53(5), 2565–2586. doi: 10.1109/TGRS.2014.2361734
  33. (2023). Panchromatic and hyperspectral image fusion: Outcome of the 2022 WHISPERS hyperspectral pansharpening challenge. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 16, 166–179. doi: 10.1109/JSTARS.2022.3220974
  34. (2020). Hyperspectral and multispectral image fusion via nonlocal low-rank tensor decomposition and spectral unmixing. IEEE Transactions on Geoscience and Remote Sensing, 58(11), 7654–7671. doi: 10.1109/TGRS.2020.2983063
  35. (2024). Nonnegative matrix functional factorization for hyperspectral unmixing with nonuniform spectral sampling. IEEE Transactions on Geoscience and Remote Sensing, 62, 1-13. doi: 10.1109/TGRS.2023.3347414
  36. (2024). A general paradigm with detail-preserving conditional invertible network for image fusion. International Journal of Computer Vision, 132(4), 1029–1054. doi: 10.1007/S11263-023-01924-5
  37. (2023). SS-INR: Spatial-spectral implicit neural representation network for hyperspectral and multispectral image fusion. IEEE Transactions on Geoscience and Remote Sensing, 61, 1-14. doi: 10.1109/TGRS.2023.3317413
  38. (2023). MCT-Net: Multi-hierarchical cross transformer for hyperspectral and multispectral image fusion. Knowledge-Based Systems, 264, 110362. doi: 10.1016/j.knosys.2023.110362
  39. (2023). A self-supervised deep denoiser for hyperspectral and multispectral image fusion. IEEE Transactions on Geoscience and Remote Sensing, 61, 1-14. doi: 10.1109/TGRS.2023.3303921
  40. (2024). Unsupervised blind spectral–spatial cross-super-resolution network for HSI and MSI fusion. IEEE Transactions on Geoscience and Remote Sensing, 62, 1-14. doi: 10.1109/TGRS.2024.3362862
  41. (2023). Revisiting implicit neural representations in low-level vision. In International Conference on Learning Representations Workshop (ICLR Workshop).
  42. (2015). Smooth nonnegative matrix and tensor factorizations for robust multi-way data analysis. Signal Processing, 113, 234–249. doi: 10.1016/j.sigpro.2015.02.003
  43. (2012). Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion. IEEE Transactions on Geoscience and Remote Sensing, 50(2), 528–537. doi: 10.1109/TGRS.2011.2161320
  44. (2022). Implicit neural representation learning for hyperspectral image super-resolution. IEEE Transactions on Geoscience and Remote Sensing, 61, 1–12. doi: 10.1109/TGRS.2022.3230204
  45. (2018). Exploiting clustering manifold structure for hyperspectral imagery super-resolution. IEEE Transactions on Image Processing, 27(12), 5969–5982. doi: 10.1109/TIP.2018.2862629
  46. (2020). Coupled convolutional neural network with adaptive response function learning for unsupervised hyperspectral super resolution. IEEE Transactions on Geoscience and Remote Sensing, 59(3), 2487–2502. doi: 10.1109/TGRS.2020.3006534
  47. (2018). Fast hyperspectral image denoising and inpainting based on low-rank and sparse representations. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 11(3), 730–742. doi: 10.1109/JSTARS.2018.2796570
  48. (2021). FastHyMix: Fast and parameter-free hyperspectral image mixed noise removal. IEEE Transactions on Neural Networks and Learning Systems, 34(8), 4702–4716. doi: 10.1109/TNNLS.2021.3112577

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets