Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Multispectral Stereo-Image Fusion for 3D Hyperspectral Scene Reconstruction (2401.09428v1)

Published 15 Dec 2023 in eess.IV and cs.CV

Abstract: Spectral imaging enables the analysis of optical material properties that are invisible to the human eye. Different spectral capturing setups, e.g., based on filter-wheel, push-broom, line-scanning, or mosaic cameras, have been introduced in the last years to support a wide range of applications in agriculture, medicine, and industrial surveillance. However, these systems often suffer from different disadvantages, such as lack of real-time capability, limited spectral coverage or low spatial resolution. To address these drawbacks, we present a novel approach combining two calibrated multispectral real-time capable snapshot cameras, covering different spectral ranges, into a stereo-system. Therefore, a hyperspectral data-cube can be continuously captured. The combined use of different multispectral snapshot cameras enables both 3D reconstruction and spectral analysis. Both captured images are demosaicked avoiding spatial resolution loss. We fuse the spectral data from one camera into the other to receive a spatially and spectrally high resolution video stream. Experiments demonstrate the feasibility of this approach and the system is investigated with regard to its applicability for surgical assistance monitoring.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (59)
  1. Sparse recovery of hyperspectral signal from natural rgb images. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part VII 14, pages 19–34. Springer.
  2. Ntire 2020 challenge on spectral reconstruction from an rgb image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops.
  3. Ntire 2022 spectral recovery challenge and data set. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 862–880.
  4. Ntire 2022 spectral demosaicing challenge and data set. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 881–895.
  5. Optical properties of the subcutaneous adipose tissue in the spectral range 400–2500 nm. Optics and spectroscopy, 99:836–842.
  6. Bayer, B. (1976). Color imaging array us patent 3 971 065.
  7. Data-fusion for multiplatform characterization of an italian craft beer aimed at its authentication. Analytica chimica acta, 820:23–31.
  8. Data fusion methodologies for food and beverage authentication and quality assessment–a review. Analytica Chimica Acta, 891:1–14.
  9. Multi-spectral material classification in landscape scenes using commodity hardware. In Computer Analysis of Images and Patterns: 15th International Conference, CAIP 2013, York, UK, August 27-29, 2013, Proceedings, Part II 15, pages 209–216. Springer.
  10. A color filter array based multispectral camera. In 12. Workshop Farbbildverarbeitung, pages 5–6. Ilmenau.
  11. A naturalistic open source movie for optical flow evaluation. In A. Fitzgibbon et al. (Eds.), editor, European Conf. on Computer Vision (ECCV), Part IV, LNCS 7577, pages 611–625. Springer-Verlag.
  12. Surgical spectral imaging. Medical image analysis, 63:101699.
  13. Intraoperative colon perfusion assessment using multispectral imaging. Biomedical Optics Express, 12(12):7556–7567.
  14. Advances in multi-sensor data fusion: Algorithms and applications. Sensors, 9(10):7771–7784.
  15. Intraoperative hyperspectral label-free imaging: from system design to first-in-patient translation. Journal of Physics D: Applied Physics, 54(29):294003.
  16. Eisert, P. (2002). Model-based camera calibration using analysis by synthesis techniques. In VMV, pages 307–314.
  17. Evaluation of different techniques for data fusion of lc/ms and 1h-nmr. Chemometrics and Intelligent Laboratory Systems, 85(1):102–109.
  18. Camera array for multi-spectral imaging. IEEE Transactions on Image Processing, 29:9234–9249.
  19. Multispectral single chip reconstruction using dnns with application to open neurosurgery. Current Directions in Biomedical Engineering, 7(2):37–40.
  20. Multispectral and hyperspectral image fusion based on regularized coupled non-negative block-term tensor decomposition. Remote Sensing, 14(21):5306.
  21. Multispectral digital camera employing both visible light and non-visible light sensing on a single image sensor. US Patent 7,460,160.
  22. Challenges in automatic sorting of construction and demolition waste by hyperspectral imaging. In Advanced Environmental, Chemical, and Biological Sensing Technologies XIII, volume 9862, pages 73–82. SPIE.
  23. High resolution multi-scale raft (robust vision challenge 2022). arXiv preprint arXiv:2210.16900.
  24. Hyperspectral technology in vegetation analysis. Progress in Agricultural Engineering Sciences, 2(1):95–117.
  25. Stereo acquisition with a filter wheel multispectral camera on a goniometric measuring setup. In 20. Workshop Farbbildverarbeitung, pages 89–96. Zentrum für Bild- und Signalverarbeitung e.V.
  26. Multispectral filter arrays: Recent advances and practical implementation. Sensors, 14(11):21626–21659.
  27. Medical hyperspectral imaging: a review. Journal of biomedical optics, 19(1):010901–010901.
  28. Lightfield hyperspectral imaging in neuro-oncology surgery: an ideal 0 and 1 study. Frontiers in Neuroscience, 17.
  29. Regression models with process variables and parallel blocks of raw material measurements. Journal of Chemometrics: A Journal of the Chemometrics Society, 22(8):443–456.
  30. A reconfigurable camera add-on for high dynamic range, multispectral, polarization, and light-field imaging. ACM Transactions on Graphics, 32(4):47–1.
  31. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4040–4048.
  32. Object scene flow for autonomous vehicles. In Conference on Computer Vision and Pattern Recognition (CVPR).
  33. Comparison of different spectral cameras for image-guided organ transplantation. Journal of Biomedical Optics, 26(7):076007.
  34. Multispectral Demosaicing Using Pseudo-Panchromatic Image. IEEE Transactions on Computational Imaging, 3(4):982–995.
  35. Plant disease detection using hyperspectral imaging. In 2017 International Conference on Digital Image Computing: Techniques and Applications (DICTA), pages 1–8. IEEE.
  36. Dictionary based hyperspectral image reconstruction captured with cs-musi. In 2018 9th Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing (WHISPERS), pages 1–5. IEEE.
  37. Multisensor image fusion in remote sensing: concepts, methods and applications. International journal of remote sensing, 19(5):823–854.
  38. Measurement of the optical properties of fruits and vegetables using spatially resolved hyperspectral diffuse reflectance imaging technique. Postharvest Biology and Technology, 49(3):355–365.
  39. Kalibrierung stereoskopischer systeme für medizinische messaufgaben. In Proceedings of the 16th Annual Conference of the German Society for Computer and Robotic Assisted Surgery (CURAC, pages 161–163.
  40. Snapshot spectropolarimetric imaging using a pair of filter array cameras. Optical Engineering, 61(4):043104–043104.
  41. Hyperspectral remote sensing of urban areas: an overview of techniques and applications. Research Journal of Applied Sciences, Engineering and Technology, 4(11):1557–1565.
  42. Intraoperative multispectral and hyperspectral label-free imaging: A systematic review of in vivo clinical studies. Journal of biophotonics, 12(9):e201800455.
  43. One-shot multispectral color imaging with a stereo camera. In Digital Photography VII, volume 7876, pages 82–92. SPIE.
  44. Spectral organ fingerprints for machine learning-based intraoperative tissue classification with hyperspectral imaging in a porcine model. Scientific Reports, 12(1):11028.
  45. Dual snapshot hyperspectral imaging system for 41-band spectral analysis and stereo reconstruction. In Advances in Visual Computing, pages 3–13, Cham. Springer International Publishing.
  46. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 402–419. Springer.
  47. A deep survey in the applications of demosaicking. In 2021 3rd International Academic Exchange Conference on Science and Technology Innovation (IAECST), pages 596–602. IEEE.
  48. Hyperspectral demosaicing of snapshot camera images using deep learning. In Pattern Recognition: 44th DAGM German Conference, DAGM GCPR 2022, Konstanz, Germany, September 27–30, 2022, Proceedings, pages 198–212. Springer.
  49. 3d hyperspectral light-field imaging: a first intraoperative implementation. Current Directions in Biomedical Engineering, 9(1):611–614.
  50. Validation of two techniques for intraoperative hyperspectral human tissue determination. Journal of Medical Imaging, 7(6):065001.
  51. Interactive and multimodal-based augmented reality for remote assistance using a digital surgical microscope. In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pages 1477–1484. IEEE.
  52. Telepresence for surgical assistance and training using extended reality during and after pandemic periods. Journal of Telemedicine and Telecare, page 1357633X231166226.
  53. Surgical guidance for removal of cholesteatoma using a multispectral 3d-endoscope. Sensors, 20(18):5334.
  54. From multispectral-stereo to intraoperative hyperspectral imaging: a feasibility study. Current Directions in Biomedical Engineering, 9(1):311–314.
  55. Intraoperative hyperspectral determination of human tissue properties. Journal of biomedical optics, 23(9):091409.
  56. An introduction to hyperspectral imaging and its application for security, surveillance and target acquisition. The Imaging Science Journal, 58(5):241–253.
  57. Zhang, J. (2010). Multi-source remote sensing data fusion: status and trends. International Journal of Image and Data Fusion, 1(1):5–24.
  58. Dip: Deep inverse patchmatch for high-resolution optical flow. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8925–8934.
  59. 3d reconstruction from hyperspectral images. In IEEE winter conference on applications of computer vision, pages 318–325. IEEE.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now