Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
158 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

SpectralWaste Dataset: Multimodal Data for Waste Sorting Automation (2403.18033v1)

Published 26 Mar 2024 in cs.CV and cs.RO

Abstract: The increase in non-biodegradable waste is a worldwide concern. Recycling facilities play a crucial role, but their automation is hindered by the complex characteristics of waste recycling lines like clutter or object deformation. In addition, the lack of publicly available labeled data for these environments makes developing robust perception systems challenging. Our work explores the benefits of multimodal perception for object segmentation in real waste management scenarios. First, we present SpectralWaste, the first dataset collected from an operational plastic waste sorting facility that provides synchronized hyperspectral and conventional RGB images. This dataset contains labels for several categories of objects that commonly appear in sorting plants and need to be detected and separated from the main trash flow for several reasons, such as security in the management line or reuse. Additionally, we propose a pipeline employing different object segmentation architectures and evaluate the alternatives on our dataset, conducting an extensive analysis for both multimodal and unimodal alternatives. Our evaluation pays special attention to efficiency and suitability for real-time processing and demonstrates how HSI can bring a boost to RGB-only perception in these realistic industrial settings without much computational overhead.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (35)
  1. M. Sukno and I. Palunko, “Hand-crafted features for floating plastic detection,” in International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2022, pp. 3378–3383.
  2. T. Linder, N. Vaskevicius, R. Schirmer, and K. O. Arras, “Cross-modal analysis of human detection for robotics: An industrial case study,” in International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2021, pp. 971–978.
  3. J. Lee, J. Hur, I. Hwang, and Y. M. Kim, “MasKGrasp: Mask-based grasping for scenes with multiple general real-world objects,” in International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2022, pp. 3137–3144.
  4. Y. Lu, Y. Fan, B. Deng, F. Liu, Y. Li, and S. Wang, “VL-Grasp: a 6-Dof interactive grasp policy for language-oriented objects in cluttered indoor scenes,” in International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2023, pp. 976–983.
  5. M. L. Henriksen, C. B. Karlsen, P. Klarskov, and M. Hinge, “Plastic classification via in-line hyperspectral camera analysis and unsupervised machine learning,” Vibrational Spectroscopy, vol. 118, 2022.
  6. N. Habili, E. Kwan, W. Li, C. Webers, J. Oorloff, M. A. Armin, and L. Petersson, “A hyperspectral and RGB dataset for building façade segmentation,” in European Conference on Computer Vision (ECCV).   Springer, 2022, pp. 258–267.
  7. S. Kodgule, A. Candela, and D. Wettergreen, “Non-myopic planetary exploration combining in situ and remote measurements,” in International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2019, pp. 536–543.
  8. Z. Liu, X. Wang, Y. Zhong, M. Shu, and C. Sun, “SiamHYPER: Learning a hyperspectral object tracker from an RGB-based tracker,” IEEE Transactions on Image Processing, vol. 31, pp. 7116–7129, 2022.
  9. D. Bashkirova, M. Abdelfattah, Z. Zhu, J. Akl, F. Alladkani, P. Hu, V. Ablavsky, B. Calli, S. A. Bargal, and K. Saenko, “ZeroWaste dataset: Towards deformable object segmentation in cluttered scenes,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 21 147–21 157.
  10. M. Yang and G. Thung, “Classification of trash for recyclability status,” CS229 project report, vol. 2016, no. 1, p. 3, 2016.
  11. P. F. Proença and P. Simões, “TACO: Trash annotations in context for litter detection,” 2020, arXiv preprint arXiv:2003.06975.
  12. Y. Cheng, J. Zhu, M. Jiang, J. Fu, C. Pang, P. Wang, K. Sankaran, O. Onabola, Y. Liu, D. Liu, and Y. Bengio, “FloW: A dataset and benchmark for floating waste detection in inland waters,” in International Conference on Computer Vision (ICCV), 2021, pp. 10 933–10 942.
  13. J. Kim, M.-H. Jeon, S. Jung, W. Yang, M. Jung, J. Shin, and A. Kim, “Transpose: Large-scale multispectral dataset for transparent object,” The International Journal of Robotics Research, 2023.
  14. D. Bashkirova, S. Mishra, D. Lteif, P. Teterwak, D. Kim, F. Alladkani, J. Akl, B. Calli, S. A. Bargal, K. Saenko, D. Kim, M. Seo, Y. Jeon, D.-G. Choi, S. Ettedgui, R. Giryes, S. Abu-Hussein, B. Xie, and S. Li, “VisDA 2022 challenge: Domain adaptation for industrial waste sorting,” in NeurIPS 2022 Competition Track.   PMLR, 2022, pp. 104–118.
  15. F. A. Kruse, A. B. Lefkoff, J. W. Boardman, K. B. Heidebrecht, A. T. Shapiro, P. J. Barloon, and A. F. H. Goetz, “The spectral image processing system (SIPS)—interactive visualization and analysis of imaging spectrometer data,” Remote Sensing of Environment, vol. 44, no. 2-3, pp. 145–163, 1993.
  16. N. Hanson, W. Lewis, K. Puthuveetil, D. Furline, A. Padmanabha, T. Padir, and Z. Erickson, “SLURP! spectroscopy of liquids using robot pre-touch sensing,” in International Conference on Robotics and Automation (ICRA).   IEEE, 2023, pp. 3786–3792.
  17. S. Seidlitz, J. Sellner, J. Odenthal, B. Özdemir, A. Studier-Fischer, S. Knödler, L. Ayala, T. J. Adler, H. G. Kenngott, M. Tizabi, et al., “Robust deep learning-based semantic organ segmentation in hyperspectral images,” Medical Image Analysis, vol. 80, 2022.
  18. A. Wendel and J. Underwood, “Self-supervised weed detection in vegetable crops using ground based hyperspectral imaging,” in International Conference on Robotics and Automation (ICRA).   IEEE, 2016, pp. 5128–5135.
  19. J. Nalepa, M. Myller, and M. Kawulok, “Validating hyperspectral image segmentation,” IEEE Geoscience and Remote Sensing Letters, vol. 16, no. 8, pp. 1264–1268, 2019.
  20. R. Grewal, S. S. Kasana, and G. Kasana, “Hyperspectral image segmentation: A comprehensive survey,” Multimedia Tools and Applications, vol. 82, pp. 20 819–10 872, 2023.
  21. M. Shiddiq, D. S. Arief, Zulfansyah, K. Fatimah, D. Wahyudi, D. A. Mahmudah, D. K. E. Putri, I. R. Husein, and S. A. Ningsih, “Plastic and organic waste identification using multispectral imaging,” Materials Today: Proceedings, vol. 87, pp. 338–344, 2023.
  22. A. C. Karaca, A. Ertürk, M. K. Güllü, M. Elmas, and S. Ertürk, “Automatic waste sorting using shortwave infrared hyperspectral imaging system,” in Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing (WHISPERS).   IEEE, 2013.
  23. Z. Chen, T. R. Scott, S. Bearman, H. Anand, D. Keating, C. Scott, J. R. Arrowsmith, and J. Das, “Geomorphological analysis using unpiloted aircraft systems, structure from motion, and deep learning,” in International Conference on Intelligent Robots and Systems (IROS).   IEEE, 2020, pp. 1276–1283.
  24. A. Valada, J. Vertens, A. Dhall, and W. Burgard, “AdapNet: Adaptive semantic segmentation in adverse environmental conditions,” in International Conference on Robotics and Automation (ICRA).   IEEE, 2017, pp. 4644–4651.
  25. H. Qin, W. Zhou, Y. Yao, and W. Wang, “Individual tree segmentation and tree species classification in subtropical broadleaf forests using UAV-based LiDAR, hyperspectral, and ultrahigh-resolution RGB data,” Remote Sensing of Environment, vol. 280, 2022.
  26. S. D. Fabiyi, H. Vu, C. Tachtatzis, P. Murray, D. Harle, T. K. Dao, I. Andonovic, J. Ren, and S. Marshall, “Varietal classification of rice seeds using RGB and hyperspectral images,” IEEE Access, vol. 8, pp. 22 493–22 505, 2020.
  27. A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, et al., “Segment anything,” in International Conference on Computer Vision (ICCV), 2023, pp. 4015–4026.
  28. I. Alonso, L. Riazuelo, and A. C. Murillo, “MiniNet: An efficient semantic segmentation ConvNet for real-time robotic applications,” IEEE Transactions on Robotics (T-RO), vol. 36, no. 4, pp. 1340–1347, 2020.
  29. E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo, “SegFormer: Simple and efficient design for semantic segmentation with transformers,” Advances in Neural Information Processing Systems (NeurIPS), vol. 34, pp. 12 077–12 090, 2021.
  30. J. Zhang, H. Liu, K. Yang, X. Hu, R. Liu, and R. Stiefelhagen, “CMX: Cross-modal fusion for RGB-X semantic segmentation with transformers,” IEEE Transactions on Intelligent Transportation Systems (TITS), vol. 24, pp. 14 679–14 694, 2023.
  31. Z. Wang, F. Colonnier, J. Zheng, J. Acharya, W. Jiang, and K. Huang, “TIRDet: Mono-modality thermal infrared object detection based on prior thermal-to-visible translation,” in ACM International Conference on Multimedia.   Association for Computing Machinery, 2023, p. 2663–2672.
  32. C. Yan, H. Zhang, X. Li, Y. Yang, and D. Yuan, “Cross-modality complementary information fusion for multispectral pedestrian detection,” Neural Computing and Applications, vol. 35, no. 14, pp. 10 361–10 386, 2023.
  33. K. Chen, J. Liu, and H. Zhang, “IGT: Illumination-guided RGB-T object detection with transformers,” Knowledge-Based Systems, vol. 268, 2023.
  34. J. Behmann, A.-K. Mahlein, S. Paulus, H. Kuhlmann, E.-C. Oerke, and L. Plümer, “Calibration of hyperspectral close-range pushbroom cameras for plant phenotyping,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 106, pp. 172–182, 2015.
  35. W. Jiang, E. Trulls, J. Hosang, A. Tagliasacchi, and K. M. Yi, “COTR: Correspondence transformer for matching across images,” in International Conference on Computer Vision (ICCV), 2021, pp. 6207–6217.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now