Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Image-based Deep Learning for Smart Digital Twins: a Review (2401.02523v1)

Published 4 Jan 2024 in cs.CV, cs.AI, cs.LG, cs.SY, and eess.SY

Abstract: Smart Digital twins (SDTs) are being increasingly used to virtually replicate and predict the behaviors of complex physical systems through continual data assimilation enabling the optimization of the performance of these systems by controlling the actions of systems. Recently, deep learning (DL) models have significantly enhanced the capabilities of SDTs, particularly for tasks such as predictive maintenance, anomaly detection, and optimization. In many domains, including medicine, engineering, and education, SDTs use image data (image-based SDTs) to observe and learn system behaviors and control their behaviors. This paper focuses on various approaches and associated challenges in developing image-based SDTs by continually assimilating image data from physical systems. The paper also discusses the challenges involved in designing and implementing DL models for SDTs, including data acquisition, processing, and interpretation. In addition, insights into the future directions and opportunities for developing new image-based DL approaches to develop robust SDTs are provided. This includes the potential for using generative models for data augmentation, developing multi-modal DL models, and exploring the integration of DL with other technologies, including 5G, edge computing, and IoT. In this paper, we describe the image-based SDTs, which enable broader adoption of the digital twin DT paradigms across a broad spectrum of areas and the development of new methods to improve the abilities of SDTs in replicating, predicting, and optimizing the behavior of complex systems.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (48)
  1. R. Rosen, G. Von Wichert, G. Lo, and K. D. Bettenhausen, “About the importance of autonomy and digital twins for the future of manufacturing,” Ifac-Papersonline, vol. 48, no. 3, pp. 567–572, 2015.
  2. A. El Saddik, “Digital twins: The convergence of multimedia technologies,” IEEE multimedia, vol. 25, no. 2, pp. 87–92, 2018.
  3. S. Yi, S. Liu, X. Xu, X. V. Wang, S. Yan, and L. Wang, “A vision-based human-robot collaborative system for digital twin,” Procedia CIRP, vol. 107, pp. 552–557, 2022, leading manufacturing systems transformation – Proceedings of the 55th CIRP Conference on Manufacturing Systems 2022. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S2212827122003080
  4. A. Mukhopadhyay, G. Rajshekar Reddy, I. Mukherjee, G. Kumar Gopa, A. Pena-Rios, and P. Biswas, “Generating synthetic data for deep learning using vr digital twin,” in Proceedings of the 2021 5th International Conference on Cloud and Big Data Computing, ser. ICCBDC ’21.   New York, NY, USA: Association for Computing Machinery, 2021, p. 52–56. [Online]. Available: https://doi-org.leo.lib.unomaha.edu/10.1145/3481646.3481655
  5. S. Jeong, H. Kim, J. Lee, S.-Y. Park, and S.-H. Ahn, “Digital twin-based cutting tool breakage detection model using synthetic depth map and deep learning,” in Proc. of the 9th Intl. Conf. of Asian Society for Precision Engg. and Nanotechnology (ASPEN 2022) 15–18 November 2022, Singapore, N. M. L. Sharon and A. S. Kumar, Eds.   CRC Press, 2022.
  6. J. Vachálek, L. Bartalskỳ, O. Rovnỳ, D. Šišmišová, M. Morháč, and M. Lokšík, “The digital twin of an industrial production line within the industry 4.0 concept,” in 2017 21st international conference on process control (PC).   IEEE, 2017, pp. 258–262.
  7. A. Mukhopadhyay, G. R. Reddy, K. S. Saluja, S. Ghosh, A. Peña-Rios, G. Gopal, and P. Biswas, “Virtual-reality-based digital twin of office spaces with social distance measurement feature,” Virtual Reality & Intelligent Hardware, vol. 4, no. 1, pp. 55–75, 2022.
  8. R. Ferdousi, F. Laamarti, C. Yang, and A. El Saddik, “Railtwin: A digital twin framework for railway,” in 2022 IEEE 18th International Conference on Automation Science and Engineering (CASE).   IEEE Press, 2022, p. 1767–1772. [Online]. Available: https://doi.org/10.1109/CASE49997.2022.9926529
  9. A. Mukhopadhyay, G. S. R. Reddy, S. Ghosh, M. L R D, and P. Biswas, “Validating social distancing through deep learning and vr-based digital twins,” in Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology, ser. VRST ’21.   New York, NY, USA: Association for Computing Machinery, 2021. [Online]. Available: https://doi-org.leo.lib.unomaha.edu/10.1145/3489849.3489959
  10. Y. Wu, K. Zhang, and Y. Zhang, “Digital twin networks: A survey,” IEEE Internet of Things Journal, vol. 8, no. 18, pp. 13 789–13 804, 2021.
  11. O. E. Marai, T. Taleb, and J. Song, “Roads infrastructure digital twin: A step toward smarter cities realization,” IEEE Network, vol. 35, no. 2, pp. 136–143, 2021.
  12. T. Y. Melesse, M. Bollo, V. D. Pasquale, F. Centro, and S. Riemma, “Machine learning-based digital twin for monitoring fruit quality evolution,” Procedia Comput. Sci., vol. 200, pp. 13–20, 2022.
  13. B. Subramanian, J. Kim, M. Maray, and A. Paul, “Digital twin model: A real-time emotion recognition system for personalized healthcare,” IEEE Access, vol. 10, pp. 81 155–81 165, 2022.
  14. Y. Wu, H. Cao, G. Yang, T. Lu, and S. Wan, “Digital twin of intelligent small surface defect detection with cyber-manufacturing systems,” ACM Trans. Internet Technol., nov 2022, just Accepted. [Online]. Available: https://doi.org/10.1145/3571734
  15. Unity Technologies, “Unity Perception package,” https://github.com/Unity-Technologies/com.unity.perception, 2020.
  16. S. Shah, D. Dey, C. Lovett, and A. Kapoor, “Airsim: High-fidelity visual and physical simulation for autonomous vehicles,” in Field and Service Robotics, 2017. [Online]. Available: https://arxiv.org/abs/1705.05065
  17. P. Martinez-Gonzalez, S. Oprea, A. Garcia-Garcia, A. Jover-Alvarez, S. Orts-Escolano, and J. Garcia-Rodriguez, “Unrealrox: An extremely photorealistic virtual reality environment for robotics simulations and synthetic data generation,” Virtual Reality, pp. 1–18, 2019, [Online; accessed 6-April-2023].
  18. R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2014, pp. 580–587.
  19. K. Alexopoulos, N. Nikolakis, and G. Chryssolouris, “Digital twin-driven supervised machine learning for the development of artificial intelligence applications in manufacturing,” International Journal of Computer Integrated Manufacturing, vol. 33, no. 5, pp. 429–439, Apr. 2020. [Online]. Available: https://doi.org/10.1080/0951192x.2020.1747642
  20. J. Lee, M. Azamfar, J. Singh, and S. Siahpour, “Integration of digital twin and deep learning in cyber-physical systems: towards smart manufacturing,” IET Collaborative Intelligent Manufacturing, vol. 2, no. 1, pp. 34–36, Mar. 2020. [Online]. Available: https://doi.org/10.1049/iet-cim.2020.0009
  21. X. Zhou, K. Sun, J. Wang, J. Zhao, C. Feng, Y. Yang, and W. Zhou, “Computer vision enabled building digital twin using building information model,” IEEE Transactions on Industrial Informatics, vol. 19, no. 3, pp. 2684–2692, 2023.
  22. J. Masci, U. Meier, D. Ciresan, J. Schmidhuber, and G. Fricout, “Steel defect classification with max-pooling convolutional neural networks,” in The 2012 International Joint Conference on Neural Networks (IJCNN), 2012, pp. 1–6.
  23. A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” Communications of the ACM, vol. 60, no. 6, pp. 84–90, 2017.
  24. Z. Zou, K. Chen, Z. Shi, Y. Guo, and J. Ye, “Object detection in 20 years: A survey,” Proceedings of the IEEE, 2023.
  25. J. Huang, V. Rathod, C. Sun, M. Zhu, A. Korattikara, A. Fathi, I. Fischer, Z. Wojna, Y. Song, S. Guadarrama et al., “Speed/accuracy trade-offs for modern convolutional object detectors,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 7310–7311.
  26. W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, “SSD: Single shot MultiBox detector,” in Computer Vision – ECCV 2016.   Springer International Publishing, 2016, pp. 21–37. [Online]. Available: https://doi.org/10.1007/978-3-319-46448-0_2
  27. R. Girshick, “Fast r-cnn,” in Proceedings of the IEEE international conference on computer vision, 2015, pp. 1440–1448.
  28. S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” Advances in neural information processing systems, vol. 28, 2015.
  29. Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel, “Backpropagation applied to handwritten zip code recognition,” Neural Computation, vol. 1, no. 4, pp. 541–551, 1989.
  30. Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998.
  31. A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification with deep convolutional neural networks,” Communications of the ACM, vol. 60, no. 6, pp. 84–90, May 2017. [Online]. Available: https://doi.org/10.1145/3065386
  32. K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
  33. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 1–9.
  34. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  35. M. Pak and S. Kim, “A review of deep learning in image recognition,” in 2017 4th International Conference on Computer Applications and Information Processing Technology (CAIPT).   IEEE, Aug. 2017. [Online]. Available: https://doi.org/10.1109/caipt.2017.8320684
  36. G. Habib and S. Qureshi, “Optimization and acceleration of convolutional neural networks: A survey,” Journal of King Saud University - Computer and Information Sciences, vol. 34, no. 7, pp. 4244–4268, 2022. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1319157820304845
  37. R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” 2014.
  38. ——, “Faster r-cnn: Towards real-time object detection with region proposal networks,” 2015. [Online]. Available: https://arxiv.org/abs/1506.01497
  39. J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” 2015. [Online]. Available: https://arxiv.org/abs/1506.02640
  40. M. Pengnoo, M. T. Barros, L. Wuttisittikulkij, B. Butler, A. Davy, and S. Balasubramaniam, “Digital twin for metasurface reflector management in 6g terahertz communications,” IEEE Access, vol. 8, pp. 114 580–114 596, 2020.
  41. G. Jocher, “Yolov5,” 2020. [Online]. Available: https://github.com/ultralytics/yolov5
  42. M. Sozzi, S. Cantalamessa, A. Cogato, A. Kayad, and F. Marinello, “Automatic bunch detection in white grape varieties using yolov3, yolov4, and yolov5 deep learning algorithms,” Agronomy, vol. 12, no. 2, p. 319, 2022.
  43. A. Mukhopadhyay, I. Mukherjee, and P. Biswas, “Comparing CNNs for non-conventional traffic participants,” in Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications: Adjunct Proceedings.   ACM, Sep. 2019. [Online]. Available: https://doi.org/10.1145/3349263.3351336
  44. C. Lugaresi, J. Tang, H. Nash, C. McClanahan, E. Uboweja, M. Hays, F. Zhang, C.-L. Chang, M. G. Yong, J. Lee et al., “Mediapipe: A framework for building perception pipelines,” arXiv preprint arXiv:1906.08172, 2019.
  45. Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 10 012–10 022.
  46. T. Wang, J. Li, Y. Deng, C. Wang, H. Snoussi, and F. Tao, “Digital twin for human-machine interaction with convolutional neural network,” International Journal of Computer Integrated Manufacturing, vol. 34, no. 7-8, pp. 888–897, May 2021. [Online]. Available: https://doi.org/10.1080/0951192x.2021.1925966
  47. S. Ji, W. Xu, M. Yang, and K. Yu, “3D convolutional neural networks for human action recognition,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 1, pp. 221–231, Jan. 2013.
  48. J. Redmon and A. Farhadi, “Yolov3: An incremental improvement,” arXiv preprint arXiv:1804.02767, 2018.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now