Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Pretrain, Prompt, and Transfer: Evolving Digital Twins for Time-to-Event Analysis in Cyber-physical Systems (2310.00032v3)

Published 29 Sep 2023 in cs.SE

Abstract: Cyber-Physical Systems (CPSs), e.g., elevator systems and autonomous driving systems, are progressively permeating our everyday lives. To ensure their safety, various analyses need to be conducted, such as anomaly detection and time-to-event analysis (the focus of this paper). Recently, it has been widely accepted that digital Twins (DTs) can serve as an efficient method to aid in the development, maintenance, and safe and secure operation of CPSs. However, CPSs frequently evolve, e.g., with new or updated functionalities, which demand their corresponding DTs be co-evolved, i.e., in synchronization with the CPSs. To that end, we propose a novel method, named PPT, utilizing an uncertainty-aware transfer learning for DT evolution. Specifically, we first pretrain PPT with a pretraining dataset to acquire generic knowledge about the CPSs, followed by adapting it to a specific CPS with the help of prompt tuning. Results highlight that PPT is effective in time-to-event analysis in both elevator and ADSs case studies, on average, outperforming a baseline method by 7.31 and 12.58 in terms of Huber loss, respectively. The experiment results also affirm the effectiveness of transfer learning, prompt tuning and uncertainty quantification in terms of reducing Huber loss by at least 21.32, 3.14 and 4.08, respectively, in both case studies.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (51)
  1. A. A. Musa, A. Hussaini, W. Liao, F. Liang, and W. Yu, “Deep Neural Networks for Spatial-Temporal Cyber-Physical Systems: A Survey,” Future Internet, vol. 15, no. 6, p. 199, May 2023. [Online]. Available: https://www.mdpi.com/1999-5903/15/6/199
  2. E. A. Lee, “CPS foundations,” in Proceedings of the 47th Design Automation Conference, ser. DAC ’10.   New York, NY, USA: Association for Computing Machinery, 2010, pp. 737–742, event-place: Anaheim, California. [Online]. Available: https://doi.org/10.1145/1837274.1837462
  3. K.-C. Li and B. B. Gupta, “Recent Advances in Security, Privacy and Trust for Internet-of-Things (IoT) and Cyber-Physical Systems (CPS),” 2020, publisher: Chapman and Hall/CRC.
  4. M. Eckhart and A. Ekelhart, “Securing Cyber-Physical Systems through Digital Twins,” Ercim News, no. 115, pp. 22–23, 2018.
  5. Q. Xu, S. Ali, and T. Yue, “Digital Twin-based Anomaly Detection in Cyber-physical Systems,” in 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST).   Porto de Galinhas, Brazil: IEEE, Apr. 2021, pp. 205–216. [Online]. Available: https://ieeexplore.ieee.org/document/9438560/
  6. ——, “Digital Twin-based Anomaly Detection with Curriculum Learning in Cyber-physical Systems,” ACM Trans. Softw. Eng. Methodol., vol. 32, no. 5, Jul. 2023, place: New York, NY, USA Publisher: Association for Computing Machinery. [Online]. Available: https://doi.org/10.1145/3582571
  7. Q. Xu, S. Ali, T. Yue, Z. Nedim, and I. Singh, “KDDT: Knowledge Distillation-Empowered Digital Twin for Anomaly Detection,” in Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ser. ESEC/FSE 2023.   New York, NY, USA: Association for Computing Machinery, 2023, pp. 1867–1878. [Online]. Available: https://doi.org/10.1145/3611643.3613879
  8. Q. Xu, S. Ali, T. Yue, and M. Arratibel, “Uncertainty-aware transfer learning to evolve digital twins for industrial elevators,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering.   Singapore Singapore: ACM, Nov. 2022, pp. 1257–1268. [Online]. Available: https://dl.acm.org/doi/10.1145/3540250.3558957
  9. C. V. Nguyen, T. Hassner, M. Seeger, and C. Archambeau, “LEEP: a new measure to evaluate transferability of learned representations,” in Proceedings of the 37th International Conference on Machine Learning, ser. ICML’20.   JMLR.org, 2020.
  10. T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language Models are Few-Shot Learners,” Jul. 2020, arXiv:2005.14165 [cs]. [Online]. Available: http://arxiv.org/abs/2005.14165
  11. T. Yue and S. Ali, “Evolve the Model Universe of a System Universe,” in 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), Sep. 2023, pp. 1726–1731, journal Abbreviation: 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE).
  12. X. Zhang, J. Tao, K. Tan, M. Törngren, J. M. G. Sánchez, M. R. Ramli, X. Tao, M. Gyllenhammar, F. Wotawa, N. Mohan, M. Nica, and H. Felbinger, “Finding Critical Scenarios for Automated Driving Systems: A Systematic Mapping Study,” IEEE Transactions on Software Engineering, vol. 49, no. 3, pp. 991–1026, Mar. 2023. [Online]. Available: https://ieeexplore.ieee.org/document/9763411/
  13. A. El Saddik, “Digital Twins: The Convergence of Multimedia Technologies,” IEEE MultiMedia, vol. 25, no. 2, pp. 87–92, Jun. 2018.
  14. T. Yue, P. Arcaini, and S. Ali, “Understanding Digital Twins for Cyber-Physical Systems: A Conceptual Model,” Physical Systems.
  15. M. Abdar, F. Pourpanah, S. Hussain, D. Rezazadegan, L. Liu, M. Ghavamzadeh, P. Fieguth, X. Cao, A. Khosravi, U. R. Acharya, V. Makarenkov, and S. Nahavandi, “A review of uncertainty quantification in deep learning: Techniques, applications and challenges,” Information Fusion, vol. 76, pp. 243–297, 2021.
  16. L. Lin, H. Bao, and N. Dinh, “Uncertainty quantification and software risk analysis for digital twins in the nearly autonomous management and control systems: A review,” Annals of Nuclear Energy, vol. 160, pp. 108 362–108 362, 2021, publisher: Elsevier Ltd. [Online]. Available: https://doi.org/10.1016/j.anucene.2021.108362
  17. T. Gneiting, F. Balabdaoui, and A. E. Raftery, “Probabilistic forecasts, calibration and sharpness,” Journal of the Royal Statistical Society Series B: Statistical Methodology, vol. 69, no. 2, pp. 243–268, 2007, publisher: Oxford University Press.
  18. G. J. Klir, “Uncertainty and information: foundations of generalized information theory,” Kybernetes, vol. 35, no. 7/8, pp. 1297–1299, 2006, publisher: Emerald Group Publishing Limited.
  19. Y. Yang and M. Loog, “Active learning using uncertainty information,” in 2016 23rd International Conference on Pattern Recognition (ICPR), Dec. 2016, pp. 2646–2651, journal Abbreviation: 2016 23rd International Conference on Pattern Recognition (ICPR).
  20. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” 2017, pp. 5998–6008.
  21. A. Farahani, S. Voghoei, K. Rasheed, and H. R. Arabnia, “A brief review of domain adaptation,” Advances in data science and information engineering: proceedings from ICDATA 2020 and IKE 2020, pp. 877–894, 2021, publisher: Springer.
  22. X. Liu, K. Ji, Y. Fu, W. Tam, Z. Du, Z. Yang, and J. Tang, “P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).   Dublin, Ireland: Association for Computational Linguistics, 2022, pp. 61–68. [Online]. Available: https://aclanthology.org/2022.acl-short.8
  23. P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig, “Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing,” Jul. 2021, arXiv:2107.13586 [cs]. [Online]. Available: http://arxiv.org/abs/2107.13586
  24. L. Han, S. Ali, T. Yue, A. Arrieta, and M. Arratibel, “Uncertainty-Aware Robustness Assessment of Industrial Elevator Systems,” ACM Trans. Softw. Eng. Methodol., vol. 32, no. 4, May 2023, place: New York, NY, USA Publisher: Association for Computing Machinery. [Online]. Available: https://doi.org/10.1145/3576041
  25. C. Lu, T. Yue, and S. Ali, “DeepScenario: An Open Driving Scenario Dataset for Autonomous Driving System Testing,” in 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR), May 2023, pp. 52–56, journal Abbreviation: 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR).
  26. C. Lu, Y. Shi, H. Zhang, M. Zhang, T. Wang, T. Yue, and S. Ali, “Learning Configurations of Operating Environment of Autonomous Vehicles to Maximize their Collisions,” IEEE Transactions on Software Engineering, vol. 49, no. 1, pp. 384–402, 2023.
  27. G. P. Meyer, “An Alternative Probabilistic Interpretation of the Huber Loss,” in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).   Los Alamitos, CA, USA: IEEE Computer Society, Jun. 2021, pp. 5257–5265. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/CVPR46437.2021.00522
  28. A. Arcuri and L. Briand, “A practical guide for using statistical tests to assess randomized algorithms in software engineering,” in Proceedings of the 33rd International Conference on Software Engineering.   Waikiki, Honolulu HI USA: ACM, May 2011, pp. 1–10. [Online]. Available: https://dl.acm.org/doi/10.1145/1985793.1985795
  29. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “PyTorch: An Imperative Style, High-Performance Deep Learning Library,” Dec. 2019, arXiv:1912.01703 [cs, stat]. [Online]. Available: http://arxiv.org/abs/1912.01703
  30. Y. Chung, I. Char, H. Guo, J. Schneider, and W. Neiswanger, “Uncertainty Toolbox: an Open-Source Library for Assessing, Visualizing, and Improving Uncertainty Quantification,” Sep. 2021, arXiv:2109.10254 [cs, stat]. [Online]. Available: http://arxiv.org/abs/2109.10254
  31. A. Humayed, J. Lin, F. Li, and B. Luo, “Cyber-physical systems security—A survey,” IEEE Internet of Things Journal, vol. 4, no. 6, pp. 1802–1831, 2017, publisher: IEEE.
  32. C. Lv, Y. Xing, J. Zhang, X. Na, Y. Li, T. Liu, D. Cao, and F.-Y. Wang, “Levenberg–Marquardt backpropagation training of multilayer neural networks for state estimation of a safety-critical cyber-physical system,” IEEE Transactions on Industrial Informatics, vol. 14, no. 8, pp. 3436–3446, 2017, publisher: IEEE.
  33. C. S. Wickramasinghe, D. L. Marino, K. Amarasinghe, and M. Manic, “Generalization of deep learning for cyber-physical system security: A survey,” Proceedings: IECON 2018 - 44th Annual Conference of the IEEE Industrial Electronics Society, vol. 1, pp. 745–751, 2018, publisher: IEEE.
  34. T. K. Lee, T. W. Wang, W. X. Wu, Y. C. Kuo, S. H. Huang, G. S. Wang, C. Y. Lin, J. J. Chen, and Y. C. Tseng, “Building a V2X Simulation Framework for Future Autonomous Driving,” 2019 20th Asia-Pacific Network Operations and Management Symposium: Management in a Cyber-Physical World, APNOMS 2019, pp. 1–6, 2019, publisher: IEICE.
  35. A. Bécue, Y. Fourastier, I. Praça, A. Savarit, C. Baron, B. Gradussofs, E. Pouille, and C. Thomas, “CyberFactory#1 — Securing the industry 4.0 with cyber-ranges and digital twins,” 2018, pp. 1–4.
  36. M. Eckhart and A. Ekelhart, “Towards security-aware virtual environments for digital twins,” 2018, pp. 61–72.
  37. R. Bitton, T. Gluck, O. Stan, M. Inokuchi, Y. Ohta, Y. Yamada, T. Yagyu, Y. Elovici, and A. Shabtai, “Deriving a Cost-Effective Digital Twin of an ICS to Facilitate Security Evaluation: 23rd European Symposium on Research in Computer Security, ESORICS 2018, Barcelona, Spain, September 3-7, 2018, Proceedings, Part I,” Aug. 2018, pp. 533–554.
  38. V. Damjanovic-Behrendt, “A Digital Twin-based Privacy Enhancement Mechanism for the Automotive Industry,” in 2018 International Conference on Intelligent Systems (IS).   IEEE Press, 2018, pp. 272–279, place: Funchal - Madeira, Portugal. [Online]. Available: https://doi.org/10.1109/IS.2018.8710526
  39. T. Wang, C. Tan, L. Huang, Y. Shi, T. Yue, and Z. Huang, “Simplexity testbed: A model-based digital twin testbed,” Computers in Industry, vol. 145, p. 103804, Feb. 2023. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0166361522002007
  40. L. Duan, D. Xu, and I. W. Tsang, “Learning with augmented features for heterogeneous domain adaptation,” in Proceedings of the 29th International Coference on International Conference on Machine Learning, ser. ICML’12.   Madison, WI, USA: Omnipress, 2012, pp. 667–674, event-place: Edinburgh, Scotland.
  41. J. Zhuang, Z. Chen, P. Wei, G. Li, and L. Lin, “Open Set Domain Adaptation By Novel Class Discovery,” Mar. 2022. [Online]. Available: http://arxiv.org/abs/2203.03329
  42. X. Gao, C. Shan, C. Hu, Z. Niu, and Z. Liu, “An Adaptive Ensemble Machine Learning Model for Intrusion Detection,” IEEE Access, vol. 7, pp. 82 512–82 521, 2019. [Online]. Available: https://ieeexplore.ieee.org/document/8740962/
  43. A. Farahani, S. Voghoei, K. Rasheed, and H. R. Arabnia, “A Brief Review of Domain Adaptation,” 2020. [Online]. Available: http://arxiv.org/abs/2010.03978
  44. K.-C. Wang, P. Vicol, J. Lucas, L. Gu, R. Grosse, and R. Zemel, “Adversarial Distillation of Bayesian Neural Network Posteriors,” Jun. 2018, arXiv:1806.10317 [cs, stat]. [Online]. Available: http://arxiv.org/abs/1806.10317
  45. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A Simple Way to Prevent Neural Networks from Overfitting.”
  46. R. Salakhutdinov and A. Mnih, “Bayesian probabilistic matrix factorization using Markov chain Monte Carlo,” in Proceedings of the 25th international conference on Machine learning - ICML ’08.   Helsinki, Finland: ACM Press, 2008, pp. 880–887. [Online]. Available: http://portal.acm.org/citation.cfm?doid=1390156.1390267
  47. M. Weiss and P. Tonella, “Uncertainty Quantification for Deep Neural Networks: An Empirical Comparison and Usage Guidelines,” 2022. [Online]. Available: http://arxiv.org/abs/2212.07118
  48. F. O. Catak, T. Yue, and S. Ali, “Uncertainty-Aware Prediction Validator in Deep Learning Models for Cyber-Physical System Data,” ACM Trans. Softw. Eng. Methodol., vol. 31, no. 4, Jul. 2022, place: New York, NY, USA Publisher: Association for Computing Machinery. [Online]. Available: https://doi.org/10.1145/3527451
  49. X. Zhang, G. Kumar, H. Khayrallah, K. Murray, J. Gwinnup, M. J. Martindale, P. McNamee, K. Duh, and M. Carpuat, “An Empirical Exploration of Curriculum Learning for Neural Machine Translation,” 2018. [Online]. Available: http://arxiv.org/abs/1811.00739
  50. J. M. Zhang, M. Harman, L. Ma, and Y. Liu, “Machine Learning Testing: Survey, Landscapes and Horizons,” IEEE Transactions on Software Engineering, vol. X, no. X, pp. 1–1, 2020.
  51. T. Shin, Y. Razeghi, R. L. Logan IV, E. Wallace, and S. Singh, “AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts,” Nov. 2020, arXiv:2010.15980 [cs]. [Online]. Available: http://arxiv.org/abs/2010.15980
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets