Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

On Extending the Automatic Test Markup Language (ATML) for Machine Learning (2404.03769v1)

Published 4 Apr 2024 in cs.SE, cs.LG, cs.SY, and eess.SY

Abstract: This paper addresses the urgent need for messaging standards in the operational test and evaluation (T&E) of ML applications, particularly in edge ML applications embedded in systems like robots, satellites, and unmanned vehicles. It examines the suitability of the IEEE Standard 1671 (IEEE Std 1671), known as the Automatic Test Markup Language (ATML), an XML-based standard originally developed for electronic systems, for ML application testing. The paper explores extending IEEE Std 1671 to encompass the unique challenges of ML applications, including the use of datasets and dependencies on software. Through modeling various tests such as adversarial robustness and drift detection, this paper offers a framework adaptable to specific applications, suggesting that minor modifications to ATML might suffice to address the novelties of ML. This paper differentiates ATML's focus on testing from other ML standards like Predictive Model Markup Language (PMML) or Open Neural Network Exchange (ONNX), which concentrate on ML model specification. We conclude that ATML is a promising tool for effective, near real-time operational T&E of ML applications, an essential aspect of AI lifecycle management, safety, and governance.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (28)
  1. T. M. King, J. Arbon, D. Santiago, D. Adamo, W. Chin, and R. Shanmugam, “AI for testing today and tomorrow: industry perspectives,” in 2019 IEEE International Conference On Artificial Intelligence Testing (AITest).   IEEE, 2019, pp. 81–88.
  2. A. Munck and J. Madsen, “Test-driven modeling and development of cloud-enabled cyber-physical smart systems,” in 2017 Annual IEEE International Systems Conference (SysCon).   IEEE, 2017, pp. 1–8.
  3. H. Guissouma, A. Lauber, A. Mkadem, and E. Sax, “Virtual test environment for efficient verification of software updates for variant-rich automotive systems,” in 2019 IEEE International Systems Conference (SysCon).   IEEE, 2019, pp. 1–8.
  4. J. Gama, I. Žliobaitė, A. Bifet, M. Pechenizkiy, and A. Bouchachia, “A survey on concept drift adaptation,” ACM Computing Surveys (CSUR), vol. 46, no. 4, pp. 1–37, 2014.
  5. T. Cody, S. Adams, and P. A. Beling, “Empirically measuring transfer distance for system design and operation,” IEEE Systems Journal, vol. 16, no. 3, pp. 4962–4973, 2022.
  6. A. Chakraborty, M. Alam, V. Dey, A. Chattopadhyay, and D. Mukhopadhyay, “A survey on adversarial attacks and defences,” CAAI Transactions on Intelligence Technology, vol. 6, no. 1, pp. 25–45, 2021.
  7. Y. Nishi, S. Masuda, H. Ogawa, and K. Uetsuki, “A test architecture for machine learning product,” in 2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW).   IEEE, 2018, pp. 273–278.
  8. T. Cody, P. Beling, and L. Freeman, “Test and evaluation harnesses for learning systems,” in 2022 IEEE AUTOTESTCON.   IEEE, 2022, pp. 1–6.
  9. T. Cody, E. Lanus, D. D. Doyle, and L. Freeman, “Systematic training and testing for machine learning using combinatorial interaction testing,” in 2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW).   IEEE, 2022, pp. 102–109.
  10. T. Cody, J. Kauffman, J. Krometis, D. Sobien, and L. Freeman, “Combinatorial coverage framework for machine learning in multi-domain operations,” in Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications IV, vol. 12113.   SPIE, 2022, pp. 467–471.
  11. C. Gorringe, M. Seavey, and T. Lopes, “An overview of the atml family and related standards,” 2011 IEEE AUTOTESTCON, pp. 117–123, 2011.
  12. M. Seavey and T. Einspanjer, “Atml: what” it” is, what” it” is not, and an example of how” it” can be applied,” in 2005 IEEE AUTOTESTCON.   IEEE, 2005, pp. 137–143.
  13. A. Jain and S. Delgado, “Implementing atml into the automatic test system development and execution workflow,” in 2008 IEEE AUTOTESTCON.   IEEE, 2008, pp. 275–281.
  14. R. Taylor, “Incorporating atml interface standards within an ate software architecture,” in 2003 IEEE AUTOTESTCON.   IEEE, 2003, pp. 48–51.
  15. A. Guazzelli, M. Zeller, W.-C. Lin, G. Williams et al., “PMML: An open standard for sharing models.” R J., vol. 1, no. 1, p. 60, 2009.
  16. J. Bai, F. Lu, K. Zhang et al., “ONNX: Open neural network exchange,” 2019.
  17. T. Cody and P. A. Beling, “A systems theory of transfer learning,” IEEE Systems Journal, vol. 17, no. 1, pp. 26–37, 2023.
  18. T. Cody, S. Adams, and P. A. Beling, “A systems theoretic perspective on transfer learning,” in 2019 IEEE International Systems Conference (SysCon).   IEEE, 2019, pp. 1–7.
  19. T. Cody, S. Adams, and P. Beling, “Motivating a systems theory of ai,” Insight, vol. 23, no. 1, pp. 37–40, 2020.
  20. T. Cody, “Mesarovician abstract learning systems,” in Artificial General Intelligence: 14th International Conference, AGI 2021, Palo Alto, CA, USA, October 15–18, 2021, Proceedings 14.   Springer, 2022, pp. 55–64.
  21. ——, “Homomorphisms between transfer, multi-task, and meta-learning systems,” in International Conference on Artificial General Intelligence.   Springer, 2022, pp. 199–208.
  22. T. Cody and P. A. Beling, “Cascading negative transfer in networks of machine learning systems,” in 2023 IEEE International Conference on Assured Autonomy (ICAA).   IEEE, 2023, pp. 141–148.
  23. M. Mitchell, S. Wu, A. Zaldivar, P. Barnes, L. Vasserman, B. Hutchinson, E. Spitzer, I. D. Raji, and T. Gebru, “Model cards for model reporting,” in Proceedings of the Conference on Fairness, Accountability, and Transparency, 2019, pp. 220–229.
  24. S. Delgado and A. Jain, “Atml as a framework for reducing development time and maintenance of next generation test systems,” in 2007 IEEE AUTOTESTCON.   IEEE, 2007, pp. 174–177.
  25. S. A. Wegener, “Practical implementation of an open system using atml,” in 2004 IEEE AUTOTESTCON.   IEEE, 2004, pp. 196–201.
  26. S. Huang, N. Papernot, I. Goodfellow, Y. Duan, and P. Abbeel, “Adversarial attacks on neural network policies,” arXiv preprint arXiv:1702.02284, 2017.
  27. M. Vardigan, P. Heus, and W. Thomas, “Data documentation initiative: Toward a standard for the social sciences,” International Journal of Digital Curation, vol. 3, no. 1, 2008.
  28. W. W. W. Consortium et al., “Data catalog vocabulary (dcat),” 2014.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Tyler Cody (23 papers)
  2. Bingtong Li (1 paper)
  3. Peter A. Beling (16 papers)
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets