Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Model Generation with LLMs: From Requirements to UML Sequence Diagrams (2404.06371v2)

Published 9 Apr 2024 in cs.SE, cs.CL, and cs.LG

Abstract: Complementing natural language (NL) requirements with graphical models can improve stakeholders' communication and provide directions for system design. However, creating models from requirements involves manual effort. The advent of generative LLMs, ChatGPT being a notable example, offers promising avenues for automated assistance in model generation. This paper investigates the capability of ChatGPT to generate a specific type of model, i.e., UML sequence diagrams, from NL requirements. We conduct a qualitative study in which we examine the sequence diagrams generated by ChatGPT for 28 requirements documents of various types and from different domains. Observations from the analysis of the generated diagrams have systematically been captured through evaluation logs, and categorized through thematic analysis. Our results indicate that, although the models generally conform to the standard and exhibit a reasonable level of understandability, their completeness and correctness with respect to the specified requirements often present challenges. This issue is particularly pronounced in the presence of requirements smells, such as ambiguity and inconsistency. The insights derived from this study can influence the practical utilization of LLMs in the RE process, and open the door to novel RE-specific prompting strategies targeting effective model generation.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (38)
  1. R. Jolak, M. Savary-Leblanc, M. Dalibor, A. Wortmann, R. Hebig, J. Vincur, I. Polasek, X. Le Pallec, S. Gérard, and M. R. Chaudron, “Software engineering whispers: The effect of textual vs. graphical software design descriptions on software design communication,” Empirical software engineering, vol. 25, pp. 4427–4471, 2020.
  2. S. Wagner, D. M. Fernández, M. Felderer, A. Vetrò, M. Kalinowski, R. Wieringa, D. Pfahl, T. Conte, M.-T. Christiansson, D. Greer et al., “Status quo in requirements engineering: A theory and a global family of surveys,” ACM TOSEM, vol. 28, no. 2, pp. 1–48, 2019.
  3. V. Ambriola and V. Gervasi, “On the systematic analysis of natural language requirements with circe,” ASE, vol. 13, pp. 107–167, 2006.
  4. C. Arora, M. Sabetzadeh, S. Nejati, and L. Briand, “An active learning approach for improving the accuracy of automated domain model extraction,” ACM TOSEM, vol. 28, no. 1, pp. 1–34, 2019.
  5. J. Horkoff, F. B. Aydemir, E. Cardoso, T. Li, A. Maté, E. Paja, M. Salnitri, L. Piras, J. Mylopoulos, and P. Giorgini, “Goal-oriented requirements engineering: an extended systematic mapping study,” REJ, vol. 24, pp. 133–160, 2019.
  6. “Unified modeling language (UML) 2.5.1 core specification,” https://www.omg.org/spec/UML, 2017.
  7. T. Yue, L. C. Briand, and Y. Labiche, “aToucan: an automated framework to derive uml analysis models from use case models,” ACM TOSEM, vol. 24, no. 3, pp. 1–52, 2015.
  8. M. Jahan, Z. S. H. Abad, and B. Far, “Generating sequence diagram from natural language requirements,” in REW’21.   IEEE, 2021, pp. 39–48.
  9. R. Saini, G. Mussbacher, J. L. Guo, and J. Kienzle, “Automated, interactive, and traceable domain modelling empowered by artificial intelligence,” SoSym, pp. 1–31, 2022.
  10. C. Arora, J. Grundy, and M. Abdelrazek, “Advancing requirements engineering through generative ai: Assessing the role of llms,” arXiv preprint arXiv:2310.13976, 2023.
  11. S. Ahmed, A. Ahmed, and N. U. Eisty, “Automatic transformation of natural to unified modeling language: A systematic review,” in SERA’22.   IEEE, 2022, pp. 112–119.
  12. B. Chen, K. Chen, S. Hassani, Y. Yang, D. Amyot, L. Lessard, G. Mussbacher, M. Sabetzadeh, and D. Varró, “On the use of gpt-4 for creating goal models: an exploratory study,” in REW’23.   IEEE, 2023, pp. 262–271.
  13. J. Cámara, J. Troya, L. Burgueño, and A. Vallecillo, “On the assessment of generative ai in modeling tasks: an experience report with chatgpt and uml,” SoSym, pp. 1–13, 2023.
  14. K. Chen, Y. Yang, B. Chen, J. A. H. López, G. Mussbacher, and D. Varró, “Automated domain modeling with large language models: A comparative study,” in MODELS’23.   IEEE, 2023, pp. 162–172.
  15. L. Zhao, W. Alhoshan, A. Ferrari, K. J. Letsholo, M. A. Ajagbe, E.-V. Chioasca, and R. T. Batista-Navarro, “Natural language processing for requirements engineering: A systematic mapping study,” CSUR, vol. 54, no. 3, pp. 1–41, 2021.
  16. L. Kof, “Scenarios: Identifying missing objects and actions by means of computational linguistics,” in RE’07.   IEEE, 2007, pp. 121–130.
  17. B. Min, H. Ross, E. Sulem, A. P. B. Veyseh, T. H. Nguyen, O. Sainz, E. Agirre, I. Heintz, and D. Roth, “Recent advances in natural language processing via large pre-trained language models: A survey,” CSUR, vol. 56, no. 2, pp. 1–40, 2023.
  18. A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, and J. M. Zhang, “Large language models for software engineering: Survey and open problems,” arXiv preprint arXiv:2310.03533, 2023.
  19. X. Hou, Y. Zhao, Y. Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engineering: A systematic literature review,” arXiv preprint arXiv:2308.10620, 2023.
  20. C. Jain, P. R. Anish, A. Singh, and S. Ghaisas, “A transformer-based approach for abstractive summarization of requirements from obligations in software engineering contracts,” in RE’23.   IEEE, 2023, pp. 169–179.
  21. A. D. Rodriguez, K. R. Dearstyne, and J. Cleland-Huang, “Prompts matter: Insights and strategies for prompt engineering in automated software traceability,” in REW’23.   IEEE, 2023, pp. 455–464.
  22. K. Ronanki, B. Cabrero-Daniel, J. Horkoff, and C. Berger, “Requirements engineering using generative ai: Prompts and prompting patterns,” arXiv preprint arXiv:2311.03832, 2023.
  23. J. White, S. Hays, Q. Fu, J. Spencer-Smith, and D. C. Schmidt, “Chatgpt prompt patterns for improving code quality, refactoring, requirements elicitation, and software design,” arXiv preprint arXiv:2303.07839, 2023.
  24. D. Kundu, D. Samanta, and R. Mall, “Automatic code generation from unified modelling language sequence diagrams,” IET Software, vol. 7, no. 1, pp. 12–28, 2013.
  25. A. Ferrari, G. O. Spagnolo, and S. Gnesi, “Pure: A dataset of public requirements documents,” in RE’17.   IEEE, 2017, pp. 502–505.
  26. F. Dalpiaz and A. Sturm, “Conceptualizing requirements using user stories and use cases: a controlled experiment,” in REFSQ’20.   Springer, 2020, pp. 221–238.
  27. V. Braun and V. Clarke, “To saturate or not to saturate? questioning data saturation as a useful concept for thematic analysis and sample-size rationales,” Qualitative research in sport, exercise and health, vol. 13, no. 2, pp. 201–216, 2021.
  28. J. White, Q. Fu, S. Hays, M. Sandborn, C. Olea, H. Gilbert, A. Elnashar, J. Spencer-Smith, and D. C. Schmidt, “A prompt pattern catalog to enhance prompt engineering with chatGPT,” arXiv:2302.11382, 2023.
  29. D. Zowghi and V. Gervasi, “On the interplay between consistency, completeness, and correctness in requirements evolution,” IST, vol. 45, no. 14, pp. 993–1009, 2003.
  30. “Iso/iec/ieee international standard - systems and software engineering – life cycle processes – requirements engineering,” ISO/IEC/IEEE 29148:2018(E), pp. 1–104, 2018.
  31. F. Alessio, A. Sallam, and A. Chetan, “Model Generation from Requirements with LLMs: an Exploratory Study - Replication Package,” Apr. 2024. [Online]. Available: https://doi.org/10.5281/zenodo.10579731
  32. J. Cohen, “A coefficient of agreement for nominal scales,” Educational and Psychological Measurement, vol. 20, no. 1, 1960.
  33. S. Abrahão, E. Insfran, J. A. Carsí, and M. Genero, “Evaluating requirements modeling methods based on user perceptions: A family of experiments,” Inf. Sci., vol. 181, no. 16, pp. 3356–3378, 2011.
  34. V. Clarke and V. Braun, “Thematic analysis,” The journal of positive psychology, vol. 12, no. 3, pp. 297–298, 2017.
  35. J. Zhang, Y. Chen, N. Niu, and C. Liu, “A preliminary evaluation of ChatGPT in requirements information retrieval,” arXiv:2304.12562, 2023.
  36. K. Ronanki, C. Berger, and J. Horkoff, “Investigating chatgpt’s potential to assist in requirements elicitation processes,” in SEAA’23.   IEEE, 2023, pp. 354–361.
  37. J. Sallou, T. Durieux, and A. Panichella, “Breaking the silence: the threats of using LLMs in software engineering,” arXiv:2312.08055, 2023.
  38. K.-J. Stol and B. Fitzgerald, “Guidelines for conducting software engineering research,” in Contemporary Empirical Methods in Software Engineering.   Springer, 2020, pp. 27–62.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Alessio Ferrari (22 papers)
  2. Sallam Abualhaija (13 papers)
  3. Chetan Arora (79 papers)
Citations (1)
View on Semantic Scholar