Development of Automated Software Design Document Review Methods Using Large Language Models

Abstract: In this study, we explored an approach to automate the review process of software design documents by using LLM. We first analyzed the review methods of design documents and organized 11 review perspectives. Additionally, we analyzed the issues of utilizing LLMs for these 11 review perspectives and determined which perspectives can be reviewed by current general-purpose LLMs instead of humans. For the reviewable perspectives, we specifically developed new techniques to enable LLMs to comprehend complex design documents that include table data. For evaluation, we conducted experiments using GPT to assess the consistency of design items and descriptions across different design documents in the design process used in actual business operations. Our results confirmed that LLMs can be utilized to identify inconsistencies in software design documents during the review process.