The Impact of Large Language Models (LLMs) on Code Review Process

Abstract: LLMs have recently gained prominence in the field of software development, significantly boosting productivity and simplifying teamwork. Although prior studies have examined task-specific applications, the phase-specific effects of LLM assistance on the efficiency of code review processes remain underexplored. This research investigates the effect of GPT on GitHub pull request (PR) workflows, with a focus on reducing resolution time, optimizing phase-specific performance, and assisting developers. We curated a dataset of 25,473 PRs from 9,254 GitHub projects and identified GPT-assisted PRs using a semi-automated heuristic approach that combines keyword-based detection, regular expression filtering, and manual verification until achieving 95% labeling accuracy. We then applied statistical modeling, including multiple linear regression and Mann-Whitney U test, to evaluate differences between GPT-assisted and non-assisted PRs, both at the overall resolution level and across distinct review phases. Our research has revealed that early adoption of GPT can substantially boost the effectiveness of the PR process, leading to considerable time savings at various stages. Our findings suggest that GPT-assisted PRs reduced median resolution time by more than 60% (9 hours compared to 23 hours for non-assisted PRs). We discovered that utilizing GPT can reduce the review time by 33% and the waiting time before acceptance by 87%. Analyzing a sample dataset of 300 GPT-assisted PRs, we discovered that developers predominantly use GPT for code optimization (60%), bug fixing (26%), and documentation updates (12%). This research sheds light on the impact of the GPT model on the code review process, offering actionable insights for software teams seeking to enhance workflows and promote seamless collaboration.

• The paper demonstrates that GPT assistance reduces PR resolution times by up to 87.5%, streamlining code review efficiency.
• It employs a robust methodology with 25,473 GitHub PRs and statistical models to validate improvements in review, merge, and waiting phases.
• GPT is primarily used for code enhancement and bug fixing, underscoring its role in optimizing collaboration and code quality.

The Impact of LLMs on Code Review Process

This study explores the integration of LLMs, particularly GPT, into the code review process within collaborative software development. By analyzing a substantial dataset of GitHub pull requests (PRs), the research examines the phase-specific influence of GPT assistance on PR efficiency, resolution times, and developer collaboration dynamics.

Dataset Curation and Methodology

The study begins with the curation of a substantial dataset encompassing 25,473 PRs across 9,254 GitHub projects. A combination of keyword-based detection and manual verification ensures 95% accuracy in distinguishing GPT-assisted PRs from non-assisted ones. This meticulous process uses heuristics focusing on project names, PR titles, file modifications, and PR body content to filter the dataset effectively. The identification process highlights discrepancies between GPT being used for project integration versus actual code review enhancement.

Phase-Specific Effects and Statistical Modeling

The core investigation centers around the PR lifecycle phases: review, waiting before change, change, and waiting after acceptance. GPT's influence exhibits a substantial reduction in merge times, with statistical models (including multiple linear regression and Mann-Whitney U tests) confirming these observations. Notably, GPT assistance reduces median resolution time by 61%, review time by 66.7%, and waiting time before acceptance by 87.5%, compared to non-assisted PRs. The statistical rigor of these findings is ensured via logarithmic transformations and robust statistical measures, validating the positive impact of LLMs in streamlining the code review process.

Developer Use of GPT Across PR Phases

Through an analysis of 310 GPT-assisted PRs, the study categorizes the primary tasks where GPT models contribute:

• Enhancement: 60.26% of activities in the review phase involve code optimization, error handling, and performance improvements.
• Bug Fixing: Predominant in 25.64% of review phase tasks, GPT aids in diagnosing and resolving code errors.
• Documentation: Occurs in 11.54% of PRs, where GPT assists in creating or refining both new and existing documentation.
• Implementation and Testing: Less frequent, highlighting GPT's support role rather than primary generation of new code components.

The systematic application of LLMs in these tasks underscores their utility in enhancing code quality and review efficiency, particularly in time-intensive phases such as review and waiting for changes.

Implications and Future Directions

For Practitioners

The results advocate for the integration of LLMs like GPT into software development workflows to improve collaboration and efficiency. By standardizing practices for GPT use in specific scenarios—such as code refactoring and documentation—teams can maximize productivity and maintain high-quality outcomes.

For Researchers

The study prompts further research on balancing GPT reliance and human judgment in code review. Exploring frameworks that ensure effective GPT integration and minimize unproductive interactions could bolster AI-driven development processes. Future studies might focus on long-term productivity impacts and develop metrics to assess LLM effectiveness in varied software engineering environments.

Conclusion

The investigation into LLM-driven code reviews reveals significant efficiencies in PR processes, particularly in reducing review and waiting times. Although still limited in some early and late lifecycle phases, GPT's contributions substantiate its role as an integral tool in software engineering, fostering improved collaboration, code quality, and development workflows. As LLMs advance, they are poised to drive innovation and efficiency across software development landscapes.