Why Do Developers Engage with ChatGPT in Issue-Tracker? Investigating Usage and Reliance on ChatGPT-Generated Code (2412.06757v2)

Abstract: LLMs like ChatGPT have shown the potential to assist developers with coding and debugging tasks. However, their role in collaborative issue resolution is underexplored. In this study, we analyzed 1,152 Developer-ChatGPT conversations across 1,012 issues in GitHub to examine the diverse usage of ChatGPT and reliance on its generated code. Our contributions are fourfold. First, we manually analyzed 289 conversations to understand ChatGPT's usage in the GitHub Issues. Our analysis revealed that ChatGPT is primarily utilized for ideation, whereas its usage for validation (e.g., code documentation accuracy) is minimal. Second, we applied BERTopic modeling to identify key areas of engagement on the entire dataset. We found that backend issues (e.g., API management) dominate conversations, while testing is surprisingly less covered. Third, we utilized the CPD clone detection tool to check if the code generated by ChatGPT was used to address issues. Our findings revealed that ChatGPT-generated code was used as-is to resolve only 5.83\% of the issues. Fourth, we estimated sentiment using a RoBERTa-based sentiment analysis model to determine developers' satisfaction with different usages and engagement areas. We found positive sentiment (i.e., high satisfaction) about using ChatGPT for refactoring and addressing data analytics (e.g., categorizing table data) issues. On the contrary, we observed negative sentiment when using ChatGPT to debug issues and address automation tasks (e.g., GUI interactions). Our findings show the unmet needs and growing dissatisfaction among developers. Researchers and ChatGPT developers should focus on developing task-specific solutions that help resolve diverse issues, improving user satisfaction and problem-solving efficiency in software development.

• The paper empirically analyzes 1,152 GitHub issue conversations, revealing how developers use ChatGPT for ideation (25.26%) and synthesis (18.69%) but rarely for validation (5.19%).
• Backend issues were the most discussed topic with ChatGPT (34.45%), while testing tasks were least frequent, highlighting areas of varying utility.
• Only 12.78% of resolved issues incorporated ChatGPT code, primarily for refactoring (69.56%), and developers reported high dissatisfaction (77.8%) for debugging but higher satisfaction (35.92%) for data analysis tasks.

Analyzing ChatGPT's Integration in GitHub's Issue-Tracking System

The paper "Why Do Developers Engage with ChatGPT in Issue-Tracker? Investigating Usage and Reliance on ChatGPT-Generated Code" presents an empirical paper aimed at discovering how developers are utilizing ChatGPT within the context of GitHub Issues. By analyzing 1,152 conversations on GitHub pertaining to 1,012 distinct issues, the paper explores the multifaceted roles that ChatGPT plays in software development, bridging unresolved gaps and enhancing problem resolution efficiency.

Methodology

The research employs a mixed-methods approach, combining qualitative and quantitative analyses to answer four pivotal research questions. The paper initially categorizes the various uses of ChatGPT in GitHub Issues through thematic analysis. It then applies BERTopic modeling to classify the key topical areas of ChatGPT-discussion. Additionally, the paper utilizes the CPD clone detection tool for identifying the extent to which ChatGPT-generated code is incorporated into real-world codebases. Finally, a RoBERTa-based sentiment analysis model is employed to gauge developers' satisfaction with ChatGPT's outputs.

Key Findings

1. Usage Patterns: Developers primarily use ChatGPT for ideation and synthesis within issue-tracking systems. Specifically, 25.26% of the conversations were for ideation, where developers utilized ChatGPT to brainstorm and generate potential solutions. Synthesis, wherein developers describe desired functionalities and ChatGPT delivers corresponding code, constituted 18.69% of usages. However, ChatGPT was seldom used for validation purposes (5.19%).
2. Topical Discussions: The analysis identified that backend issues, at 34.45%, were the most frequently discussed topics with ChatGPT. Conversations often revolved around backend frameworks, API management, and security configurations. Testing tasks were the least discussed, highlighting an area where ChatGPT's application remains limited.
3. Code Reliance: Of the issues resolved using ChatGPT-generated code, only 12.78% incorporated the generated code, and merely 5.83% of these instances utilized the code without modification. Refactoring tasks, accounting for 69.56% of usage cases, emerged as the primary reason for integrating ChatGPT-generated code, indicating its effectiveness in refining rather than creating code from scratch.
4. Satisfaction Metrics: Developers expressed dissatisfaction in 77.8% of cases about ChatGPT's solutions, particularly when debugging or addressing automation tasks. In contrast, data analysis tasks elicited higher positive responses (35.92%) due to the self-contained nature of the data involved, indicating areas where ChatGPT provides significant value.

Implications and Future Directions

The research underscores the potential for targeted improvements in ChatGPT's capability to handle complex, domain-specific issues, particularly in areas like backend development and automation. By enhancing context understanding, such as through conversation refinements and integration with other domain-specific APIs, ChatGPT's real-world applicability and satisfaction levels could be significantly improved.

For researchers, the findings call for development of task-specific models within ITS environments that leverage contextual data more effectively. Developers can enhance ChatGPT's usability by crafting detailed prompts that offer comprehensive context, thus facilitating more accurate and relevant responses.

In summary, the paper provides critical insights into ChatGPT's current role and efficacy in software issue management while charting a course for future research and development to bridge the existing functionality gaps. As the landscape of LLMs like ChatGPT evolves, their strategic integration into ITS represents both a challenge and an opportunity to enhance collaborative software development.