Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

An Empirical Study of ChatGPT-Related Projects and Their Issues on GitHub (2403.17437v2)

Published 26 Mar 2024 in cs.SE

Abstract: Since the launch of ChatGPT in 2022, an increasing number of ChatGPT-related projects are being published on GitHub, sparking widespread discussions. However, GitHub does not provide a detailed classification of these projects to help users effectively explore interested projects. Additionally, the issues raised by users for these projects cover various aspects, e.g., installation, usage, and updates. It would be valuable to help developers prioritize more urgent issues and improve development efficiency. We retrieved 71,244 projects from GitHub using the keyword `ChatGPT' and selected the top 200 representative projects with the highest numbers of stars as our dataset. By analyzing the project descriptions, we identified three primary categories of ChatGPT-related projects, namely ChatGPT Implementation & Training, ChatGPT Application, ChatGPT Improvement & Extension. Next, we applied a topic modeling technique to 23,609 issues of those projects and identified ten issue topics, e.g., model reply and interaction interface. We further analyzed the popularity, difficulty, and evolution of each issue topic within the three project categories. Our main findings are: 1) The increase in the number of projects within the three categories is closely related to the development of ChatGPT; and 2) There are significant differences in the popularity, difficulty, and evolutionary trends of the issue topics across the three project categories. Based on these findings, we finally provided implications for project developers and platform managers on how to better develop and manage ChatGPT-related projects.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (66)
  1. Chatgpt for cybersecurity: practical applications, challenges, and future directions. Cluster Computing 26, 6 (aug 2023), 3421–3436. https://doi.org/10.1007/s10586-023-04124-5
  2. Abeer Alessa and Hend Al-Khalifa. 2023. Towards Designing a ChatGPT Conversational Companion for Elderly People. In Proceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments (¡conf-loc¿, ¡city¿Corfu¡/city¿, ¡country¿Greece¡/country¿, ¡/conf-loc¿) (PETRA ’23). Association for Computing Machinery, New York, NY, USA, 667–674. https://doi.org/10.1145/3594806.3596572
  3. Mehdi Bagherzadeh and Raffi Khatchadourian. 2019. Going big: a large-scale study on what big data developers ask. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Tallinn, Estonia) (ESEC/FSE 2019). Association for Computing Machinery, New York, NY, USA, 432–442. https://doi.org/10.1145/3338906.3338939
  4. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. ArXiv abs/2204.05862 (2022). https://api.semanticscholar.org/CorpusID:248118878
  5. What are developers talking about? An analysis of topics and trends in Stack Overflow. Empirical Softw. Engg. 19, 3 (jun 2014), 619–654. https://doi.org/10.1007/s10664-012-9231-y
  6. Steven Bird and Edward Loper. 2004. NLTK: The Natural Language Toolkit. In Proceedings of the ACL Interactive Poster and Demonstration Sessions. Association for Computational Linguistics, Barcelona, Spain, 214–217. https://aclanthology.org/P04-3031
  7. Latent dirichlet allocation. Journal of machine Learning research 3, Jan (2003), 993–1022.
  8. Automatic Core-Developer Identification on GitHub: A Validation Study. ACM Trans. Softw. Eng. Methodol. 32, 6, Article 138 (sep 2023), 29 pages. https://doi.org/10.1145/3593803
  9. Justus Bogner and Manuel Merkel. 2022. To type or not to type? a systematic comparison of the software quality of JavaScript and typescript applications on GitHub. In Proceedings of the 19th International Conference on Mining Software Repositories (Pittsburgh, Pennsylvania) (MSR ’22). Association for Computing Machinery, New York, NY, USA, 658–669. https://doi.org/10.1145/3524842.3528454
  10. Hudson Borges and Marco Tulio Valente. 2018. What’s in a GitHub Star? Understanding Repository Starring Practices in a Social Coding Platform. Journal of Systems and Software 146 (2018), 112–129. https://doi.org/10.1016/j.jss.2018.09.016
  11. Kira: A Financial Chatbot Using ChatGPT and Data Obfuscation. J. Comput. Sci. Coll. 39, 3 (oct 2023), 277–294.
  12. Joyjit Chatterjee and Nina Dethlefs. 2023. This new conversational AI model can be your friend, philosopher, and guide … and even your worst enemy. Patterns 4, 1 (2023), 100676. https://doi.org/10.1016/j.patter.2022.100676
  13. All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text. In Annual Meeting of the Association for Computational Linguistics. https://api.semanticscholar.org/CorpusID:235694265
  14. Kate Crawford and Trevor Paglen. 2021. Excavating AI: the politics of images in machine learning training sets. AI & SOCIETY 36, 4 (Dec. 2021), 1105–1116. https://doi.org/10.1007/s00146-021-01162-8
  15. Too long; didn’t read: Automatic summarization of GitHub README.MD with Transformers. In Proceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering (Oulu, Finland) (EASE ’23). Association for Computing Machinery, New York, NY, USA, 267–272. https://doi.org/10.1145/3593434.3593448
  16. Mehmet Firat. 2023a. How Chat GPT Can Transform Autodidactic Experiences and Open Education? https://doi.org/10.31219/osf.io/9ge8m
  17. Mehmet Firat. 2023b. What ChatGPT means for universities: Perceptions of scholars and students. 6 (04 2023), 1–22. https://doi.org/10.37074/jalt.2023.6.1.22
  18. Mehmet Firat and Saniye Kuleli. 2023. What if GPT4 Became Autonomous: The Auto-GPT Project and Use Cases. Journal of Emerging Computer Technologies 3, 1 (2023), 1–6. https://doi.org/10.57020/ject.1297961
  19. Joseph Fleiss. 1971. Measuring Nominal Scale Agreement Among Many Raters. Psychological Bulletin 76 (11 1971), 378–. https://doi.org/10.1037/h0031619
  20. On the Variability of Software Engineering Needs for Deep Learning: Stages, Trends, and Application Types. IEEE Transactions on Software Engineering 49, 2 (2023), 760–776. https://doi.org/10.1109/TSE.2022.3163576
  21. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.14375 (2022).
  22. The State of the ML-universe: 10 Years of Artificial Intelligence & Machine Learning Software Development on GitHub. In Proceedings of the 17th International Conference on Mining Software Repositories (Seoul, Republic of Korea) (MSR ’20). Association for Computing Machinery, New York, NY, USA, 431–442. https://doi.org/10.1145/3379597.3387473
  23. Roberto Gozalo-Brizuela and Eduardo C. Garrido-Merchan. 2023. ChatGPT is not all you need. A State of the Art Review of large Generative AI models. arXiv:2301.04655 [cs.LG]
  24. What do Programmers Discuss about Deep Learning Frameworks. Empirical Software Engineering 25 (07 2020). https://doi.org/10.1007/s10664-020-09819-6
  25. Deep Speech: Scaling up end-to-end speech recognition. ArXiv abs/1412.5567 (2014). https://api.semanticscholar.org/CorpusID:16979536
  26. Md. Asraful Haque and Shuai Li. 2023. The Potential Use of ChatGPT for Debugging and Bug Fixing. EAI Endorsed Transactions on AI and Robotics (2023). https://api.semanticscholar.org/CorpusID:258657620
  27. Challenges in Docker Development: A Large-scale Study Using Stack Overflow. In Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) (Bari, Italy) (ESEM ’20). Association for Computing Machinery, New York, NY, USA, Article 7, 11 pages. https://doi.org/10.1145/3382494.3410693
  28. Liangjie Hong and Brian D. Davison. 2010. Empirical study of topic modeling in Twitter. In Proceedings of the First Workshop on Social Media Analytics (Washington D.C., District of Columbia) (SOMA ’10). Association for Computing Machinery, New York, NY, USA, 80–88. https://doi.org/10.1145/1964858.1964870
  29. Characterizing and Predicting Good First Issues. In Proceedings of the 15th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) (Bari, Italy) (ESEM ’21). Association for Computing Machinery, New York, NY, USA, Article 13, 12 pages. https://doi.org/10.1145/3475716.3475789
  30. Predicting the objective and priority of issue reports in software repositories. Empirical Softw. Engg. 27, 2 (mar 2022), 37 pages. https://doi.org/10.1007/s10664-021-10085-3
  31. Ticket Tagger: Machine Learning Driven Issue Classification. In 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME). 406–409. https://doi.org/10.1109/ICSME.2019.00070
  32. Artificial intelligence foundation and pre-trained models: Fundamentals, applications, opportunities, and social impacts. Simulation Modelling Practice and Theory 126 (2023), 102754. https://doi.org/10.1016/j.simpat.2023.102754
  33. Anis Koubaa. 2023. GPT-4 vs. GPT-3.5: A Concise Showdown. Preprints (March 2023). https://doi.org/10.20944/preprints202303.0422.v1
  34. Diana Kozachek. 2023. Investigating the Perception of the Future in GPT-3, -3.5 and GPT-4. In Proceedings of the 15th Conference on Creativity and Cognition (¡conf-loc¿, ¡city¿Virtual Event¡/city¿, ¡country¿USA¡/country¿, ¡/conf-loc¿) (C&C ’23). Association for Computing Machinery, New York, NY, USA, 282–287. https://doi.org/10.1145/3591196.3596827
  35. Studying software logging using topic models. Empirical Softw. Engg. 23, 5 (oct 2018), 2655–2694. https://doi.org/10.1007/s10664-018-9595-8
  36. Cleyciane Lima and Daricelio Soares. 2022. On the Nature of Duplicate Pull Requests: An Empirical Study Using Association Rules. In Proceedings of the 16th Brazilian Symposium on Software Components, Architectures, and Reuse (¡conf-loc¿, ¡city¿Uberlandia¡/city¿, ¡country¿Brazil¡/country¿, ¡/conf-loc¿) (SBCARS ’22). Association for Computing Machinery, New York, NY, USA, 68–75. https://doi.org/10.1145/3559712.3559722
  37. Chung Kwan Lo. 2023. What Is the Impact of ChatGPT on Education? A Rapid Review of the Literature. Education Sciences 13 (04 2023), 410. https://doi.org/10.3390/educsci13040410
  38. Brady Lund and Ting Wang. 2023. Chatting about ChatGPT: How may AI and GPT impact academia and libraries? Library Hi Tech News 40 (02 2023). https://doi.org/10.1108/LHTN-01-2023-0009
  39. Introduction to Information Retrieval. Cambridge University Press. https://books.google.com.sg/books?id=t1PoSh4uwVcC
  40. George A. Miller. 1995. WordNet: a lexical database for English. Commun. ACM 38, 11 (nov 1995), 39–41. https://doi.org/10.1145/219717.219748
  41. Meredith Ringel Morris. 2023. Scientists’ Perspectives on the Potential for Generative AI in their Fields. Technical Report. https://arxiv.org/abs/2304.01420
  42. Artificial Intelligence vs. Software Engineers: An Empirical Study on Performance and Efficiency using ChatGPT. In Proceedings of the 33rd Annual International Conference on Computer Science and Software Engineering (¡conf-loc¿, ¡city¿Las Vegas, NV¡/city¿, ¡country¿USA¡/country¿, ¡/conf-loc¿) (CASCON ’23). IBM Corp., USA, 24–33.
  43. Automatic evaluation of topic coherence. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics (Los Angeles, California) (HLT ’10). Association for Computational Linguistics, USA, 100–108.
  44. Peter Adebowale Olujimi and Abejide Ade-Ibijola. 2023. NLP techniques for automating responses to customer queries: a systematic review. Discover Artificial Intelligence 3, 1 (March 2023). https://doi.org/10.1007/s44163-023-00065-5
  45. OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL]
  46. Requirements and GitHub Issues: An Automated Approach for Quality Requirements Classification. Program. Comput. Softw. 47, 8 (dec 2021), 704–721. https://doi.org/10.1134/S0361768821080193
  47. Martin F. Porter. 2001. Snowball: A language for stemming algorithms. https://api.semanticscholar.org/CorpusID:59634627
  48. Alec Radford and Karthik Narasimhan. 2018. Improving Language Understanding by Generative Pre-Training. https://api.semanticscholar.org/CorpusID:49313245
  49. Language Models are Unsupervised Multitask Learners. https://api.semanticscholar.org/CorpusID:160025533
  50. Exploring the Space of Topic Coherence Measures. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining (Shanghai, China) (WSDM ’15). Association for Computing Machinery, New York, NY, USA, 399–408. https://doi.org/10.1145/2684822.2685324
  51. Christoffer Rosen and Emad Shihab. 2016. What are mobile developers asking about? A large scale study using stack overflow. Empirical Softw. Engg. 21, 3 (jun 2016), 1192–1223. https://doi.org/10.1007/s10664-015-9379-3
  52. Latent DIRICHLET allocation (LDA) based information modelling on BLOCKCHAIN technology: a review of trends and research patterns used in integration. Multimedia Tools Appl. 81, 25 (oct 2022), 36805–36831. https://doi.org/10.1007/s11042-022-13500-z
  53. Md. Manjurul Shourov and Ishtiak Mahmud. 2019. pyMannKendall: a python package for non parametric Mann Kendall family of trend tests. Journal of Open Source Software 4 (07 2019), 1556. https://doi.org/10.21105/joss.01556
  54. Mohammed Latif Siddiq and Joanna C. S. Santos. 2023. BERT-based GitHub issue report classification. In Proceedings of the 1st International Workshop on Natural Language-Based Software Engineering (Pittsburgh, Pennsylvania) (NLBSE ’22). Association for Computing Machinery, New York, NY, USA, 33–36. https://doi.org/10.1145/3528588.3528660
  55. Learning to summarize from human feedback. In Proceedings of the 34th International Conference on Neural Information Processing Systems (Vancouver, BC, Canada) (NIPS’20). Curran Associates Inc., Red Hook, NY, USA, Article 253, 14 pages.
  56. Modeling the evolution of topics in source code histories. In Proceedings of the 8th Working Conference on Mining Software Repositories (Waikiki, Honolulu, HI, USA) (MSR ’11). Association for Computing Machinery, New York, NY, USA, 173–182. https://doi.org/10.1145/1985441.1985467
  57. Is ChatGPT the Ultimate Programming Assistant – How far is it? arXiv:2304.11938 [cs.SE]
  58. Stefan Tilkov and Steve Vinoski. 2010. Node.js: Using JavaScript to Build High-Performance Network Programs. IEEE Internet Computing 14, 6 (nov 2010), 80–83. https://doi.org/10.1109/MIC.2010.145
  59. Christoph Treude and Markus Wagner. 2019. Predicting good configurations for GitHub and stack overflow topic models. In Proceedings of the 16th International Conference on Mining Software Repositories (Montreal, Quebec, Canada) (MSR ’19). IEEE Press, 84–95. https://doi.org/10.1109/MSR.2019.00022
  60. What Do Programmers Discuss About Blockchain? A Case Study on the Use of Balanced LDA and the Reference Architecture of a Domain to Capture Online Discussions About Blockchain Platforms Across Stack Exchange Communities. IEEE Transactions on Software Engineering 47, 7 (2021), 1331–1349. https://doi.org/10.1109/TSE.2019.2921343
  61. ChatGPT Prompt Patterns for Improving Code Quality, Refactoring, Requirements Elicitation, and Software Design. arXiv:2303.07839 [cs.SE]
  62. Towards Automated Detection of Unethical Behavior in Open-Source Software Projects. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (¡conf-loc¿, ¡city¿San Francisco¡/city¿, ¡state¿CA¡/state¿, ¡country¿USA¡/country¿, ¡/conf-loc¿) (ESEC/FSE 2023). Association for Computing Machinery, New York, NY, USA, 644–656. https://doi.org/10.1145/3611643.3616314
  63. What Security Questions Do Developers Ask? A Large-Scale Study of Stack Overflow Posts. Journal of Computer Science and Technology 31 (09 2016), 910–924. https://doi.org/10.1007/s11390-016-1672-0
  64. An empirical study of blockchain system vulnerabilities: modules, types, and patterns. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (¡conf-loc¿, ¡city¿Singapore¡/city¿, ¡country¿Singapore¡/country¿, ¡/conf-loc¿) (ESEC/FSE 2022). Association for Computing Machinery, New York, NY, USA, 709–721. https://doi.org/10.1145/3540250.3549105
  65. An Empirical Study of the Dynamics of GitHub Repository and Its Impact on Distributed Software Development. In Proceedings of the Confederated International Workshops on On the Move to Meaningful Internet Systems: OTM 2014 Workshops - Volume 8842. Springer-Verlag, Berlin, Heidelberg, 457–466. https://doi.org/10.1007/978-3-662-45550-0_46
  66. ChatGPT: potential, prospects, and limitations. Frontiers of Information Technology & Electronic Engineering 25, 1 (Jan. 2024), 6–11. https://doi.org/10.1631/FITEE.2300089
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Zheng Lin (104 papers)
  2. Neng Zhang (7 papers)
  3. Chao Liu (358 papers)
  4. Zibin Zheng (194 papers)
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets