Voices from the Frontier: A Comprehensive Analysis of the OpenAI Developer Forum (2408.01687v1)
Abstract: OpenAI's advanced LLMs have revolutionized natural language processing and enabled developers to create innovative applications. As adoption grows, understanding the experiences and challenges of developers working with these technologies is crucial. This paper presents a comprehensive analysis of the OpenAI Developer Forum, focusing on (1) popularity trends and user engagement patterns, and (2) a taxonomy of challenges and concerns faced by developers. We first employ a quantitative analysis of the metadata from 29,576 forum topics, investigating temporal trends in topic creation, the popularity of topics across different categories, and user contributions at various trust levels. We then qualitatively analyze content from 9,301 recently active topics on developer concerns. From a sample of 886 topics, we construct a taxonomy of concerns in the OpenAI Developer Forum. Our findings uncover critical concerns raised by developers in creating AI-powered applications and offer targeted recommendations to address them. This work not only advances AI-assisted software engineering but also empowers developer communities to shape the responsible evolution and integration of AI technology in society.
- A. Abdellatif, D. Costa, K. Badran, R. Abdalkareem, and E. Shihab, “Challenges in chatbot development: A study of stack overflow posts,” in Proceedings of the 17th international conference on mining software repositories, 2020, pp. 174–185.
- J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al., “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023.
- S. Ahmed and M. Bagherzadeh, “What do concurrency developers ask about? a large-scale study using stack overflow,” in Proceedings of the 12th ACM/IEEE international symposium on empirical software engineering and measurement, 2018, pp. 1–10.
- J. Atwood, “Discourse trust levels,” https://blog.discourse.org/2018/06/understanding-discourse-trust-levels/, 2018.
- Z. Chen, Y. Cao, Y. Liu, H. Wang, T. Xie, and X. Liu, “A comprehensive study on challenges in deploying deep learning based software,” in Proceedings of the 28th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, 2020, pp. 750–762.
- J. Cui, W.-L. Chiang, I. Stoica, and C.-J. Hsieh, “Or-bench: An over-refusal benchmark for large language models,” arXiv preprint arXiv:2405.20947, 2024.
- J. Han, E. Shihab, Z. Wan, S. Deng, and X. Xia, “What do programmers discuss about deep learning frameworks,” Empirical Software Engineering, vol. 25, pp. 2694–2747, 2020.
- X. Hou, Y. Zhao, Y. Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engineering: A systematic literature review,” arXiv e-prints, pp. arXiv–2308, 2023.
- OpenAI, “Assistants api overview,” https://platform.openai.com/docs/assistants/overview, 2024.
- ——, “Chatgpt,” https://chat.openai.com/, 2024.
- ——, “Introducing the gpt store,” https://openai.com/index/introducing-the-gpt-store/, 2024.
- ——, “.net library,” https://platform.openai.com/docs/libraries/dotnet-library, 2024.
- ——, “Openai,” https://openai.com, 2024.
- ——, “Openai developer forum,” https://community.openai.com, 2024.
- ——, “Openai developer platform,” https://platform.openai.com/docs/overview, 2024.
- ——, “Vector databases,” https://cookbook.openai.com/examples/vector_databases/readme, 2024.
- C. Rosen and E. Shihab, “What are mobile developers asking about? a large scale study using stack overflow,” Empirical Software Engineering, vol. 21, pp. 1192–1223, 2016.
- C. B. Seaman, “Qualitative methods in empirical studies of software engineering,” IEEE Transactions on software engineering, vol. 25, no. 4, pp. 557–572, 1999.
- P. K. Venkatesh, S. Wang, F. Zhang, Y. Zou, and A. E. Hassan, “What do client developers concern when using web apis? an empirical study on developer forums and stack overflow,” in 2016 IEEE International Conference on Web Services (ICWS). IEEE, 2016, pp. 131–138.
- Z. Wan, X. Xia, and A. E. Hassan, “What do programmers discuss about blockchain? a case study on the use of balanced lda and the reference architecture of a domain to capture online discussions about blockchain platforms across stack exchange communities,” IEEE Transactions on Software Engineering, vol. 47, no. 7, pp. 1331–1349, 2019.
- T. Xie, X. Qi, Y. Zeng, Y. Huang, U. M. Sehwag, K. Huang, L. He, B. Wei, D. Li, Y. Sheng et al., “Sorry-bench: Systematically evaluating large language model safety refusal behaviors,” arXiv preprint arXiv:2406.14598, 2024.
- T. Zhang, C. Gao, L. Ma, M. Lyu, and M. Kim, “An empirical study of common challenges in developing deep learning applications,” in 2019 IEEE 30th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 2019, pp. 104–115.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.