Papers
Topics
Authors
Recent
2000 character limit reached

AllHands: Ask Me Anything on Large-scale Verbatim Feedback via Large Language Models (2403.15157v2)

Published 22 Mar 2024 in cs.SE

Abstract: Verbatim feedback constitutes a valuable repository of user experiences, opinions, and requirements essential for software development. Effectively and efficiently extracting valuable insights from such data poses a challenging task. This paper introduces Allhands , an innovative analytic framework designed for large-scale feedback analysis through a natural language interface, leveraging LLMs. Allhands adheres to a conventional feedback analytic workflow, initially conducting classification and topic modeling on the feedback to convert them into a structurally augmented format, incorporating LLMs to enhance accuracy, robustness, generalization, and user-friendliness. Subsequently, an LLM agent is employed to interpret users' diverse questions in natural language on feedback, translating them into Python code for execution, and delivering comprehensive multi-modal responses, including text, code, tables, and images. We evaluate Allhands across three diverse feedback datasets. The experiments demonstrate that Allhands achieves superior efficacy at all stages of analysis, including classification and topic modeling, eventually providing users with an "ask me anything" experience with comprehensive, correct and human-readable response. To the best of our knowledge, Allhands stands as the first comprehensive feedback analysis framework that supports diverse and customized requirements for insight extraction through a natural language interface.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (89)
  1. Visual opinion analysis of customer feedback data. In 2009 IEEE symposium on visual analytics science and technology, pages 187–194. IEEE, 2009.
  2. Sentiment analysis on customer feedback data: Amazon product reviews. In 2019 International conference on machine learning, big data, cloud and parallel computing (COMITCon), pages 320–322. IEEE, 2019.
  3. Unveiling the life cycle of user feedback: Best practices from software practitioners. arXiv preprint arXiv:2309.07345, 2023.
  4. A method for prioritizing end-user feedback for requirements engineering. In 2012 5th International Workshop on Co-operative and Human Aspects of Software Engineering (CHASE), pages 47–49. IEEE, 2012.
  5. Feedback-driven requirements engineering: the heuristic requirements assistant. In 2009 IEEE 31st international conference on software engineering, pages 587–590. IEEE, 2009.
  6. Natural language processing for requirements engineering: A systematic mapping study. ACM Computing Surveys (CSUR), 54(3):1–41, 2021.
  7. Software engineering in start-up companies: An analysis of 88 experience reports. Empirical Software Engineering, 24:68–102, 2019.
  8. Understanding meanings in multilingual customer feedback. arXiv preprint arXiv:1806.01694, 2018.
  9. Stephen Withall. Software requirement patterns. Pearson Education, 2007.
  10. How can i improve my app? classifying user reviews for software maintenance and evolution. In 2015 IEEE international conference on software maintenance and evolution (ICSME), pages 281–290. IEEE, 2015.
  11. User feedback in the appstore: An empirical study. In 2013 21st IEEE international requirements engineering conference (RE), pages 125–134. IEEE, 2013.
  12. Software release planning: an evolutionary and iterative approach. Information and software technology, 46(4):243–253, 2004.
  13. Android apps and user feedback: a dataset for software evolution and quality improvement. In Proceedings of the 2nd ACM SIGSOFT international workshop on app market analytics, pages 8–11, 2017.
  14. The potential of machine learning algorithms for sentiment classification of students’ feedback on mooc. In Intelligent Systems and Applications: Proceedings of the 2021 Intelligent Systems Conference (IntelliSys) Volume 3, pages 11–22. Springer, 2022.
  15. Latent dirichlet allocation (lda) and topic modeling: models, applications, a survey. Multimedia Tools and Applications, 78:15169–15211, 2019.
  16. Infar: Insight extraction from app reviews. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 904–907, 2018.
  17. Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171–4186, 2019.
  18. Abstractive vs. extractive summarization: An experimental review. Applied Sciences, 13(13):7620, 2023.
  19. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837, 2022.
  20. Quality evaluation of product reviews using an information quality framework. Decision Support Systems, 50(4):755–768, 2011.
  21. On the automatic classification of app reviews. Requirements Engineering, 21:311–331, 2016.
  22. Nathan Baker. Employee feedback technologies in the human performance system. Human Resource Development International, 13(4):477–485, 2010.
  23. Short text topic modeling techniques, applications, and performance: a survey. IEEE Transactions on Knowledge and Data Engineering, 34(3):1427–1445, 2020.
  24. A review of topic modeling methods. Information Systems, 94:101582, 2020.
  25. An overview of user feedback classification approaches. In REFSQ Workshops, volume 3, pages 357–369, 2019.
  26. Interactive topic modeling. Machine learning, 95:423–469, 2014.
  27. Evaluating pre-trained models for user feedback analysis in software engineering: A study on classification of app-reviews. Empirical Software Engineering, 28(4):88, 2023.
  28. Integrating document clustering and topic modeling. In Proceedings of the Twenty-Ninth Conference on Uncertainty in Artificial Intelligence, pages 694–703, 2013.
  29. Emerging app issue identification from user feedback: Experience on wechat. In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pages 279–288. IEEE, 2019.
  30. Emerging app issue identification via online joint sentiment-topic tracing. IEEE Transactions on Software Engineering, 48(8):3025–3043, 2021.
  31. Too many user-reviews! what should app developers look at first? IEEE Transactions on Software Engineering, 47(2):367–378, 2019.
  32. How do users like this feature? a fine grained sentiment analysis of app reviews. In 2014 IEEE 22nd international requirements engineering conference (RE), pages 153–162. Ieee, 2014.
  33. Causal impact analysis for app releases in google play. In Proceedings of the 2016 24th ACM SIGSOFT international symposium on foundations of software engineering, pages 435–446, 2016.
  34. Helpfulness prediction for vr application reviews: Exploring topic signals for causal inference. In 2022 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), pages 17–21. IEEE, 2022.
  35. Mobile app evolution analysis based on user reviews. In New Trends in Intelligent Software Methodologies, Tools and Techniques, pages 773–786. IOS Press, 2018.
  36. “what apps did you use?”: Understanding the long-term evolution of mobile app usage. In Proceedings of the web conference 2020, pages 66–76, 2020.
  37. OpenAI. GPT-4 technical report, 2023.
  38. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  39. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019.
  40. Database meets artificial intelligence: A survey. IEEE Transactions on Knowledge and Data Engineering, 34(3):1096–1116, 2020.
  41. Distance weighted cosine similarity measure for text classification. In Intelligent Data Engineering and Automated Learning–IDEAL 2013: 14th International Conference, IDEAL 2013, Hefei, China, October 20-23, 2013. Proceedings 14, pages 611–618. Springer, 2013.
  42. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  43. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942, 2019.
  44. Understanding bag-of-words model: a statistical framework. International journal of machine learning and cybernetics, 1:43–52, 2010.
  45. Maarten Grootendorst. Bertopic: Neural topic modeling with a class-based tf-idf procedure. arXiv preprint arXiv:2203.05794, 2022.
  46. Benchmarking neural topic models: An empirical study. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4363–4368, 2021.
  47. A novel neural topic model and its supervised extension. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
  48. Parth Mehta. From extractive to abstractive summarization: A journey. In ACL (Student Research Workshop), pages 100–106. Springer, 2016.
  49. Movie review mining and summarization. In Proceedings of the 15th ACM international conference on Information and knowledge management, pages 43–50, 2006.
  50. Benchmarking large language models for news summarization. arXiv preprint arXiv:2301.13848, 2023.
  51. Daniel Müllner. Modern hierarchical, agglomerative clustering algorithms. arXiv preprint arXiv:1109.2378, 2011.
  52. Bartscore: Evaluating generated text as text generation. Advances in Neural Information Processing Systems, 34:27263–27277, 2021.
  53. TaskWeaver: A code-first agent framework. arXiv preprint arXiv:2311.17541, 2023.
  54. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837, 2022.
  55. Everything of thoughts: Defying the law of penrose triangle for thought generation. arXiv preprint arXiv:2311.04254, 2023.
  56. UFO: A UI-Focused Agent for Windows OS Interaction. arXiv preprint arXiv:2402.07939, 2024.
  57. Online app review analysis for identifying emerging issues. In Proceedings of the 40th International Conference on Software Engineering, pages 48–58, 2018.
  58. The multipurpose application wechat: a review on recent research. Frontiers in psychology, 9:2247, 2018.
  59. Unsupervised summarization of privacy concerns in mobile application reviews. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, pages 1–12, 2022.
  60. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.11366, 2023.
  61. Lorena A Barba. The python/jupyter ecosystem: Today’s problem-solving environment for computational science. Computing in Science & Engineering, 23(3):5–9, 2021.
  62. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
  63. AR-miner: mining informative reviews for developers from mobile app marketplace. In Proceedings of the 36th international conference on software engineering, pages 767–778, 2014.
  64. Can a conversation paint a picture? mining requirements in software forums. In 2019 IEEE 27th International Requirements Engineering Conference (RE), pages 17–27. IEEE, 2019.
  65. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019.
  66. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations, 2020.
  67. Ro{bert}a: A robustly optimized {bert} pretraining approach, 2020.
  68. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451, 2020.
  69. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
  70. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):993–1022, 2003.
  71. Sharing clusters among related groups: Hierarchical dirichlet processes. Advances in neural information processing systems, 17, 2004.
  72. Algorithms for non-negative matrix factorization. Advances in neural information processing systems, 13, 2000.
  73. Autoencoding variational inference for topic models. arXiv preprint arXiv:1703.01488, 2017.
  74. Cross-lingual contextualized topic models with zero-shot learning. arXiv preprint arXiv:2004.07737, 2020.
  75. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551, 2020.
  76. Using word embedding to evaluate the coherence of topics from twitter data. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, pages 1057–1060, 2016.
  77. Analysing app reviews for software engineering: a systematic literature review. Empirical Software Engineering, 27(2):43, 2022.
  78. Demystifying “removed reviews” in ios app store. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 1489–1499, 2022.
  79. A survey on the use of topic models when mining software repositories. Empirical Software Engineering, 21:1843–1919, 2016.
  80. Mining user feedback for software engineering: Use cases and reference architecture. In 2022 IEEE 30th International Requirements Engineering Conference (RE), pages 114–126. IEEE, 2022.
  81. Evaluating unsupervised text embeddings on software user feedback. In 2021 IEEE 29th International Requirements Engineering Conference Workshops (REW), pages 87–95. IEEE, 2021.
  82. Listening to users’ voice: Automatic summarization of helpful app reviews. IEEE Transactions on Reliability, 2022.
  83. Empowering practical root cause analysis by large language models for cloud incidents. arXiv preprint arXiv:2305.15778, 2023.
  84. Assess and summarize: Improve outage understanding with large language models. arXiv preprint arXiv:2305.18084, 2023.
  85. Xpert: Empowering incident management with query recommendations via large language models. arXiv preprint arXiv:2312.11988, 2023.
  86. Where is your app frustrating users? In Proceedings of the 44th International Conference on Software Engineering, pages 2427–2439, 2022.
  87. Advancing requirements engineering through generative ai: Assessing the role of llms. arXiv preprint arXiv:2310.13976, 2023.
  88. Automated question answering for improved understanding of compliance requirements: A multi-document study. In 2022 IEEE 30th International Requirements Engineering Conference (RE), pages 39–50. IEEE, 2022.
  89. PersonaGen: A tool for generating personas from user feedback. In 2023 IEEE 31st International Requirements Engineering Conference (RE), pages 353–354. IEEE, 2023.
Citations (1)
View on Semantic Scholar

Summary

  • The paper introduces AllHands, a framework that employs LLMs to classify, model topics, and answer queries on large-scale feedback.
  • It utilizes few-shot learning and a human-in-the-loop process to achieve high classification accuracy and coherent abstractive topic summaries.
  • The system integrates a natural language QA agent that translates user queries into executable code, offering multi-modal responses and surpassing traditional methods.

AllHands: Leveraging LLMs for Comprehensive Analysis of Large-scale Verbatim Feedback

Introduction

The exponential growth of user-generated content, such as product reviews and feedback, presents both opportunities and challenges for data analysis. Traditional approaches for extracting insights from verbatim feedback often require substantial human effort and domain-specific model training, presenting barriers to efficiency and generalization. The paper introduces AllHands, a novel analytic framework utilizing LLMs to address these challenges. By integrating classification, abstractive topic modeling, and a natural language interface for question answering, AllHands facilitates an end-to-end solution for insight extraction from large-scale feedback.

Feedback Classification and Abstractive Topic Modeling

Classification with LLMs

AllHands employs LLMs for feedback classification, leveraging few-shot learning with in-context examples to categorize feedback into predefined dimensions. This approach minimizes the need for extensive labeled data and domain-specific model training, enhancing the framework's applicability across various contexts.

Abstractive Topic Modeling

AllHands introduces a novel method for abstractive topic modeling, utilizing LLMs to summarize feedback into human-readable topics. This method, enhanced by a human-in-the-loop refinement process, not only increases the relevancy and coherence of topics but also ensures that they adhere to user-defined criteria. The abstractive approach represents a significant improvement over traditional extractive methods, which often struggle with polysemy and lack human readability.

"Ask Me Anything" with an LLM-based QA Agent

The integration of an LLM-based QA Agent is a pivotal component of AllHands. This agent interprets user queries in natural language, translating them into executable Python code to query the structured feedback database, and returns comprehensive answers in various formats including text, code, tables, and images. This multi-modal response capability enables a truly "ask me anything" experience for users, significantly broadening the scope and depth of feedback analysis.

System Evaluation

AllHands was systematically evaluated across three diverse feedback datasets, demonstrating superior performance in feedback classification, abstractive topic modeling, and question answering. Employing GPT-4, AllHands consistently outperformed various state-of-the-art baselines in classification accuracy and topic modeling. Furthermore, user queries spanning a wide range of topics and complexities were effectively addressed, showcasing the system's robustness and versatility.

Implications and Future Directions

The implications of incorporating LLMs into feedback analysis are profound. AllHands not only streamlines the analysis workflow but also democratizes access to comprehensive data insights by lowering the barrier to entry for users without coding expertise. Looking ahead, future research could explore the integration of more advanced LLMs or refine the human-in-the-loop refinement process to further enhance topic modeling quality. Additionally, expanding AllHands to include real-time feedback streams could open new avenues for dynamic and adaptive product development insights.

Conclusion

AllHands represents a significant advancement in the use of LLMs for feedback analysis, offering a seamless, comprehensive, and user-friendly solution for extracting valuable insights from vast quantities of verbatim feedback. Its robust classification, innovative abstractive topic modeling, and flexible QA capabilities set a new standard for feedback analysis frameworks, promising to revolutionize how organizations and researchers approach user-generated content analysis.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown

Whiteboard

Open Problems

We found no open problems mentioned in this paper.

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Tweets

Sign up for free to view the 8 tweets with 118 likes about this paper.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up for Free