Papers
Topics
Authors
Recent
Search
2000 character limit reached

Focus Agent: LLM-Powered Virtual Focus Group

Published 3 Sep 2024 in cs.HC | (2409.01907v1)

Abstract: In the domain of Human-Computer Interaction, focus groups represent a widely utilised yet resource-intensive methodology, often demanding the expertise of skilled moderators and meticulous preparatory efforts. This study introduces the ``Focus Agent,'' a LLM powered framework that simulates both the focus group (for data collection) and acts as a moderator in a focus group setting with human participants. To assess the data quality derived from the Focus Agent, we ran five focus group sessions with a total of 23 human participants as well as deploying the Focus Agent to simulate these discussions with AI participants. Quantitative analysis indicates that Focus Agent can generate opinions similar to those of human participants. Furthermore, the research exposes some improvements associated with LLMs acting as moderators in focus group discussions that include human participants.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (66)
  1. Let the llms talk: Simulating human-to-human conversational qa via zero-shot llm-to-llm interactions. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining. 8–17.
  2. WhisperX: Time-Accurate Speech Transcription of Long-Form Audio. INTERSPEECH 2023 (2023).
  3. On the dangers of stochastic parrots: Can language models be too big?. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency. 610–623.
  4. Narelle Biedermann. 2018. The use of Facebook for virtual asynchronous focus groups in qualitative research. Contemporary nurse 54, 1 (2018), 26–34.
  5. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901.
  6. Elisabeth Brüggen and Pieter Willems. 2009. A critical comparison of offline focus groups, online focus groups and e-Delphi. International Journal of Market Research 51, 3 (2009), 1–15.
  7. Jean Carletta. 2006. Announcing the AMI meeting corpus. The ELRA Newsletter 11, 1 (2006), 3–5.
  8. Julienne Chen and Pearlyn Neo. 2019. Texting the waters: An assessment of focus groups conducted via the WhatsApp smartphone messaging application. Methodological Innovations 12, 3 (2019), 2059799119884276.
  9. Simulating Opinion Dynamics with Networks of LLM-based Agents. arXiv preprint arXiv:2311.09618 (2023).
  10. Edward J Ciaccio. 2023. Use of artificial intelligence in scientific paper writing. , 101253 pages.
  11. OpenCompass Contributors. 2023. OpenCompass: A Universal Evaluation Platform for Foundation Models. https://github.com/InternLM/OpenCompass.
  12. STEER: Factors to consider when designing online focus groups using audiovisual technology in health research. International Journal of Qualitative Methods 18 (2019), 1609406919885786.
  13. Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification. arXiv preprint arXiv:2005.07143 (2020).
  14. Towards next-generation intelligent assistants leveraging llm techniques. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 5792–5793.
  15. Virtual reality games for people using wheelchairs. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–11.
  16. Asynchronous online focus groups for health research: case study and lessons learned. International journal of qualitative methods 20 (2021), 1609406921990489.
  17. Marie-France Gratton and Susan O’Donnell. 2011. Communication technologies for focus groups with remote communities: a case study of research with First Nations in Canada. Qualitative Research 11, 2 (2011), 159–175.
  18. How many focus groups are enough? Building an evidence base for nonprobability sample sizes. Field methods 29, 1 (2017), 3–22.
  19. Claudia E Haupt and Mason Marks. 2023. AI-generated medical advice—GPT and beyond. Jama 329, 16 (2023), 1349–1350.
  20. End-to-end speaker diarization for an unknown number of speakers with encoder-decoder based attractors. arXiv preprint arXiv:2005.09921 (2020).
  21. End-to-end speaker diarization as post-processing. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 7188–7192.
  22. End-to-end speaker diarization as post-processing. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 7188–7192.
  23. Kristina Jokinen and Michael McTear. 2022. Spoken dialogue systems. Springer Nature.
  24. Kristina Jokinen and Michael McTear. 2022. Spoken dialogue systems. Springer Nature.
  25. Evaluation of GPT-3 AI language model in research paper writing. Turkish Journal of Science and Technology 18, 2 (2023), 311–318.
  26. Evaluation of GPT-3 AI language model in research paper writing. Turkish Journal of Science and Technology 18, 2 (2023), 311–318.
  27. Gpt-4 passes the bar exam. Philosophical Transactions of the Royal Society A 382, 2270 (2024), 20230254.
  28. From challenge to opportunity: virtual qualitative research during COVID-19 and beyond. International Journal of Qualitative Methods 21 (2022), 16094069221105075.
  29. Jenny Kitzinger. 1994. The methodology of focus groups: the importance of interaction between research participants. Sociology of health & illness 16, 1 (1994), 103–121.
  30. Jenny Kitzinger. 1995. Qualitative research: introducing focus groups. Bmj 311, 7000 (1995), 299–302.
  31. Anis Koubaa. 2023. GPT-4 vs. GPT-3.5: A concise showdown. (2023).
  32. Can large language models provide useful feedback on research papers? A large-scale empirical analysis. arXiv preprint arXiv:2310.01783 (2023).
  33. Riccardo Mazza. 2006. Evaluating information visualization applications with focus groups: the CourseVis experience. In Proceedings of the 2006 AVI workshop on BEyond time and errors: novel evaluation methods for information visualization. 1–6.
  34. Target-speaker voice activity detection: a novel approach for multi-speaker diarization in a dinner party scenario. arXiv preprint arXiv:2005.07272 (2020).
  35. Barry Nagle and Nichelle Williams. 2013. Methodology brief: Introduction to focus groups. Center for Assessment, Planning and Accountability 1-12 (2013).
  36. Large language models as tax attorneys: a case study in legal capabilities emergence. Philosophical Transactions of the Royal Society A 382, 2270 (2024), 20230159.
  37. Contrasting internet and face-to-face focus groups for children with chronic health conditions: Outcomes and participant experiences. International Journal of Qualitative Methods 9, 1 (2010), 105–121.
  38. R OpenAI. 2023. GPT-4 technical report. arXiv (2023), 2303–08774.
  39. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442 (2023).
  40. Robust Speech Recognition via Large-Scale Weak Supervision. arXiv:2212.04356 [eess.AS]
  41. Laria Reynolds and Kyle McDonell. 2021. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. 1–7.
  42. A guide to conducting online focus groups via Reddit. International journal of qualitative methods 20 (2021), 16094069211012217.
  43. Focus groups in HCI: wealth of information or waste of resources?. In CHI’02 extended abstracts on human factors in computing systems. 702–703.
  44. Daniel Rough and Benjamin Cowan. 2020. Don’t Believe The Hype! White Lies of Conversational User Interface Creation Tools. In Proceedings of the 2nd Conference on Conversational User Interfaces. 1–3.
  45. Malik Sallam. 2023. ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. In Healthcare, Vol. 11. MDPI, 887.
  46. End-of-life decisions: A focus group study with German health professionals from human and veterinary medicine. Frontiers in Veterinary Science 10 (2023), 1044561.
  47. Role play with large language models. Nature 623, 7987 (2023), 493–498.
  48. Using focus groups in medical education research: AMEE Guide No. 91. Medical teacher 36, 11 (2014), 923–939.
  49. David W Stewart and Prem Shamdasani. 2017. Online focus groups. Journal of Advertising 46, 1 (2017), 48–60.
  50. David W Stewart and Prem N Shamdasani. 2014. Focus groups: Theory and practice. Vol. 20. Sage publications.
  51. Yashar Talebirad and Amirhossein Nadiri. 2023. Multi-agent collaboration: Harnessing the power of intelligent llm agents. arXiv preprint arXiv:2306.03314 (2023).
  52. Systematic biases in LLM simulations of debates. arXiv preprint arXiv:2402.04049 (2024).
  53. Do we trust in AI? Role of anthropomorphism and intelligence. Journal of Computer Information Systems 61, 5 (2021), 481–491.
  54. Lyn Turney and Catherine Pocknee. 2005. Virtual focus groups: New frontiers in research. International Journal of Qualitative Methods 4, 2 (2005), 32–43.
  55. Braian Veloso. 2020. WHATSAPP COMO FERRAMENTA PARA A ORGANIZAÇÃO DE GRUPOS FOCAIS ONLINE NA PESQUISA DA EDUCAÇÃO: UM RELATO DE EXPERIÊNCIA. In Anais do CIET: EnPED: 2020-(Congresso Internacional de Educação e Tecnologias— Encontro de Pesquisadores em Educação a Distância).
  56. A survey on large language model based autonomous agents. arXiv preprint arXiv:2308.11432 (2023).
  57. Aligning large language models with human: A survey. arXiv preprint arXiv:2307.12966 (2023).
  58. CCNet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint arXiv:1911.00359 (2019).
  59. Recommendations for internet-based qualitative health research with hard-to-reach populations. Qualitative health research 24, 4 (2014), 561–574.
  60. Computer-mediated communication to facilitate synchronous online focus group discussions: feasibility study for qualitative HIV research among transgender women across the United States. Journal of medical Internet research 21, 3 (2019), e12569.
  61. Achieving human parity in conversational speech recognition. arXiv preprint arXiv:1610.05256 (2016).
  62. Exploring large language models for communication games: An empirical study on werewolf. arXiv preprint arXiv:2309.04658 (2023).
  63. Hallucination is inevitable: An innate limitation of large language models. arXiv preprint arXiv:2401.11817 (2024).
  64. Mirror: A Multiple-perspective Self-Reflection Method for Knowledge-rich Reasoning. arXiv preprint arXiv:2402.14963 (2024).
  65. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601 (2023).
  66. Large language models for information retrieval: A survey. arXiv preprint arXiv:2308.07107 (2023).

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 3 tweets with 2 likes about this paper.

Sign Up for Free