Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
158 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

QuaLLM: An LLM-based Framework to Extract Quantitative Insights from Online Forums (2405.05345v1)

Published 8 May 2024 in cs.CL and cs.HC

Abstract: Online discussion forums provide crucial data to understand the concerns of a wide range of real-world communities. However, the typical qualitative and quantitative methods used to analyze those data, such as thematic analysis and topic modeling, are infeasible to scale or require significant human effort to translate outputs to human readable forms. This study introduces QuaLLM, a novel LLM-based framework to analyze and extract quantitative insights from text data on online forums. The framework consists of a novel prompting methodology and evaluation strategy. We applied this framework to analyze over one million comments from two Reddit's rideshare worker communities, marking the largest study of its type. We uncover significant worker concerns regarding AI and algorithmic platform decisions, responding to regulatory calls about worker insights. In short, our work sets a new precedent for AI-assisted quantitative data analysis to surface concerns from online forums.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (55)
  1. Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In Proceedings of the 2018 CHI conference on human factors in computing systems. 1–18.
  2. From sparse to dense: GPT-4 summarization with chain of density prompting. arXiv preprint arXiv:2309.04269 (2023).
  3. The illusion of artificial inclusion. arXiv preprint arXiv:2401.08572 (2024).
  4. Hussam Alkaissi and Samy I McFarlane. 2023. Artificial hallucinations in ChatGPT: implications in scientific writing. Cureus 15, 2 (2023).
  5. Meor Amer. 2023. Evaluating LLM Outputs. https://txt.cohere.com/evaluating-llm-outputs/
  6. Exploring Qualitative Research Using LLMs. arXiv preprint arXiv:2306.13298 (2023).
  7. On the dangers of stochastic parrots: Can language models be too big?. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency. 610–623.
  8. Data and Algorithms at Work. (2021). https://laborcenter.berkeley.edu/wp-content/uploads/2021/11/Data-and-Algorithms-at-Work.pdf
  9. Virginia Braun and Victoria Clarke. 2012. Thematic analysis. American Psychological Association.
  10. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901.
  11. LLM-assisted content analysis: Using large language models to support deductive coding. arXiv preprint arXiv:2306.14924 (2023).
  12. Harry M Collins. 2005. What is tacit knowledge? In The practice turn in contemporary theory. Routledge, 115–128.
  13. LLM-in-the-loop: Leveraging large language model for thematic analysis. arXiv preprint arXiv:2310.15100 (2023).
  14. No echo in the chambers of political interactions on Reddit. Scientific reports 11, 1 (2021), 2818.
  15. LLMs to the Moon? Reddit Market Sentiment Analysis with Large Language Models. In Companion Proceedings of the ACM Web Conference 2023. 1014–1019.
  16. Norman K Denzin. 2001. Interpretive interactionism. Vol. 16. Sage.
  17. Veena Dubal. 2023. On algorithmic wage discrimination. Available at SSRN 4331080 (2023).
  18. Zackary Okun Dunivin. 2024. Scalable Qualitative Coding with LLMs: Chain-of-Thought Reasoning Matches Human Performance in Some Hermeneutic Tasks. arXiv preprint arXiv:2401.15170 (2024).
  19. “So what if ChatGPT wrote it?” Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy. International Journal of Information Management 71 (2023), 102642.
  20. Statistical methods for rates and proportions. john wiley & sons.
  21. Mwenda J Gichuru. 2017. The interpretive research paradigm: A critical review of is research methodologies. International Journal of Innovative Research and Advanced Studies (IJIRAS) 4, 2 (2017), 1–5.
  22. The discovery of grounded theory; strategies for qualitative research. Nursing research 17, 4 (1968), 364.
  23. Stigma of HIV testing on online HIV forums: self-stigma and the unspoken. Journal of Psychosocial Nursing and Mental Health Services 55, 12 (2017), 34–43.
  24. Jodi Kantor and Arya Sundaram. 2022. The Rise of the Worker Productivity Score (New York Times). https://www.nytimes.com/interactive/2022/08/14/business/worker-productivity-tracking.html
  25. Meng Li and Fei Gao. 2003. Why Nonaka highlights tacit knowledge: a critical review. Journal of knowledge management 7, 4 (2003), 6–14.
  26. Using stakeholder theory to examine drivers’ stake in Uber. In Proceedings of the 2018 CHI conference on human factors in computing systems. 1–12.
  27. Marisa Alise Madsen and Dag Øivind Madsen. 2022. Communication between Parents and Teachers of Special Education Students: A Small Exploratory Study of Reddit Posts. Social Sciences 11, 11 (2022), 518.
  28. Bronislaw Malinowski. 1929. Practical anthropology. Africa 2, 1 (1929), 22–38.
  29. Nahema Marchal. 2020. The polarizing potential of intergroup affect in online political discussions: Evidence from reddit r/politics. Politics (July 10, 2020) (2020).
  30. Can generalist foundation models outcompete special-purpose tuning? case study in medicine. arXiv preprint arXiv:2311.16452 (2023).
  31. OpenAI. 2023. Prompt Engineering. https://platform.openai.com/docs/guides/prompt-engineering
  32. Upvotes? Downvotes? No Votes? Understanding the relationship between reaction mechanisms and political discourse on Reddit. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–28.
  33. Harnessing reddit to understand the written-communication challenges experienced by individuals with mental health disorders: analysis of texts from mental health communities. Journal of medical Internet research 20, 4 (2018), e8219.
  34. The positivism paradigm of research. Academic medicine 95, 5 (2020), 690–694.
  35. Christina G Paxman. 2021. “Everyone thinks I am just lazy”: Legitimacy narratives of Americans suffering from fibromyalgia. Health 25, 1 (2021), 121–137.
  36. TopicGPT: A Prompt-based Topic Modeling Framework. arXiv preprint arXiv:2311.01449 (2023).
  37. Studying reddit: A systematic overview of disciplines, approaches, methods, and ethics. Social Media+ Society 7, 2 (2021), 20563051211019004.
  38. Large language models are effective text rankers with pairwise ranking prompting. arXiv preprint arXiv:2306.17563 (2023).
  39. Alex Rosenblat and Luke Stark. 2016. Algorithmic labor and information asymmetries: A case study of Uber’s drivers. International journal of communication 10 (2016), 27.
  40. ChatGPT: Bullshit spewer or the end of traditional assessments in higher education? Journal of Applied Learning and Teaching 6, 1 (2023).
  41. Johnny Saldaña. 2021. The coding manual for qualitative researchers. The coding manual for qualitative researchers (2021), 1–440.
  42. Privacy, surveillance, and power in the gig economy. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 1–15.
  43. An Examination of the Use of Large Language Models to Aid Analysis of Textual Data. bioRxiv (2023), 2023–07.
  44. Grammar Prompting for Domain-Specific Language Generation with Large Language Models. arXiv preprint arXiv:2305.19234 (2023).
  45. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 (2022).
  46. Elizabeth Anne Watkins. 2022. “Have you learned your lesson?” Communities of practice under algorithmic competition. New Media & Society 24, 7 (2022), 1567–1590.
  47. Max Weber. 1949. ” Objectivity” in social science and social policy. The methodology of the social sciences (1949), 49–112.
  48. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems 35 (2022), 24824–24837.
  49. White House (OSTP). 2023. Request for Information; Automated Worker Surveillance and Management. https://www.federalregister.gov/documents/2023/05/03/2023-09353/request-for-information-automated-worker-surveillance-and-management
  50. Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding. In Companion Proceedings of the 28th International Conference on Intelligent User Interfaces. 75–78.
  51. Technology-Mediated Strategies for Coping with Mental Health Challenges: Insights from People with Bipolar Disorder. Proceedings of the ACM on Human-Computer Interaction 7, CSCW2 (2023), 1–31.
  52. Together but alone: Atomization and peer support among gig workers. Proceedings of the ACM on Human-Computer Interaction 5, CSCW2 (2021), 1–29.
  53. Samaneh Zamanifard and Andrew Robb. 2023. Social Virtual Reality Is My Therapist: Overcoming Social Anxiety Disorder Through Using Social Virtual Reality. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems. 1–6.
  54. Algorithmic management reimagined for workers and by workers: Centering worker well-being in gig work. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 1–20.
  55. A survey of large language models. arXiv preprint arXiv:2303.18223 (2023).
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now