Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
110 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Stance Detection: A Practical Guide to Classifying Political Beliefs in Text (2305.01723v2)

Published 2 May 2023 in cs.CL

Abstract: Stance detection is identifying expressed beliefs in a document. While researchers widely use sentiment analysis for this, recent research demonstrates that sentiment and stance are distinct. This paper advances text analysis methods by precisely defining stance detection and presenting three distinct approaches: supervised classification, natural language inference, and in-context learning with generative LLMs. I discuss how document context and trade-offs between resources and workload should inform your methods. For all three approaches I provide guidance on application and validation techniques, as well as coding tutorials for implementation. Finally, I demonstrate how newer classification approaches can replicate supervised classifiers.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (67)
  1. (2022), “Superglue leaderboard.” URL https://super.gluebenchmark.com/leaderboard. Leaderboard accessed August 25, 2022.
  2. (2023a), “Prompt engineering.” Https://platform.openai.com/docs/guides/prompt-engineering.
  3. (2023b), “What are tokens and how to count them?” Https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them.
  4. Ackley, David H, Geoffrey E Hinton, and Terrence J Sejnowski (1985), “A learning algorithm for boltzmann machines.” Cognitive science, 9, 147–169.
  5. Aldayel, Abeer and Walid Magdy (2019), “Assessing sentiment of the expressed stance on social media.” In Social Informatics: 11th International Conference, SocInfo 2019, Doha, Qatar, November 18–21, 2019, Proceedings 11, 277–286, Springer.
  6. AlDayel, Abeer and Walid Magdy (2021), “Stance detection on social media: State of the art and trends.” Information Processing & Management, 58, 102597.
  7. Ash, Elliott and Stephen Hansen (2023), “Text algorithms in economics.” Annual Review of Economics, 15, 659–688.
  8. Barberá, Pablo (2015), “Birds of the same feather tweet together: Bayesian ideal point estimation using twitter data.” Political analysis, 23, 76–91.
  9. Barberá, Pablo (2016), “Less is more? how demographic sample weights can improve public opinion estimates based on twitter data.” Work Pap NYU.
  10. Barberá, Pablo, Amber E Boydstun, Suzanna Linn, Ryan McMahon, and Jonathan Nagler (2021), “Automated text classification of news articles: A practical guide.” Political Analysis, 29, 19–42.
  11. Bestvater, Samuel E and Burt L Monroe (2022), “Sentiment is not stance: Target-aware opinion classification for political text analysis.” Political Analysis, 1–22.
  12. Biber, Douglas and Edward Finegan (1988), “Adverbial stance types in english.” Discourse processes, 11, 1–34.
  13. Block Jr, Ray, Michael Burnham, Kayla Kahn, Rachel Peng, Jeremy Seeman, and Christopher Seto (2022), “Perceived risk, political polarization, and the willingness to follow covid-19 mitigation guidelines.” Social Science & Medicine, 115091.
  14. Bor, Alexander, Frederik Jørgensen, and Michael Bang Petersen (2023), “Discriminatory attitudes against unvaccinated people during the pandemic.” Nature, 613, 704–711.
  15. Brown, Tom, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. (2020), “Language models are few-shot learners.” Advances in neural information processing systems, 33, 1877–1901.
  16. Chalkidis, Ilias, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos (2020), “Legal-bert: The muppets straight out of law school.” arXiv preprint arXiv:2010.02559.
  17. Chicco, Davide and Giuseppe Jurman (2023), “The matthews correlation coefficient (mcc) should replace the roc auc as the standard metric for assessing binary classification.” BioData Mining, 16, 1–23.
  18. Cormack, Lindsey (2017), “Dcinbox—capturing every congressional constituent e-newsletter from 2009 onwards.” The Legislative Scholar, 2, 27–34.
  19. Cranmer, Skyler J and Bruce A Desmarais (2017), “What can we learn from predictive modeling?” Political Analysis, 25, 145–166.
  20. Dagan, Ido, Oren Glickman, and Bernardo Magnini (2005), “The pascal recognising textual entailment challenge.” In Machine Learning Challenges Workshop, 177–190, Springer.
  21. Denny, Matthew J and Arthur Spirling (2018), “Text preprocessing for unsupervised learning: Why it matters, when it misleads, and what to do about it.” Political Analysis, 26, 168–189.
  22. Eisenstein, Jacob and Regina Barzilay (2008), “Bayesian unsupervised topic segmentation.” In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, 334–343.
  23. Felmlee, Diane H, Justine I Blanford, Stephen A Matthews, and Alan M MacEachren (2020), “The geography of sentiment towards the women’s march of 2017.” Plos one, 15, e0233994.
  24. Gilardi, F, M Alizadeh, and M Kubli (2023), “Chatgpt outperforms crowd-workers for text-annotation tasks(arxiv: 2303.15056). arxiv.”
  25. Häffner, Sonja, Martin Hofer, Maximilian Nagl, and Julian Walterskirchen (2023), “Introducing an interpretable deep learning approach to domain-specific dictionary creation: A use case for conflict prediction.” Political Analysis, 1–19.
  26. He, Xingwei, Zhenghao Lin, Yeyun Gong, Alex Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, Weizhu Chen, et al. (2023), “Annollm: Making large language models to be better crowdsourced annotators.” arXiv preprint arXiv:2303.16854.
  27. Holtzman, Ari, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi (2020), “The curious case of neural text degeneration.”
  28. Hutto, Clayton and Eric Gilbert (2014), “Vader: A parsimonious rule-based model for sentiment analysis of social media text.” In Proceedings of the international AAAI conference on web and social media, volume 8, 216–225.
  29. Joseph, Kenneth, Lisa Friedland, William Hobbs, Oren Tsur, and David Lazer (2017), “Constance: Modeling annotation contexts to improve stance classification.” arXiv preprint arXiv:1708.06309.
  30. Kawintiranon, Kornraphop and Lisa Singh (2021), “Knowledge enhanced masked language model for stance detection.” In Proceedings of the 2021 conference of the north american chapter of the association for computational linguistics: human language technologies.
  31. Kawintiranon, Kornraphop and Lisa Singh (2022), “Polibertweet: A pre-trained language model for analyzing political content on twitter.” In Proceedings of the Language Resources and Evaluation Conference, European Language Resources Association.
  32. Koshorek, Omri, Adir Cohen, Noam Mor, Michael Rotman, and Jonathan Berant (2018), “Text segmentation as a supervised learning task.” arXiv preprint arXiv:1803.09337.
  33. Kristensen-McLachlan, Ross Deans, Miceal Canavan, Márton Kardos, Mia Jacobsen, and Lene Aarøe (2023), “Chatbots are not reliable text annotators.” arXiv preprint arXiv:2311.05769.
  34. Laurer, Moritz, Wouter van Atteveldt, Andreu Casas, and Kasper Welbers (2023), “Building efficient universal classifiers with natural language inference.”
  35. Laurer, Moritz, Wouter Van Atteveldt, Andreu Casas, and Kasper Welbers (2024), “Less annotating, more classifying: Addressing the data scarcity issue of supervised machine learning with deep transfer learning and bert-nli.” Political Analysis, 32, 84–100.
  36. Lee, Ernesto, Furqan Rustam, Patrick Bernard Washington, Fatima El Barakaz, Wajdi Aljedaani, and Imran Ashraf (2022), “Racism detection by analyzing differential opinions through sentiment analysis of tweets using stacked ensemble gcr-nn model.” IEEE Access, 10, 9717–9728.
  37. Liu, Alisa, Swabha Swayamdipta, Noah A Smith, and Yejin Choi (2022), “Wanli: Worker and ai collaboration for natural language inference dataset creation.” arXiv preprint arXiv:2201.05955.
  38. Liu, Bing et al. (2010), “Sentiment analysis and subjectivity.” Handbook of natural language processing, 2, 627–666.
  39. Liu, Yinhan, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov (2019), “Roberta: A robustly optimized bert pretraining approach.” arXiv preprint arXiv:1907.11692.
  40. Lu, Yao, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp (2021), “Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity.” arXiv preprint arXiv:2104.08786.
  41. Margatina, Katerina, Loic Barrault, and Nikolaos Aletras (2021), “On the importance of effectively adapting pretrained language models for active learning.” arXiv preprint arXiv:2104.08320.
  42. Miller, Blake, Fridolin Linder, and Walter R Mebane (2020), “Active learning approaches for labeling text: review and assessment of the performance of active learning approaches.” Political Analysis, 28, 532–551.
  43. Miyajiwala, Aamir, Arnav Ladkat, Samiksha Jagadale, and Raviraj Joshi (2022), “On sensitivity of deep learning based text classification algorithms to practical input perturbations.” In Science and Information Conference, 613–626, Springer.
  44. Mohammad, Saif, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry (2016), “Semeval-2016 task 6: Detecting stance in tweets.” In Proceedings of the 10th international workshop on semantic evaluation (SemEval-2016), 31–41.
  45. Nguyen, Dat Quoc, Thanh Vu, and Anh Tuan Nguyen (2020), “BERTweet: A pre-trained language model for English Tweets.” In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 9–14.
  46. Nie, Yixin, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela (2020), “Adversarial nli: A new benchmark for natural language understanding.” In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics.
  47. OpenAI (2023), “Gpt-4 technical report.” ArXiv, abs/2303.08774, URL https://arxiv.org/abs/2303.08774.
  48. Osmundsen, Mathias, Alexander Bor, Peter Bjerregaard Vahlstrup, Anja Bechmann, and Michael Bang Petersen (2021), “Partisan polarization is the primary psychological motivation behind political fake news sharing on twitter.” American Political Science Review, 115, 999–1015.
  49. Perez, Ethan, Douwe Kiela, and Kyunghyun Cho (2021), “True few-shot learning with language models.” Advances in neural information processing systems, 34, 11054–11070.
  50. Prabhu, Sumanth, Moosa Mohamed, and Hemant Misra (2021), “Multi-class text classification using bert-based active learning.” arXiv preprint arXiv:2104.14289.
  51. Riedl, Martin and Chris Biemann (2012), “Topictiling: a text segmentation algorithm based on lda.” In Proceedings of ACL 2012 student research workshop, 37–42.
  52. Sasaki, Akira, Junta Mizuno, Naoaki Okazaki, and Kentaro Inui (2016), “Stance classification by recognizing related events about targets.” In 2016 IEEE/WIC/ACM International Conference on Web Intelligence (WI), 582–587, IEEE.
  53. Sobhani, Parinaz, Diana Inkpen, and Xiaodan Zhu (2017), “A dataset for multi-target stance detection.” In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, 551–557.
  54. Spirling, Arthur (2023), “Why open-source generative ai models are an ethical way forward for science.” Nature, 616, 413–413.
  55. Stine, Robert A (2019), “Sentiment analysis.” Annual review of statistics and its application, 6, 287–308.
  56. Terechshenko, Zhanna, Fridolin Linder, Vishakh Padmakumar, Michael Liu, Jonathan Nagler, Joshua A Tucker, and Richard Bonneau (2020), “A comparison of methods in political science text classification: Transfer learning language models for politics.” Available at SSRN 3724644.
  57. Terman, Rochelle (2017), “Islamophobia and media portrayals of muslim women: A computational text analysis of us news coverage.” International Studies Quarterly, 61, 489–502.
  58. Törnberg, Petter (2023a), “Chatgpt-4 outperforms experts and crowd workers in annotating political twitter messages with zero-shot learning.” arXiv preprint arXiv:2304.06588.
  59. Törnberg, Petter (2023b), “How to use llms for text analysis.” arXiv preprint arXiv:2307.13106.
  60. Tunstall, Lewis, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. (2023), “Zephyr: Direct distillation of lm alignment.” arXiv preprint arXiv:2310.16944.
  61. van den Berg, Esther, Katharina Korfhage, Josef Ruppenhofer, Michael Wiegand, and Katja Markert (2019), “Not my president: How names and titles frame political figures.” In Proceedings of the Third Workshop on Natural Language Processing and Computational Social Science, 1–6.
  62. Wang, Yaqing, Quanming Yao, James T Kwok, and Lionel M Ni (2020), “Generalizing from a few examples: A survey on few-shot learning.” ACM computing surveys (csur), 53, 1–34.
  63. Wei, Jason, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. (2022), “Chain-of-thought prompting elicits reasoning in large language models.” Advances in Neural Information Processing Systems, 35, 24824–24837.
  64. Welleck, Sean, Ilia Kulikov, Jaedeok Kim, Richard Yuanzhe Pang, and Kyunghyun Cho (2020), “Consistency of a recurrent language model with respect to incomplete decoding.” arXiv preprint arXiv:2002.02492.
  65. Yin, Wenpeng, Jamaal Hay, and Dan Roth (2019), “Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach.” arXiv preprint arXiv:1909.00161.
  66. Zellers, Rowan, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi (2019), “Hellaswag: Can a machine really finish your sentence?” arXiv preprint arXiv:1905.07830.
  67. Zhao, Zihao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh (2021), “Calibrate before use: Improving few-shot performance of language models.” In International Conference on Machine Learning, 12697–12706, PMLR.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (1)
  1. Michael Burnham (4 papers)
Citations (5)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now