Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Human-in-the-loop Evaluation for Early Misinformation Detection: A Case Study of COVID-19 Treatments (2212.09683v4)

Published 19 Dec 2022 in cs.CL

Abstract: We present a human-in-the-loop evaluation framework for fact-checking novel misinformation claims and identifying social media messages that support them. Our approach extracts check-worthy claims, which are aggregated and ranked for review. Stance classifiers are then used to identify tweets supporting novel misinformation claims, which are further reviewed to determine whether they violate relevant policies. To demonstrate the feasibility of our approach, we develop a baseline system based on modern NLP methods for human-in-the-loop fact-checking in the domain of COVID-19 treatments. We make our data and detailed annotation guidelines available to support the evaluation of human-in-the-loop systems that identify novel misinformation directly from raw user-generated content.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (84)
  1. Jon Agley and Yunyu Xiao. 2021. Misinformation about COVID-19: evidence for differential latent profiles and a strong association with trust in science. BMC Public Health.
  2. Multitask instruction-based prompting for fallacy recognition. In Proceedings of Empirical Methods in Natural Language Processing.
  3. The fact extraction and VERification over unstructured and structured information (FEVEROUS) shared task. In Proceedings of the Fourth Workshop on Fact Extraction and VERification (FEVER).
  4. Rami Aly and Andreas Vlachos. 2022. Natural logic-guided autoregressive multi-hop document retrieval for fact verification. In Proceedings of Empirical Methods in Natural Language Processing.
  5. Ron Artstein and Massimo Poesio. 2008. Survey article: Inter-coder agreement for computational linguistics. Computational Linguistics.
  6. Powergrading: a clustering approach to amplify human effort for short answer grading.
  7. Twitter-COMMs: Detecting climate, COVID, and military multimodal misinformation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics.
  8. Rumor detection on social media with bi-directional graph convolutional networks. In AAAI Conference on Artificial Intelligence.
  9. CDC. 2021. How to protect yourself and others.
  10. Generating literal and implied subquestions to fact-check complex claims. In Proceedings of Empirical Methods in Natural Language Processing.
  11. Yuanzhi Chen and Mohammad Hasan. 2021. Navigating the kaleidoscope of COVID-19 misinformation using deep learning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.
  12. Towards automatic fake news detection: Cross-level stance detection in news articles. In Proceedings of the First Workshop on Fact Extraction and VERification (FEVER), Brussels, Belgium. Association for Computational Linguistics.
  13. SemEval-2017 task 8: RumourEval: Determining rumour veracity and support for rumours. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017).
  14. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
  15. Climate-FEVER: A dataset for verification of real-world climate claims. In Proceedings of the NeurIPS 2020 Workshop: Tackling Climate Change with Machine Learning.
  16. Tweetscov19 - a knowledge base of semantically annotated tweets about the covid-19 pandemic. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management.
  17. Xinya Du and Claire Cardie. 2020. Event extraction by answering (almost) natural questions. In Proceedings of Empirical Methods in Natural Language Processing.
  18. QA-driven zero-shot slot filling with weak supervision pretraining. In Proceedings of the Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing.
  19. Neural end-to-end learning for computational argumentation mining. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
  20. Ronald A. Fisher. 1922. On the interpretation of χ𝜒\chiitalic_χ2 from contingency tables, and the calculation of P.
  21. Stance Detection in COVID-19 Tweets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing.
  22. Missing counter-evidence renders nlp fact-checking unrealistic for misinformation. Proceedings of Empirical Methods in Natural Language Processing.
  23. SemEval-2019 task 7: RumourEval, determining rumour veracity and support for rumours. In Proceedings of the 13th International Workshop on Semantic Evaluation.
  24. PASTA: Table-operations aware fact verification via sentence-table cloze pre-training. In Proceedings of Empirical Methods in Natural Language Processing.
  25. A Survey on Automated Fact-Checking. In Transactions of the Association for Computational Linguistics.
  26. Neural check-worthiness ranking with weak supervision: Finding sentences for fact-checking. Association for Computing Machinery.
  27. Toward automated fact-checking: Detecting check-worthy factual claims by claimbuster.
  28. COVIDLies: Detecting COVID-19 misinformation on social media. In Proceedings of the 1st Workshop on NLP for COVID-19 at EMNLP.
  29. ClaimRank: Detecting check-worthy claims in Arabic and English. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
  30. HoVer: A dataset for many-hop fact extraction and claim verification. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online. Association for Computational Linguistics.
  31. Improving translation quality by discarding most of the phrasetable. In Proceedings of Empirical Methods in Natural Language Processing.
  32. Can you verifi this? studying uncertainty and decision-making about misinformation using visual analytics. In Twelfth international AAAI conference on web and social media.
  33. Toward automated factchecking: Developing an annotation schema and benchmark for consistent automated claim detection. Digital threats: research and practice.
  34. Crowd, expert & ai: A human-ai interactive approach towards natural language explanation based covid-19 misinformation detection. In International Joint Conferences on Artificial Intelligence Organization.
  35. Klaus Krippendorff. 2011. Computing Krippendorff’s alpha-reliability.
  36. On unifying misinformation detection. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
  37. Stephan Lewandowsky and Sander Van Der Linden. 2021. Countering misinformation and fake news through inoculation and prebunking. European Review of Social Psychology.
  38. COVID-19 claim radar: A structured claim extraction and tracking system. In Proceedings of the Association for Computational Linguistics.
  39. Rumor detection by exploiting user credibility information, attention and multi-task learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy. Association for Computational Linguistics.
  40. Real-time rumor debunking on twitter. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, New York, NY, USA. Association for Computing Machinery.
  41. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
  42. A general framework for information extraction using dynamic span graphs. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
  43. Detecting rumors from microblogs with recurrent neural networks. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence.
  44. Detect rumors in microblog posts using propagation structure via kernel learning. In Proceedings of the Association for Computational Linguistics. Association for Computational Linguistics.
  45. Fact-checking polarized politics: Does the fact-check industry provide consistent guidance on disputed realities? The Forum, 13(4):577–596.
  46. Robert C. Moore. 2004. On log-likelihood-ratios and the significance of rare events. In Proceedings of Empirical Methods in Natural Language Processing.
  47. COVID-Twitter-BERT: A natural language processing model to analyse COVID-19 content on Twitter. In arXiv preprint arXiv:2005.07503.
  48. Overview of the clef–2022 checkthat! lab on fighting the covid-19 infodemic and fake news detection. In Experimental IR Meets Multilinguality, Multimodality, and Interaction, Cham. Springer International Publishing.
  49. Automated fact-checking for assisting human fact-checkers. In International Joint Conference on Artificial Intelligence.
  50. Believe it or not: Designing a human-ai partnership for mixed-initiative fact-checking. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology. Association for Computing Machinery.
  51. Overview of constraint 2021 shared tasks: Detecting english covid-19 fake news and hindi hostile posts. In Combating Online Hostile Posts in Regional Languages during Emergency Situation.
  52. Gordon Pennycook and David G. Rand. 2019. Fighting misinformation on social media using crowdsourced judgments of news source quality. Proceedings of the National Academy of Sciences.
  53. What to pre-train on? Efficient intermediate task selection. In Proceedings of Empirical Methods in Natural Language Processing.
  54. Intermediate-task transfer learning with pretrained language models: When and why does it work? In Proceedings of the Association for Computational Linguistics.
  55. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research.
  56. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the Association for Computational Linguistics.
  57. Citation needed: A taxonomy and algorithmic assessment of wikipedia’s verifiability.
  58. Data-driven response generation in social media. In Proceedings of Empirical Methods in Natural Language Processing.
  59. Open domain event extraction from twitter. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining.
  60. CSI: A hybrid deep model for fake news detection. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. Association for Computing Machinery.
  61. Improving evidence retrieval for automated explainable fact-checking. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Demonstrations, Online. Association for Computational Linguistics.
  62. The risk of racial bias in hate speech detection. In Proceedings of the Association for Computational Linguistics.
  63. SAMS: Human-in-the-loop approach to combat the sharing of digital misinformation. In Proceedings of the AAAI 2021 Spring Symposium on Combining Machine Learning and Knowledge Engineering.
  64. An exploratory study of covid-19 misinformation on twitter. Online Social Networks and Media.
  65. Self-attention with relative position representations. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
  66. Fakenewsnet: A data repository with news content, social context, and spatiotemporal information for studying fake news on social media. Big Data.
  67. Matching the blanks: Distributional similarity for relation learning. In Proceedings of the Association for Computational Linguistics.
  68. Empowering the fact-checkers! automatic identification of claim spans on twitter. In Proceedings of Empirical Methods in Natural Language Processing.
  69. Fake news stance detection using stacked ensemble of classifiers. In Proceedings of the 2017 EMNLP Workshop: Natural Language Processing meets Journalism.
  70. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics.
  71. Twitter. 2021. Covid-19 misleading information policy.
  72. Rumor gauge: Predicting the veracity of rumors on twitter. ACM Transactions on Knowledge Discovery from Data.
  73. A DQN-based approach to finding precise evidences for fact verification. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online. Association for Computational Linguistics.
  74. Congcong Wang and David Lillis. 2020. UCD-CS at W-NUT 2020 shared task-3: A text to text approach for COVID-19 event extraction on social media. In Proceedings of the Sixth Workshop on Noisy User-generated Text.
  75. Dustin Wright and Isabelle Augenstein. 2020. Claim check-worthiness detection as positive unlabelled learning. In Findings of the Association for Computational Linguistics: EMNLP, Online. Association for Computational Linguistics.
  76. Cross-document misinformation detection based on event graph reasoning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
  77. A state-independent and time-evolving network for early rumor detection in social media. In Proceedings of Empirical Methods in Natural Language Processing.
  78. A convolutional approach for misinformation identification. In Proceedings of the 26th International Joint Conference on Artificial Intelligence.
  79. Contrastive domain adaptation for early misinformation detection: A case study on covid-19. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management.
  80. Fengzhu Zeng and Wei Gao. 2022. Early rumor detection using neural Hawkes process with a new benchmark dataset. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
  81. Early rumour detection. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota. Association for Computational Linguistics.
  82. Coronavirus drug and treatment tracker.
  83. Extracting a knowledge base of COVID-19 events from social media. Proceedings of the 29th International Conference on Computational Linguistics.
  84. Learning reporting dynamics during breaking news for rumour detection in social media. ArXiv, abs/1610.07363.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Ethan Mendes (4 papers)
  2. Yang Chen (535 papers)
  3. Wei Xu (535 papers)
  4. Alan Ritter (57 papers)
Citations (11)
View on Semantic Scholar