Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
153 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Reliability Criteria for News Websites (2407.03865v1)

Published 4 Jul 2024 in cs.HC

Abstract: Misinformation poses a threat to democracy and to people's health. Reliability criteria for news websites can help people identify misinformation. But despite their importance, there has been no empirically substantiated list of criteria for distinguishing reliable from unreliable news websites. We identify reliability criteria, describe how they are applied in practice, and compare them to prior work. Based on our analysis, we distinguish between manipulable and less manipulable criteria and compare politically diverse laypeople as end users and journalists as expert users. We discuss 11 widely recognized criteria, including the following 6 criteria that are difficult to manipulate: content, political alignment, authors, professional standards, what sources are used, and a website's reputation. Finally, we describe how technology may be able to support people in applying these criteria in practice to assess the reliability of websites.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (88)
  1. Foreign affairs and issue voting: Do presidential candidates “waltz before a blind audience?”. American Political Science Review 83, 1 (1989), 123–141.
  2. Birds of a Feather Don’t Fact-Check Each Other: Partisanship and the Evaluation of News in Twitter’s Birdwatch Crowdsourced Fact-Checking Program. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 245, 19 pages. https://doi.org/10.1145/3491102.3502040
  3. A survey of expert views on misinformation: Definitions, determinants, solutions, and future of the field. Harvard Kennedy School Misinformation Review 4, 4 (2023).
  4. “If This account is True, It is Most Enormously Wonderful”: Interestingness-If-True and the Sharing of True and False News. Digital Journalism 0, 0 (2021), 1–22. https://doi.org/10.1080/21670811.2021.1941163 arXiv:https://doi.org/10.1080/21670811.2021.1941163
  5. Understanding and reducing online misinformation across 16 countries on six continents. PsyArXiv.
  6. Fatemeh Torabi Asr and Maite Taboada. 2019. Big Data and quality data for fake news and misinformation detection. Big Data & Society 6, 1 (2019), 2053951719843310. https://doi.org/10.1177/2053951719843310 arXiv:https://doi.org/10.1177/2053951719843310
  7. Investigating Differences in Crowdsourced News Credibility Assessment: Raters, Tasks, and Expert Criteria. Proc. ACM Hum.-Comput. Interact. 4, CSCW2, Article 93 (Oct. 2020), 26 pages. https://doi.org/10.1145/3415164
  8. danah boyd. 2018. You Think You Want Media Literacy… Do You? https://points.datasociety.net/you-think-you-want-media-literacy-do-you-7cad6af18ec2
  9. Sourcing and automation of political news and information over social media in the United States, 2016-2018. Political Communication 37, 2 (2020), 173–193.
  10. Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology 3, 2 (jan 2006), 77–101. https://doi.org/10.1191/1478088706qp063oa
  11. Virginia Braun and Victoria Clarke. 2019. Reflecting on reflexive thematic analysis. Qualitative Research in Sport, Exercise and Health 11, 4 (2019), 589–597. https://doi.org/10.1080/2159676X.2019.1628806 arXiv:https://doi.org/10.1080/2159676X.2019.1628806
  12. Information Credibility on Twitter. In Proceedings of the 20th International Conference on World Wide Web (Hyderabad, India) (WWW ’11). Association for Computing Machinery, New York, NY, USA, 675–684. https://doi.org/10.1145/1963405.1963500
  13. A calibrated measure to compare fluctuations of different entities across timescales. Scientific reports 10, 1 (2020), 1–16.
  14. Juliet Corbin and Anselm Strauss. 2014. Basics of qualitative research: Techniques and procedures for developing grounded theory. Sage publications.
  15. M.X. Delli Carpini and S. Keeter. 1997. What Americans Know About Politics and Why it Matters. Yale University Press, New Haven.
  16. Beyond Red vs. Blue: The Political Typology.
  17. Judith Donath. 2007a. Signals, cues and meaning. Signals, Truth and Design (2007).
  18. Judith Donath. 2007b. Signals in Social Supernets. Journal of Computer-Mediated Communication 13, 1 (10 2007), 231–251. https://doi.org/10.1111/j.1083-6101.2007.00394.x arXiv:https://academic.oup.com/jcmc/article-pdf/13/1/231/22317022/jjcmcom0231.pdf
  19. Josefina Erikson and Cecilia Josefsson. 2019. Does higher education matter for MPs in their parliamentary work? Evidence from the Swedish Parliament. Representation 55, 1 (2019), 65–80.
  20. We only believe in news that we doctored ourselves. Social Psychology (2019).
  21. Falling for Fake News: Investigating the Consumption of News via Social Media. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–10. https://doi.org/10.1145/3173574.3173950
  22. What Makes Web Sites Credible? A Report on a Large Quantitative Study. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Seattle, Washington, USA) (CHI ’01). Association for Computing Machinery, New York, NY, USA, 61–68. https://doi.org/10.1145/365024.365037
  23. How Do Users Evaluate the Credibility of Web Sites? A Study with over 2,500 Participants. In Proceedings of the 2003 Conference on Designing for User Experiences (San Francisco, California) (DUX ’03). Association for Computing Machinery, New York, NY, USA, 1–15. https://doi.org/10.1145/997078.997097
  24. NELA-GT-2019: A Large Multi-Labelled News Dataset for The Study of Misinformation in News Articles. arXiv:2003.08444 [cs.CY]
  25. A digital media literacy intervention increases discernment between mainstream and false news in the United States and India. Proceedings of the National Academy of Sciences 117, 27 (2020), 15536–15545. https://doi.org/10.1073/pnas.1920498117 arXiv:https://www.pnas.org/doi/pdf/10.1073/pnas.1920498117
  26. A digital media literacy intervention increases discernment between mainstream and false news in the United States and India. Proceedings of the National Academy of Sciences 117, 27 (2020), 15536–15545.
  27. Hendrik Heuer and Elena Leah Glassman. 2022. A Comparative Evaluation of Interventions Against Misinformation: Augmenting the WHO Checklist. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 241, 21 pages. https://doi.org/10.1145/3491102.3517717
  28. Machine learning in tutorials – Universal applicability, underinformed application, and other misconceptions. Big Data & Society 8, 1 (2021), 20539517211017593. https://doi.org/10.1177/20539517211017593 arXiv:https://doi.org/10.1177/20539517211017593
  29. Resilience to Online Disinformation: A Framework for Cross-National Comparative Research. The International Journal of Press/Politics 25, 3 (2020), 493–516. https://doi.org/10.1177/1940161219900126 arXiv:https://doi.org/10.1177/1940161219900126
  30. Exploring Lightweight Interventions at Posting Time to Reduce the Sharing of Misinformation on Social Media. Proc. ACM Hum.-Comput. Interact. 5, CSCW1, Article 18 (apr 2021), 42 pages. https://doi.org/10.1145/3449092
  31. Shan Jiang and Christo Wilson. 2018. Linguistic Signals under Misinformation and Fact-Checking: Evidence from User Comments on Social Media. Proc. ACM Hum.-Comput. Interact. 2, CSCW, Article 82 (Nov. 2018), 23 pages. https://doi.org/10.1145/3274351
  32. Dan M Kahan. 2013. Ideology, motivated reasoning, and cognitive reflection. Judgment and Decision making 8, 4 (2013), 407–424.
  33. Jan Kirchner and Christian Reuter. 2020. Countering Fake News: A Comparison of Possible Solutions Regarding User Acceptance and Effectiveness. Proc. ACM Hum.-Comput. Interact. 4, CSCW2, Article 140 (Oct. 2020), 27 pages. https://doi.org/10.1145/3415211
  34. Mike Kuniavsky. 2003. Observing the user experience: a practitioner’s guide to user research. Elsevier.
  35. Hannu Kuusela and Pallab Paul. 2000. A comparison of concurrent and retrospective verbal protocol analysis. American journal of psychology 113, 3 (2000), 387–404.
  36. Airi Lampinen. 2015. Deceptively Simple: Unpacking the Notion of “Sharing”. Social Media + Society 1, 1 (2015), 2056305115578135. https://doi.org/10.1177/2056305115578135 arXiv:https://doi.org/10.1177/2056305115578135
  37. The science of fake news. Science 359, 6380 (2018), 1094–1096.
  38. Misinformation and Its Correction: Continued Influence and Successful Debiasing. Psychological Science in the Public Interest 13, 3 (2012), 106–131. https://doi.org/10.1177/1529100612451018 arXiv:https://doi.org/10.1177/1529100612451018 PMID: 26173286.
  39. Philipp Mayring. 2021. Qualitative content analysis: A step-by-step guide. Qualitative Content Analysis (2021), 1–100.
  40. Facebook Privacy-Protected Full URLs Data Set. https://doi.org/10.7910/DVN/TDOAPG
  41. Can Americans tell factual from opinion statements in the news. Pew Research Center’s Journalism Project. (2018). https://www.journalism.org/2018/06/18/distinguishing-between-factual-and-opinion-statements-in-the-news
  42. Many Americans Say Made-Up News Is a Critical Problem That Needs To Be Fixed. https://www.journalism.org/2019/06/05/many-americans-say-made-up-news-is-a-critical-problem-that-needs-to-be-fixed/
  43. Nic Newman. 2020. Overview and Key Findings of the 2020 Digital News Report. https://www.digitalnewsreport.org/survey/2020/overview-key-findings-2020/
  44. Reuters Institute Digital News Report 2022. (2022). https://reutersinstitute.politics.ox.ac.uk/digital-news-report/2022
  45. NewsGuard. 2022. Rating process and criteria. https://www.newsguardtech.com/ratings/rating-process-criteria/
  46. David Lynn Painter and Juliana Fernandes. 2022. “The Big Lie”: How Fact Checking Influences Support for Insurrection. American Behavioral Scientist (2022), 00027642221103179.
  47. Tackling misinformation: What researchers could do with social media data. The Harvard Kennedy School Misinformation Review (2020).
  48. Trustworthiness Criteria for Supporting Users to Assess the Credibility of Web Information. In Proceedings of the 22nd International Conference on World Wide Web (Rio de Janeiro, Brazil) (WWW ’13 Companion). Association for Computing Machinery, New York, NY, USA, 1123–1130. https://doi.org/10.1145/2487788.2488132
  49. Cyrus Peikari and Anton Chuvakin. 2004. Security Warrior: Know Your Enemy. ” O’Reilly Media, Inc.”.
  50. Prior exposure increases perceived accuracy of fake news. Journal of experimental psychology: general 147, 12 (2018), 1865.
  51. Shifting attention to accuracy can reduce misinformation online. Nature 592, 7855 (01 Apr 2021), 590–595. https://doi.org/10.1038/s41586-021-03344-2
  52. Gordon Pennycook and David G Rand. 2019a. Fighting misinformation on social media using crowdsourced judgments of news source quality. Proceedings of the National Academy of Sciences 116, 7 (2019), 2521–2526.
  53. Gordon Pennycook and David G Rand. 2019b. Lazy, not biased: Susceptibility to partisan fake news is better explained by lack of reasoning than by motivated reasoning. Cognition 188 (2019), 39–50.
  54. Gordon Pennycook and David G Rand. 2020. Who falls for fake news? The roles of bullshit receptivity, overclaiming, familiarity, and analytic thinking. Journal of personality 88, 2 (2020), 185–200.
  55. Identity concerns drive belief: The impact of partisan identity on the belief and dissemination of true and false news. Group Processes & Intergroup Relations 26, 1 (2023), 24–47.
  56. Automatic Detection of Fake News. In Proceedings of the 27th International Conference on Computational Linguistics. Association for Computational Linguistics, Santa Fe, New Mexico, USA, 3391–3401. https://www.aclweb.org/anthology/C18-1287
  57. A Stylometric Inquiry into Hyperpartisan and Fake News. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Melbourne, Australia, 231–240. https://doi.org/10.18653/v1/P18-1022
  58. Clickbait Detection. In Advances in Information Retrieval, Nicola Ferro, Fabio Crestani, Marie-Francine Moens, Josiane Mothe, Fabrizio Silvestri, Giorgio Maria Di Nunzio, Claudia Hauff, and Gianmaria Silvello (Eds.). Springer International Publishing, Cham, 810–817.
  59. RPC-Lex: A dictionary to measure German right-wing populist conspiracy discourse online. Convergence 0, 0 (0), 13548565221109440. https://doi.org/10.1177/13548565221109440 arXiv:https://doi.org/10.1177/13548565221109440
  60. George Rabinowitz and Stuart Elaine Macdonald. 1989. A directional theory of issue voting. American political science review 83, 1 (1989), 93–121.
  61. Kim Salazar. 2020. Contextual inquiry: inspire design by observing and interviewing users in their context. Nielsen Norman Group 6 (2020).
  62. Dietram A. Scheufele and Nicole M. Krause. 2019. Science audiences, misinformation, and fake news. Proceedings of the National Academy of Sciences 116, 16 (2019), 7662–7669. https://doi.org/10.1073/pnas.1805871115 arXiv:https://www.pnas.org/content/116/16/7662.full.pdf
  63. ”It Matches My Worldview”: Examining Perceptions and Attitudes Around Fake Videos. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 255, 15 pages. https://doi.org/10.1145/3491102.3517646
  64. Combating Fake News: A Survey on Identification and Mitigation Techniques. ACM Trans. Intell. Syst. Technol. 10, 3, Article 21 (April 2019), 42 pages. https://doi.org/10.1145/3305260
  65. Studying Fake News via Network Analysis: Detection and Mitigation. In Emerging Research Challenges and Opportunities in Computational Social Network Analysis and Mining, Nitin Agarwal, Nima Dokoohaki, and Serpil Tokdemir (Eds.). Springer International Publishing, Cham, 43–65. https://doi.org/10.1007/978-3-319-94105-9_3
  66. Fake News Detection on Social Media: A Data Mining Perspective. SIGKDD Explor. Newsl. 19, 1 (Sept. 2017), 22–36. https://doi.org/10.1145/3137597.3137600
  67. Learning From Revisions: Quality Assessment of Claims in Argumentation at Scale. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. Association for Computational Linguistics, Online, 1718–1729. https://doi.org/10.18653/v1/2021.eacl-main.147
  68. S. Sloman and P. Fernbach. 2017. The Knowledge Illusion: Why We Never Think Alone. Penguin Publishing Group. https://books.google.dk/books?id=2xuMDAAAQBAJ
  69. Michael Spence. 1978. Job market signaling. In Uncertainty in economics. Elsevier, 281–306.
  70. Disinformation as Collaborative Work: Surfacing the Participatory Nature of Strategic Information Operations. Proc. ACM Hum.-Comput. Interact. 3, CSCW, Article 127 (Nov. 2019), 26 pages. https://doi.org/10.1145/3359229
  71. The Trust Project. 2017. The 8 Trust Indicators. https://thetrustproject.org/trust-indicators/
  72. Stuart A. Thompson and Davey Alba. 2022. Fact and mythmaking blend in Ukraine’s Information War. https://www.nytimes.com/2022/03/03/technology/ukraine-war-misinfo.html?action=click&module=RelatedLinks&pgtype=Article
  73. The spread of true and false news online. Science 359, 6380 (2018), 1146–1151.
  74. W3C Community. 2020. Reviewed Credibility Signals. https://credweb.org/reviewed-signals-20200224
  75. Computational Argumentation Quality Assessment in Natural Language. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers. Association for Computational Linguistics, Valencia, Spain, 176–187. https://aclanthology.org/E17-1017
  76. William Yang Wang. 2017. “Liar, Liar Pants on Fire”: A New Benchmark Dataset for Fake News Detection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, Vancouver, Canada, 422–426. https://doi.org/10.18653/v1/P17-2067
  77. Claire Wardle. 2018. Fake news. it’s complicated. https://medium.com/1st-draft/fake-news-its-complicated-d0f773766c79
  78. Thinking about ‘information disorder’: formats of misinformation, disinformation, and mal-information. Ireton, Cherilyn; Posetti, Julie. Journalism,‘fake news’& disinformation. Paris: Unesco (2018), 43–54.
  79. Wikipedia contributors. 2021. List of political parties in Germany — Wikipedia, The Free Encyclopedia. https://en.wikipedia.org/w/index.php?title=List_of_political_parties_in_Germany&oldid=1030256103. [Online; accessed 1-July-2021].
  80. Wikipedia contributors. 2022a. Five whys — Wikipedia, The Free Encyclopedia. https://en.wikipedia.org/w/index.php?title=Five_whys&oldid=1102936260 [Online; accessed 23-August-2022].
  81. Wikipedia contributors. 2022b. Impressum — Wikipedia, The Free Encyclopedia. https://en.wikipedia.org/w/index.php?title=Impressum&oldid=1092341470 [Online; accessed 1-September-2022].
  82. Wikipedia contributors. 2022c. List of newspapers in the United States — Wikipedia, The Free Encyclopedia. https://en.wikipedia.org/w/index.php?title=List_of_newspapers_in_the_United_States&oldid=1101330486 [Online; accessed 4-August-2022].
  83. Lateral reading on the open Internet: A district-wide field study in high school government classes. Journal of Educational Psychology (2022).
  84. Sam Wineburg and Sarah McGrew. 2019. Lateral Reading and the Nature of Expertise: Reading Less and Learning More When Evaluating Digital Information. Teachers College Record 121, 11 (2019), 1–40. https://doi.org/10.1177/016146811912101102 arXiv:https://doi.org/10.1177/016146811912101102
  85. Thomas Wood and Ethan Porter. 2019. The elusive backfire effect: Mass attitudes’ steadfast factual adherence. Political Behavior 41, 1 (2019), 135–163.
  86. World Health Organization. 2020. Managing the COVID-19 infodemic: Promoting healthy behaviours and mitigating the harm from misinformation and disinformation. https://www.who.int/news/item/23-09-2020-managing-the-covid-19-infodemic-promoting-healthy-behaviours-and-mitigating-the-harm-from-misinformation-and-disinformation
  87. Amotz Zahavi. 1975. Mate selection—a selection for a handicap. Journal of theoretical Biology 53, 1 (1975), 205–214.
  88. A structured response to misinformation: Defining and annotating credibility indicators in news articles. In Companion Proceedings of the The Web Conference 2018. 603–612.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now