Independent fact-checking organizations exhibit a departure from political neutrality

Abstract: Independent fact-checking organizations have emerged as the crusaders to debunk fake news. However, they may not always remain neutral, as they can be selective in the false news they choose to expose and in how they present the information. They can deviate from neutrality by being selective in what false news they debunk and how the information is presented. Prompting the now popular LLM, GPT-3.5, with journalistic frameworks, we establish a longitudinal measure (2018-2023) for political neutrality that looks beyond the left-right spectrum. Specified on a range of -1 to 1 (with zero being absolute neutrality), we establish the extent of negative portrayal of political entities that makes a difference in the readers' perception in the USA and India. Here, we observe an average score of -0.17 and -0.24 in the USA and India, respectively. The findings indicate how seemingly objective fact-checking can still carry distorted political views, indirectly and subtly impacting the perception of consumers of the news.

• The paper finds that independent fact-checkers exhibit measurable political bias, deviating notably from neutrality on a -1 to +1 scale.
• It employs an interdisciplinary approach with the 5W1H framework, GPT-3.5, claim normalization, and Sentence Transformer embeddings to quantify media bias.
• The findings urge consumers and policymakers to critically assess fact-checking sources due to the demonstrated selective portrayal of political entities.

Independent Fact-Checking Organizations and Political Neutrality: An Analytical Overview

The paper "Independent fact-checking organizations exhibit a departure from political neutrality" provides a critical examination of media bias prevalent within independent fact-checking organizations. The study is grounded in a longitudinal analysis that spans from 2018 to 2023 across six key fact-checking entities in the USA and India. These include PolitiFact, Snopes, and Check Your Fact in the US, and Alt News, Boom, and OpIndia in India.

Methodology

This study leverages the 5W1H journalistic framework in conjunction with the capabilities of LLMs, specifically GPT-3.5, to evaluate political neutrality. By employing this interdisciplinary approach, the research deviates from the traditional left-right dichotomy to measure the portrayal of political entities on a granular scale from -1 to 1, which represents the spectrum of negative to positive portrayal. The scale facilitates a nuanced understanding of political bias as communicated through these organizations. The research further integrates LLM-based claim normalization and Sentence Transformer embeddings to measure thematic overlap, providing a multi-dimensional perspective on the political neutrality score of each organization.

Key Findings

The results reveal substantial evidence of deviation from political neutrality by all studied organizations. In terms of political neutrality, entities like PolitiFact, Snopes, and Check Your Fact exhibit scores of -0.10, -0.28, and -0.12, respectively, while Alt News, Boom, and OpIndia exhibit scores of -0.28, -0.19, and -0.25, respectively. These scores indicate that selectivity in the portrayal of political figures exists across geographies, with fact-checking organizations sometimes presenting a biased narrative. Notably, while topical similarity among articles across organizations stands at approximately 0.8, indicating a shared focus on similar content, the similarity in portrayals of political entities registers at nearly 1, emphasizing a consistent focus on particular political figures, often heads of state or prominent leaders.

Implications

The implications of this research are multifaceted. Practically, the study signals to media consumers and policy-makers the potential biases that may exist within ostensibly neutral fact-checking outlets. By understanding these biases, consumers can be more discerning in their consumption of fact-checked content. Theoretically, the study introduces the possibility of quantifying political leanings through the analysis of polarity in media portrayals, which can redefine our understanding of media bias beyond traditional spectrums.

Future Directions

This research opens several avenues for further exploration. Future inquiries may explore the dynamics of specific high-profile political events and assess the temporal skewness of biases. Additionally, examining how the perception bias measured by this study influences reader belief systems could offer insights into the broader societal implications of media consumption. Another intriguing dimension could involve investigating how these findings align or differ when examining fact-checking bodies in regions exhibiting different political climates than those analyzed in this study.

This paper contributes a vital computational method to the media bias discourse, advocating for a more methodical and metric-driven approach to evaluating the perceived objectivity of fact-checking institutions. As media plays an ever-increasing role in shaping public discourse, such analyses remain necessary to bridge the gap between information dissemination and unbiased journalism.