Entity-Based Evaluation of Political Bias in Automatic Summarization
Abstract: Growing literature has shown that NLP systems may encode social biases; however, the political bias of summarization models remains relatively unknown. In this work, we use an entity replacement method to investigate the portrayal of politicians in automatically generated summaries of news articles. We develop an entity-based computational framework to assess the sensitivities of several extractive and abstractive summarizers to the politicians Donald Trump and Joe Biden. We find consistent differences in these summaries upon entity replacement, such as reduced emphasis of Trump's presence in the context of the same article and a more individualistic representation of Trump with respect to the collective US government (i.e., administration). These summary dissimilarities are most prominent when the entity is heavily featured in the source article. Our characterization provides a foundation for future studies of bias in summarization and for normative discussions on the ideal qualities of automatic summaries.
- Eugene Bagdasaryan and Vitaly Shmatikov. 2021. Spinning Language Models: Risks of Propaganda-As-A-Service and Countermeasures.
- Truth, Lies, and Automation.
- Dennis Chong and James N Druckman. 2007. A theory of framing and opinion formation in competitive elite environments. Journal of communication, 57(1):99–118.
- Evaluating Factuality in Text Simplification. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7331–7345, Dublin, Ireland. Association for Computational Linguistics.
- BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation. FAccT.
- Automatic text summarization: A comprehensive survey. Expert Systems with Applications, 165:113679.
- Robert M Entman. 1993. Framing: Towards clarification of a fractured paradigm. McQuail’s reader in mass communication theory, pages 390–397.
- CausaLM: Causal Model Explanation Through Counterfactual Language Models. Computational Linguistics, pages 1–54.
- From pretraining data to language models to downstream tasks: Tracking the trails of political biases leading to unfair NLP models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11737–11762, Toronto, Canada. Association for Computational Linguistics.
- He is very intelligent, she is very beautiful? On Mitigating Social Biases in Language Modelling and Generation. In FINDINGS.
- News summarization and evaluation in the era of gpt-3. ArXiv, abs/2209.12356.
- Measuring Media Bias via Masked Language Modeling. Proceedings of the International AAAI Conference on Web and Social Media, 16:1404–1408.
- Teaching Machines to Read and Comprehend. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc.
- The factual inconsistency problem in abstractive text summarization: A survey. ArXiv, abs/2104.14839.
- Survey of hallucination in natural language generation. arXiv preprint arXiv:2202.03629.
- Evaluating the Factual Consistency of Abstractive Text Summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9332–9346, Online. Association for Computational Linguistics.
- NeuS: Neutral Multi-News Summarization for Mitigating Framing Bias. ArXiv.
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. arXiv:1910.13461 [cs, stat].
- Towards Understanding and Mitigating Social Biases in Language Models. In ICML.
- Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
- Yang Liu and Mirella Lapata. 2019. Text Summarization with Pretrained Encoders. arXiv:1908.08345 [cs].
- On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906–1919, Online. Association for Computational Linguistics.
- On Faithfulness and Factuality in Abstractive Summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906–1919, Online. Association for Computational Linguistics.
- Fightin’ Words: Lexical Feature Selection and Evaluation for Identifying the Content of Political Conflict. Political Analysis, 16.
- Entity-level factual consistency of abstractive text summarization. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2727–2733, Online. Association for Computational Linguistics.
- Ani Nenkova and Kathleen McKeown. 2011. Automatic summarization. Now Publishers Inc.
- QUOTUS: The Structure of Political Media Coverage as Revealed by Quoting Patterns. WWW.
- Roma Patel and Ellie Pavlick. 2021. “was it “stated” or was it “claimed”?: How linguistic bias affects generative language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10080–10095, Online and Punta Cana,
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
