Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Understanding Position Bias Effects on Fairness in Social Multi-Document Summarization (2405.01790v1)

Published 3 May 2024 in cs.CL and cs.AI

Abstract: Text summarization models have typically focused on optimizing aspects of quality such as fluency, relevance, and coherence, particularly in the context of news articles. However, summarization models are increasingly being used to summarize diverse sources of text, such as social media data, that encompass a wide demographic user base. It is thus crucial to assess not only the quality of the generated summaries, but also the extent to which they can fairly represent the opinions of diverse social groups. Position bias, a long-known issue in news summarization, has received limited attention in the context of social multi-document summarization. We deeply investigate this phenomenon by analyzing the effect of group ordering in input documents when summarizing tweets from three distinct linguistic communities: African-American English, Hispanic-aligned Language, and White-aligned Language. Our empirical analysis shows that although the textual quality of the summaries remains consistent regardless of the input document order, in terms of fairness, the results vary significantly depending on how the dialect groups are presented in the input data. Our results suggest that position bias manifests differently in social multi-document summarization, severely impacting the fairness of summarization models.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (43)
  1. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150.
  2. Demographic dialectal variation in social media: A case study of african-american english. arXiv preprint arXiv:1608.08868.
  3. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in neural information processing systems, 29.
  4. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258.
  5. Quantifying social biases in nlp: A generalization and empirical comparison of extrinsic fairness metrics. arXiv preprint arXiv:2106.14574.
  6. Summarizing user-generated textual content: Motivation and methods for fairness in algorithmic summaries. Proceedings of the ACM on Human-Computer Interaction, 3(CSCW):1–28.
  7. Evaluation of african american language bias in natural language generation.
  8. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  9. Improving extractive document summarization with sentence centrality. PloS one, 17(7):e0268278.
  10. Countering the effects of lead bias in news summarization via multi-stage training and auxiliary losses. arXiv preprint arXiv:1909.04028.
  11. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. arXiv preprint arXiv:1804.11283.
  12. Kai Hong and Ani Nenkova. 2014. Improving the estimation of word importance for news multi-document summarization. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 712–721.
  13. Earlier isn’t always better: Sub-aspect analysis on corpus and system biases in summarization. arXiv preprint arXiv:1908.11723.
  14. Content selection in deep learning models of summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1818–1828, Brussels, Belgium. Association for Computational Linguistics.
  15. Vijay Keswani and L Elisa Celis. 2021. Dialect diversity in text summarization on twitter. In Proceedings of the Web Conference 2021, pages 3802–3814.
  16. Abstractive summarization of Reddit posts with multi-level memory networks. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2519–2531, Minneapolis, Minnesota. Association for Computational Linguistics.
  17. When do pre-training biases propagate to downstream tasks? a case study in text summarization. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3198–3211.
  18. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461.
  19. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.
  20. Chin-Yew Lin and Eduard Hovy. 1997. Identifying topics by position. In Fifth conference on applied natural language processing, pages 283–290.
  21. Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. arXiv preprint arXiv:1908.08345.
  22. Rada Mihalcea and Paul Tarau. 2004. Textrank: Bringing order into text. In Proceedings of the 2004 conference on empirical methods in natural language processing, pages 404–411.
  23. Derek Miller. 2019. Leveraging bert for extractive text summarization on lectures. arXiv preprint arXiv:1906.04165.
  24. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. In Proceedings of the AAAI conference on artificial intelligence, volume 31.
  25. Analyzing the dialect diversity in multi-document summaries. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6208–6221.
  26. Unsupervised extractive summarization using pointwise mutual information. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2505–2512, Online. Association for Computational Linguistics.
  27. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683.
  28. Echoes from alexandria: A large resource for multilingual book summarization. In Findings of the Association for Computational Linguistics: ACL 2023, pages 853–867, Toronto, Canada. Association for Computational Linguistics.
  29. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073–1083, Vancouver, Canada. Association for Computational Linguistics.
  30. Fairness of extractive text summarization. In Companion Proceedings of the The Web Conference 2018, pages 97–98.
  31. BIGPATENT: A large-scale dataset for abstractive and coherent summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2204–2213, Florence, Italy. Association for Computational Linguistics.
  32. MentSum: A resource for exploring summarization of mental health online posts. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2682–2692, Marseille, France. European Language Resources Association.
  33. Exploring domain shift in extractive text summarization. arXiv preprint arXiv:1908.11664.
  34. Primera: Pyramid-based masked sentence pre-training for multi-document summarization. arXiv preprint arXiv:2110.08499.
  35. Demoting the lead bias in news summarization via alternating adversarial learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 948–954, Online. Association for Computational Linguistics.
  36. TED: A pretrained unsupervised summarization model with theme modeling and denoising. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1865–1874, Online. Association for Computational Linguistics.
  37. Bartscore: Evaluating generated text as text generation. In Advances in Neural Information Processing Systems, volume 34, pages 27263–27277. Curran Associates, Inc.
  38. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In International Conference on Machine Learning, pages 11328–11339. PMLR.
  39. Attention temperature matters in abstractive summarization distillation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 127–141, Dublin, Ireland. Association for Computational Linguistics.
  40. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations.
  41. Neural latent extractive document summarization. arXiv preprint arXiv:1808.07187.
  42. Towards a unified multi-dimensional evaluator for text generation. arXiv preprint arXiv:2210.07197.
  43. Make lead bias in your favor: Zero-shot abstractive news summarization. In International Conference on Learning Representations.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (2)
  1. Olubusayo Olabisi (1 paper)
  2. Ameeta Agrawal (23 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets