Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

On Hate Scaling Laws For Data-Swamps (2306.13141v2)

Published 22 Jun 2023 in cs.CY

Abstract: Scale the model, scale the data, scale the GPU-farms' is the reigning sentiment in the world of generative AI today. While model scaling has been extensively studied, data scaling and its downstream impacts remain under explored. This is especially of critical importance in the context of visio-linguistic datasets whose main source is the World Wide Web, condensed and packaged as the CommonCrawl dump. This large scale data-dump, which is known to have numerous drawbacks, is repeatedly mined and serves as the data-motherlode for large generative models. In this paper, we: 1) investigate the effect of scaling datasets on hateful content through a comparative audit of the LAION-400M and LAION-2B-en, containing 400 million and 2 billion samples respectively, and 2) evaluate the downstream impact of scale on visio-linguistic models trained on these dataset variants by measuring racial bias of the models trained on them using the Chicago Face Dataset (CFD) as a probe. Our results show that 1) the presence of hateful content in datasets, when measured with a Hate Content Rate (HCR) metric on the inferences of the Pysentimiento hate-detection NLP model, increased by nearly $12\%$ and 2) societal biases and negative stereotypes were also exacerbated with scale on the models we evaluated. As scale increased, the tendency of the model to associate images of human faces with thehuman being' class over 7 other offensive classes reduced by half. Furthermore, for the Black female category, the tendency of the model to associate their faces with the `criminal' class doubled, while quintupling for Black male faces. We present a qualitative and historical analysis of the model audit results, reflect on our findings and its implications for dataset curation practice, and close with a summary of our findings and potential future work to be done in this area.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (108)
  1. Persistent anti-muslim bias in large language models. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society (2021), pp. 298–306.
  2. Alexander, M. The new Jim Crow: Mass incarceration in the age of colorblindness. The New Press, 2020.
  3. GPT-NeoX: Large scale autoregressive language modeling in pytorch, 2021.
  4. Apel, D. Just joking? chimps, obama and racial stereotype. Journal of Visual Culture 8, 2 (2009), 134–142.
  5. Bardes, J. K. Redefining vagrancy: Policing freedom and disorder in reconstruction new orleans, 1862–1868. Journal of Southern History 84, 1 (2018), 69–112.
  6. Inspecting the geographical representativeness of images from text-to-image models, 2023.
  7. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (2021), pp. 610–623.
  8. Climbing towards nlu: On meaning, form, and understanding in the age of data. In Proceedings of the 58th annual meeting of the association for computational linguistics (2020), pp. 5185–5198.
  9. Benjamin, R. Race after technology: Abolitionist tools for the new jim code. John Wiley & Sons, 2019.
  10. Bey, M. “bring out your dead” understanding the historical persistence of the criminalization of black bodies. Cultural Studies? Critical Methodologies 16, 3 (2016), 271–277.
  11. Easily accessible text-to-image generation amplifies demographic stereotypes at large scale. arXiv preprint arXiv:2211.03759 (2022).
  12. The values encoded in machine learning research. arXiv preprint arXiv:2106.15590 (2021).
  13. Multimodal datasets: misogyny, pornography, and malignant stereotypes. arXiv preprint arXiv:2110.01963 (2021).
  14. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, Mar. 2021. If you use this software, please cite it using these metadata.
  15. Language (technology) is power: A critical survey of” bias” in nlp. arXiv preprint arXiv:2005.14050 (2020).
  16. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 (2020).
  17. Browne, S. Dark matters: On the surveillance of blackness. Duke University Press, 2015.
  18. Extracting training data from diffusion models. arXiv preprint arXiv:2301.13188 (2023).
  19. Do# blacklivesmatter? implicit bias, institutional racism and fear of the black body. Ralph Bunche Journal of Public Affairs 6, 1 (2017), 2.
  20. A literature survey on multimodal and multilingual automatic hate speech identification. Multimedia Systems (2023), 1–28.
  21. Citron, D. K. Hate crimes in cyberspace. Harvard University Press, 2014.
  22. Racial bias in hate speech and abusive language detection datasets. arXiv preprint arXiv:1905.12516 (2019).
  23. Automated hate speech detection and the problem of offensive language. In Proceedings of the international AAAI conference on web and social media (2017), vol. 11, pp. 512–515.
  24. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition (2009), pp. 248–255.
  25. EDWARDS, B. Flooded with ai-generated images, some art communities ban them completely — ars technica. https://arstechnica.com/information-technology/2022/09/flooded-with-ai-generated-images-some-art-communities-ban-them-completely/, Sep 2022. (Accessed on 04/22/2023).
  26. Computer age statistical inference, student edition: algorithms, evidence, and data science, vol. 6. Cambridge University Press, 2021.
  27. Eric Sheridan, K. R. Are we on the cusp of a generative ai revolution? https://www.goldmansachs.com/insights/podcasts/episodes/02-21-2023-sheridan-rangan.html, Feb 2023. (Accessed on 04/21/2023).
  28. Viral hate: Containing its spread on the Internet. Macmillan, 2013.
  29. The latino eyelid: anthropometric analysis of a spectrum of findings. Ophthalmic plastic and reconstructive surgery 33, 6 (2017), 440.
  30. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 (2020).
  31. Uncurated image-text datasets: Shedding light on demographic bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023), pp. 6957–6966.
  32. Handling bias in toxic speech detection: A survey. ACM Computing Surveys (2022).
  33. Gaston, S. Enforcing race: A neighborhood-level explanation of black–white differences in drug arrests. Crime & Delinquency 65, 4 (2019), 499–526.
  34. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009.11462 (2020).
  35. The essence of innocence: consequences of dehumanizing black children. Journal of personality and social psychology 106, 4 (2014), 526.
  36. Google’s photo app still can’t find gorillas. and neither can apple’s., 2023.
  37. Race and wrongful convictions in the united states 2022. Available at SSRN 4245863 (2022).
  38. Gross, T. F. Own-ethnicity bias in the recognition of black, east asian, hispanic, and white faces. Basic and Applied Social Psychology 31, 2 (2009), 128–135.
  39. Against scale: Provocations and resistances to scale thinking. arXiv preprint arXiv:2010.08850 (2020).
  40. Detoxify. Github. https://github.com/unitaryai/detoxify, 2020.
  41. Heaven, W. D. Generative ai is changing everything. but what’s left when the hype is gone? https://www.technologyreview.com/2022/12/16/1065005/generative-ai-revolution-art/, Dec 2022. (Accessed on 04/21/2023).
  42. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409 (2017).
  43. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 (2022).
  44. A survey and taxonomy of graph sampling. arXiv preprint arXiv:1308.5865 (2013).
  45. Huang, H. The generative ai revolution has begun—how did we get here? — ars technica. https://arstechnica.com/gadgets/2023/01/the-generative-ai-revolution-has-begun-how-did-we-get-here/, Jan 2023. (Accessed on 04/21/2023).
  46. Openclip, July 2021. If you use this software, please cite it as below.
  47. A systematic review of hate speech automatic detection using natural language processing. arXiv preprint arXiv:2106.00742 (2021).
  48. Hiding in plain sight: Dehumanization as a foundation of white racial prejudice. Sociology Compass 15, 9 (2021), e12913.
  49. Scaling up visual and vision-language representation learning with noisy text supervision. arXiv preprint arXiv:2102.05918 (2021).
  50. Joshua Lu, R. G. The generative ai revolution will enable anyone to create games — andreessen horowitz. https://a16z.com/2023/03/17/the-generative-ai-revolution/, Mar 2023. (Accessed on 04/21/2023).
  51. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 (2020).
  52. Fairface: Face attribute dataset for balanced race, gender, and age. arXiv preprint arXiv:1908.04913 (2019).
  53. Reduced, reused and recycled: The life of a dataset in machine learning research. arXiv preprint arXiv:2112.01716 (2021).
  54. LEE, T. B. Stable diffusion copyright lawsuits could be a legal earthquake for ai — ars technica. https://arstechnica.com/tech-policy/2023/04/stable-diffusion-copyright-lawsuits-could-be-a-legal-earthquake-for-ai/, Apr 2023. (Accessed on 04/22/2023).
  55. Sampling from large graphs. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining (2006), pp. 631–636.
  56. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. arXiv preprint arXiv:2201.12086 (2022).
  57. Lomas, N. Shutterstock to integrate OpenAI’s DALL-E 2 and launch fund for contributor artists, 2022.
  58. Lott, T. Racist discourse and the negro-ape metaphor. The Invention of Race: Black Culture and the Politics of Representation (1999), 7–13.
  59. Stable bias: Analyzing societal representations in diffusion models. arXiv preprint arXiv:2303.11408 (2023).
  60. What’s in the box? an analysis of undesirable content in the common crawl corpus. arXiv preprint arXiv:2105.02732 (2021).
  61. The chicago face database: A free stimulus set of faces and norming data. Behavior research methods 47 (2015), 1122–1135.
  62. Mantilla, K. Gendertrolling: How misogyny went viral: How misogyny went viral. ABC-CLIO, 2015.
  63. McQuillan, D. Resisting AI: an anti-fascist approach to artificial intelligence. Policy Press, 2022.
  64. The delaware pain database: A set of painful expressions and corresponding norming data. Pain reports 5, 6 (2020).
  65. Auditing algorithms: Understanding algorithmic systems from the outside in. Foundations and Trends® in Human–Computer Interaction 14, 4 (2021), 272–344.
  66. Montagu, M. A. The genetical theory of race, and anthropological method. American Anthropologist (1942), 369–375.
  67. Nagel, S. Common crawl’s first in-house web graph – common crawl. https://commoncrawl.org/2017/05/hostgraph-2017-feb-mar-apr-crawls/, May 2017. (Accessed on 03/15/2023).
  68. ‘alllooksame’? mediating asian american visual cultures of race on the web. East main street: Asian American popular culture (2005), 262–272.
  69. Noble, S. U. Algorithms of oppression. In Algorithms of oppression. New York University Press, 2018.
  70. Orselli, B. Stable diffusion ai has mastered the female form - niche gamer. https://nichegamer.com/stable-diffusion-ai-has-mastered-the-female-form/, Oct 2022. (Accessed on 04/22/2023).
  71. Pacheco Jr, G. Rhetoric with humor: An analysis of Hispanic/Latino comedians’ uses of humor. The University of Southern Mississippi, 2008.
  72. pysentimiento: A python toolkit for sentiment analysis and socialnlp tasks. arXiv preprint arXiv:2106.09462 (2021).
  73. The politics of scaling. Social Studies of Science 52, 1 (2022), 3–34.
  74. Combined scaling for open-vocabulary image classification. arXiv e-prints (2021), arXiv–2111.
  75. Resources and benchmark corpora for hate speech detection: a systematic review. Language Resources and Evaluation 55 (2021), 477–523.
  76. Online abuse and human rights: Woah satellite session at rightscon 2020. In Proceedings of the Fourth Workshop on Online Abuse and Harms (2020), pp. 1–6.
  77. Project, S. Report to the united nations on racial disparities in the us criminal justice system.
  78. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020 (2021).
  79. Actionable auditing: Investigating the impact of publicly naming biased performance results of commercial ai products. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society (2019), pp. 429–435.
  80. Zero-shot text-to-image generation. arXiv preprint arXiv:2102.12092 (2021).
  81. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2022), pp. 10684–10695.
  82. Rosich, K. J. Race, ethnicity, and the criminal justice system.
  83. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems 35 (2022), 36479–36494.
  84. Saini, A. Superior: the return of race science. Beacon Press, 2019.
  85. The bogazici face database: Standardized photographs of turkish faces with supporting materials. PloS one 13, 2 (2018), e0192018.
  86. How we’ve taught algorithms to see identity: Constructing race and gender in image databases for facial analysis. Proceedings of the ACM on Human-computer Interaction 4, CSCW1 (2020), 1–35.
  87. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402 (2022).
  88. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 (2021).
  89. Innovation and scaling for impact: How effective social enterprises do it. Stanford university press, 2017.
  90. From “brute” to “thug:” the demonization and criminalization of unarmed black male victims in america. Journal of human behavior in the social environment 26, 3-4 (2016), 350–366.
  91. Diffusion art or digital forgery? investigating data replication in diffusion models. arXiv preprint arXiv:2212.03860 (2022).
  92. Measuring trustworthiness or automating physiognomy? a comment on safra, chevallier, gr\\\backslash\ezes, and baumard (2020). arXiv preprint arXiv:2202.08674 (2022).
  93. Physiognomic artificial intelligence. Fordham Intell. Prop. Media & Ent. LJ 32 (2021), 922.
  94. The mr2: A multi-racial, mega-resolution database of facial stimuli. Behavior research methods 48 (2016), 1197–1204.
  95. Thompson, W. When the beautiful game turns ugly. ESPN The Magazine 6 (2013).
  96. Manifestations of xenophobia in ai systems, 2022.
  97. Van Miltenburg, E. Stereotyping and bias in the flickr30k dataset. arXiv preprint arXiv:1605.06083 (2016).
  98. Algorithmic auditing and social justice: Lessons from the history of audit studies. In Equity and Access in Algorithms, Mechanisms, and Optimization. 2021, pp. 1–9.
  99. Learning from the worst: Dynamically generated datasets to improve online hate detection. arXiv preprint arXiv:2012.15761 (2020).
  100. Learning from the worst: Dynamically generated datasets to improve online hate detection. In ACL (2021).
  101. Villalobos, P. Scaling laws literature review, 2023. Accessed: 2023-4-22.
  102. A survey of toxic comment classification methods. arXiv preprint arXiv:2112.06412 (2021).
  103. Understanding abuse: A typology of abusive language detection subtasks. arXiv preprint arXiv:1705.09899 (2017).
  104. On the de-duplication of laion-2b. arXiv preprint arXiv:2303.12733 (2023).
  105. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359 (2021).
  106. Wilson, E. B. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association 22, 158 (1927), 209–212.
  107. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10789 (2022).
  108. Isiea: An image database of social inclusion and exclusion in young asian adults. Behavior Research Methods (2021), 1–13.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Abeba Birhane (24 papers)
  2. Vinay Prabhu (5 papers)
  3. Sang Han (3 papers)
  4. Vishnu Naresh Boddeti (48 papers)
Citations (26)
View on Semantic Scholar

Summary

An Analysis of "On Hate Scaling Laws for Data-Swamps"

This paper, authored by Birhane et al., rigorously analyzes the implications of dataset scaling on the prevalence of hateful content and biases in visio-linguistic models. The paper underscores a critical oversight in generative AI, where the focus has predominantly been on model scaling, while data scaling has been inadequately examined.

Core Contributions and Findings

  1. Data Scaling and Hateful Content: The authors investigate the LAION family of datasets, specifically comparing LAION-400M and its successor, LAION-2B-en. Using the Hate Content Rate (HCR) metric in conjunction with the Pysentimiento model, the paper demonstrates a concerning increase in hateful content by approximately 12% as the dataset scales from 400 million to 2 billion samples. This increase is manifest across hateful, targeted, and aggressive content categories.
  2. Visio-Linguistic Model Bias: The downstream effects of these datasets are scrutinized through their impact on visio-linguistic models such as CLIP. A critical experiment involving the Chicago Face Dataset (CFD) highlights exacerbated racial biases in models trained on larger datasets. Notably, for Black individuals, the propensity to be associated with derogatory classifications such as "criminal" increased substantially—doubling for Black females and quintupling for Black males—when the dataset expanded.
  3. Qualitative and Historical Analysis: The paper contextualizes these findings historically, linking the negative biases to stereotypical and dehumanizing narratives historically imposed on marginalized communities. Such biases are argued to be perpetuated and amplified in scaled datasets.

Implications

The research raises significant concerns about the current trajectory of AI dataset scaling, suggesting that it might reinforce and spread societal biases rather than mitigate them. It calls for a re-evaluation of dataset curation practices, urging the community to focus not just on size but also on diversity and representational fairness.

Future Directions

The authors advocate for more transparent and accountable dataset curation practices, emphasizing the need for standardized metrics that consider ethical implications alongside model performance. They propose that future research could investigate alternative langual and racial representations in datasets to mitigate bias propagation.

Conclusion

This paper offers a comprehensive critique of the "scale is all you need" paradigm in AI dataset development. By exposing the risks inherent in merely expanding data size without assessing quality, Birhane et al. provide a crucial contribution to the discourse on ethical AI, urging for balanced development that foregrounds equity and accountability.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube