Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Models (2102.04130v3)

Abstract: The capabilities of natural LLMs trained on large-scale data have increased immensely over the past few years. Open source libraries such as HuggingFace have made these models easily available and accessible. While prior research has identified biases in LLMs, this paper considers biases contained in the most popular versions of these models when applied `out-of-the-box' for downstream tasks. We focus on generative LLMs as they are well-suited for extracting biases inherited from training data. Specifically, we conduct an in-depth analysis of GPT-2, which is the most downloaded text generation model on HuggingFace, with over half a million downloads per month. We assess biases related to occupational associations for different protected categories by intersecting gender with religion, sexuality, ethnicity, political affiliation, and continental name origin. Using a template-based data collection pipeline, we collect 396K sentence completions made by GPT-2 and find: (i) The machine-predicted jobs are less diverse and more stereotypical for women than for men, especially for intersections; (ii) Intersectional interactions are highly relevant for occupational associations, which we quantify by fitting 262 logistic models; (iii) For most occupations, GPT-2 reflects the skewed gender and ethnicity distribution found in US Labor Bureau data, and even pulls the societally-skewed distribution towards gender parity in cases where its predictions deviate from real labor market observations. This raises the normative question of what LLMs should learn - whether they should reflect or correct for existing inequalities.

An Analysis of Intersectional Occupational Biases in Generative LLMs

Generative LLMs have gained widespread adoption due to their impressive capabilities in natural language processing tasks. Among these, GPT-2 has emerged as one of the most utilized models for text generation, available via platforms such as HuggingFace, enabling access to pretrained models for various applications. This paper offers a detailed empirical analysis of intersectional occupational biases inherent in GPT-2 when used 'out-of-the-box'.

Methodology Overview

The research investigates GPT-2's bias by examining the intersectional occupational associations tied to gender and five protected categories: ethnicity, religion, sexuality, political affiliation, and continent name origin. The authors employed a template-based data collection strategy to prompt GPT-2, producing 396,000 sentence completions to analyze its stereotyping behaviors. Logistic regression models were developed to quantify interaction effects, focusing on occupation prediction based on gender intersections.

Key Findings

1. Gender Bias: The paper reveals significant gender bias in occupation prediction, with GPT-2 associating men with a broader range of professions as compared to women. Jobs typically allocated to women were narrower and stereotypical, such as roles in caregiving and domestic services.
2. Intersectional Bias: Intersectional effects were prominent. For instance, certain combinations of gender and ethnicity yielded distinct occupation predictions, suggesting strong stereotypical associations. Significantly, the interactions of gender with factors such as religion and sexuality showed heightened predictive importance, demonstrating the intricate bias encoded within GPT-2.
3. Comparison to Real-World Data: When GPT-2's outputs were compared to US Labor Bureau statistics, interesting patterns emerged. For several occupations, GPT-2 approximated real gender and ethnicity distributions but also displayed tendencies to alter skewed societal distributions towards more balanced gender proportions.

Implications

The paper raises compelling discussions about the normative objectives for generative models—whether they should correct societal biases or merely reflect them. GPT-2’s inclination to modulate gender biases by broadening the representation of women in traditionally male-dominated professions suggests an emergent capability to resist exacerbating societal skew.

Speculative Outlook and Future Directions

The implications of this research are profound for applications relying on generative models in sensitive domains such as hiring and automated job matchmaking. As AI recognition in decision-making processes accelerates, addressing embedded biases becomes urgent to prevent perpetuating inappropriate stereotypes. The paper advocates for transparency about model biases and calls for enhanced frameworks that encompass various gender identities beyond binary constructs. Future endeavors may explore the biases within newer models and extend analyses globally, incorporating a broader range of intersections.

In conclusion, while LLMs like GPT-2 demonstrate impressive linguistic aptitude, their underlying biases, especially regarding occupational stereotypes, necessitate critical examination. Researchers and practitioners should prioritize developing methods to audit and mitigate these biases, aiming for responsible and equitable AI deployment in real-world applications.