What's in a Name? Auditing Large Language Models for Race and Gender Bias

Abstract: We employ an audit design to investigate biases in state-of-the-art LLMs, including GPT-4. In our study, we prompt the models for advice involving a named individual across a variety of scenarios, such as during car purchase negotiations or election outcome predictions. We find that the advice systematically disadvantages names that are commonly associated with racial minorities and women. Names associated with Black women receive the least advantageous outcomes. The biases are consistent across 42 prompt templates and several models, indicating a systemic issue rather than isolated incidents. While providing numerical, decision-relevant anchors in the prompt can successfully counteract the biases, qualitative details have inconsistent effects and may even increase disparities. Our findings underscore the importance of conducting audits at the point of LLM deployment and implementation to mitigate their potential for harm against marginalized communities.

• The paper systematically audits LLMs for race and gender bias using name-based prompts and quantitative metrics.
• It employs 42 prompt templates across five scenarios to reveal disparities in model outputs tied to racial and gender stereotypes.
• Numerical anchoring in prompts often nullified biases, suggesting a promising mitigation strategy for LLM deployment.

Evaluation of Race and Gender Bias in LLMs

The paper "What's in a Name? Auditing LLMs for Race and Gender Bias" by Amit Haim, Alejandro Salinas, and Julian Nyarko, presents a meticulous exploration of biases embedded within state-of-the-art LLMs. By employing a systemic audit, the researchers examined how LLMs respond to scenario prompts involving individuals with names that are strongly associated with different racial and gender identities. Importantly, the study investigates the correlation between names and biased model outputs, specifically focusing on names significant within the U.S. context as proxies for race and gender.

Methodological Insights

To uncover implicit biases, the authors conducted extensive prompting experiments across several prevalent LLMs, including GPT-4, GPT-3.5, and Google's PaLM-2. Their methodology encompassed 42 distinct prompt templates spread over five central scenarios: purchasing decisions, chess competitions, public office elections, sports rankings, and initial hiring offers. The prompts incorporated names recognized by their racial and gender associations, thereby simulating real-world advisory use cases that LLMs might encounter.

The design of the prompts was structured around three levels of context: low, high, and numeric. This differentiation allowed the study to probe how context influences the expression of bias by the models. The researchers applied a quantitative approach to analyze the LLM responses, ensuring consistency and agility in calculating disparities. The extensive dataset, encompassing 168,000 responses, enabled a robust statistical analysis of biases.

Numerical Findings

The results elucidate significant disparities that align with racial and gender stereotypes. For instance, in scenarios involving financial transactions like the purchase of a car or bicycle, names conventionally identified with Black individuals were suggested lower initial offers compared to those associated with White individuals. Similarly, male-associated names were generally afforded more beneficial outcomes than female-associated names. The effect persisted across variations of prompts in certain contexts, revealing an ingrained bias that resisted simple mitigation through qualitative contextual information.

Notably, the introduction of numerical anchors in prompts often nullified these disparities, indicating a potential pathway for reducing biases during LLM deployment. The researchers found no significant difference in bias levels between different model iterations or their training quality, suggesting these biases might be a prevalent attribute of training data or architectural limitations rather than specific to a particular model type.

Practical and Theoretical Implications

The evidence from this study underscores the systematic bias embedded in LLMs that can influence model outputs in consequential and disparate ways. From a practical standpoint, organizations deploying LLMs in environments that utilize individual names for personalization must recognize the potential for biased outputs to propagate, possibly through policies like personalized customer interaction via LLMs. Such recognition is especially crucial as commercial applications of LLMs proliferate.

Adopting mitigation strategies such as embedding numerical context, enhancing audit processes, and revisiting training datasets to filter bias propagation channels in LLMs becomes critical. This research pushes for naming-based audits to be integrated as standard practice at both the deployment and implementation phases of LLM utilization, representing a necessary step for ensuring fairness and equity in AI technologies.

Future Directions

This work opens avenues for more specialized research into mitigation techniques that extend beyond numerical anchoring, making qualitative insights also effective. Regulatory bodies and policymakers could engage with results from such studies to establish frameworks that standardize LLM auditing practices in key sectors like financial services, legal systems, and other domains of high societal importance. While this paper navigated the biases within a U.S.-centric perspective, future studies could scale this research internationally to identify localized biases in diverse global contexts.

In conclusion, while LLMs hold significant potential in transforming numerous industries, their fair and equitable application hinges on our ability to understand and mitigate inherent biases. Thus, the work by Haim, Salinas, and Nyarko contributes valuable expertise to the discourse on ethical responsibilities in developing and deploying AI technologies.