Who Does the Giant Number Pile Like Best: Analyzing Fairness in Hiring Contexts (2501.04316v1)

Abstract: LLMs are increasingly being deployed in high-stakes applications like hiring, yet their potential for unfair decision-making and outcomes remains understudied, particularly in generative settings. In this work, we examine the fairness of LLM-based hiring systems through two real-world tasks: resume summarization and retrieval. By constructing a synthetic resume dataset and curating job postings, we investigate whether model behavior differs across demographic groups and is sensitive to demographic perturbations. Our findings reveal that race-based differences appear in approximately 10% of generated summaries, while gender-based differences occur in only 1%. In the retrieval setting, all evaluated models display non-uniform selection patterns across demographic groups and exhibit high sensitivity to both gender and race-based perturbations. Surprisingly, retrieval models demonstrate comparable sensitivity to non-demographic changes, suggesting that fairness issues may stem, in part, from general brittleness issues. Overall, our results indicate that LLM-based hiring systems, especially at the retrieval stage, can exhibit notable biases that lead to discriminatory outcomes in real-world contexts.

Analyzing Fairness in LLM-Based Hiring

The integration of LLMs in high-stakes applications like hiring underscores the imperative to scrutinize the fairness of these technologies, an area which remains insufficiently explored, especially in generative contexts. The paper "Who Does the Giant Number Pile Like Best: Analyzing Fairness in Hiring Contexts" undertakes a critical examination of fairness in LLM-based hiring systems, focusing on resume summarization and retrieval tasks. Through a synthetic resume dataset and job postings, the paper investigates the differential behavior of models across demographics and their sensitivity to demographic perturbations.

Key Findings

The paper reveals significant findings related to race and gender biases:

• Summarization Bias: Approximately 10% of race-related summarizations exhibit meaningful differences, while only 1% of gender-related cases show such disparities. This finding indicates a notable racial bias in how summaries are generated by the LLMs, albeit at low proportions.
• Retrieval Bias: The retrieval tasks show non-uniform selection patterns across demographics, with high sensitivity to both gender and race perturbations. This suggests that retrieval models are considerably impacted by demographic signals, raising concerns about fairness in resume screening systems.
• General Sensitivity: Surprisingly, the models display sensitivity comparable to non-demographic perturbations, indicating that fairness issues might partly stem from the general brittleness of these models, rather than solely demographic biases.

Methodology

The research adopts a two-pronged approach to paper resume retrieval and summarization:

1. Synthetic Resumes and Job Postings: By generating synthetic resumes and carefully curating job postings, the paper sets the stage for a controlled examination of LLM behaviors under demographic perturbations.
2. Metrics for Fairness: The paper introduces metrics to measure fairness in both generative and retrieval settings, validating these metrics through an expert human preference paper.

The choices in the paper design, including diverse demographic perturbations using name and extracurricular content, provide a comprehensive basis for assessing the fairness of LLM operations in hiring.

Implications and Future Work

The implications of the findings are twofold:

• Practical Implications: In real-world contexts, biased LLM behavior can lead to discriminatory outcomes in hiring, adversely affecting already marginalized groups. Addressing these biases in early hiring stages is critical to ensuring equitable employment opportunities.
• Theoretical Implications: The paper highlights an interplay between brittleness and fairness, suggesting that improvements in model robustness could mitigate some bias issues. This opens new avenues for research into the root causes of bias beyond representational factors in LLMs.

Future research should explore understanding how to mitigate these biases through model improvements and explore fairness in a broader spectrum of demographic categories beyond race and gender. Additionally, assessing fairness considerations in multilingual and multicultural contexts remains a pivotal area for further exploration.

Conclusion

This paper provides a nuanced exploration of the potential for bias in LLM-powered hiring tools, illustrating the necessity of rigorous fairness evaluations. The insights contribute to the theoretical understanding and practical mitigation of algorithmic bias in automated decision-making systems, particularly in critical applications such as hiring. Future advancements in AI must consider these findings to enhance fairness and equity in automated systems.