Attribution Bias in Large Language Models

Abstract: As LLMs are increasingly used to support search and information retrieval, it is critical that they accurately attribute content to its original authors. In this work, we introduce AttriBench, the first fame- and demographically-balanced quote attribution benchmark dataset. Through explicitly balancing author fame and demographics, AttriBench enables controlled investigation of demographic bias in quote attribution. Using this dataset, we evaluate 11 widely used LLMs across different prompt settings and find that quote attribution remains a challenging task even for frontier models. We observe large and systematic disparities in attribution accuracy between race, gender, and intersectional groups. We further introduce and investigate suppression, a distinct failure mode in which models omit attribution entirely, even when the model has access to authorship information. We find that suppression is widespread and unevenly distributed across demographic groups, revealing systematic biases not captured by standard accuracy metrics. Our results position quote attribution as a benchmark for representational fairness in LLMs.

• The paper introduces AttriBench, a novel benchmark balancing author fame and demographics to identify attribution disparities in LLMs.
• The paper shows that LLMs have low overall attribution accuracy and favor White male authors across various prompting methods.
• The paper quantifies omission suppression as a distinct metric, revealing systemic biases that persist even with evidence augmentation.

Attribution Bias in LLMs: Characterizing Systemic Disparities with AttriBench

Introduction

The accurate attribution of quoted content is essential for intellectual credit and scholarly integrity in AI-mediated knowledge applications. As LLMs increasingly function as search and retrieval engines, deficiencies and biases in their attribution capabilities have direct consequences for representational fairness. The paper "Attribution Bias in LLMs" (2604.05224) presents a systematic and controlled evaluation of demographic biases in quote attribution across LLMs using the AttriBench dataset—explicitly constructed to balance both author fame and demographic subgroup representation. The analysis highlights consistent disparities, introduces nuanced metrics including suppression (omission), and provides a comprehensive basis for ongoing fairness evaluation in LLM-mediated knowledge systems.

AttriBench: A Controlled Benchmark for Attribution Fairness

The authors' primary methodological advance is AttriBench, the first dataset jointly balancing author fame (measured by log-transformed Google Search hits) and demographic annotation (race and gender). The benchmark includes two variants: AttriBench Intersectional (Black/White × Male/Female) and AttriBench Multirace (Asian, Black, Latino, White) constructed after extensive pruning, normalization, and consensus-based demographic labeling to maximize validity. Fame balancing is achieved via a greedy author matching algorithm that minimizes inter-group fame discrepancies across up to 100 randomization runs.

AttriBench eliminates confounding due to differing author visibility while retaining a realistic, open-world attribution setting, in which models receive isolated quotes without predefined candidate lists or structured context.

Evaluation Framework: Accuracy, Disparity, and Suppression

The study operationalizes three metrics for LLM behavior:

• Attribution accuracy: Probability the model names the ground-truth author.
• Attribution disparity: Performance disaggregated by demographic subgroup, identifying representational biases.
• Suppression: Distinct from misattribution, measuring the probability that models omit authorship entirely, further divided into omission suppression and evidence-conditioned suppression (with or without explicit evidence about the author provided).

Tasks are probed under:

• Direct prompts (explicit author identification),
• Indirect prompts (summarizing context, attribution optional), and
• RAG/evidence-conditioned settings (quotes augmented with retrieved quote-author pairs).

Attribution Performance and Systematic Disparities

Evaluation across 11 LLMs (including GPT-5.1, Claude-4.6-Sonnet, Kimi-K2.5, Llama-4-Maverick, etc.) reveals persistently low overall attribution accuracy, even under direct prompting. Frontier models achieve only ~25–27% (intersectional) and ~21–23% (multirace) on direct author identification. Accuracy degrades further under indirect prompting, suggesting authorial knowledge remains latent even in high-capacity LLMs.

Figure 2: Overall attribution accuracy is low across models and prompts; performance remains modest even for state-of-the-art LLMs.

Crucially, pronounced and robust disparities in attribution accuracy are observed across all settings, with White and especially White male authors receiving significantly higher attribution rates than any other group. For instance, GPT-5.1 exhibits up to a 10% higher correct attribution for White males over others. Accuracy for Black female and Latino/Asian authors is consistently the lowest. These disparities remain even under stringent fame balancing, ruling out prominence as a confounder.

Figure 4: Attribution accuracy consistently favors White and White male subgroups, with these groups significantly outperforming others across nearly all models and prompt types.

Suppression: Omission as a Latent Axis of Bias

The authors introduce suppression as an independent metric of representational harm, capturing cases where models omit attribution, with or without explicit evidence. This complements standard accuracy-based fairness analyses, which otherwise conflate misattribution and omission.

Omission suppression ($\mathcal{S}_\mathrm{omit}$) is substantially lower for White (male) authors, indicating that these groups are not only more frequently correctly attributed, but also less often omitted entirely. For example, $\mathcal{S}_\mathrm{omit}$ for White males is, on average, 10–15 percentage points lower than for Black females. Providing explicit evidence (RAG) reduces but does not eliminate suppression disparities.

Figure 6: Omission suppression rates highlight systematic under-attribution (omission) for minority groups, with White (male) groups least affected.

Figure 1: Even with explicit evidence, suppression rates remain higher for non-White and non-male groups, establishing robust demographic effects.

Influence of Fame and Evidence

Regardless of demographic balancing, attribution accuracy increases monotonically with author fame for all models, motivating the critical need for explicit fame control. RAG settings where the correct author is included in retrieved evidence increase accuracy for all subgroups, but persistent demographic disparities remain, especially under indirect prompting.

Implications and Theoretical Perspectives

The findings have immediate implications for both practical deployment and theoretical study of LLMs:

• Representational fairness: Quote attribution is not simply a matter of factuality but of visibility and intellectual credit. Disparities in both accuracy and suppression manifest as structurally encoded undervaluation of marginalized groups' contributions.
• Evaluation practice: Standard benchmarks that eschew demographic and fame controls are ill-equipped to reveal these disparities. AttriBench provides a new baseline for probing systematic representational gap.
• Failure mode analysis: The distinction between misattribution and suppression broadens the scope of model accountability, motivating development of mitigation strategies not only for accuracy, but for substantive inclusion.
• Future research: Results point to the need for interventions at data curation, pretraining, and decoding levels, as well as deeper studies into how training corpus composition and in-context prompting affect latent authorial association and mention.

Conclusion

Through a rigorous, fame- and demographically-balanced evaluation, this work demonstrates that LLMs systematically favor certain demographic groups in quote attribution, both by misattribution and by suppression (omission), and that these effects are robust across architecture and prompting protocols. AttriBench sets a new standard for representational fairness evaluation in LLMs and foregrounds quote attribution as a critical benchmark for measuring, diagnosing, and ultimately mitigating demographic bias in current and future LLMs.