Demographic Prompting at Scale: When More Attributes Hurt LLM--Human Agreement

Abstract: We investigate how annotator demographic attributes, supplied as prompt cues, shape the alignment between LLM predictions and human annotations across five tasks. Using five open-source LLMs, we systematically vary the number and composition of demographic components in the prompt, spanning every combination from single-attribute through full-attribute configurations. Our experiments reveal three principal findings. First, alignment consistently peaks with one to three high-signal attributes and degrades under the full attribute set, establishing a clear over-specification threshold. Second, the overall magnitude of demographic influence on human annotations does not predict which attributes improve LLM alignment; instead, both the learnability and the directional coherence of each attribute's annotation signal need to be considered jointly. Third, neuron probing reveals that specialized activation correlates with alignment gains only under coherent annotation signals, and that activation volume alone does not imply steerability. Together, these results demonstrate that demographic prompting is not a monolithic intervention: its utility is highly context-dependent, shaped by attribute signal quality, task characteristics, and model architecture.

• The paper finds that LLM-human alignment peaks when using one to three high-signal demographic attributes while extra cues degrade performance.
• The methodology employs combinatorial prompt experiments across five tasks and neuron probing to link demographic cues with alignment outcomes.
• Key implications include guiding optimal attribute selection using learnability and subgroup coherence, cautioning against full demographic over-specification.

Demographic Prompting at Scale: Attribute Selection, Over-Specification, and Neuron-Level Mechanisms in LLM-Human Alignment

Introduction

This paper presents a comprehensive analysis of demographic prompting as an alignment strategy between LLM outputs and human annotations on subjective language tasks. Motivated by increasing use of LLMs for automated annotation in socially sensitive tasks (toxicity detection, sentiment analysis, etc.), the authors interrogate how the inclusion, combination, and structural properties of demographic prompt cues modulate alignment with human-labeled data. Unlike prior work, which is limited to single- or all-attribute comparisons, the study conducts a granular combinatorial enumeration of demographic prompts across five tasks and model architectures, and further integrates dataset-level analyses of annotation signal and direct probing of neuron activations to identify conditions under which demographic prompting succeeds or fails.

Experimental Setup

The study benchmarks five open-source instruction-tuned LLMs (Llama-3.2-3B, Mistral-7B, Gemma-3-12B, Qwen2.5-7B, DeepSeek-R1-7B) on five subjective language understanding tasks: toxicity, sentiment, politeness, offensiveness, and emotion attribution. Each underlying dataset is annotated by human raters with rich demographic metadata, enabling construction of prompts conditioned on any subset of available demographic attributes.

Model-human alignment is quantified via quadratic-weighted Cohen's $\kappa$ (for ordinal-labeled tasks) and accuracy (for categorical emotion), using bootstrapped confidence intervals. For each model-task pair, all possible single and multi-attribute prompt permutations, including the baseline (no demographic context), are tested to enumerate the full alignment landscape. Robustness is ensured by averaging across paraphrased prompt templates and randomization of label orders, mitigating positional biases.

Figure 1: Best quadratic-weighted Cohen's $\kappa$ on the Toxicity task by the number of demographic prompt components across five LLMs.

Figure 2: Best quadratic-weighted Cohen's $\kappa$ on the Sentiment task by the number of demographic prompt components across five LLMs.

Figure 3: Best quadratic-weighted Cohen's $\kappa$ on the Politeness task by the number of demographic prompt components across five LLMs.

Figure 4: Best quadratic-weighted Cohen's $\kappa$ on the Offensiveness task by the number of demographic prompt components across five LLMs.

Figure 5: Best accuracy on the Emotion task by the number of demographic prompt components across five LLMs.

Key Findings: Alignment Peaks and Over-Specification

The empirical results surface a non-monotonic trajectory for model-human alignment as a function of demographic prompt complexity. Across all tasks and models, alignment typically peaks with prompts containing one to three high-signal demographic components. Critically, the inclusion of additional attributes causes a statistically significant and consistent degradation in $\kappa$, with the full attribute set never outperforming targeted compact configurations (Figures 1–5). This over-specification threshold robustly holds across four of five tested models for toxicity, with model-specificity modulating the inflection point and identity of optimal attributes.

On toxicity, Qwen2.5-7B, Llama-3.2-3B, and Mistral-7B achieve substantial $\kappa$ gains over baseline using two or three attributes (e.g., combinations involving lgbtq_status, race, or political_affiliation); DeepSeek-R1-7B, by contrast, exhibits maximal alignment in the baseline condition with all demographic cueing detrimental. In sentiment and emotion, demographic prompting offers muted, model- and attribute-specific improvements—frequently, no configuration outperforms the baseline for Gemma or Mistral, and gains for Qwen or Llama hinge on very specific attribute sets. Politeness and offensiveness display broad resistance to demographic prompting; performance is often best at the unconditioned baseline and degrades with demographic cues.

Dataset-Signal Analysis: Importance, Learnability, and Directional Coherence

To diagnose the underpinnings of these combinatorial effects, the study introduces a three-level framework distinguishing between (1) the magnitude of a demographic attribute’s influence on human annotation labels (SHAP importance), (2) the learnability of attribute-conditioned lexical patterns (measured with word-demographic interaction LSVC models), and (3) the directional coherence of annotation signals across attribute subgroups (quantified by Fisher-averaged Spearman correlations on interaction weights).

Empirically, SHAP importance is uninformative for predicting whether a demographic attribute, when prompted, will improve alignment. High SHAP attributes can carry conflicting subgroup signals, defeating persona-based steering. LSVC learnability correlates positively with alignment only when the underlying annotation signal is directionally coherent—i.e., when attribute subgroups agree lexically on what constitutes the target label. The critical insight is that attributes with high learnability but directionally opposed subgroup signals (e.g., gender on sentiment/offensiveness) systematically degrade alignment due to irreconcilable prompt constraints.

This framework enables principled selection of demographic attributes: compact prompts leveraging attributes with high LSVC learnability and positive Fisher coherence maximize alignment. Conversely, stacking attributes with diverging subgroup signals or low coherence leads to over-specification penalties.

Neuron Probing: Specialized Activation and the High-Volume Paradox

The paper applies neuron probing to link demographic prompting and alignment outcomes at the circuit level. By recording and comparing the distribution of high-magnitude MLP neuron activations across baseline and persona-prompted generations, the study tests whether the internal engagement of specialized neurons predicts alignment gains.

For toxicity, the hypothesis holds: LLMs (notably Llama, Qwen, and Mistral) that activate higher proportions of specialized neurons in response to coherent demographic prompts show substantial gains in $\kappa$. However, this relationship is not universal. DeepSeek-R1-7B demonstrates the high-volume paradox: while activating the most specialized neurons across all tasks/attributes, its demographic-prompt performance never improves over baseline, suggesting that raw activation volume reflects undifferentiated representational spread rather than targeted, annotation-relevant processing. Furthermore, on tasks and models where demographic signals are opposed (e.g., Gemma on Sentiment and Emotion), attributes with the highest specialized-neuron counts induce the largest drops in alignment.

Figure 6: Ablation study for Toxicity and Sentiment Task—masking the top-k specialized neurons impacts LLM-task performance, verifying neuron selectivity.

Thus, specialized neuron activation is a conditional correlate of steerability: it denotes effective engagement only when annotation signals are homogeneous and prompt-exploitable.

Implications and Future Directions

This work demonstrates, with unprecedented combinatorial resolution, that demographic prompting is not a uniformly effective or trivial intervention for LLM alignment. Its efficacy is highly context-dependent, governed by the structural quality of the annotation signal, the underlying model’s internal representations, and interaction effects across demographic attributes. The prevalent intuition that more demographic specificity yields better alignment is empirically invalidated. Instead, attribute selection should be guided by diagnostic indicators of learnability and coherence, and practitioners must avoid full over-specification.

From a mechanistic perspective, the findings caution against naïve interpretations of neuron activation as a marker for steerability or targeted knowledge engagement. The high-volume paradox exposes a failure mode where broad neuron recruitment degrades, rather than enhances, alignment.

Open research questions include extension to other languages and more diverse sociocultural settings, fine-grained intersectional subgroup modeling, and the causal manipulation of candidate neurons to directly test steerability. There is also opportunity to design data-driven or interpretable frameworks for automatic attribute selection in demographic prompting, as well as systematic analysis of the impact of training procedures (e.g., distillation, RLHF) on persona sensitivity.

Conclusion

This paper provides definitive empirical evidence that the utility of demographic prompting is narrowly circumscribed by the interaction between task, model, and the structural quality of annotation signals. Alignment consistently peaks under compact, coherent attribute sets and degrades with over-specification. The relationship between dataset-side demographic signal and LLM steerability is nontrivial, requiring both learnability and subgroup consensus. Neuron-level analysis confirms that specialized activation parallels alignment only under coherent signaling; otherwise, it can induce performance deterioration. The findings synthesize methodological rigor with fine-grained mechanistic analysis, yielding actionable guidance for principled demographic prompting and laying the groundwork for deeper exploration of social-contextual alignment in LLMs.

Reference:

"Demographic Prompting at Scale: When More Attributes Hurt LLM--Human Agreement" (2607.10590)