- The paper identifies two consistent alignment clusters: models correlate positively with explicit behaviors such as insult and violence, but systematically invert evaluative attributes such as respect, sentiment, status, and hatespeech.
- The paper shows that poorly aligned evaluative attributes receive the highest model confidence, while demographic persona conditioning lowers confidence without meaningfully improving alignment.
- The paper’s confidence-weighted Ridge pipeline reconstructs continuous human hate-speech scores with R² up to 0.71 and approximately 84% accuracy, outperforming direct prompting baselines and producing more balanced predictions.
This paper presents a systematic, attribute-level analysis of how LLMs align with human hate speech annotations, using the Measuring Hate Speech (MHS) corpus as the evaluation substrate. Rather than treating annotation simulation as a single holistic label prediction, the authors decompose hate speech into ten theoretically grounded ordinal attributes—insult, humiliate, dehumanize, violence, genocide, attack/defend, status, respect, sentiment, and hatespeech—and prompt four open-weight LLMs (Llama-3.1-8B/70B-Instruct and Qwen2.5-7B/72B-Instruct) to rate each attribute independently. The central contributions are (1) the identification of a consistent two-cluster alignment pattern across all models, and (2) a confidence-weighted Ridge regression pipeline that reconstructs the continuous IRT-derived hate speech score from attribute-level predictions, outperforming direct prompting baselines (2605.27025).
Motivation and positioning
Hate speech annotation is costly, subjective, and prone to systematic annotator disagreement that cannot be resolved by adjudication. This has motivated a growing line of work on LLMs as scalable substitutes for human annotators, but existing approaches collapse subjective constructs into single labels or scalar scores, offering no insight into which facets of the construct the model understands and where it diverges. The MHS corpus—with 39,565 comments annotated by 7,912 crowd workers on ten ordinal attributes, aggregated via Item Response Theory into a continuous score while accounting for annotator reliability—is uniquely suited to attribute-level diagnosis. The paper positions itself against prior work showing that LLMs struggle in no-consensus tasks, that naive aggregation of LLM outputs can collapse meaningful variation, and that persona-based LLM biases differ systematically from human annotator biases; its contribution is to extend this line from the label level to the attribute level.
Method
For each comment and each of the ten attributes, the model is prompted with the attribute definition and ordinal scale anchors drawn verbatim from the original MHS annotation guidelines, under two conditions: a vanilla condition with no demographic context, and a persona condition conditioned on the real demographic profile (gender, age, race, religion, political ideology) of the corresponding MHS annotator, producing one prediction per (comment, annotator, attribute) triple. Decoding is deterministic (temperature 0, single token), enabling extraction of confidence scores from token-level log probabilities renormalized via softmax over label tokens. Attribute-level alignment is measured by Spearman rank correlation between LLM predictions and mean human ratings.
For score reconstruction, per-attribute predictions are multiplied by their extracted confidence to form weighted features xn,i​=Sn,i​⋅Cn,i​, which feed a Ridge regression trained to predict the continuous IRT hate speech score y^​n​, evaluated with 5-fold cross-validation. Four direct binary prompting baselines (zero-shot, few-shot, definition-based, attribute-aware) are compared on classification performance.
The two-cluster alignment pattern
The headline finding is a consistent split across all four models and both prompting conditions:
| Cluster |
Attributes |
Spearman range (large models) |
| Behaviorally explicit |
insult, humiliate, attack/defend, violence, dehumanize, genocide |
+53 to +72 |
| Evaluative |
respect, sentiment, status, hatespeech |
−52 to −75 |
Behaviorally explicit dimensions correlate strongly and positively with human judgments, while evaluative dimensions are systematically inverted—the models apply a different judgment schema for dimensions requiring holistic or contextual assessment. This pattern holds regardless of model family or scale, though smaller models show deviations: Llama-3.1-8B's sentiment correlation collapses to near zero (−1.56) and status flips positive (+38.68) under vanilla prompting, indicating that small models struggle to maintain consistent evaluative judgments.
A more striking result concerns calibration: the poorly aligned evaluative attributes attract the highest model confidence (0.88–0.95), clustering in the bottom-right quadrant of the confidence–correlation plane. The models are most certain precisely where they diverge most from human perception—a property that would make naive use of these models as annotators particularly hazardous, since standard confidence-based filtering or human-in-the-loop triage schemes would not flag these errors.
Persona conditioning reduces confidence without improving alignment. Demographic persona prompts consistently shift attribute confidence leftward across both large models while leaving Spearman correlations essentially unchanged. This confirms that demographic conditioning affects model certainty but not conceptual understanding, reinforcing prior evidence that sociodemographic prompting is not yet an effective approach for simulating subjective judgments.
Score reconstruction results
The confidence-weighted Ridge pipeline achieves R2 of up to 70.57 (Llama-70B vanilla) and 70.71 (Llama-70B persona), and 68.85/68.65 for Qwen-72B, substantially outperforming all direct prompting baselines on accuracy (~84% vs. 57–70%) with competitive F1. Notably, Qwen baselines achieve high F1 only through extreme prediction bias—recall up to 99% with precision as low as 34%, effectively flagging nearly every comment as hate speech—whereas the pipeline produces calibrated predictions with balanced precision (~71%) and recall (~66%).
An instructive discrepancy emerges: Qwen shows marginally stronger per-attribute Spearman correlations than Llama, yet Llama achieves higher reconstruction R2. Aggregate reconstruction quality therefore does not follow directly from per-attribute alignment—an observation the paper leaves unexplained mechanistically.
The ablation study isolates three findings. First, Ridge regression is essential: replacing it with Spearman- or confidence-weighted sums collapses R2 to strongly negative values (as low as −13.24), because without learned weights the inverted evaluative attributes subtract from the score in an uncontrolled way. Second, confidence weighting yields a modest but consistent gain over unweighted Ridge (e.g., 70.57 vs. 68.63 for Llama-70B). Third, Spearman pre-scaling is redundant—Ridge absorbs it into its learned weights automatically. Inspection of the learned Ridge weights confirms that the model exploits the inversion signal rather than discarding it: evaluative attributes receive large negative weights, turning a misalignment into a useful predictive feature.
Smaller models retain the pattern at reduced strength (R2 up to 62.55 for Llama-8B and 60.53 for Qwen-7B), with better accuracy and precision than baselines, though some baselines achieve slightly higher F1 through the same high-recall/low-precision bias; few-shot prompting for Llama-8B nearly collapses (99.54% recall), suggesting instability of few-shot prompting for small models on this task.
Qualitative examples illustrate the mechanism concretely: on an extreme hate speech comment (human score 6.30), Llama correctly rates all behavioral attributes at maximum but rates sentiment, respect, and status at zero where humans rate them at 4, causing the pipeline to severely underestimate the score (1.88). Conversely, on a supportive comment, behavioral attributes are correctly zeroed but sentiment and respect are maximized where humans rate them at 0.
Limitations and open questions
The paper is candid about scope constraints. The analysis relies exclusively on the MHS corpus, which the authors note is the only dataset combining multi-attribute annotation, continuous IRT scoring, and annotator demographics; extension to other corpora requires comparable annotation depth. Only two model families in two size regimes are evaluated, constrained by compute, though cross-family consistency suggests the patterns are not idiosyncratic. Persona conditioning is limited to the demographic fields available in MHS; richer signals such as annotation history or cultural context cannot be incorporated. The Ridge aggregation is supervised, requiring a labeled training set—an inherent constraint the authors acknowledge, with unsupervised alternatives left unexplored. Two substantive questions remain open: why evaluative attributes are systematically inverted (the paper identifies the pattern but does not explain its cause), and why aggregate reconstruction quality decouples from per-attribute alignment between model families.
Conclusion
This work demonstrates that diagnosing LLM annotation alignment at the attribute level yields findings invisible to end-to-end evaluation: behaviorally explicit hate speech dimensions align well with human judgments across four models, while evaluative dimensions are confidently and systematically inverted, and demographic persona conditioning modulates certainty without correcting understanding. Exploiting this structure, a simple confidence-weighted Ridge regression over attribute predictions reconstructs the continuous IRT hate speech score with R2 up to 0.71, outperforming direct prompting in accuracy and calibration. The practical implication is twofold: structured decomposition recovers more human-aligned signal than holistic label prediction, yet the confident inversion of evaluative attributes argues strongly for using such systems to assist rather than replace human annotators in content moderation.
