Ambiguous Intentions Hostility Questionnaire

• AIHQ is a 15-item instrument that evaluates hostile attribution bias by presenting realistic ambiguous social scenarios.
• It combines structured ratings (anger, blame, intentionality) with free-text responses to capture nuanced interpretations.
• Automated scoring with fine-tuned language models shows high alignment with human ratings, enhancing research scalability.

Searching arXiv for the specified paper and closely related context. The Ambiguous Intentions Hostility Questionnaire (AIHQ) is a 15-item, vignette-based instrument for assessing hostile attribution bias, defined as the tendency to interpret other people’s actions, especially in negative or ambiguous situations, as intentionally hostile. Originally developed by Combs et al. (2007), the AIHQ is used in research and clinical practice to quantify how participants interpret others’ intentions, how much blame and anger they report, and how aggressively they expect to respond. A 2025 study on automated scoring showed that fine-tuned LLMs can reproduce the most labor-intensive part of the instrument—the scoring of open-ended responses—with alignment comparable to human raters, while preserving established group differences and construct-relevant associations (Lyu et al., 5 Aug 2025).

1. Conceptual scope and instrument structure

The AIHQ was designed to measure hostile attribution bias in ecologically realistic negative social situations. It has been widely used in psychiatric populations such as schizophrenia, paranoia, depression, and anxiety; in traumatic brain injury (TBI) samples, where elevated hostile attributions and aggression are common; and in nonclinical populations such as undergraduates. AIHQ scores correlate with paranoia, depression, anxiety, anger, aggression, and poorer social and functional outcomes, and the instrument is often used to evaluate interventions intended to reduce hostile attribution bias (Lyu et al., 5 Aug 2025).

The questionnaire comprises 15 short written vignettes, grouped into three scenario types with 5 scenarios each: intentional scenarios, in which the negative act is clearly deliberate; ambiguous scenarios, in which intent is unclear; and accidental scenarios, in which the negative outcome is clearly accidental. Participants are asked to imagine that the event happened to them and then provide both structured ratings and open-ended responses.

Component	Prompt focus	Scale
Anger	“How angry would you feel?”	5-point scale
Blame	“How much would you blame the other person?”	5-point scale
Intentionality	“How intentional was the other person’s behavior?”	6-point scale
Attribution of hostility	“Why did the person act that way toward you?”	Open-ended, later scored 1 to 5
Aggression response	“What would you do in that situation?”	Open-ended, later scored 1 to 5

The structured ratings are straightforward self-report measures. The open-ended responses are more distinctive: they capture the content of hostile interpretations and anticipated behavior, and therefore occupy a central role in the instrument’s interpretive value. The same feature also creates the major practical bottleneck, because the free-text responses must traditionally be evaluated by trained human raters.

2. Traditional scoring and psychometric burden

Human scoring of the AIHQ focuses on two open-ended dimensions. Attribution of hostility is scored from 1 to 5, where 1 indicates that the participant clearly interprets the act as accidental or benign, 3 indicates that the behavior is seen as partly intentional but not clearly malicious, and 5 indicates a clear belief that the other person acted on purpose to harm or insult. Aggression response is also scored from 1 to 5, with anchors ranging from 1, passive or no response, through 2, mild information-seeking, and 3, assertive but not overtly aggressive, to 4, verbal aggression, and 5, physical aggression (Lyu et al., 5 Aug 2025).

Raters are trained to apply these criteria consistently, often by scoring practice responses and calibrating against a “gold-standard” rater until acceptable reliability is achieved. Once item-level scores are assigned, raters compute mean scores across the five scenarios within each scenario type and derive overall mean hostility and aggression scores across all 15 scenarios. These aggregated scores are the primary outcome variables used to quantify hostile attribution and aggressive response tendencies.

The psychometric benchmark is high but operationally expensive. In Dataset 1—a TBI plus healthy control sample—the reported human inter-rater reliability was approximately 0.80–0.88 for attribution of hostility and 0.73–0.93 for aggression response. In Dataset 2—an undergraduate sample—human reliability was $r = 0.896$ for hostility and $r = 0.918$ for aggression. The scoring burden is substantial because each participant generates 15 vignettes × 2 open-ended responses × 2 raters, and raters must be trained and monitored for drift. The 2025 study treats this combination of time-intensiveness and rater-variability as the primary motivation for automated scoring.

3. Automated scoring with fine-tuned LLMs

The automated-scoring study evaluated whether LLMs can reproduce human scoring of the AIHQ’s open-ended responses by learning the same coding rubric. Two models were used: GPT-3.5-Turbo and Flan-T5-Large. For each item, the model receives the scenario text and one open-ended response—either the “why” response for attribution of intent or the “what would you do” response for anticipated behavior—and is prompted to output a single number from 1 to 5 corresponding exactly to the human scoring scales (Lyu et al., 5 Aug 2025).

The study used two datasets. Dataset 1, from Neumann et al. (2020), included 170 adults, with 85 participants with traumatic brain injury and 85 healthy controls. TBI severity ranged from complicated mild to severe; participants were at least 6 months post-injury; all spoke fluent English and had adequate language comprehension; and major comorbid neurological or severe psychiatric conditions affecting social cognition were excluded. This dataset was randomly split 50/50, preserving the TBI/healthy control balance, into a training set of 42 TBI + 42 healthy controls and a test set of 43 TBI + 43 healthy controls. Dataset 2, from Combs et al. (2013), included 146 undergraduates and was reserved for out-of-sample generalization.

The fine-tuning objective was to make the model output the human rating for each response given the scenario and the answer text. GPT-3.5-Turbo was fine-tuned in a chat-style conversational format, with temperature = 0 and max tokens = 10 to enforce deterministic, minimal outputs. Flan-T5-Large was fine-tuned in an instruction prompt / question–answer style, with training monitored using loss and ROUGE metrics; epoch 3 yielded the best performance, with the lowest training and validation loss and the highest ROUGE-1, ROUGE-2, and ROUGE-L, and was therefore selected as the final fine-tuned model. Evaluation compared model ratings with mean human ratings using Pearson correlations, human-vs-human agreement, analyses by scenario type and group, replication of TBI-versus-control differences, and convergent validity with intentionality, anger, and blame.

4. Empirical performance and validity

On the held-out half of Dataset 1, the fine-tuned models closely matched human scoring. Human inter-rater reliability was approximately $r \approx 0.88$ for hostility and $r \approx 0.93$ for aggression. Against mean human ratings, fine-tuned GPT-3.5-Turbo achieved $r = 0.934,\ t(84) = 23.99,\ p < .001$ for hostility and $r = 0.962,\ t(84) = 32.28,\ p < .001$ for aggression. Fine-tuned Flan-T5-Large achieved $r = 0.874,\ t(84) = 16.52,\ p < .001$ for hostility and $r = 0.925,\ t(84) = 22.31,\ p < .001$ for aggression. Pre-trained, non-fine-tuned models performed worse, especially Flan-T5 for hostility, where $r = 0.476$, whereas base GPT-3.5-Turbo still achieved $r = 0.840$ for hostility and $r = 0.918$0 for aggression. Fine-tuning therefore improved alignment, especially for the weaker base model (Lyu et al., 5 Aug 2025).

Performance remained high across scenario types and diagnostic groups. For fine-tuned GPT-3.5-Turbo, hostility correlations with human ratings in the TBI group ranged from 0.882 to 0.948 across scenario types, and in the healthy control group from 0.691 to 0.863. For aggression response, correlations ranged from 0.852 to 0.977 in the TBI group and 0.863 to 0.938 in healthy controls. Flan-T5-Large was somewhat lower but still typically above 0.75 across most splits. The reported conclusion is that there was no systematic drop in performance for TBI relative to healthy controls, and that agreement was maintained for ambiguous, intentional, and accidental scenarios.

A more stringent validity test was whether automated scores reproduce known group differences. Using human ratings, prior work had shown that TBI participants exhibit higher hostile attributions and higher aggressive response scores than healthy controls, especially in ambiguous and accidental scenarios. Fine-tuned GPT-3.5-Turbo reproduced these differences with nearly identical values. For all scenarios, hostility, human means were 1.90 for TBI versus 1.71 for healthy controls with $r = 0.918$1, and model means were 1.90 versus 1.71 with $r = 0.918$2. For ambiguous hostility, human means were 1.97 versus 1.69 with $r = 0.918$3, and model means were 1.98 versus 1.68 with $r = 0.918$4. For all scenarios, aggression, human means were 2.17 versus 1.91 with $r = 0.918$5, and model means were 2.17 versus 1.91 with $r = 0.918$6. For ambiguous aggression, human means were 2.07 versus 1.84 with $r = 0.918$7, and model means were 2.04 versus 1.86 with $r = 0.918$8. Flan-T5-Large also replicated these group differences, although with slightly reduced effect sizes in some cells.

The study also examined whether model-derived scores show the same relations to other AIHQ subscales as human scores. In the TBI group of Dataset 1, fine-tuned GPT-3.5-Turbo hostility scores correlated strongly with intentionality for all scenarios at $r = 0.918$9, for ambiguous scenarios at $r \approx 0.88$0, and for accidental scenarios at $r \approx 0.88$1. Aggression scores correlated with intentionality at $r \approx 0.88$2 for all scenarios and $r \approx 0.88$3 for accidental scenarios. Hostility was non-significantly related to anger across all scenarios, similar to Neumann et al. (2020), but correlated for ambiguous scenarios at $r \approx 0.88$4. Aggression correlated with anger at $r \approx 0.88$5 for all scenarios, $r \approx 0.88$6 for ambiguous scenarios, and $r \approx 0.88$7 for accidental scenarios. Hostility correlated with blame at $r \approx 0.88$8 for all scenarios and $r \approx 0.88$9 for ambiguous scenarios, while aggression correlated with blame at $r \approx 0.93$0 for all scenarios and $r \approx 0.93$1 for accidental scenarios. In Dataset 2, model-derived scores again showed similar relations to intentionality, anger, and blame, including hostility versus intentionality at $r \approx 0.93$2 and aggression versus blame at $r \approx 0.93$3. The pattern was reported as closely matching both human scores and prior publications.

5. Generalization and practical deployment

Generalization to unseen data was tested on Dataset 2, which was not used in training and involved a different, nonclinical population. On this undergraduate sample, fine-tuned GPT-3.5-Turbo achieved $r \approx 0.93$4 for hostility and $r \approx 0.93$5 for aggression. Fine-tuned Flan-T5-Large achieved $r \approx 0.93$6 for hostility and $r \approx 0.93$7 for aggression. By scenario type, hostility correlations ranged from 0.704 to 0.770, while aggression was lower for ambiguous scenarios at roughly 0.48–0.54 and higher for intentional and accidental scenarios at roughly 0.84–0.92. Human-human reliability was also lower for ambiguous aggression in this sample, at $r \approx 0.93$8, which contextualizes the weaker model-human alignment (Lyu et al., 5 Aug 2025).

The paper frames these results as evidence that automated scoring can make the AIHQ more feasible in research and clinical settings. Once deployed, scoring is described as near-instantaneous and potentially useful for large research datasets, clinical assessments where staffing is limited, and longitudinal or multi-site studies where consistency is important. A plausible implication is that the AIHQ’s open-ended component, previously constrained by human labor, can be retained rather than omitted when studies scale up.

Several concrete implementation routes are provided. The authors released an open-source fine-tuned Flan-T5-Large model on Hugging Face at https://huggingface.co/lyulouisaa/flant5-finetuned-aihqrating. They also provide a Google Colab notebook that can run the fine-tuned Flan-T5-Large model or call the base GPT-3.5-Turbo via API without local installation, although data then go to Google servers. In addition, they provide a local browser-based interface that accepts a CSV of AIHQ free-text responses and returns model-generated ratings for hostility and aggression. With Flan-T5-Large, this interface can run entirely locally, so no data leave the machine; it also supports GPT-3.5-Turbo if an OpenAI API key is supplied. Documentation is available at https://aihqrating.readthedocs.io/en/latest/index.html.

6. Limitations, controversies, and broader research context

The principal limitation identified by the authors is rater lens dependence. The models do not infer a universal concept of hostile intent independently of human coding practice; rather, they are trained to reproduce the coding style and standards of the raters in the training dataset, which were U.S.-based and shaped by particular training protocols. A common misconception is therefore that automated scoring removes subjectivity altogether. The study instead suggests that automated scoring can standardize a given scoring regime, but only within the boundaries of the human rubric it has learned (Lyu et al., 5 Aug 2025).

The paper also highlights cross-cultural generalizability as an open issue. Hostile attributions vary across cultures, including different patterns of attributing hostility to friends versus strangers in the United States, Poland, and Japan. Models trained on U.S. data may therefore internalize U.S.-specific patterns and may require local fine-tuning on culturally appropriate data. Another limitation is slight degradation in new samples: correlations in the undergraduate dataset were about 0.10 lower than in the held-out TBI plus healthy-control sample, suggesting sensitivity to sample composition or rater differences.

A further limitation is the reduction to a single number. Although the current approach faithfully reproduces the traditional AIHQ coding scheme, it compresses each response to one hostility score and one aggression score. The authors explicitly note possible future extensions, including multidimensional extraction of emotional tone, cognitive themes such as paranoia or self-blame versus other-blame, and types of aggression such as reactive versus proactive aggression. They also suggest integration with neuroimaging, physiological data, and other neuropsychological tests, as well as multilingual and cross-cultural versions trained on translated vignettes and culturally adapted scoring datasets.

Within the broader hostile-attribution literature, the AIHQ occupies a distinctive position. Personality scales such as the Cook-Medley Hostility Scale assess dispositional hostility but lack contextual detail. Simplified stimulus tasks, including ambiguous facial expressions or laughter, capture basic bias but not rich social context. Other scenario-based tools include the Epps Scenarios, SIP-AEQ, and V-SEIP. The AIHQ is especially valued because it combines ecologically realistic scenarios with both quantitative ratings and content-rich free responses, and because it has a substantial evidence base in schizophrenia, TBI, paranoia, aggression, and intervention studies. The automated-scoring work extends that tradition by making the open-ended component more usable at scale without discarding the original interpretive framework.