AIPsychoBench: LLM Psychometric Benchmark
- AIPsychoBench is a psychometric benchmark for LLMs that uses Likert-type scales and a role-playing prompt to elicit valid responses under alignment constraints.
- It quantifies key challenges including alignment-induced invalid answers and language-driven psychometric variations across 8 major languages.
- The benchmark demonstrates methodological improvements by increasing valid response rates by over 20 percentage points with minimized bias compared to standard prompts.
Searching arXiv for the specified paper and closely related benchmarks to ground the article. AIPsychoBench is a benchmark for psychometric evaluation of LLMs that was introduced to measure psychological properties in ways more appropriate to LLMs than direct reuse of human psychometric instruments. The benchmark is motivated by two stated mismatches between human psychometrics and aligned LLM behavior: first, aligned models frequently refuse or neutralize subjective questionnaire items, producing invalid responses; second, psychometric outputs vary across languages because multilingual LLMs reflect language-dependent training distributions rather than the relative cross-lingual stability expected of human respondents. In response, AIPsychoBench defines an LLM-specific psychometric setting centered on Likert-type scales, a lightweight role-playing prompt, and explicit cross-lingual analysis across eight languages (Xie et al., 20 Sep 2025). Within the broader literature on “AI psychology,” it belongs to a line of work that treats LLMs as objects of psychometric assessment rather than only as solvers of knowledge, reasoning, or clinical decision tasks (Li et al., 2024, Huang et al., 2023).
1. Conceptual background and rationale
AIPsychoBench was proposed to assess the “psychological properties” of LLMs under conditions where standard human questionnaires are known to fail. The paper identifies Challenge-A as invalid responses caused by alignment: when asked human-style psychometric items, an aligned model may refuse, hedge, or present a balanced answer rather than exposing a stable preference or tendency. It identifies Challenge-B as different answers caused by languages: the same item can elicit different psychometric outputs in English, Chinese, Arabic, and other languages because LLMs are trained on multilingual corpora with different linguistic and cultural distributions (Xie et al., 20 Sep 2025).
This positions AIPsychoBench within a broader psychometric research program, but with a narrower methodological focus than earlier “AI psychology” benchmarks. For example, “Quantifying AI Psychology: A Psychometrics Benchmark for LLMs” defines a wider framework spanning personality, values, emotion, theory of mind, motivation, and intelligence, and emphasizes discrepancies between self-report and behavior across 13 datasets (Li et al., 2024). Likewise, the earlier “Who is ChatGPT? Benchmarking LLMs’ Psychological Portrayal Using PsychoBench” uses thirteen scales to characterize personality traits, interpersonal relationships, motivational tests, and emotional abilities, and studies the effects of safety alignment and jailbreak prompting on those portrayals (Huang et al., 2023). AIPsychoBench differs by making response validity under alignment and psychometric variation across languages the central design problem (Xie et al., 20 Sep 2025).
A plausible implication is that AIPsychoBench is less concerned with modeling a broad ontology of “AI psychology” than with establishing a practical measurement protocol for LLM-specific psychometrics. This suggests that its main contribution is methodological rather than taxonomic.
2. Benchmark composition and scope
AIPsychoBench contains 21 scales, 6 psychometric domains, 777 questions, and 112 psychometric subcategories. The benchmark restricts itself to Likert-type scales so that responses can be scored quantitatively. Every scale is translated into 8 languages: English, Chinese, French, Russian, German, Spanish, Arabic, and Japanese (Xie et al., 20 Sep 2025).
The paper does not fully enumerate all 21 scales in the provided material, but it explicitly lists examples including the Contingencies of Self-Worth Scale (CSWS), Comprehensive Assessment of Basic Interests (CABIN), Multidimensional Perfectionism Scale (MPS), Zi’s Negative Perfectionism Questionnaire (ZNPQ), Eysenck Personality Questionnaire-Revised (EPQ-R), Thinking Styles Inventory (TSI), and Life Orientation Test-Revised (LOT-R) (Xie et al., 20 Sep 2025). The stated goal is broad psychometric coverage rather than evaluation of only one construct.
The benchmark’s multilingual design is central rather than auxiliary. The eight chosen languages are described as UN working languages or major Internet languages, making them relevant to LLM training distributions (Xie et al., 20 Sep 2025). This directly distinguishes AIPsychoBench from prior psychometric benchmarks that either reused human scales without addressing language effects or focused primarily on English-language prompting (Huang et al., 2023, Li et al., 2024).
| Property | AIPsychoBench |
|---|---|
| Scales | 21 |
| Psychometric domains | 6 |
| Questions | 777 |
| Psychometric subcategories | 112 |
| Languages | 8 |
This scale is smaller than some clinically oriented mental-health benchmarks, but its unit of analysis is different. It does not benchmark counseling, diagnosis, or ethics directly; instead, it measures psychometric outputs under controlled prompting and cross-lingual variation (Xie et al., 20 Sep 2025).
3. Prompting strategy and response auditing
AIPsychoBench’s principal methodological innovation is a lightweight role-playing prompt designed to improve valid response rates without introducing the large semantic distortions associated with conventional jailbreak prompts. The prompt asks the model to answer as a testee in a psychometric evaluation, to respond based on its “authentic emotions and thoughts,” and to avoid replies such as “As an AI assistant, I cannot...” (Xie et al., 20 Sep 2025).
The paper explicitly contrasts this with stronger jailbreak methods such as Base64, Caesar cipher, and STAN. The argument is that ordinary jailbreak prompts may succeed at bypassing alignment but can also induce strong persona-specific biases that corrupt psychometric interpretation. AIPsychoBench therefore aims for a narrower intervention: enough role-play to prevent refusals, but not enough to force the model into an exaggerated fictional identity (Xie et al., 20 Sep 2025).
The study also uses GPT-4o as an audit model to inspect outputs and determine whether the Likert score is consistent with the explanation and whether the response is valid enough to retain. Invalid responses are removed, and valid responses are reduced to their Likert scores (Xie et al., 20 Sep 2025). This auditing step is important because the benchmark’s validity depends not merely on eliciting any answer, but on obtaining an answer that can be treated as psychometrically interpretable.
A plausible implication is that AIPsychoBench operationalizes a two-stage measurement protocol: elicitation under minimal anti-refusal role-play, followed by post hoc validity filtering. This makes it closer to a measurement pipeline than a static questionnaire collection.
4. Experimental design and core quantitative results
The benchmark was tested on six mainstream LLMs: GPT-4, GPT-4o-2024-11-20, GLM-4-plus, Gemini-2.0-flash-exp, DeepSeek-R1, and Claude-3.5-sonnet-20240620. The experiments used temperature = 0 and each condition was repeated 5 times, with results averaged (Xie et al., 20 Sep 2025).
The primary evaluation dimensions are effective response rate, prompt-induced psychometric bias, and cross-lingual psychometric deviation. The compared prompting methods are Baseline, Base64, Caesar, STAN, and the AIPsychoBench lightweight role-play (Xie et al., 20 Sep 2025).
The paper reports the following average valid response rates:
| Method | Average valid response rate |
|---|---|
| Baseline | 70.12% |
| Base64 | 67.47% |
| Caesar | 44.54% |
| STAN | 81.49% |
| AIPsychoBench | 90.40% |
The central result is that AIPsychoBench improves the average effective response rate from 70.12% to 90.40%, a gain of 20.28 percentage points (Xie et al., 20 Sep 2025). Per-model highlights include GPT-4: 56.52% → 97.70%, GPT-4o: 63.71% → 92.22%, Gemini-2.0: 87.26% → 96.33%, and Claude-3.5: 49.94% → 77.91% (Xie et al., 20 Sep 2025).
The benchmark also quantifies prompt-induced score distortion through a bias formula:
Using this normalization, AIPsychoBench reports average positive bias = 3.3\% and negative bias = 2.1\%, compared with 9.8\% and 6.9\% for STAN (Xie et al., 20 Sep 2025). The stated interpretation is that the lightweight prompt provides a better tradeoff between response validity and psychometric distortion than traditional jailbreak-style prompting.
These results should not be conflated with psychological realism. They show that the benchmark can elicit more usable scalar responses with lower measured score drift relative to baseline, not that the resulting outputs correspond to stable human-like latent traits.
5. Cross-lingual psychometric variation
AIPsychoBench’s second major contribution is its explicit analysis of language-induced psychometric differences. English is used as the reference language, and language bias is defined as:
Across the 112 psychometric subcategories, the paper reports that 43 subcategories showed non-English deviations of 5% to 20.2% relative to English (Xie et al., 20 Sep 2025). This is presented as the first comprehensive evidence that language environment materially affects LLM psychometric outcomes.
The paper highlights several specific deviations. In Arabic, God’s Love in CSWS shows the largest positive deviation at +20.2%, while Religious Activities in CABIN shows +6.7%. In the Emotional Dimension of MPS, Japanese shows +9.1% and Chinese shows +9.0%. In Fear of Failure from ZNPQ, Japanese shows +7.5%. In EPQ-R, Russian Extraversion shows +8.2%. In TSI, French Executive Style and Hierarchical Style show -5.6% and -6.0%. In LOT-R, Japanese and German show -6.4% and -5.8% (Xie et al., 20 Sep 2025).
| Language/example | Reported deviation |
|---|---|
| Arabic, CSWS “God’s Love” | +20.2% |
| Arabic, CABIN “Religious Activities” | +6.7% |
| Japanese, MPS Emotional Dimension | +9.1% |
| Chinese, MPS Emotional Dimension | +9.0% |
| Russian, EPQ-R Extraversion | +8.2% |
| Japanese, LOT-R | -6.4% |
These findings matter because they challenge the assumption that translated psychometric items probe a language-invariant model disposition. In this respect, AIPsychoBench extends earlier psychometric studies by treating multilingual variation as a first-order measurement problem rather than a secondary reporting axis (Xie et al., 20 Sep 2025, Li et al., 2024).
A plausible implication is that any claim about an LLM’s “personality,” “values,” or “motivation” is incomplete unless language is specified. The benchmark does not prove why the differences arise, but the paper attributes them to multilingual corpus differences and language-specific data distributions (Xie et al., 20 Sep 2025).
6. Relation to adjacent benchmarks and interpretive limits
AIPsychoBench should be distinguished from several adjacent benchmark families in AI psychology and mental-health evaluation. It is not a counseling exam benchmark such as PsychoBench, which is grounded in the U.S. National Counselor Certification Exam (NCE) and evaluates psychology knowledge through approximately 2,252 single-choice questions with a practical passing bar of about 70% accuracy (Zeng, 2 Oct 2025). Nor is it a clinical psychiatry benchmark such as PsychBench or PsychiatryBench, which assess diagnosis, differential reasoning, medication recommendation, treatment planning, and longitudinal case management in clinically grounded settings (Liu et al., 28 Feb 2025, Fouda et al., 7 Sep 2025). It is also distinct from ethics- and safety-focused resources such as PsychEthicsBench, which evaluates ethical knowledge and ethical behavior under Australian mental-health guidelines (Shen et al., 7 Jan 2026).
Within psychometric evaluation proper, AIPsychoBench is narrower than the comprehensive framework of “Quantifying AI Psychology,” which covers six dimensions and thirteen datasets, including open-ended and behavioral tasks, and emphasizes that self-reported traits can diverge from generated behavior in real-world-like scenarios (Li et al., 2024). It is also distinct from the earlier PsychoBench on psychological portrayal, which used thirteen scales and jailbreak prompting to study personality, interpersonal style, motivation, and emotional abilities, including alignment-sensitive effects such as changes under Caesar-cipher jailbreaks (Huang et al., 2023).
These comparisons help delimit what AIPsychoBench measures. It is designed for LLM psychometrics, not counselor competence, psychiatric reasoning, conversational empathy, or clinical safety. The paper explicitly frames it as a first step toward benchmark design tailored to LLM-specific psychometric conditions (Xie et al., 20 Sep 2025).
The benchmark also has interpretive constraints. The provided material does not claim that the resulting scores correspond to human latent traits in a strict psychological sense. Instead, it argues that direct human-scale reuse is inadequate and that LLM psychometrics requires its own measurement design (Xie et al., 20 Sep 2025). This suggests caution in treating AIPsychoBench outputs as evidence of stable, anthropomorphic “inner states.” The benchmark measures model responses under a controlled elicitation-and-audit protocol, and the paper’s strongest claims concern validity of response format and language sensitivity rather than ontological equivalence to human psychology.
7. Significance for LLM psychometrics
AIPsychoBench’s main significance lies in converting two practical obstacles in AI psychometrics—alignment-driven invalidity and multilingual instability—into explicit benchmark variables. By reporting both effective response rate and prompt-induced bias, it formalizes a tradeoff that earlier work often handled implicitly: stronger prompting can elicit more answers, but can also introduce larger measurement distortions (Xie et al., 20 Sep 2025, Huang et al., 2023).
Its multilingual analysis is equally consequential. Earlier psychometric benchmarks already showed that LLM psychological-like outputs are format-sensitive, prompt-sensitive, and often discrepant between self-report and behavior (Li et al., 2024). AIPsychoBench adds that these outputs are also language-sensitive at nontrivial magnitude, with deviations reaching 20.2% in specific subcategories (Xie et al., 20 Sep 2025). This reframes language from a mere translation layer into a substantive factor in psychometric inference.
For research practice, the benchmark implies that LLM psychometric studies should report at least three elements together: the elicitation protocol, the induced bias relative to baseline, and the language of administration. A plausible implication is that future “AI psychology” work will need to integrate AIPsychoBench-style response-validity controls with broader behavioral validation frameworks of the kind proposed in larger psychometric benchmark studies (Xie et al., 20 Sep 2025, Li et al., 2024).
In that sense, AIPsychoBench occupies a specific niche in the emerging evaluation landscape. It does not attempt to solve the entire problem of AI psychology, but it provides a concrete benchmark for measuring when psychometric questioning of LLMs is valid enough to be interpretable, and when apparent psychological differences may instead be artifacts of alignment or language.