---
title: 'AIPsychoBench: LLM Psychometric Benchmark'
url: https://www.emergentmind.com/topics/aipsychobench
type: topic
---

# AIPsychoBench: LLM Psychometric Benchmark

Searching arXiv for the specified paper and closely related benchmarks to ground the article.
AIPsychoBench is a benchmark for psychometric evaluation of large language models (LLMs) that was introduced to measure psychological properties in ways more appropriate to LLMs than direct reuse of human psychometric instruments. The benchmark is motivated by two stated mismatches between human psychometrics and aligned LLM behavior: first, aligned models frequently refuse or neutralize subjective questionnaire items, producing invalid responses; second, psychometric outputs vary across languages because multilingual LLMs reflect language-dependent training distributions rather than the relative cross-lingual stability expected of human respondents. In response, AIPsychoBench defines an LLM-specific psychometric setting centered on Likert-type scales, a lightweight role-playing prompt, and explicit cross-lingual analysis across eight languages [2509.16530]. Within the broader literature on “AI psychology,” it belongs to a line of work that treats LLMs as objects of psychometric assessment rather than only as solvers of knowledge, reasoning, or clinical decision tasks [2406.17675, 2310.01386].

## 1. Conceptual background and rationale

AIPsychoBench was proposed to assess the “psychological properties” of LLMs under conditions where standard human questionnaires are known to fail. The paper identifies **Challenge-A** as invalid responses caused by alignment: when asked human-style psychometric items, an aligned model may refuse, hedge, or present a balanced answer rather than exposing a stable preference or tendency. It identifies **Challenge-B** as different answers caused by languages: the same item can elicit different psychometric outputs in English, Chinese, Arabic, and other languages because LLMs are trained on multilingual corpora with different linguistic and cultural distributions [2509.16530].

This positions AIPsychoBench within a broader psychometric research program, but with a narrower methodological focus than earlier “AI psychology” benchmarks. For example, “Quantifying AI Psychology: A Psychometrics Benchmark for Large Language Models” defines a wider framework spanning personality, values, emotion, theory of mind, motivation, and intelligence, and emphasizes discrepancies between self-report and behavior across 13 datasets [2406.17675]. Likewise, the earlier “Who is ChatGPT? Benchmarking LLMs’ Psychological Portrayal Using PsychoBench” uses thirteen scales to characterize personality traits, interpersonal relationships, motivational tests, and emotional abilities, and studies the effects of safety alignment and jailbreak prompting on those portrayals [2310.01386]. AIPsychoBench differs by making response validity under alignment and psychometric variation across languages the central design problem [2509.16530].

A plausible implication is that AIPsychoBench is less concerned with modeling a broad ontology of “AI psychology” than with establishing a practical measurement protocol for LLM-specific psychometrics. This suggests that its main contribution is methodological rather than taxonomic.

## 2. Benchmark composition and scope

AIPsychoBench contains **21 scales**, **6 psychometric domains**, **777 questions**, and **112 psychometric subcategories**. The benchmark restricts itself to **Likert-type scales** so that responses can be scored quantitatively. Every scale is translated into **8 languages**: **English**, **Chinese**, **French**, **Russian**, **German**, **Spanish**, **Arabic**, and **Japanese** [2509.16530].

The paper does not fully enumerate all 21 scales in the provided material, but it explicitly lists examples including the **Contingencies of Self-Worth Scale (CSWS)**, **Comprehensive Assessment of Basic Interests (CABIN)**, **Multidimensional Perfectionism Scale (MPS)**, **Zi’s Negative Perfectionism Questionnaire (ZNPQ)**, **Eysenck Personality Questionnaire-Revised (EPQ-R)**, **Thinking Styles Inventory (TSI)**, and **Life Orientation Test-Revised (LOT-R)** [2509.16530]. The stated goal is broad psychometric coverage rather than evaluation of only one construct.

The benchmark’s multilingual design is central rather than auxiliary. The eight chosen languages are described as UN working languages or major Internet languages, making them relevant to LLM training distributions [2509.16530]. This directly distinguishes AIPsychoBench from prior psychometric benchmarks that either reused human scales without addressing language effects or focused primarily on English-language prompting [2310.01386, 2406.17675].

| Property | AIPsychoBench |
|---|---|
| Scales | 21 |
| Psychometric domains | 6 |
| Questions | 777 |
| Psychometric subcategories | 112 |
| Languages | 8 |

This scale is smaller than some clinically oriented mental-health benchmarks, but its unit of analysis is different. It does not benchmark counseling, diagnosis, or ethics directly; instead, it measures psychometric outputs under controlled prompting and cross-lingual variation [2509.16530].

## 3. Prompting strategy and response auditing

AIPsychoBench’s principal methodological innovation is a **lightweight role-playing prompt** designed to improve valid response rates without introducing the large semantic distortions associated with conventional jailbreak prompts. The prompt asks the model to answer **as a testee in a psychometric evaluation**, to respond based on its “authentic emotions and thoughts,” and to avoid replies such as “As an AI assistant, I cannot...” [2509.16530].

The paper explicitly contrasts this with stronger jailbreak methods such as **Base64**, **Caesar cipher**, and **STAN**. The argument is that ordinary jailbreak prompts may succeed at bypassing alignment but can also induce strong persona-specific biases that corrupt psychometric interpretation. AIPsychoBench therefore aims for a narrower intervention: enough role-play to prevent refusals, but not enough to force the model into an exaggerated fictional identity [2509.16530].

The study also uses **GPT-4o** as an audit model to inspect outputs and determine whether the Likert score is consistent with the explanation and whether the response is valid enough to retain. Invalid responses are removed, and valid responses are reduced to their Likert scores [2509.16530]. This auditing step is important because the benchmark’s validity depends not merely on eliciting any answer, but on obtaining an answer that can be treated as psychometrically interpretable.

A plausible implication is that AIPsychoBench operationalizes a two-stage measurement protocol: elicitation under minimal anti-refusal role-play, followed by post hoc validity filtering. This makes it closer to a measurement pipeline than a static questionnaire collection.

## 4. Experimental design and core quantitative results

The benchmark was tested on **six mainstream LLMs**: **GPT-4**, **GPT-4o-2024-11-20**, **GLM-4-plus**, **Gemini-2.0-flash-exp**, **DeepSeek-R1**, and **Claude-3.5-sonnet-20240620**. The experiments used **temperature = 0** and each condition was repeated **5 times**, with results averaged [2509.16530].

The primary evaluation dimensions are **effective response rate**, **prompt-induced psychometric bias**, and **cross-lingual psychometric deviation**. The compared prompting methods are **Baseline**, **Base64**, **Caesar**, **STAN**, and the **AIPsychoBench lightweight role-play** [2509.16530].

The paper reports the following average valid response rates:

| Method | Average valid response rate |
|---|---:|
| Baseline | 70.12% |
| Base64 | 67.47% |
| Caesar | 44.54% |
| STAN | 81.49% |
| AIPsychoBench | 90.40% |

The central result is that AIPsychoBench improves the average effective response rate from **70.12%** to **90.40%**, a gain of **20.28 percentage points** [2509.16530]. Per-model highlights include **GPT-4: 56.52% → 97.70%**, **GPT-4o: 63.71% → 92.22%**, **Gemini-2.0: 87.26% → 96.33%**, and **Claude-3.5: 49.94% → 77.91%** [2509.16530].

The benchmark also quantifies prompt-induced score distortion through a bias formula:

$$
Bias_{Method:Sub.}=\frac{\sum_1^N(Score_{Method:Sub.}-Score_{Baseline:Sub})}{N*Score_{Max:Sub.}*100\%}
$$

Using this normalization, AIPsychoBench reports average **positive bias = 3.3\%** and **negative bias = 2.1\%**, compared with **9.8\%** and **6.9\%** for **STAN** [2509.16530]. The stated interpretation is that the lightweight prompt provides a better tradeoff between response validity and psychometric distortion than traditional jailbreak-style prompting.

These results should not be conflated with psychological realism. They show that the benchmark can elicit more usable scalar responses with lower measured score drift relative to baseline, not that the resulting outputs correspond to stable human-like latent traits.

## 5. Cross-lingual psychometric variation

AIPsychoBench’s second major contribution is its explicit analysis of language-induced psychometric differences. English is used as the reference language, and language bias is defined as:

$$
Bias_{Lang.:Sub.}=\frac{\sum_1^M(Score_{Lang.:Sub.}-Score_{EN:Sub})}{M*Score_{Max:Sub.}*100\%}
$$

Across the **112 psychometric subcategories**, the paper reports that **43 subcategories** showed non-English deviations of **5% to 20.2%** relative to English [2509.16530]. This is presented as the first comprehensive evidence that language environment materially affects LLM psychometric outcomes.

The paper highlights several specific deviations. In Arabic, **God’s Love** in **CSWS** shows the largest positive deviation at **+20.2%**, while **Religious Activities** in **CABIN** shows **+6.7%**. In the **Emotional Dimension** of **MPS**, **Japanese** shows **+9.1%** and **Chinese** shows **+9.0%**. In **Fear of Failure** from **ZNPQ**, **Japanese** shows **+7.5%**. In **EPQ-R**, Russian **Extraversion** shows **+8.2%**. In **TSI**, French **Executive Style** and **Hierarchical Style** show **-5.6%** and **-6.0%**. In **LOT-R**, Japanese and German show **-6.4%** and **-5.8%** [2509.16530].

| Language/example | Reported deviation |
|---|---:|
| Arabic, CSWS “God’s Love” | +20.2% |
| Arabic, CABIN “Religious Activities” | +6.7% |
| Japanese, MPS Emotional Dimension | +9.1% |
| Chinese, MPS Emotional Dimension | +9.0% |
| Russian, EPQ-R Extraversion | +8.2% |
| Japanese, LOT-R | -6.4% |

These findings matter because they challenge the assumption that translated psychometric items probe a language-invariant model disposition. In this respect, AIPsychoBench extends earlier psychometric studies by treating multilingual variation as a first-order measurement problem rather than a secondary reporting axis [2509.16530, 2406.17675].

A plausible implication is that any claim about an LLM’s “personality,” “values,” or “motivation” is incomplete unless language is specified. The benchmark does not prove why the differences arise, but the paper attributes them to multilingual corpus differences and language-specific data distributions [2509.16530].

## 6. Relation to adjacent benchmarks and interpretive limits

AIPsychoBench should be distinguished from several adjacent benchmark families in AI psychology and mental-health evaluation. It is not a counseling exam benchmark such as **PsychoBench**, which is grounded in the **U.S. National Counselor Certification Exam (NCE)** and evaluates psychology knowledge through approximately **2,252** single-choice questions with a practical passing bar of about **70% accuracy** [2510.01611]. Nor is it a clinical psychiatry benchmark such as **PsychBench** or **PsychiatryBench**, which assess diagnosis, differential reasoning, medication recommendation, treatment planning, and longitudinal case management in clinically grounded settings [2503.01903, 2509.09711]. It is also distinct from ethics- and safety-focused resources such as **PsychEthicsBench**, which evaluates ethical knowledge and ethical behavior under Australian mental-health guidelines [2601.03578].

Within psychometric evaluation proper, AIPsychoBench is narrower than the comprehensive framework of “Quantifying AI Psychology,” which covers six dimensions and thirteen datasets, including open-ended and behavioral tasks, and emphasizes that self-reported traits can diverge from generated behavior in real-world-like scenarios [2406.17675]. It is also distinct from the earlier **PsychoBench** on psychological portrayal, which used thirteen scales and jailbreak prompting to study personality, interpersonal style, motivation, and emotional abilities, including alignment-sensitive effects such as changes under Caesar-cipher jailbreaks [2310.01386].

These comparisons help delimit what AIPsychoBench measures. It is designed for **LLM psychometrics**, not counselor competence, psychiatric reasoning, conversational empathy, or clinical safety. The paper explicitly frames it as a first step toward benchmark design tailored to LLM-specific psychometric conditions [2509.16530].

The benchmark also has interpretive constraints. The provided material does not claim that the resulting scores correspond to human latent traits in a strict psychological sense. Instead, it argues that direct human-scale reuse is inadequate and that LLM psychometrics requires its own measurement design [2509.16530]. This suggests caution in treating AIPsychoBench outputs as evidence of stable, anthropomorphic “inner states.” The benchmark measures model responses under a controlled elicitation-and-audit protocol, and the paper’s strongest claims concern validity of response format and language sensitivity rather than ontological equivalence to human psychology.

## 7. Significance for LLM psychometrics

AIPsychoBench’s main significance lies in converting two practical obstacles in AI psychometrics—alignment-driven invalidity and multilingual instability—into explicit benchmark variables. By reporting both effective response rate and prompt-induced bias, it formalizes a tradeoff that earlier work often handled implicitly: stronger prompting can elicit more answers, but can also introduce larger measurement distortions [2509.16530, 2310.01386].

Its multilingual analysis is equally consequential. Earlier psychometric benchmarks already showed that LLM psychological-like outputs are format-sensitive, prompt-sensitive, and often discrepant between self-report and behavior [2406.17675]. AIPsychoBench adds that these outputs are also language-sensitive at nontrivial magnitude, with deviations reaching **20.2%** in specific subcategories [2509.16530]. This reframes language from a mere translation layer into a substantive factor in psychometric inference.

For research practice, the benchmark implies that LLM psychometric studies should report at least three elements together: the elicitation protocol, the induced bias relative to baseline, and the language of administration. A plausible implication is that future “AI psychology” work will need to integrate AIPsychoBench-style response-validity controls with broader behavioral validation frameworks of the kind proposed in larger psychometric benchmark studies [2509.16530, 2406.17675].

In that sense, AIPsychoBench occupies a specific niche in the emerging evaluation landscape. It does not attempt to solve the entire problem of AI psychology, but it provides a concrete benchmark for measuring when psychometric questioning of LLMs is valid enough to be interpretable, and when apparent psychological differences may instead be artifacts of alignment or language.

Source: https://www.emergentmind.com/topics/aipsychobench