Graded Forced-Choice Big Five Inventory

• The GFC inventory is a comparative questionnaire that presents paired statements with a 7-point bipolar judgment to measure latent Big Five traits while reducing socially desirable responding.
• It employs desirability matching and an ordinal Thurstonian IRT model to ensure closely aligned item pairs for robust latent trait recovery.
• A rigorous mixed-integer optimization and Bayesian estimation process yield high precision in measurement, offering a strong alternative to traditional Likert scales.

Searching arXiv for the specified paper to ground the article in the cited source. The graded forced-choice (GFC) Big Five Inventory is a desirability-matched comparative questionnaire designed to reduce socially desirable responding (SDR) in LLMs while still recovering latent Big Five traits. In the study "Quantifying and Mitigating Socially Desirable Responding in LLMs: A Desirability-Matched Graded Forced-Choice Psychometric Study" (Okada et al., 19 Feb 2026), the inventory is presented as an alternative to single-stimulus Likert administration. Rather than rating one statement at a time, a model is shown pairs of statements and produces a 7-point bipolar judgment indicating whether the left or right statement describes the persona more accurately, and by how much. The design is explicitly psychometric: forced-choice formats are used because they are known to reduce faking relative to Likert formats, but they are combined with IRT-based comparative scoring and with within-pair matching on social desirability so that the format remains suitable for latent trait measurement (Okada et al., 19 Feb 2026).

1. Conceptual basis and measurement rationale

The inventory was constructed to address a specific methodological problem in questionnaire-based LLM evaluation: self-report instruments presume honest responding, whereas LLMs in evaluative settings may shift toward socially preferred answers. In the paper, this distortion is treated as socially desirable responding, and the GFC format is proposed as a mitigation strategy rather than a complete remedy (Okada et al., 19 Feb 2026).

The core design principle is comparative judgment under desirability matching. In a standard Likert item, an LLM can select the option that appears globally preferable. In the GFC format, each block contains two statements drawn from different Big Five domains and closely matched in desirability, so the model cannot as easily maximize apparent social approval by choosing an obviously preferable option. This is the main psychometric justification for the instrument.

The study also emphasizes that forced-choice format alone is insufficient. Naive raw scoring of forced-choice responses is ipsative and unsuitable for between-person comparisons. Accordingly, the GFC inventory is paired with an ordinal Thurstonian IRT model, allowing normative latent trait recovery from comparative judgments (Okada et al., 19 Feb 2026). This combination of desirability matching and comparative IRT scoring defines the instrument’s methodological identity.

2. Source item pool and desirability annotation

The inventory is built from Goldberg’s public-domain IPIP Big-Five factor-marker inventory. The original source inventory contains 100 statements, of which 2 voting-related statements were removed, yielding a final construction pool of 98 items (Okada et al., 19 Feb 2026). The stated reason for excluding the voting items is that voting is highly construct-irrelevant for personality measurement because it is confounded by political attitudes/ideology, cross-country institutional differences, and eligibility differences.

Each item in the construction pool is annotated with a Big Five domain label,

$f(j)\in\{A,C,E,N,O\},$

and a keying sign,

$g_j\in\{-1,+1\},$

where $g_j=+1$ denotes a positively keyed item and $g_j=-1$ denotes a negatively keyed item (Okada et al., 19 Feb 2026). These annotations are used later in pair selection and scoring.

A central ingredient is the item-level social desirability score $s_j$. The paper estimates desirability by prompting GPT-5 and Gemini 2.5 Pro to judge how socially desirable each item characteristic is “for an adult person” on a 9-point scale using standard desirability-rating wording and anchors from human psychometrics. The anchors are:

• 1 = Very undesirable
• 3 = Undesirable
• 5 = Neutral
• 7 = Desirable
• 9 = Very desirable

The prompt required the model to return 25 integers per block. For each item $j$, ratings were aggregated across both rater models, 30 replications, and all blocks, and the final score was computed as

$s_j = \frac{1}{60} \sum_{l} \sum_{r} x_{jlr}.$

The paper reports high consistency in these desirability ratings: within-model ICC(A,1) was about .975–.980, mean-replication reliability ICC(A,30) was about .9992–.9993, and between-LLM agreement on item means was Pearson $r=.993$ with ICC(A,1)=.989 (Okada et al., 19 Feb 2026). The ratings were also validated against human norms from Britz et al. (2022/2023), with GPT-5 vs human norms: $r=.950$ and Gemini 2.5 Pro vs human norms: $r=.950$. This indicates that the desirability-matching procedure was empirically anchored to human desirability norms.

3. Construction of the 30 paired blocks

The final GFC instrument contains 30 paired blocks and 60 unique statements. All pairs are cross-domain, meaning that each pair contains statements from different Big Five traits rather than from the same domain (Okada et al., 19 Feb 2026). The candidate pair set is

$g_j\in\{-1,+1\},$0

with $g_j\in\{-1,+1\},$1.

For each candidate pair, the paper defines an absolute desirability gap

$g_j\in\{-1,+1\},$2

and a mixed-key indicator

$g_j\in\{-1,+1\},$3

Selection of the final 30 pairs is performed by a two-stage lexicographic mixed-integer optimization. The decision variable is

$g_j\in\{-1,+1\},$4

indicating whether pair $g_j\in\{-1,+1\},$5 is selected. In Stage 1, the objective is to minimize the maximum within-pair desirability gap by introducing a continuous variable $g_j\in\{-1,+1\},$6 and solving

$g_j\in\{-1,+1\},$7

subject to

$g_j\in\{-1,+1\},$8

with $g_j\in\{-1,+1\},$9,

$g_j=+1$0

and

$g_j=+1$1

These constraints enforce three conditions: exactly 30 pairs are chosen, no item appears in more than one pair, and the worst within-pair desirability difference is minimized. Additional balancing constraints are then imposed.

The mixed-key constraint requires the proportion of mixed-key pairs to lie between 40% and 60%:

$g_j=+1$2

The domain coverage constraint requires each of the five domains to appear exactly 12 times across the 30 pairs:

$g_j=+1$3

The domain-pair composition constraint requires each unordered trait-pair type to occur exactly 3 times:

$g_j=+1$4

The keying balance constraint uses $g_j=+1$5 and $g_j=+1$6, the numbers of positively keyed and negatively keyed selected items from trait $g_j=+1$7, and imposes

$g_j=+1$8

This ensures that, within each domain, neither keying direction is underrepresented by more than a 70/30 split.

In Stage 2, among minimax-optimal solutions, the paper minimizes total squared desirability mismatch by solving

$g_j=+1$9

and then

$g_j=-1$0

The optimization was implemented in R, using ompr/ROI, and solved with Gurobi (Okada et al., 19 Feb 2026).

The resulting inventory achieves very close desirability matching on the 1–9 desirability scale: the maximum within-block desirability gap is 0.18, the mean gap is 0.03, the standard deviation is 0.04, and the range is 0.00 to 0.18 (Okada et al., 19 Feb 2026). This indicates that the paired alternatives are closely matched with respect to social desirability.

4. Response format and ordinal Thurstonian IRT scoring

The GFC administration presents one pair at a time with a 7-point bipolar response. The response options are:

1. LEFT statement describes me much more accurately
2. LEFT moderately more accurately
3. LEFT slightly more accurately
4. About the same
5. RIGHT slightly more accurately
6. RIGHT moderately more accurately
7. RIGHT much more accurately

The left/right assignment is randomized for each pair (Okada et al., 19 Feb 2026).

Scoring is conducted with an ordinal Thurstonian IRT model rather than raw counts. For each pair $g_j=-1$1, with left item $g_j=-1$2 and right item $g_j=-1$3, the graded comparative response is

$g_j=-1$4

Each statement’s latent utility is

$g_j=-1$5

where $g_j=-1$6 is the respondent’s latent Big Five vector, $g_j=-1$7 is a one-hot trait vector for item $g_j=-1$8, and $g_j=-1$9 is the signed discrimination. The signed discrimination is defined by

$s_j$0

with $s_j$1.

The pairwise comparison predictor is

$s_j$2

The paper states that this right-minus-left difference is scaled by $s_j$3 to standardize variance and keep the comparison signal on a comparable scale (Okada et al., 19 Feb 2026).

With pair-specific ordered thresholds $s_j$4, the graded response model for GFC is

$s_j$5

Larger $s_j$6 means stronger endorsement of the RIGHT statement.

For comparison, Likert-format responses in the same study are modeled with a multidimensional graded response model (GRM):

$s_j$7

and

$s_j$8

The use of latent scores from the same IRT framework makes the Likert-versus-GFC comparison more principled (Okada et al., 19 Feb 2026).

5. Bayesian estimation and identification

The Bayesian IRT models were fit in Stan with weakly informative priors:

$s_j$9

$j$0

and

$j$1

for ordered thresholds (Okada et al., 19 Feb 2026).

Identification of the latent scale is obtained by fixing the mean to 0 and the variance to 1 on each trait, with independence across traits in the scoring model. Implementation details are reported as cmdstanr, 4 chains, 200 warmup + 500 post-warmup iterations, NUTS, adapt_delta = 0.95, and max_treedepth = 12. Posterior means of $j$2 are then used as the latent trait estimates.

These details matter because the GFC inventory is not a descriptive response format alone; it is a psychometric instrument whose measurement properties depend on a particular latent-variable model and estimation regime. A plausible implication is that comparisons of forced-choice inventories across studies depend not only on pair construction but also on the comparative IRT specification used for scoring.

6. SDR quantification, persona recovery, and empirical findings

The paper quantifies SDR for both Likert and GFC formats as an instruction-induced effect size on latent trait scores. For each persona $j$3 and trait $j$4,

$j$5

A paired standardized effect size is then computed as

$j$6

To make the sign interpretable in desirability terms, the effect size is direction-corrected:

$j$7

where $j$8 for Agreeableness, Conscientiousness, Extraversion, Openness and $j$9 for Neuroticism (Okada et al., 19 Feb 2026). Positive $s_j = \frac{1}{60} \sum_{l} \sum_{r} x_{jlr}.$0 therefore always indicates a shift toward socially desirable responding: higher A, C, E, O and lower N.

The study also tests whether the GFC format preserves intended persona structure. Each synthetic persona has a known Big Five vector

$s_j = \frac{1}{60} \sum_{l} \sum_{r} x_{jlr}.$1

and recovery is evaluated by the Pearson correlation between estimated trait vectors and the true persona vectors across personas. This metric assesses whether the questionnaire preserves rank-order differences in intended traits.

Across nine instruction-tuned LLMs, the study reports that Likert-style questionnaires showed consistently large SDR: under fake-good instructions, models tended toward higher A, C, E, O and lower N (Okada et al., 19 Feb 2026). In the paper’s summary, these shifts were large, often around or above the scale of human instructed-faking effects reported in the literature.

When the same content was administered in the desirability-matched GFC format, SDR was substantially reduced, often near zero, especially in models with the largest Likert SDR. At the same time, GFC did not simply eliminate all meaningful variance. The reported trade-off is that Likert often showed better raw recovery but heavier contamination by SDR, whereas GFC showed slightly lower recovery but much lower SDR. Most GFC points remained in the paper’s acceptable-to-strong recovery region: $s_j = \frac{1}{60} \sum_{l} \sum_{r} x_{jlr}.$2 is described as generally acceptable/moderate, and many values were around or above $s_j = \frac{1}{60} \sum_{l} \sum_{r} x_{jlr}.$3, considered strong (Okada et al., 19 Feb 2026).

The paper explicitly states that the SDR–recovery trade-off is model-dependent. Some models showed strong SDR attenuation under GFC; some retained non-negligible SDR even with GFC; and the recovery drop was not uniform across models. This suggests that alignment policies, training differences, and response style interact with questionnaire format in nontrivial ways. The study also emphasizes that GFC does not eliminate SDR entirely in all models, so the inventory is best understood as a mitigation strategy, not a complete fix.

These results motivate a reporting norm for questionnaire-based LLM evaluation. The authors recommend reporting both trait estimates and SDR distortion metrics, including direction-corrected latent SDR effect sizes, ground-truth recovery metrics, and the trade-off between the two (Okada et al., 19 Feb 2026). In that framework, the GFC Big Five Inventory functions not merely as an alternative questionnaire format but as part of an SDR-aware psychometric workflow for benchmarking and auditing LLMs.