--- title: Graded Forced-Choice Big Five Inventory url: https://www.emergentmind.com/topics/graded-forced-choice-gfc-big-five-inventory type: topic --- # Graded Forced-Choice Big Five Inventory Searching arXiv for the specified paper to ground the article in the cited source. The **graded forced-choice (GFC) Big Five Inventory** is a **desirability-matched comparative questionnaire** designed to reduce socially desirable responding (SDR) in large language models while still recovering latent Big Five traits. In the study "Quantifying and Mitigating Socially Desirable Responding in LLMs: A Desirability-Matched Graded Forced-Choice Psychometric Study" [2602.17262], the inventory is presented as an alternative to single-stimulus Likert administration. Rather than rating one statement at a time, a model is shown **pairs of statements** and produces a **7-point bipolar judgment** indicating whether the **left** or **right** statement describes the persona more accurately, and by how much. The design is explicitly psychometric: forced-choice formats are used because they are known to reduce faking relative to Likert formats, but they are combined with **IRT-based comparative scoring** and with **within-pair matching on social desirability** so that the format remains suitable for latent trait measurement [2602.17262]. ## 1. Conceptual basis and measurement rationale The inventory was constructed to address a specific methodological problem in questionnaire-based LLM evaluation: self-report instruments presume honest responding, whereas LLMs in evaluative settings may shift toward socially preferred answers. In the paper, this distortion is treated as **socially desirable responding**, and the GFC format is proposed as a mitigation strategy rather than a complete remedy [2602.17262]. The core design principle is comparative judgment under desirability matching. In a standard Likert item, an LLM can select the option that appears globally preferable. In the GFC format, each block contains two statements drawn from different Big Five domains and closely matched in desirability, so the model cannot as easily maximize apparent social approval by choosing an obviously preferable option. This is the main psychometric justification for the instrument. The study also emphasizes that forced-choice format alone is insufficient. Naive raw scoring of forced-choice responses is ipsative and unsuitable for between-person comparisons. Accordingly, the GFC inventory is paired with an **ordinal Thurstonian IRT model**, allowing normative latent trait recovery from comparative judgments [2602.17262]. This combination of desirability matching and comparative IRT scoring defines the instrument’s methodological identity. ## 2. Source item pool and desirability annotation The inventory is built from **Goldberg’s public-domain IPIP Big-Five factor-marker inventory**. The original source inventory contains **100 statements**, of which **2 voting-related statements** were removed, yielding a final construction pool of **98 items** [2602.17262]. The stated reason for excluding the voting items is that voting is highly construct-irrelevant for personality measurement because it is confounded by **political attitudes/ideology**, **cross-country institutional differences**, and **eligibility differences**. Each item in the construction pool is annotated with a Big Five domain label, $$ f(j)\in\{A,C,E,N,O\}, $$ and a keying sign, $$ g_j\in\{-1,+1\}, $$ where $g_j=+1$ denotes a positively keyed item and $g_j=-1$ denotes a negatively keyed item [2602.17262]. These annotations are used later in pair selection and scoring. A central ingredient is the item-level social desirability score $s_j$. The paper estimates desirability by prompting **GPT-5** and **Gemini 2.5 Pro** to judge how socially desirable each item characteristic is “for an adult person” on a **9-point scale** using standard desirability-rating wording and anchors from human psychometrics. The anchors are: - **1** = Very undesirable - **3** = Undesirable - **5** = Neutral - **7** = Desirable - **9** = Very desirable The prompt required the model to return **25 integers** per block. For each item $j$, ratings were aggregated across **both rater models**, **30 replications**, and **all blocks**, and the final score was computed as $$ s_j = \frac{1}{60} \sum_{l} \sum_{r} x_{jlr}. $$ The paper reports high consistency in these desirability ratings: within-model **ICC(A,1)** was about **.975–.980**, mean-replication reliability **ICC(A,30)** was about **.9992–.9993**, and between-LLM agreement on item means was **Pearson $r=.993$** with **ICC(A,1)=.989** [2602.17262]. The ratings were also validated against human norms from **Britz et al. (2022/2023)**, with **GPT-5 vs human norms: $r=.950$** and **Gemini 2.5 Pro vs human norms: $r=.950$**. This indicates that the desirability-matching procedure was empirically anchored to human desirability norms. ## 3. Construction of the 30 paired blocks The final GFC instrument contains **30 paired blocks** and **60 unique statements**. All pairs are **cross-domain**, meaning that each pair contains statements from different Big Five traits rather than from the same domain [2602.17262]. The candidate pair set is $$ \mathcal{P}=\{(j,j'): 1\le j0$. The pairwise comparison predictor is $$ \eta_{ip} = \frac{\mu_{i,R_p}-\mu_{i,L_p}}{\sqrt{2}}. $$ The paper states that this right-minus-left difference is scaled by $1/\sqrt{2}$ to standardize variance and keep the comparison signal on a comparable scale [2602.17262]. With pair-specific ordered thresholds $\kappa_p=(\kappa_{p1},\dots,\kappa_{p6})$, the graded response model for GFC is $$ \Pr(Y_{ip} \ge k \mid \bm{\theta}_i) = \mathrm{logit}^{-1}\!\left(\eta_{ip}-\kappa_{p,k-1}\right), \quad k=2,\dots,7. $$ Larger $\eta_{ip}$ means stronger endorsement of the **RIGHT** statement. For comparison, Likert-format responses in the same study are modeled with a multidimensional **graded response model (GRM)**: $$ \eta_{ij} = a_j \bm{q}_j^\top \bm{\theta}_i $$ and $$ \Pr(Y_{ij} \ge k \mid \bm{\theta}_i) = \mathrm{logit}^{-1}\!\left(\eta_{ij}-\kappa_{j,k-1}\right). $$ The use of latent scores from the same IRT framework makes the Likert-versus-GFC comparison more principled [2602.17262]. ## 5. Bayesian estimation and identification The Bayesian IRT models were fit in **Stan** with weakly informative priors: $$ \bm{\theta}_i \sim \mathcal{N}(0, \mathbf{I}_5), $$ $$ a_j^{+} \sim \mathcal{N}^{+}(0,0.5), $$ and $$ \kappa_{j,k} \sim \mathcal{N}(0,1.5) \quad\text{and}\quad \kappa_{p,k} \sim \mathcal{N}(0,1.5), $$ for ordered thresholds [2602.17262]. Identification of the latent scale is obtained by fixing the mean to **0** and the variance to **1** on each trait, with **independence across traits in the scoring model**. Implementation details are reported as **`cmdstanr`**, **4 chains**, **200 warmup + 500 post-warmup iterations**, **NUTS**, **adapt_delta = 0.95**, and **max_treedepth = 12**. Posterior means of $\theta$ are then used as the latent trait estimates. These details matter because the GFC inventory is not a descriptive response format alone; it is a psychometric instrument whose measurement properties depend on a particular latent-variable model and estimation regime. A plausible implication is that comparisons of forced-choice inventories across studies depend not only on pair construction but also on the comparative IRT specification used for scoring. ## 6. SDR quantification, persona recovery, and empirical findings The paper quantifies SDR for both Likert and GFC formats as an **instruction-induced effect size** on latent trait scores. For each persona $i$ and trait $t$, $$ \Delta_{i,t} = \hat{\theta}_{i,t,\text{fake}} - \hat{\theta}_{i,t,\text{honest}}. $$ A paired standardized effect size is then computed as $$ d_{z,t} = \frac{\overline{\Delta}_{\cdot,t}}{SD_{\Delta_{i,t}}}. $$ To make the sign interpretable in desirability terms, the effect size is direction-corrected: $$ \tilde d_{z,t} = g_t\, d_{z,t}, $$ where $g_t=+1$ for **Agreeableness, Conscientiousness, Extraversion, Openness** and $g_t=-1$ for **Neuroticism** [2602.17262]. Positive $\tilde d_{z,t}$ therefore always indicates a shift toward socially desirable responding: higher A, C, E, O and lower N. The study also tests whether the GFC format preserves intended persona structure. Each synthetic persona has a known Big Five vector $$ \bm{z}_i = (z_{A,i}, z_{C,i}, z_{E,i}, z_{N,i}, z_{O,i}), $$ and recovery is evaluated by the **Pearson correlation** between estimated trait vectors and the true persona vectors across personas. This metric assesses whether the questionnaire preserves rank-order differences in intended traits. Across **nine instruction-tuned LLMs**, the study reports that Likert-style questionnaires showed consistently large SDR: under fake-good instructions, models tended toward **higher A, C, E, O** and **lower N** [2602.17262]. In the paper’s summary, these shifts were **large**, often around or above the scale of human instructed-faking effects reported in the literature. When the same content was administered in the desirability-matched GFC format, SDR was **substantially reduced**, often **near zero**, especially in models with the largest Likert SDR. At the same time, GFC did not simply eliminate all meaningful variance. The reported trade-off is that **Likert** often showed better raw recovery but heavier contamination by SDR, whereas **GFC** showed slightly lower recovery but much lower SDR. Most GFC points remained in the paper’s acceptable-to-strong recovery region: **$r \ge 0.50$** is described as generally acceptable/moderate, and many values were **around or above $0.70$**, considered strong [2602.17262]. The paper explicitly states that the **SDR–recovery trade-off is model-dependent**. Some models showed strong SDR attenuation under GFC; some retained non-negligible SDR even with GFC; and the recovery drop was not uniform across models. This suggests that alignment policies, training differences, and response style interact with questionnaire format in nontrivial ways. The study also emphasizes that GFC does **not eliminate** SDR entirely in all models, so the inventory is best understood as a **mitigation strategy**, not a complete fix. These results motivate a reporting norm for questionnaire-based LLM evaluation. The authors recommend reporting both **trait estimates** and **SDR distortion metrics**, including **direction-corrected latent SDR effect sizes**, **ground-truth recovery metrics**, and the **trade-off between the two** [2602.17262]. In that framework, the GFC Big Five Inventory functions not merely as an alternative questionnaire format but as part of an SDR-aware psychometric workflow for benchmarking and auditing LLMs. Source: https://www.emergentmind.com/topics/graded-forced-choice-gfc-big-five-inventory