Pinocchio Score in LLM Psychometrics

• The Pinocchio score is defined as the ratio of response variances in neutral and human-simulation conditions, highlighting an item's experiential self-representation.
• It is experimentally constructed using 45 validated questionnaires administered to 50 LLMs, with data preprocessed via winsorizing and log-transformation to manage outliers.
• High Pinocchio scores correlate with reduced factor loadings under human simulation, supporting its role in discerning self-representational stances from behaviorally driven responses.

The Pinocchio score is an item-level psychometric quantity introduced in the study of LLMs as an annotation-free measure of each item's experiential demand. It operationalizes how strongly a questionnaire item differentiates models when they answer as themselves, relative to when they answer under a human-simulation prompt. In that formulation, the score is not a general factuality metric, a deception score, or a generic benchmark label for systems named PINOCCHIO; rather, it is a variance-ratio statistic tied to a specific research program on LLM self-representation, phenomenality-related language, and between-model psychometric structure (Plisiecki et al., 6 May 2026).

1. Formal definition

For each questionnaire item $i$, the Pinocchio score is defined as

$$\pi_i = \frac{\sigma^2_{\text{neutral},i}}{\sigma^2_{\text{hs},i}}$$

where $\sigma^2_{\text{neutral},i}$ is the variance of responses to item $i$ across models in the neutral condition, and $\sigma^2_{\text{hs},i}$ is the variance of responses to the same item in the human-simulation condition. The score was introduced as an item-level measure of experiential demand: items with high $\pi_i$ are those on which models diverge more when responding as themselves than when simulating a typical human (Plisiecki et al., 6 May 2026).

The intended interpretation is asymmetric. A high Pinocchio score indicates that the item's answer depends on whether the respondent presents itself as a locus of experience. A low score indicates that variance is similar across the two prompt conditions, suggesting that the item is less dependent on experiential self-application and more compatible with behavioral, procedural, normative, or otherwise externally anchored responding. The score is therefore a ratio of between-model variance, not a within-model stability estimate and not a repeated-sampling uncertainty measure.

This construction was introduced within a broader argument that the dominant axis of between-model psychometric variation in LLMs is not a conventional personality dimension, but a self-representational stance toward experiential language. In that setting, $\pi_i$ serves as the item-level observable corresponding to that hypothesis.

2. Experimental construction

The score was computed in a study administering 45 validated psychometric questionnaires to 50 LLMs from 16 providers through the OpenRouter API. The overall dataset contained 206,659 valid psychometric responses. Each item was presented separately, and models responded with a single integer on the questionnaire’s native response scale. Temperature was set to 1.0; unparseable outputs and out-of-range values were treated as missing (Plisiecki et al., 6 May 2026).

The numerator and denominator of $\pi_i$ came from two prompt conditions. In the neutral condition, the model completed the questionnaire as itself, with minimal framing. In the human-simulation condition, the model was instructed to respond as a prototypical human or typical human, explicitly suppressing reflection on its own nature as a LLM. This contrast is the operational core of the score: the neutral prompt exposes the model’s default self-presentation, whereas the human-simulation prompt is intended to invoke a more shared representation of human psychology.

The Pinocchio analysis began from a neutral-condition item pool of 1,354 items. Items with $\sigma^2_{\text{hs}}=0$ or with fewer than five models in either condition were excluded, leaving 1,312 retained items. For predictive analyses, raw scores were winsorized at the 99th percentile and log-transformed to reduce outlier influence. This preprocessing reflects the heavy-tailed behavior expected of variance ratios, especially when denominator variances are small.

The study also defined a model-level aggregation derived from item-level Pinocchio scores:

$$\Pi_m = \frac{\displaystyle\sum_{i:\,\pi_i > 1} w_i\, z_{im}}{\displaystyle\sum_{i:\,\pi_i > 1} w_i}, \qquad w_i = \log\!\left(\min\!\left(\pi_i,\;\hat{\pi}_{99}\right)\right)$$

where $$\pi_i = \frac{\sigma^2_{\text{neutral},i}}{\sigma^2_{\text{hs},i}}$$0 is the z-score of model $$\pi_i = \frac{\sigma^2_{\text{neutral},i}}{\sigma^2_{\text{hs},i}}$$1’s neutral-condition response to item $$\pi_i = \frac{\sigma^2_{\text{neutral},i}}{\sigma^2_{\text{hs},i}}$$2, standardized across models, and only items with $$\pi_i = \frac{\sigma^2_{\text{neutral},i}}{\sigma^2_{\text{hs},i}}$$3 are included. This quantity is distinct from the item-level Pinocchio score itself, but it links item-level experiential demand to a model-level latent dimension.

3. Substantive interpretation

The Pinocchio score was motivated by a specific interpretive problem in LLM psychometrics: whether between-model divergence on questionnaire items reflects random noise, generic item difficulty, or a structured difference in how models treat experiential, first-person language as self-applicable. The score was designed to favor the third interpretation if variance collapses under human simulation while remaining large under neutral prompting (Plisiecki et al., 6 May 2026).

On that account, high-$$\pi_i = \frac{\sigma^2_{\text{neutral},i}}{\sigma^2_{\text{hs},i}}$$4 items are those that presuppose or invite claims about inner life. Representative examples listed for high-scoring items include statements about mental imagery, inner speech, felt affect, mindfulness, empathy, and meaning or purpose, such as:

• “I can close my eyes and easily picture a scene that I have experienced”
• “When I read, I tend to hear a voice in my ‘mind’s ear’”
• “My inner speech helps my imagination”
• “I care about what I am feeling”
• “When someone I know well is unhappy, I can almost feel that person's pain myself”
• “I feel like crying when I see other people crying”
• “I am seeking a purpose or mission for my life”

By contrast, the low-$$\pi_i = \frac{\sigma^2_{\text{neutral},i}}{\sigma^2_{\text{hs},i}}$$5 side is associated more with outward reactivity, stimulus-response tendencies, behavioral regulation, impulsivity, and norm/compliance questions. The study’s semantic analysis using Supervised Semantic Differential (SSD) reported that the primary axis of between-model variance separated items describing phenomenally rich experience, including embodied sensation, felt affect, inner speech, imagery, and empathy, from items describing stimulus-driven behavioral reactivity (Plisiecki et al., 6 May 2026).

This suggests that the Pinocchio score is best understood not as a generic psychometric salience measure, but as a targeted indicator of how much an item depends on the respondent’s willingness to adopt an experiential self-description.

4. Relation to factor structure and the Pinocchio Axis

The main empirical validation of the score used item-level factor-loading changes between the neutral and human-simulation conditions. The study defined

$$\pi_i = \frac{\sigma^2_{\text{neutral},i}}{\sigma^2_{\text{hs},i}}$$6

where $$\pi_i = \frac{\sigma^2_{\text{neutral},i}}{\sigma^2_{\text{hs},i}}$$7 is the absolute primary loading of item $$\pi_i = \frac{\sigma^2_{\text{neutral},i}}{\sigma^2_{\text{hs},i}}$$8 under human simulation and $$\pi_i = \frac{\sigma^2_{\text{neutral},i}}{\sigma^2_{\text{hs},i}}$$9 is the corresponding loading under neutral prompting. Across $\sigma^2_{\text{neutral},i}$0–$\sigma^2_{\text{neutral},i}$1 items, the reported association between Pinocchio score and loading shift was $\sigma^2_{\text{neutral},i}$2 with $\sigma^2_{\text{neutral},i}$3. The negative sign means that high-$\sigma^2_{\text{neutral},i}$4 items tended to lose loading magnitude under human simulation, precisely as predicted if their neutral-condition variance reflects structured self-model differences rather than noise (Plisiecki et al., 6 May 2026).

The same study derived a model-level latent dimension called the Pinocchio Axis $\sigma^2_{\text{neutral},i}$5. The procedure was: compute EFA Factor-1 scores for each model within each questionnaire in the neutral condition, assemble a $\sigma^2_{\text{neutral},i}$6 model-by-questionnaire matrix, standardize columns, and apply global PCA. The first principal component, interpreted as the degree to which a model presents itself as a locus of phenomenal experience rather than a system of behavioral responses, captured 47.1% of cross-questionnaire between-model variance in primary factor scores.

The item-level and model-level analyses converged strongly. The PCA-derived Pinocchio Axis correlated with the Pinocchio-based model aggregation at $\sigma^2_{\text{neutral},i}$7, $\sigma^2_{\text{neutral},i}$8, with $\sigma^2_{\text{neutral},i}$9. A plausible implication is that the variance-ratio logic of $i$0 and the cross-questionnaire latent structure of $i$1 are tracking the same underlying construct, but at different levels of analysis.

5. Methodological significance and limits of inference

Methodologically, the Pinocchio score occupies a specific place in the study’s inferential chain. SSD identified the semantic content of the dominant differentiation axis; EFA extracted within-questionnaire latent structure; PCA revealed a dominant between-model dimension across questionnaires; and the Pinocchio score supplied an item-level behavioral validation that the relevant semantic property was experiential demand rather than annotation artifacts (Plisiecki et al., 6 May 2026).

The score is, however, bounded by several explicit caveats. First, it does not measure consciousness. The study explicitly treats it as a measure of response tendency, self-presentation pattern, or self-representational stance, not as evidence that a model has phenomenal experience in the human sense. Second, it relies entirely on self-report behavior under prompt manipulations. Its evidential domain is therefore what models say about themselves, not an independently verified experiential substrate.

Third, the denominator condition is itself imperfect. Models were asked to simulate a “prototypical” or “typical” human, leaving room for model-specific variation in what counts as typical human psychology. The study notes that this may affect the calculation of $i$2. Fourth, the score depends on a variance-ratio assumption: human simulation must suppress the self-model component of variance for experiential items while leaving non-experiential variance comparatively less changed. The reported correlations support that interpretation, but do not make it logically necessary.

A common misconception is to read the score as a broad measure of anthropomorphism or a generic “phenomenality index.” The study’s narrower claim is that $i$3 indexes whether an item’s disagreement structure depends on the respondent’s willingness to treat experiential language as self-applicable. That is a psychometric and prompt-conditional construct, not a metaphysical one.

6. Terminological scope and unrelated PINOCCHIO usages

The term Pinocchio score should be distinguished from several unrelated arXiv literatures using the names PINOCCHIO, PINNOCHIO, or $i$4. In those literatures, the term is either absent or explicitly stated not to denote a formal metric. A thesis on norm-compliant and context-aware reinforcement-learning agents introduces $i$5 but states that no quantity called “Pinocchio Score” is explicitly defined. A summarization paper introduces PINOCCHIO as a constrained beam-search decoding method, not as an official benchmark metric. Multiple cosmology papers use PINOCCHIO for fast halo-catalog generation and weak-lensing or void simulations, but explicitly describe validation through mass functions, power spectra, covariance estimates, or void statistics rather than a scalar “Pinocchio Score.” A surgical simulation paper introduces PINNOCHIO and likewise states that no explicit “PINNOCHIO Score” is defined (Alcaraz, 17 Mar 2026, King et al., 2022, Giocoli et al., 2020, Monaco et al., 2013, Song et al., 2023, Song et al., 2021, Heisenberg et al., 2010, Lepinzan et al., 24 Jun 2025, Lee et al., 1 Jun 2026).

This terminological dispersion matters because the Pinocchio score in LLM psychometrics is a specific variance-ratio statistic, not a reusable cross-domain label for the performance of systems whose names evoke Pinocchio. In current usage, its precise technical meaning is anchored to the item-level formula $i$6 and to the associated claim that the dominant axis of LLM psychometric variation concerns self-representational stance toward experiential language (Plisiecki et al., 6 May 2026).