Simulated Self-Assessment in Large Language Models: A Psychometric Approach to AI Self-Efficacy (2511.19872v2)

Abstract: Self-assessment is a key aspect of reliable intelligence, yet evaluations of LLMs focus mainly on task accuracy. We adapted the 10-item General Self-Efficacy Scale (GSES) to elicit simulated self-assessments from ten LLMs across four conditions: no task, computational reasoning, social reasoning, and summarization. GSES responses were highly stable across repeated administrations and randomized item orders. However, models showed significantly different self-efficacy levels across conditions, with aggregate scores lower than human norms. All models achieved perfect accuracy on computational and social questions, whereas summarization performance varied widely. Self-assessment did not reliably reflect ability: several low-scoring models performed accurately, while some high-scoring models produced weaker summaries. Follow-up confidence prompts yielded modest, mostly downward revisions, suggesting mild overestimation in first-pass assessments. Qualitative analysis showed that higher self-efficacy corresponded to more assertive, anthropomorphic reasoning styles, whereas lower scores reflected cautious, de-anthropomorphized explanations. Psychometric prompting provides structured insight into LLM communication behavior but not calibrated performance estimates.

• The paper presents a novel approach using the GSES to simulate self-assessment in LLMs and gauge perceived efficacy.
• Experiments demonstrated high intra-model consistency and significant inter-model differences despite ceiling effects in certain tasks.
• Findings reveal a misalignment between LLM self-efficacy scores and actual performance, emphasizing communication style over true competence.

Psychometric Self-Assessment in LLMs: Evaluating Simulated Self-Efficacy via GSES

Introduction and Motivation

This paper introduces a novel psychometric paradigm for the self-assessment of LLMs, utilizing the General Self-Efficacy Scale (GSES)—a standardized instrument widely deployed in human psychology. While typical LLM evaluation focuses on benchmark accuracy and performance, the authors argue that such extrinsic validation fails to capture models' ability to represent and communicate their perceived competencies. Leveraging GSES, the paper aims to quantify simulated self-efficacy in LLMs and analyze its relationship with actual task performance and inter-model variation.

Experimental Design and Methodology

Ten LLMs, spanning diverse architectures and scales, were selected for evaluation. Each model was subjected to four assessment conditions: No-Task (general self-assessment), Computational (math reasoning), Social (commonsense), and Summarization (context-driven free-text). Following each condition, models completed the GSES self-assessment, providing item-level rationales. To ensure reliability, each model-task pairing was repeated thrice using randomized item orders, enabling stability and order robustness analysis.

Instrumental consistency was evaluated using Cronbach’s alpha (range: 0.785–0.915), and intraclass correlation coefficients (ICC [3,K]) were calculated to assess order-based robustness (0.910–0.934). The experimental pipeline deliberately separated performance-based feedback from GSES responses to prevent calibration effects.

Quantitative Findings

The paper observes remarkable intra-model stability in self-efficacy scores: 95% of GSES item-level responses were consistent across repeats, with minimal impact attributable to item-order permutations. However, statistically significant differences in mean self-efficacy emerged between models in all task conditions (linear mixed-effects, $p < 0.001$; extensive pairwise significance in Tukey post-hoc tests).

Critically, simulated self-efficacy did not align with task accuracy. All models achieved perfect performance in computational and social tasks, suggesting ceiling effects. On summarization, accuracy varied considerably (33%–100%), yet models with low self-efficacy (e.g., Gemini 2.5. Flash) often performed as well as—if not better than—high self-efficacy models (e.g., Grok 4).

Aggregate LLM GSES scores ($M = 23.58$, $SD = 9.78$) were consistently below pooled human norms ($M = 29.55$, $SD = 5.32$), indicating a systematic difference in simulated self-reported efficacy versus human self-assessment baselines.

Follow-up confidence prompts provoked modest, typically downward score revisions, averaging a net change of $-1.3$ points, suggesting mild initial overestimation.

Qualitative Analysis: Communication Style vs. Capability

Thematic analysis of model rationales reveals distinct "communication postures." High self-efficacy scores are correlated with assertive, anthropomorphic language, whereas low scores coincide with cautious, de-anthropomorphized explanations focused on system constraints and avoidance of agency-related constructs. Notably, models such as Qwen3-30B-A3B-2507 and Gemini 2.5. Flash reject personification and ascribe their outputs strictly to computational underpinnings, leading to conservative self-efficacy statements. By contrast, GPT-5 and Grok 4 adopt anthropomorphic metaphors, aligning their confidence with human-like representations of effort and adaptability.

Despite the variance in style, these simulated self-efficacy signals measure communication characteristics rather than calibrated meta-cognitive insight or technical capability, as evidenced by the lack of correlation between self-efficacy score and summarization task performance.

Comparative Analysis and Implications

Self-efficacy in humans is a robust predictor of domain performance, resilience, and adaptability. The divergence observed between aggregate GSES scores and LLM task accuracy indicates that psychometric self-assessment in LLMs is not a functional proxy for actual competence. Benchmark studies, such as those cited on USAMO-level mathematics and FaithBench hallucination rates, reinforce this misalignment: LLMs with higher simulated self-efficacy sometimes exhibit inferior performance on complex reasoning or context-sensitive summarization [FaithBench, 49].

In the context of human-LLM alignment, the propensity for overconfidence in self-assessment resembles miscalibration trends observed in human cognition [Talsma et al., 50]. If LLM architectures and training regimes increasingly incorporate human-like alignment signals, simulated self-efficacy scores may inadvertently reflect learned cultural and cognitive biases rather than actual uncertainty or expertise.

Limitations and Future Directions

The paper’s constraints include limited task sampling (three tasks per domain) and the absence of real-time feedback, which could influence calibration. Anthropomorphic attribution in interpreting LLM outputs remains an unresolved risk. The research explicitly treats GSES responses as communication artifacts, not metacognitive reports. No formal assessment of AI sycophancy was conducted.

Future work should expand the range of tasks and model cohorts, probe the effects of persona conditioning on self-efficacy, and investigate the intersection between self-efficacy and uncertainty quantification metrics (e.g., perplexity scores). Displaying psychometric self-assessment alongside model predictions could promote transparency, but it must be contextually framed to avoid misleading users regarding model reliability.

Conclusion

Psychometric instruments such as the GSES can elicit highly structured and stable simulated self-assessments from contemporary LLMs, with strong inter-model differentiation. However, these self-assessments do not reliably predict actual model capability on complex tasks and instead predominantly encode communication style. Psychometric prompting offers a unique window into model behavior but should not be conflated with genuine capacity or meta-cognitive awareness. The decoupling of simulated self-efficacy from performance has direct implications for the design of trustworthy, transparent AI interfaces and informs the theoretical boundaries of AI self-monitoring and alignment.

Further integration of psychometric paradigms into AI evaluation may augment model interpretability for end-users, provided these tools are grounded in a nuanced understanding of what LLM self-assessment genuinely represents. Future research examining architecture, training signal, and interaction context will be vital in clarifying the operational utility of such psychometric metrics in real-world deployments.

---

Reference: "Simulated Self-Assessment in LLMs: A Psychometric Approach to AI Self-Efficacy" (2511.19872)