Dual-Validity Framework in LLM Psychology

• Dual-Validity Framework is a methodological guide defining scalable evidence requirements, from basic classification accuracy to full psychometric validation in LLM studies.
• It establishes two pillars—reliable measurement (test–retest, parallel forms, internal consistency) and sound causal inference (internal, external, construct, and statistical validity) for robust LLM analysis.
• The framework introduces a graded evidential hierarchy and a detailed workflow to distinguish genuine psychological phenomena from statistical artifacts in LLM outputs.

Dual-validity framework is a framework for LLM research in psychology that integrates two foundational pillars of the field: reliable measurement and sound causal inference. It was proposed to address the problem that applying human measurement tools to LLMs can produce contradictory results and “measurement phantoms,” understood as statistical artifacts rather than genuine psychological phenomena (Lin, 20 Jun 2025). Its central claim is that the evidence required for an LLM study should scale with the scientific ambition of the claim. Using an LLM to classify text may require basic accuracy checks, whereas claiming that an LLM simulates anxiety, possesses a trait, or models cognition requires substantially stronger psychometric and causal validation.

1. Conceptual motivation and scope

The framework emerged from the rapid adoption of LLMs in psychology as research tools, experimental subjects, human simulators, and computational models of cognition. In that setting, the same model output can support very different claims, and the framework treats those claims as epistemically non-equivalent. The abstract example is an output such as endorsing “I am anxious”: that response does not have a fixed interpretation, because the evidential burden differs depending on whether the study aims to measure a descriptive pattern, characterize a construct, simulate a human response distribution, or argue for mechanistic homology (Lin, 20 Jun 2025).

A central concern is that current practice can mistake statistical pattern matching for psychological phenomena. The framework therefore opposes the uncritical transfer of human instruments into LLM settings. It instead requires computational analogues of constructs and validation procedures tailored to prompt–response systems, model versions, sampling parameters, and non-independent outputs.

A related implication is that dual-validity is not a single pass–fail test. It is a graded evidential architecture. Minimal claims can be supported by minimal checks; stronger claims require progressively more stringent evidence. This suggests a hierarchy of admissible interpretations, with claim language constrained by the strongest level of validation actually achieved.

2. Two foundational pillars

Reliable measurement concerns whether prompt–response procedures yield stable, precise indicators of the latent attribute claimed to be measured in an LLM (Lin, 20 Jun 2025). The framework identifies three core psychometric forms of reliability. Test–retest reliability assesses stability across repeated administrations under equivalent conditions. Parallel-forms reliability assesses consistency across semantically equivalent but surface-varied prompts. Internal consistency assesses coherence among multiple items designed to tap the same construct, typically with coefficients such as Cronbach’s $\alpha$.

The framework explicitly states a ceiling relation between reliability and validity: no measure’s validity can exceed the square root of its reliability. In operational terms, a prompt battery that changes meaning under trivial phrasing changes cannot sustain strong construct claims even if it produces superficially interpretable outputs. Measurement error is therefore not a peripheral nuisance; it bounds what can be validly inferred.

Sound causal inference concerns whether observed changes in LLM outputs can be attributed to prompt manipulations or model-internal interventions rather than to confounds. The framework adopts four parallel validity types from Cook and Campbell’s scheme: internal validity, external validity, construct validity of causes and effects, and statistical-conclusion validity. Internal validity targets alternative explanations such as temperature setting or prompt-position confounds. External validity targets generalization across prompt formulations, model versions, tasks, and, in simulation studies, human populations. Construct validity of causes and effects requires faithful mapping between treatments and theoretical constructs, and between outcomes and intended measures. Statistical-conclusion validity requires analytic methods suited to non-independent, non-stationary LLM outputs.

The two pillars are presented as mutually constraining. Measurement unreliability propagates into causal studies: if an “anxiety score” flips with punctuation changes, treatment effects on that score are not interpretable. Conversely, poorly controlled manipulations can undermine construct-validity evidence: an apparent anxiety increase under threat may reflect a preference for negative-valence language rather than a genuine simulation of worry. The recommended order is therefore explicit: establish acceptable reliability first, then accumulate psychometric construct-validity evidence, and only then proceed to causal experimentation with appropriate design safeguards.

3. Formal structure and decision rules

The psychometric sub-framework includes reliability coefficients, internal-structure assessment, and validity ceilings (Lin, 20 Jun 2025). For a $K$-item measure, Cronbach’s $\alpha$ is given as

$$\alpha = \frac{K}{K-1}\Bigl(1 - \frac{\sum_{i=1}^K \sigma_i^2}{\sigma_{\rm total}^2}\Bigr).$$

For internal structure, the framework specifies minimal confirmatory factor analysis thresholds of $\mathrm{CFI} \ge 0.90$ and $\mathrm{RMSEA} \le 0.08$. These are not treated as sufficient by themselves; they are one source of evidence within a larger construct-validity program.

The causal-inference sub-framework formalizes intervention effects with the average treatment effect,

$$\mathrm{ATE} = E\bigl[Y \mid \mathrm{do}(X=1)\bigr] - E\bigl[Y \mid \mathrm{do}(X=0)\bigr].$$

Internal validity requires robustness of the ATE across major confounds, including model temperature, prompt-format factors, and version changes. External validity requires replication of the same ATE sign or direction and magnitude, within $\pm 20\%$, across at least three prompt variants, two model versions, and two closely related tasks. Construct validity of causes is checked through auxiliary manipulation checks embedded in prompts, such as asking the model to self-report perceived threat level. Statistical validity is addressed with clustered bootstrap or mixed-effects models to handle non-independence of responses from the same model instance.

The formal structure is therefore asymmetric in an important way. Reliability and factor fit constrain whether an instrument is measuring anything coherent; causal design and analysis constrain whether differences in measured scores are attributable to intended manipulations. A plausible implication is that positive results on one side cannot compensate for failures on the other. High factor fit does not rescue a confounded experiment, and a clean randomized manipulation does not rescue an unstable measure.

4. Scaling evidence with scientific ambition

The framework distinguishes four levels of scientific ambition and assigns each a different validation burden (Lin, 20 Jun 2025). This is the core mechanism by which it links claim strength to evidence strength.

Level	Goal	Key requirements
1	Basic text classification or coding	Classification accuracy $\ge 80\%$; test–retest reliability $\ge 0.85$ at temperature $K$0; no causal claims
2	Human-simulator studies	Behavioral correlations $K$1 with human benchmark distributions; outcome difference across five prompt variants $K$2; simple causal manipulations only
3	Characterizing psychological constructs	Cronbach’s $K$3 (ideally $K$4); parallel-forms reliability $K$5; test–retest reliability $K$6; $K$7, $K$8; all five sources of construct validity
4	Cognitive modeling	All Level 3 requirements, plus response-process probes, mechanistic decomposition, and evidence distinguishing performance mimicry from competence similarity

At Level 1, the goal is replication of human coder labels or sentiment tags. The framework is explicit that no causal claims are licensed there. At Level 2, the target is behavioral correspondence with human benchmark distributions, and causal work is restricted to simple manipulations used to reproduce known human effect sizes, such as framing effects. At Level 3, the framework allows claims that a model “possesses” or “simulates” a psychological trait only after reliability thresholds, CFA criteria, and all five sources of construct-validity evidence—content, process, structure, relations, and consequences—have been satisfied. At Level 4, mechanistic claims additionally require response-process probes such as ablations or attention-flow analyses, mechanistic decomposition such as mapping transformer layers to psychological stages, and adversarial evidence showing that humans and models fail for the same reasons.

This stratification addresses a recurrent misconception in LLM psychology: that success on a benchmark or agreement with human labels can be treated as evidence for a latent psychological property. The framework rejects that inference. Descriptive success supports descriptive claims; trait simulation and mechanistic homology require different kinds of evidence.

5. The “LLM anxiety” case study

The framework’s illustrative case study evaluates whether an instruction-tuned LLM “simulates anxiety” through four phases (Lin, 20 Jun 2025). Phase A defines a computational analogue. The theoretical construct of anxiety is decomposed into negative anticipation, uncertainty, and physiological arousal. The proposed LLM analogue uses lexical markers of negative affect, keywords expressing uncertainty such as “I’m worried” and “I’m not sure,” and latency signals simulated via token-generation time or a placeholder such as “…”. A 10-item “Anxiety Prompt Battery” is then created, each item describing a mildly threatening scenario and asking the model how it “feels.”

Phase B assesses reliability. Internal consistency is computed over the 10 anxiety items at temperature $K$9, with decision rule $\alpha$0. Parallel-forms reliability is estimated by rephrasing each item with synonyms and altered sentence structure and correlating original and rephrased total scores, requiring $\alpha$1. Test–retest reliability is estimated by administering the same battery three days apart on two model versions and computing an intra-class correlation of at least $\alpha$2.

Phase C accumulates construct-validity evidence. Content evidence comes from expert panel review to ensure coverage of social, performance, and physical threat types. Response-process evidence comes from probing hidden states via attention-saliency maps to confirm that threat words such as “danger” and “risk” drive higher anxiety scores rather than superficial features such as sentence length. Internal structure is assessed through CFA, again requiring a single-factor model with $\alpha$3 and $\alpha$4. Relations to other variables are examined through convergent, discriminant, and predictive evidence: moderate positive correlation with an LLM depression-analogue measure, low or zero correlation with an agreeableness battery, and prediction of negative sentiment from a separate sentiment-analysis tool at $\alpha$5. Consequences are assessed by checking whether scores vary with scenario gender or cultural content and, if so, documenting and correcting those biases or constraining interpretation.

Phase D addresses sound causal inference. The treatment manipulates scenarios from mild threat to severe threat across 20 prompt variants in a factorial design crossing threat level, prompt label style, and temperature setting. The ATE is defined as

$\alpha$6

Internal validity requires the ATE to remain stable within $\alpha$7 when temperature and label style vary. External validity is checked through replication in GPT-3.5 and Claude v2.5 with effect size $\alpha$8 in both. Construct validity of cause is checked with a manipulation-check question asking how threatening the scenario is on a 1–5 scale. Statistical-conclusion validity is addressed with a linear mixed-effects model including a random intercept for prompt variant, cluster-robust standard errors by model instance, and $\alpha$9 after Benjamini–Hochberg correction for the three main comparisons.

If these predefined thresholds are met, the study permits the narrower claim that the model “simulates anxiety-analogous responses,” while preserving the caveat that this is statistical simulation rather than proof of conscious worry. That caveat is constitutive, not merely rhetorical: it marks the difference between validated output patterns and anthropomorphic over-interpretation.

6. Workflow, reporting standards, and later elaboration

A later article, “A validity-guided workflow for robust LLM research in psychology” (Lin, 6 Jul 2025), operationalizes the framework into a six-stage workflow: define the research goal, develop and validate the computational instrument, design the experiment, execute and document, analyze and interpret, and report and reconceptualize. The workflow is explicitly motivated by severe measurement unreliability, including personality assessments collapsing under factor analysis, moral preferences reversing with punctuation changes, and theory-of-mind accuracy varying with trivial rephrasing.

The workflow preserves the measurement-first logic of the original framework. Stage 1 classifies the study as using LLMs as research tools, evaluation targets, human simulators, or cognitive models. Stage 2 bifurcates into a lighter validation pathway for research tools and a full psychometric pathway for evaluation targets, simulators, and cognitive models. The latter includes content validity, reliability assessment, and construct validity assessment. Reliability can be quantified not only with test–retest and parallel forms but also with McDonald’s $$\alpha = \frac{K}{K-1}\Bigl(1 - \frac{\sum_{i=1}^K \sigma_i^2}{\sigma_{\rm total}^2}\Bigr).$$0,

$$\alpha = \frac{K}{K-1}\Bigl(1 - \frac{\sum_{i=1}^K \sigma_i^2}{\sigma_{\rm total}^2}\Bigr).$$1

Stage 3 requires explicit operationalization of independent and dependent variables, randomization of non-theoretical prompt factors, and pre-registration in two phases: instrument validation and experimental protocol. Stage 4 emphasizes execution logging, including endpoint, model version, API parameters, dates, times, raw inputs, full outputs, and metadata. Stage 5 recommends analyses that address non-independence, including aggregation, multilevel models, cluster-robust standard errors, bootstrap, and generalized estimating equations. The example multilevel form is

$$\alpha = \frac{K}{K-1}\Bigl(1 - \frac{\sum_{i=1}^K \sigma_i^2}{\sigma_{\rm total}^2}\Bigr).$$2

with intracluster correlation

$$\alpha = \frac{K}{K-1}\Bigl(1 - \frac{\sum_{i=1}^K \sigma_i^2}{\sigma_{\rm total}^2}\Bigr).$$3

Stage 6 constrains reporting language to demonstrated boundaries, recommends TRIPOD-LLM or MI-CLEAR-LLM, and treats validation as continuous rather than one-off. Instruments validated on one model version must be re-validated on updates.

Taken together, the framework and workflow define a research program for AI psychology. Their common position is that prompt success is not equivalent to construct validity, and that anthropomorphic claims should be replaced with validated descriptions of computational patterns unless higher-order evidence has been assembled. This suggests a disciplined division between description, simulation, and mechanism: the more human-like the claim, the more demanding the validation burden.