SP-ABCBench: S&P Attitude–Behavior–Coherence Benchmark

• SP-ABCBench is a benchmark that defines 30 tests based on 15 empirical security and privacy studies to simulate human attitudes, behaviors, and coherence.
• It operationalizes quantitative measures on a 0–100 scale to assess how well LLM-driven personas mimic validated human responses in security and privacy contexts.
• The benchmark highlights the influence of persona strategies and theory-informed prompting on simulation fidelity, informing red teaming and risk evaluation practices.

SP-ABCBench, the Security and Privacy Attitude–Behavior–Coherence Benchmark, is a benchmark introduced to test whether LLM agents endowed with personas can approximate how populations of real people think about, feel about, and act on security and privacy risks. It is presented as an empirical interrogation of the assumption that LLM-driven “personas” or agent populations can function as stand-ins for real users in security and privacy and responsible-AI workflows. The benchmark is built from 15 human-subject S&P studies and defines 30 tests evaluated on a unified 0–100 ascending scale, where higher scores indicate better alignment between simulations and published human-subject findings across the three dimensions of Attitude, Behavior, and Coherence (Li et al., 6 Feb 2026).

1. Motivation and problem setting

SP-ABCBench is positioned against a fast-growing practice in security and privacy research: using LLM-driven synthetic users as proxies for human populations. In the formulation associated with the benchmark, such agents are proposed both as “crash dummies for S&P” and as risk-envisioning tools. The first usage treats synthetic users as instruments for red teaming and security testing; the second treats them as scalable simulators of user reactions to new AI products or interfaces (Li et al., 6 Feb 2026).

The benchmark is motivated by the claim that security and privacy decision-making is a particularly difficult target for simulation. The underlying rationale is that security and privacy are often secondary to users’ primary goals, are preventive rather than immediately rewarding, and concern harms that are abstract, probabilistic, and often invisible. The paper further frames human S&P behavior as shaped by heuristics and bounded rationality, with context-dependent cost–benefit tradeoffs described through frameworks such as Privacy Calculus, Bounded Rationality, SPAF, and Contextual Integrity (Li et al., 6 Feb 2026).

Three research questions structure the benchmark. RQ1 asks whether current LLMs reproduce population-level S&P patterns from validated human studies, and whether bigger, newer, or more reasoning-capable models perform better. RQ2 asks how persona construction affects alignment, comparing demographics-only personas to richer narratives and S&P-specific priming. RQ3 asks whether theory-informed prompting, especially prompts that encourage bounded-rational and tradeoff-based reasoning, improves alignment, particularly on behavior (Li et al., 6 Feb 2026).

A plausible implication is that SP-ABCBench is not merely a model leaderboard. It is also a methodological benchmark for the use of LLM agents as substitutes or supplements for human-subject studies in pre-deployment evaluation, scalable risk forecasting, and early-stage product design.

2. Benchmark composition and the Attitude–Behavior–Coherence framework

SP-ABCBench is constructed from 15 human-subject S&P studies. These include four psychometric or attitudinal instruments—IUIPC, SeBIS, SA-6, and OPC—and eleven behavioral effects, many drawn from Acquisti et al.’s “Privacy and Human Behavior” corpus. The behavioral source studies include privacy premium and willingness to pay for privacy, confidentiality assurances and survey participation, gift card anonymity and endowment effects, interface formality and disclosure, social normative information and sensitive disclosure, unintended audience on social networks and visibility restrictions, personalized versus generic ads and privacy settings, visible privacy policy links and disclosure, government regulation assurances for location-based service adoption, and the control paradox (Li et al., 6 Feb 2026).

From these sources, the benchmark defines 30 tests: 4 Attitude tests, 11 Behavior tests, and 15 Coherence tests. Selection requires that a study report quantitative, population-level findings, that the experimental conditions be reproducible via text or images, and that the study be well-cited / foundational, often replicated, so that its findings can serve as credible ground truth (Li et al., 6 Feb 2026).

The three dimensions correspond to distinct targets of simulation:

Dimension	Target	Operationalization
Attitude	What people believe, intend, or feel	Distributions of responses to validated Likert-scale instruments
Behavior	What people actually do in specific scenarios	Choice distributions or rates under controlled manipulations
Coherence	Whether relationships among S&P constructs resemble human data	Correlations, SEM paths, and demographic contrasts

Within Attitude, the benchmark covers IUIPC subscale distributions for collection, control, awareness, trust, risk, and intention; SeBIS per-item distributions; the SA-6 overall security attitude distribution; and OPC subscale distributions for concern and protection. Within Behavior, each test is conceptualized as an effect size or ratio, such as privacy premium in shopping choices, disclosure under casual versus professional interfaces, social norm effects on sensitive disclosure, policy-link effects on real versus fake information submission, adoption of location-based services with versus without government regulation assurances, and disclosure under high versus low perceived control (Li et al., 6 Feb 2026).

The Coherence dimension addresses whether simulated responses preserve structural relations seen in human data. These tests include IUIPC structural paths linking concern, trust, risk, and behavioral intention; SeBIS correlations with DoSpeRT, GDMS, NFC, BIS, and CFC; SA-6 convergent validity with SeBIS, BIS, GSE, and SSE; SA-6 demographic contrasts using Welch’s $t$-statistics for age, gender, education, and income; and OPC subscale intercorrelations (Li et al., 6 Feb 2026).

This decomposition is one of the benchmark’s defining features. It distinguishes between matching what people say, matching what they do, and matching how attitudes, traits, and demographics relate to one another.

3. Operationalization, scoring, and evaluation protocol

Each SP-ABCBench test is transformed into a prompted task for LLM agents. For each source study, a synthetic sample is drawn to match the original demographic distribution as closely as possible using US Census PUMS data and stratified sampling over sex, age, education, and income. When exact demographics are unavailable, proxies such as replication papers or Pew Internet data are used. The synthetic sample size matches the original human study, and each sampled demographic profile becomes one agent persona under a chosen persona-construction strategy (Li et al., 6 Feb 2026).

In survey simulations, agents are instructed to respond as a person rather than as an AI. They receive persona descriptions plus demographics and answer full questionnaires, including IUIPC, SeBIS, SA-6, OPC, and companion scales needed for coherence analysis. Responses are scored with the same rules as the original papers, including reverse-coding items and aggregation into subscales or totals. In scenario simulations, agents read scenario descriptions, often with screenshots matching original interfaces, and choose among discrete actions. Behavioral effects are administered as two or three conditions mirroring the original manipulations, with each agent randomly assigned to a condition (Li et al., 6 Feb 2026).

The central metric is simulation quality (SQ), defined as the alignment between simulated and human-population results on a 0–100 ascending scale. The scale is interpreted as follows: 0 indicates approximately no alignment, often wrong direction or extremely off magnitude; 50 indicates that the effect direction is correct but the magnitude or distribution is substantially off; 100 indicates perfect alignment within tolerance (Li et al., 6 Feb 2026).

Because the benchmark targets heterogeneous quantities, the scoring is dimension-specific. For Attitude, the benchmark computes Cohen’s $d$ between simulated and published score distributions for each subscale or item, averages these values, and uses a tolerance $\tau = 2.0$:

$$s = \text{clip}_{[0,1]}\left(1 - \frac{|d|}{\tau}\right)$$

The resulting $s$ is then scaled by 100. For Behavior, each test is converted into a ratio of behavioral rates across conditions, and the score is defined piecewise with $R_{\max}=100$:

$$s = \begin{cases} 0, & r \leq 0 \ 0.5\, r, & 0 < r \leq 1 \ 0.5 + 0.5\;\dfrac{\ln(\min\{r,\, R_{\max}\})}{\ln R_{\max}}, & r > 1 \end{cases}$$

For Coherence, deviations between simulated and published correlations or Welch’s $t$-statistics are scored via

$$s = \text{clip}_{[0,1]}\left(1 - \frac{|x - x^*|}{\tau}\right),$$

with $\tau = 0.5$ for correlations and $d$0 for $d$1-statistics (Li et al., 6 Feb 2026).

Test-level scores are averaged to form dimension-level scores, and overall SQ is the average across all 30 tests. Robustness is assessed through tolerance sensitivity by varying $d$2 by $d$3. The reported result is that Kendall’s $d$4 exceeds 0.90 and mean absolute rank shift is below 1.5 across all persona strategies, indicating robustness to reasonable tolerance choices (Li et al., 6 Feb 2026).

The evaluation protocol uses single runs per configuration with temperature = 1.0 for persona generation and temperature = 0.3 for task inference. There is exactly one response per synthetic agent and no repeated sampling. The benchmark reports observed averages as actual performance rather than confidence intervals or p-values, and supplements the quantitative analysis with open coding of sampled rationales from high- and low-SQ configurations (Li et al., 6 Feb 2026).

4. Models, persona strategies, and prompting methods

The benchmark evaluates twelve models from four families. These are GPT-4.1-Nano, GPT-4.1-Mini, GPT-4.1, GPT-5-Nano (Minimal Reasoning), GPT-5-Mini (Minimal Reasoning), GPT-5 (Minimal Reasoning), GPT-5 (Medium Reasoning), Gemini-2.5-Flash-Lite, Gemini-2.5-Flash, Gemini-3.0-Flash, Qwen3-Next-80B, and Llama-4-Maverick. The evaluation spans proprietary and open-source systems, smaller and larger variants, and different reasoning modes; all are accessed through APIs and none are fine-tuned (Li et al., 6 Feb 2026).

Four persona construction strategies are compared. Strategy I: Demographic uses only demographic attributes and no narrative. Strategy II: Raw Persona adds a 120–150 word LLM-generated biographical narrative including name, occupation, hobbies, personality, life experiences, and goals, but no explicit mention of security or privacy attitudes. Strategy III: S&P-Primed Persona uses the same narrative structure but explicitly instructs the model to describe the person’s security and privacy attitudes, behaviors, and decisions. Strategy IV: Scenario-Primed Persona conditions the persona on the specific test or scenario through a <TASK_SPECIFIC_INFO> block; for example, IUIPC-related traits for IUIPC tests, SeBIS subdomains for SeBIS tests, or scenario-conditioned tendencies for behavioral tasks (Li et al., 6 Feb 2026).

Two task prompting conditions are studied. The Baseline prompt asks the model to respond authentically as the given persona would in real life, not as an AI, and permits light chain-of-thought. The theory-informed condition appends a block that explicitly encourages bounded-rational, tradeoff-based reasoning, invoking Privacy Calculus and bounded rationality by emphasizing benefit, trust, effort, risk, convenience, and social normalcy rather than idealized optimization (Li et al., 6 Feb 2026).

The experimental design thereby isolates three axes of variation: model family and scale, persona specification, and prompting method. This structure makes it possible to evaluate not only whether a given LLM can simulate S&P populations, but also under what methodological conditions such simulation is more or less aligned.

5. Empirical findings

Across all models and configurations, average SQ across models ranges from 50 to 64 on the 0–100 scale. Under a baseline setup using Demographic personas, the overall score distribution has mean $d$5 and SD $d$6, with scores spanning the full 0–100 range. By dimension, Attitude has mean $d$7 and SD $d$8, Behavior has mean $d$9 and SD $\tau = 2.0$0 with the lowest mean and median, and Coherence has mean $\tau = 2.0$1 and SD $\tau = 2.0$2 with a bimodal distribution. Overall, 26.94% of configuration–test pairs score at least 80, and 13.33% score at least 90 (Li et al., 6 Feb 2026).

Model rankings are relatively tight. The reported overall ranking runs from GPT-4.1 at $\tau = 2.0$3 to Gemini-2.5-Flash-Lite at $\tau = 2.0$4, a spread of about 13 points. Under Demographic personas, the reported dimension leaders are GPT-4.1-Nano on Attitude with mean $\tau = 2.0$5, Gemini-3.0-Flash on Behavior with mean $\tau = 2.0$6, and Gemini-2.5-Flash-Lite on Coherence with mean $\tau = 2.0$7 (Li et al., 6 Feb 2026).

A central result is that newer, bigger, and smarter models do not reliably do better and sometimes do worse. The paper reports that smaller OpenAI variants average $\tau = 2.0$8 versus $\tau = 2.0$9 for larger variants; GPT-4.1-Mini ($$s = \text{clip}_{[0,1]}\left(1 - \frac{|d|}{\tau}\right)$$0) exceeds GPT-4.1 ($$s = \text{clip}_{[0,1]}\left(1 - \frac{|d|}{\tau}\right)$$1) by more than 10 points; Gemini-2.5-Flash-Lite ($$s = \text{clip}_{[0,1]}\left(1 - \frac{|d|}{\tau}\right)$$2) exceeds Gemini-2.5-Flash ($$s = \text{clip}_{[0,1]}\left(1 - \frac{|d|}{\tau}\right)$$3) by about five points; GPT-5 variants average $$s = \text{clip}_{[0,1]}\left(1 - \frac{|d|}{\tau}\right)$$4 versus $$s = \text{clip}_{[0,1]}\left(1 - \frac{|d|}{\tau}\right)$$5 for GPT-4.1 variants; and GPT-5 (Medium Reasoning) ($$s = \text{clip}_{[0,1]}\left(1 - \frac{|d|}{\tau}\right)$$6) does not outperform GPT-5 (Minimal Reasoning) ($$s = \text{clip}_{[0,1]}\left(1 - \frac{|d|}{\tau}\right)$$7). Proprietary models average $$s = \text{clip}_{[0,1]}\left(1 - \frac{|d|}{\tau}\right)$$8 versus $$s = \text{clip}_{[0,1]}\left(1 - \frac{|d|}{\tau}\right)$$9 for open-source models, but Llama-4-Maverick ($s$0) exceeds several proprietary systems (Li et al., 6 Feb 2026).

Persona strategy produces dimension-specific tradeoffs. Overall means across all dimensions are similar, around 58.7–59.5, but Attitude tests are best with S&P-Primed Persona ($s$1), followed by Raw Persona ($s$2), while Scenario-Primed Persona ($s$3) performs worst. In contrast, Behavior tests are best with Scenario-Primed Personas ($s$4), followed by Demographic ($s$5), while Coherence scores cluster around 58.6–59.9 across strategies with minimal differences (Li et al., 6 Feb 2026).

Theory-informed prompting yields an overall SQ gain of +2.55 points on average across four representative models: GPT-4.1, GPT-5 Minimal, Gemini-2.5-Flash-Lite, and Gemini-2.5-Flash. By dimension, the reported average changes are +3.05 for Behavior, +3.36 for Coherence, and $s$6 for Attitude. The gains differ by persona strategy: overall improvements are +3.64 for Demographic, +3.58 for Raw Persona, +0.95 for S&P-Primed, and +2.02 for Scenario-Primed. The benchmark characterizes the pattern as one in which theory-informed prompting helps most where baseline performance is weak, while occasionally harming already strong configurations (Li et al., 6 Feb 2026).

The benchmark also identifies high-alignment cases. One example is the IUIPC structural paths coherence test, where GPT-5 with theory-informed prompting reaches SQ $s$7 and reproduces the pattern that higher privacy concern leads to higher perceived risk and lower disclosure intention. Several Behavior tests also exceed 95 under combinations of Scenario-Primed personas, theory-informed prompting, and recent models such as Gemini-3.0-Flash and GPT-5 Minimal (Li et al., 6 Feb 2026).

6. Failure modes, interpretation, and benchmark significance

Qualitative analysis identifies several recurring failure modes. One is over-idealized attitudes, where agents express extreme concern or diligence and thereby compress distributions rather than reflecting heterogeneous human data. Another is the replacement of contextual judgment with global policies, such as blanket rules never to share personal information online, which causes the model to ignore experimental manipulations. A third is misinterpretation of cues; in the interface-formality task, some configurations treat casual interfaces as more risky and professional interfaces as safer, thereby reversing the human effect. A fourth is structural incoherence, where plausible scale responses fail to preserve relationships such as “higher concern $s$8 higher perceived risk” (Li et al., 6 Feb 2026).

The paper interprets these findings through a distinction between believability and accuracy. LLM agents often produce outputs that appear plausible in isolation, but quantitative alignment with population-level human findings remains only moderate on average. The benchmark’s mean performance, near 60, is taken to indicate that models often recover effect direction but miss effect magnitude, distributional spread, or structural relationships (Li et al., 6 Feb 2026).

This leads to a constrained view of current LLM-as-user simulation in security and privacy. The benchmark supports using LLM agents for hypothesis generation, for surfacing plausible concerns and edge cases, and for functioning as “crash dummies” in red teaming or safety simulation. It does not support using simulated rates as reliable forecasts, nor does it support replacing user studies for design questions that hinge on calibrated effect sizes (Li et al., 6 Feb 2026).

The paper further emphasizes several cautions. Hyper-accuracy distortion is used to describe the possibility that larger or more capable models become too normatively rational or risk-focused, thereby missing heuristic and boundedly rational human choices. Over-priming is identified as a risk for Scenario-Primed personas, especially in Attitude tests, because scenario conditioning can compress variance and make the synthetic population unrealistically homogeneous. The benchmark also notes cultural and temporal biases, since source studies are largely West / US-based and span specific historical periods, as well as limitations in capturing affective and contextual drivers of behavior through text prompts alone (Li et al., 6 Feb 2026).

Within the broader literature, SP-ABCBench is described as occupying the intersection of LLM-as-human simulation / Turing Experiments, agent-based social simulations, and S&P risk and safety tooling. It differs from generic psychological or economic simulations by being S&P-specific, by drawing exclusively on validated security and privacy empirical studies, by decomposing evaluation into Attitude, Behavior, and Coherence, and by targeting population-level alignment rather than single-subject imitation (Li et al., 6 Feb 2026).

The benchmark is released with its definition, prompts, demographic sampling code, evaluation pipeline, and generated agent responses and reasoning traces, while explicitly not releasing raw human data. It is intended as a reproducible foundation for future work on non-US and cross-cultural datasets, more recent empirical studies, multi-step interactive tasks, and alternative simulation methods such as fine-tuning on behavioral data or using other theoretical frameworks as prompting scaffolds (Li et al., 6 Feb 2026). This suggests that SP-ABCBench functions both as a measurement instrument and as a methodological constraint: it formalizes what it would mean for LLM agents to simulate human security and privacy populations with quantitative fidelity, while demonstrating that current systems remain inconsistent across models, persona strategies, and prompting regimes.