Calibrating the Instrument: Controllability of an LLM-Driven Synthetic Population

Abstract: Generative Synthetic Populations (GSP) -- the convergence of population synthesis, agent-based modelling, and LLM agents -- are attracting growing interest for urban simulation and institutional communication research. Before any GSP instrument is used on a real population, a more basic question must be answered: does it respond to stimuli of known valence in an ordered, replicable, group-structured way? We call this controllability. We ask not whether a synthetic population tracks humans, but whether it tracks itself: whether the latent structure we impose on it is recovered in its own responses. This internal-validity question is logically prior to any claim about external validity, just as characterising an instrument's response function must precede using it to test a theory. We report SIVE (Synthetic Instrument Validation Experiment): a fictional municipality (Montelago) with 120 synthetic personas of known latent structure, exposed to seven conditions spanning strongly positive to strongly negative institutional communications about a water network. Seven pre-registered criteria, evaluated across a temperature sweep, jointly assess fidelity, stability, noise floor, specificity, sensitivity, and ordering. All seven pass at every temperature. A central finding turns a calibration failure into a diagnostic success: a message designed as "weakly positive" was identified by the instrument as functionally negative, traced to unresolved problems, uncertainty, and institutional passivity in its text; a redesigned version restored the expected ordering and interacts with agents' latent trust in unanticipated ways. A noise sub-experiment shows the instrument's intrinsic noise is roughly half the cross-agent estimate and stable across temperatures. Individual trajectories reveal coherent micro-dynamics that summary statistics obscure. Full data are available via an interactive explorer.

• The paper demonstrates that SIVE achieves high persona fidelity (r ≈ 0.89–0.91) by accurately recovering imposed latent trust structures through controlled experimental stimuli.
• It employs a multi-stage survey and diverse stimulus conditions to rigorously evaluate noise, specificity, and sensitivity in synthetic population responses.
• Findings reveal that positive and negative stimuli amplify existing trust dynamics, underscoring the necessity for precise instrument calibration before real-world application.

Controllability in LLM-Driven Synthetic Populations: Insights from SIVE

Introduction and Motivation

The emergence of Generative Synthetic Populations (GSPs)—which integrate advances in maximum-entropy population synthesis, agent-based modelling (ABM), and LLM-based agent simulation—marks a shift in urban simulation and institutional communication research. However, before their adoption for real-world analysis, rigorous instrument validation is required to ensure that synthetic populations are interpretable, responsive, and internally valid. The paper "Calibrating the Instrument: Controllability of an LLM-Driven Synthetic Population" (2607.00910) introduces a principled experimental system, SIVE (Synthetic Instrument Validation Experiment), to probe the foundational property of controllability: whether a synthetic population, when subjected to stimuli of known valence, produces ordered, replicable, and group-structured responses as predicted by its latent structure.

This reframes internal validity as the logically prior concern—demanding that the instrument tracks its own imposed latent structure, before any claim regarding alignment with real human populations (external validity) can be entertained. The approach is positioned as analogous to characterizing an instrument's response function in physics, separated from testing any substantive theory.

Experimental System Design

Population and Latent Structure

A synthetic, explicitly structured population of 120 personas (the "Montelago" population) was constructed. Each persona encodes three numeric attitude scores on $[1, 10]$: institutional trust (fiducia_istituzione), source credibility (credibilita), and perceived information adequacy (adeguatezza_info). Importantly, only the demographic profile and free-form background story are accessible to the LLM; numeric attitudes are available exclusively for validation and are withheld during inference. The core population is segmented into three equal-sized latent trust groups (low, medium, high).

Stimulus Design

Each persona is exposed to one of seven experimental conditions, instantiated as municipal messages with controlled valence spanning a spectrum from strongly positive to strongly negative. These conditions include:

• POS/POS2: Strong positive message, repeated to estimate the noise floor.
• POSW: Intentionally “weak positive” but post hoc identified as functionally negative due to textual content.
• POSW2: Recalibrated weak positive, correcting the defects of POSW.
• NEG: Strong negative message.
• PLA: Orthogonal placebo (non-relevant event).
• CTRL: No message (control).

This design enables precise deconvolution of sensitivity, specificity, and noise properties.

Measurement Protocol

A fixed sequence is applied: agents answer pre-stimulus surveys, react in three temporal stages to the stimulus, then respond to the same battery post-stimulus. The primary quantitative measure is the change in institutional trust; secondary metrics capture credibility, information adequacy, categorical emotion, and behavioral intention. Thirteen API calls per agent/condition ensure rigorous comparability and cost control.

Quantitative Validation Criteria and Results

Seven pre-registered criteria operationalize instrument characterization:

1. Persona Fidelity: Pearson correlation ($r>0.8$) between encoded trust and pre-stimulus trust; consistently met ($r\approx 0.89$–$0.91$).
2. Cross-Replica Stability: Baseline trust scores are invariant across condition orderings ($r_{\text{min}}>0.85$).
3. Noise Floor: Estimated via differential responses to identical stimuli (POS/POS2), with cross-agent $\sigma \approx 1.4$ and within-profile $\sigma \approx 0.77$—establishing actual instrument precision.
4. Specificity: Placebo stimulus produces null effect indistinguishable from control ($|\Delta| < 0.5$).
5. Sensitivity: Condition means are ordered per known valence; particularly, recalibration of POSW (to POSW2) resolved a salient misalignment.
6. Ordering: Kendall’s $\tau$ achieves perfect/near-perfect concordance; the ordering criterion is strictly satisfied only after correcting the stimulus miscalibration.
7. Receipt Check: On-topic messages reliably register as information receipt in adequacy ratings, regardless of valence.

All seven criteria are satisfied at all tested temperatures ($t \in \{0.2, 0.5, 0.7\}$). One robustly diagnostic finding is that a stimulus originally labelled “weak positive” was classified by the instrument as effectively negative, a result not due to LLM noise but to latent textual cues of uncertainty and passivity. Redesigning the message restored the expected ordering and revealed complex agent-by-message interactions, notably an amplification effect: positive signals reinforced existing trust—increasing polarization—while negative or ambiguous signals differentially penalized high-trust and low-trust personas.

Micro-Level Dynamics

Aggregate statistics (mean trust shift per group/condition) obscure substantial heterogeneity in reaction trajectories. Micro-analyses of agent responses under negative valence conditions demonstrate that even agents with identical trust shifts employ qualitatively distinct coping mechanisms: resignation, systematic investigation, internal mediation, or methodical civic alarm. The synthetic population framework thus recovers not only ordered population-level signals but also psychologically consistent, agent-level variation.

This supports the claim that LLM-driven agents capture not only summary response gradients but also internally plausible micro-dynamics—crucial for downstream applications in simulation and scenario analysis.

Broader Implications and Limitations

Practical Implications

By systematically quantifying controllability and characterizing noise properties, the SIVE protocol establishes a framework for vetting synthetic populations prior to case-driven applications. In applied institutional communication studies, this makes it possible to empirically validate the valence and impact of designed interventions—e.g., to assess whether institutional messages act as intended in heterogeneous audiences, especially when negative or ambiguous cues may backfire.

Theoretical Implications

The distinction between instrument controllability and external validity is sharply delineated. SIVE addresses the logically prior question: Does the instrument recover its imposed latent structure under controlled, known interventions? Only after this property is established should one proceed to measure alignment with human data or real populations.

Limitations

• Model and language specificity: All results are under DeepSeek (Italian). Replication across architectures and languages remains outstanding.
• The within-population sample (n=120) supports robust estimation but limits inference at intersectional subgroup levels.
• No claim is made about external validity or real-world policy interpretability; the experiment is explicitly synthetic.

Conclusion

SIVE provides an authoritative framework and demonstrative evidence for controllability in LLM-driven synthetic populations. The protocol rigorously quantifies and interprets internal validity prior to any external-facing conclusions—measuring stimulus calibration, noise, and response function directly.

The approach elucidates the necessity of instrument validation in agent-based simulation pipelines leveraging LLMs for psychology and behavior generation. These results substantiate the relevance of formal internal validity checks—analogous to physical calibration tests—for generative instruments intended for use as digital twins or social simulators. The capacity to recover imposed latent semantics is a minimal but necessary precondition for trustworthy downstream application and for further research on the human-alignment problem in socio-technical settings (2607.00910).