The Illusion of Intervention: Your LLM-Simulated Experiment is an Observational Study

Abstract: LLMs show potential as simulators of human behavior, offering a scalable way to study responses to interventions. However, because LLMs are trained largely on observational data, interventions in experiments with LLM-simulated synthetic users can induce unintended shifts in latent user attributes, causing user drift where the implicit simulated population differs across treatment conditions, potentially distorting effect estimates. We formalize the confounding or selection bias that can arise due to user drift and show how intervention-dependent shifts can inflate or attenuate observed differences in user responses under intervention. To diagnose confounding, we propose using negative control outcomes--attributes that should remain invariant under intervention--to identify distribution shifts across intervention conditions, providing evidence of user drift. To mitigate drift, we study adjusting the persona specification by eliciting additional confounders, finding that targeted, setting-relevant confounders can substantially reduce bias across survey-style and multi-turn agent evaluations.

• The paper reveals that LLM-simulated experiments induce substantial user drift, resulting in selection bias that challenges causal inference.
• It employs negative control outcomes and an iterative confounder adjustment scheme to diagnose and partially mitigate bias.
• The findings stress that such synthetic user interventions must be interpreted as observational studies, cautioning against straightforward causal claims.

Synthetic User Experiments with LLMs: An Observational, Not Interventional, Framework

Problem Formalization: User Drift as Selection Bias

This paper interrogates the foundational assumption underlying LLM-simulated user studies, namely that randomized assignment of interventions to LLM-initialized personas—often termed “synthetic users”—is equivalent to a randomized controlled trial (RCT) over a fixed population. The authors demonstrate rigorously that, due to the abductive inference nature of generative LLMs and their training on observational data, interventions (such as divergent system messages or agent behaviors) induce systematic shifts in the latent attributes of the “user” as simulated by the model. This drift means that, effectively, the instantiation of an identical persona in treatment and control groups results in responses that are generated by different (unobserved) user distributions, rather than counterfactual outcomes for the same underlying user.

Formally, when specifying a persona via observed attributes $L$ and presenting intervention $A$, the model samples from $P(Y|A=a, X, L) P(X|A=a, L)$, with $X$ encompassing latent attributes not fixed in $L$. In a true RCT, $P(X|A=1, L) = P(X|A=0, L)$ holds; in LLM simulations, this is violated, and therefore synthetic user interventions must be analyzed as observational studies subject to confounding/selection bias, rather than as interventional experiments. The paper provides an explicit decomposition of the treatment effect, showing that observed differences ($T_\text{obs}$) are generally biased estimators of causal effect due to drift-driven selection bias.

Diagnostic and Mitigation: Negative Controls and Persona Augmentation

To empirically diagnose the extent of user drift and resultant bias, the authors introduce the use of negative control outcomes. These are attributes or responses that would be invariant to the intervention in a real RCT (e.g., race, prior political affiliation) but depend on the unspecified latent user attributes that may drift. Variability in negative control outcome distributions across intervention arms indicates population shift in simulated users and hence confounding. The study operationalizes total variation distance (TVD) between negative control outcome distributions as a metric of induced bias.

To mitigate this bias, the paper proposes an iterative confounder adjustment scheme: by eliciting additional confounding/relevant user attributes ($L'$) and incorporating them into the persona specification, researchers can partially block the backdoor paths from intervention to outcome via the latent space. These attributes are chosen to be highly relevant to both the assigned intervention and the primary outcome, and the process is repeated until TVD across arms is acceptably reduced. Notably, the adjustment is not guaranteed to eliminate all bias, as prompt-based conditioning is only an approximation to perfect statistical conditioning.

Empirical Results: LLM-Simulated User Validity and Adjustment

The analysis spans survey-style and multi-turn agent evaluation scenarios, using a diverse set of instruction-tuned and fine-tuned LLMs. Empirical findings reveal that:

• Substantial user drift occurs even when personas are identically specified, leading to significant TVD between negative control outcomes in different intervention arms. This is observed across open-source and proprietary LLMs, with the strongest abductive models exhibiting the most pronounced drift and bias.
• Iterative confounder augmentation typically decreases TVD and stabilizes observed treatment effects, particularly when targeted, outcome- and intervention-relevant attributes are elicited. In many cases, generic demographic attributes alone are insufficient and can even increase bias before targeted attributes are incorporated.
• Observed estimates of intervention effects can shift considerably as confounders are added, confirming that naïve use of LLM-based synthetic user interventions may produce misleading causal or even qualitative conclusions.

Some models are notably less responsive to adjustment, and prompt refusal or model-specific “neutralization” training can obscure detection of drift.

Implications and Future Research

The major implication is that LLM-simulated user studies can only be interpreted as observational, not experimental. Abductive reasoning inherent to LLMs, particularly when exposed to underspecified prompt personas, enables post hoc modification of latent user identity conditional on intervention context. Future synthetic evaluation frameworks must treat such experiments with the methodological rigor of observational causal inference: including drift diagnostics, confounder identification and adjustment, and explicit caveats about unadjustable or residual bias.

Practically, for AI evaluation and iterative agent development, this work warns that causal claims about improvement, safety, or fairness based on synthetic users are tenuous unless confounding bias is explicitly diagnosed and minimized. For higher-fidelity evaluation, joint human and synthetic user studies should be compared, and mitigation strategies—e.g., leveraging doubly robust approaches or constraining LLM inference via strong persona binding and context augmentation—should be further developed.

Theoretically, this work reframes the generative use of LLMs in simulation as fundamentally non-causal unless new training or inference paradigms are developed that enforce consistency of latent user traits with “upstream” causal ordering. The necessity for task- and outcome-specific confounder specification suggests inherent limitations of prompt-based user simulation for robust social or behavioral research.

Conclusion

This paper exposes and formally characterizes selection bias induced by user drift in LLM-simulated interventions, situating such synthetic experiments within the framework of observational rather than randomized designs. Using negative control diagnostics and targeted persona augmentation, drift can be partially mitigated—but not eliminated—rendering all claims from LLM user simulation contingent on careful bias quantification and adjustment. Robust AI evaluation must internalize this distinction to avoid misattributed or spurious causal conclusions in agent and system development.

Citation: "The Illusion of Intervention: Your LLM-Simulated Experiment is an Observational Study" (2605.20767)