When prompt perturbations break your A/B test: A valid statistical test for generative surveying

Abstract: Generative surveying -- where collections of LLM-based personas provide feedback on messages -- has emerged as a cheap and scalable alternative to traditional market research. However, LLMs are sensitive to small variations in prompt design and conclusions drawn from generative surveys may depend on arbitrary phrasing choices. Controlling for this sensitivity requires including semantically equivalent perturbations in the analysis. In this paper, we show that standard hypothesis tests, including the sign test and Wilcoxon signed-rank test, are invalid under a statistical model for generative surveying that includes realistic perturbation structure. We propose a permutation test that is valid under this model and formally characterize the conditions under which standard tests fail. Applying our framework to a simple generative surveying problem, we estimate relevant parameters, characterize the power of the permutation test under realistic conditions, and provide practical guidance on budget allocation across personas, perturbations, and replicates. Finally, we show that both the magnitude and direction of the estimated effect are sensitive to the choice of model, even within the same model family.

• The paper proposes a novel permutation test that addresses cross-persona correlations induced by semantically equivalent prompt perturbations.
• It employs a hierarchical generative model accounting for baseline preferences, perturbation effects, and replicate structure to ensure valid inference.
• Empirical validation across LLM families demonstrates improved power strategies while highlighting the pitfalls of standard paired tests in generative surveys.

Valid Hypothesis Testing for Generative Surveying with LLM Prompt Sensitivity

Motivation and Problem Formulation

Generative surveying, wherein populations of LLM-based personas are interrogated for simulated feedback, is increasingly utilized for rapid, scalable market research, preference elicitation, and policy simulation. However, intrinsic sensitivity of LLMs to prompt perturbations fundamentally undermines the validity of conclusions drawn from such surveys. Standard paired hypothesis tests—specifically the sign test and the Wilcoxon signed-rank test—presuppose independence across sampled units. In generative surveying, shared, semantically equivalent prompt perturbations across personas induce correlated preference shifts, violating this independence assumption and resulting in severely inflated Type-I error rates.

Figure 1: Persona-based generative message testing: The aggregation process across personas induces cross-persona correlations; paired tests exhibit inflated Type-I error under these correlations.

The paper develops a formal statistical model for generative surveying that explicitly incorporates cross-persona correlation via a perturbation-level random effect. Under this model, the authors demonstrate the invalidity of standard paired tests and propose a valid permutation test for generative survey inference.

Statistical Model and Hypothesis Testing

The generative model organizes responses hierarchically: personas, perturbations, and replicates. Each persona possesses a latent baseline preference, perturbed by semantically equivalent prompt variants, with the degree of shared perturbation-induced variance controlled by parameter $\rho$.

Figure 2: Generative model structure—persona-level Beta priors, perturbation-level correlations (parameter $\rho$), and empirical CDFs of p-values illustrating oversized sign tests and validity of permutation tests for $\rho > 0$.

• Persona Level: Baseline preference $p_i \sim \text{Beta}(\alpha_0, \beta_0)$.
• Perturbation Level: Logit-scale perturbation effect $u_j$ induces cross-persona correlation; higher $\rho$ correlates responses across personas.
• Replicate Level: Multiple observations per (persona, perturbation) pair are sampled to estimate preferences.

The standard hypothesis of interest tests for mean preference difference between two messages (e.g., sneakers vs. boots):

$$H_0: \mathbb{E}[\delta_i] = 0 \quad \text{vs} \quad H_A: \mathbb{E}[\delta_i] \neq 0$$

Within this structure, the sign test and Wilcoxon signed-rank test, which assume independence, are shown to be invalid when $\rho > 0$. The shared $u_j$ induces cross-persona dependence such that critical values are underestimated—leading to inflated null rejection rates.

The proposed permutation test operates at the perturbation level, sign-flipping message labels to construct the exact null distribution, thus controlling for cross-persona correlation irrespective of $\rho$. Theoretical results establish exact size $\rho$0 under $\rho$1, and consistency under $\rho$2.

Empirical Validation and Numerical Results

Experiments were conducted across ten LLMs from the Mistral-3 and Qwen-3 families, using simulated persona-based purchase-intent queries (sneakers vs. boots). The empirical CDFs of p-values under the null (sneakers vs. sneakers split) substantiate that the sign test is oversized across all configurations, whereas the permutation test remains valid for $\rho$3 and mildly conservative for $\rho$4 due to permutation distribution coarseness at small $\rho$5.

Figure 3: Empirical CDFs of p-values under the null condition for sign and permutation tests; sign test exhibits substantial oversizing, the permutation test tracks the diagonal.

Power analysis reveals large inter-model variance, with highest power observed in Mistral-3-14B and Qwen-3-14B (achieving power $\rho$6 at moderate budgets), and near-degenerate response distributions (e.g., Qwen-3-0.6B and Qwen-3-Large) demonstrating uniformly low power. Budget allocation strongly favors perturbation count ($\rho$7) over replicates ($\rho$8) or personas ($\rho$9); allocating additional query bandwidth toward perturbations maximizes power.

Figure 4: Power curves for permutation test by budget allocation; maximizing perturbations ($\rho > 0$0) yields highest power. Effect of increasing $\rho > 0$1 is substantial downward shift in achievable power.

Parameter estimates confirm that cross-persona correlation is ubiquitous ($\rho > 0$2 for most models), reinforcing general invalidity of paired tests. Estimates also indicate poor identifiability in degenerate response regimes.

Model Sensitivity and Downstream Implications

The direction and magnitude of A/B effects (e.g., preference of sneakers vs. boots) are highly model-conditional. Effect size varies non-monotonically within model families; same-scale models across families generate effect estimates of opposite sign. For instance, Mistral-3-8B and Qwen-3-8B yield contradictory preferences for the same product comparison.

Figure 5: Effect size ($\rho > 0$3) versus model scale—effect estimates are non-monotonic in scale and often contradictory across families.

These findings necessitate explicit reporting of the underlying LLMs and replication across models; conclusions from generative surveys should be regarded as conditional on both architecture and prompt design.

Practical Recommendations and Theoretical Implications

• Always use perturbations and apply the permutation test: Practitioners should use $\rho > 0$4 semantically equivalent prompt variants and avoid paired tests; the permutation test corrects for correlation and is robust.
• Allocate query budget toward perturbations ($\rho > 0$5): Power scaling is most efficient when resources are directed to perturbation diversity rather than replicates or personas.
• Treat results as model-conditional: Substantial variation in effect size and sign across LLMs demands explicit documentation of model and prompt structure; conclusions should not be generalized across architectures or scales without validation.

Theoretically, this work highlights the necessity of careful statistical modeling for generative surveys, accounting for intra-group dependencies induced by prompt perturbations. The permutation test framework can be generalized to any response structure with appropriate perturbation-level statistics.

Conclusion

This work formally establishes that standard paired tests (sign test, Wilcoxon signed-rank) are invalid for generative surveying when semantically equivalent prompt perturbations induce cross-persona correlation. A simple permutation test is proposed and proven valid, with empirical results substantiating both the necessity and efficacy of the method. Model choice prescribes survey outcome directionality, underlining the importance of rigorous reporting and cross-model validation. The analysis delivers actionable guidelines for generative survey design and motivates future extensions to broader response modalities and more sophisticated persona generation protocols.