When Surveys Become Conversations: Adaptive Matrix Validation for AI-Assisted Interviews

Abstract: AI-assisted interviews promise to reduce respondent burden in surveys by allowing respondents to describe experiences naturally while an AI system noisily maps those accounts into structured survey variables. That mapping is a measurement process that is fallible, versioned, adaptive, and potentially behaves differently across subgroups. This paper proposes Adaptive Matrix Validation (AMV), a design in which each respondent completes an AI-assisted interview, which is then mapped into tabular data by the AI. Respondents are also asked a small, randomized set of structured questions, which are used for statistical adjustment. The estimator first calibrates the mapped values using validation answers from other respondents, then corrects the remaining error with the validation answers observed for the target respondent. The paper develops estimators for item means, subgroup estimates, and regression coefficients when outcomes, predictors, or both are mapped from interviews. It also gives planning formulas the number of validation questions required and the sample size. A design-calibration simulation, an American Time Use Survey emulation, and a CHAMPS verbal-autopsy narrative study show when sparse validation can improve precision and when it cannot

• The paper's main contribution is presenting the AMV framework that calibrates AI-mapped narrative interviews using sparse, randomized validation questions to correct measurement error.
• The methodology employs a two-phase sample design with cross-fold calibration and augmented inverse-probability weighting to ensure unbiased estimation for means, subgroups, and regression coefficients.
• Empirical simulations and a CHAMPS study demonstrate that AMV significantly reduces bias and RMSE compared to mapping-only or validation-only approaches, contingent on high mapping quality.

Adaptive Matrix Validation for AI-Assisted Interviews: Measurement, Calibration, and Statistical Tradeoffs

Motivation and Problem Formulation

AI-assisted conversational interviewing is proposed as a means to reduce respondent burden in surveys by allowing respondents to express experiences conversationally, while an AI system generates structured variables via mapping from narrative accounts. This mapping process, however, is inherently fallible, subject to model versioning, adaptability, and may exhibit differential accuracy across subpopulations. The absence of perfect mapping necessitates statistical machinery to correct for measurement error and to ensure valid inference for population parameters, subgroups, and regression effects.

The paper introduces Adaptive Matrix Validation (AMV), a survey design and estimation framework in which all respondents complete an AI-assisted conversational interview, after which a sparse random subset of structured survey questions (“validation questions”) is asked. These structured items are used for statistical correction and calibration. The framework supports estimation of means, subgroup means, and regression coefficients—where outcome, predictors, or both can be mapped by the AI rather than directly observed—while providing principled formulae for sample size and validation burden planning.

Statistical Design and Estimation

AMV is conceptually a two-phase sample design: every respondent's conversation is mapped to an entire matrix of structured survey items by the AI, but only a sparse, respondent-specific, random subset is validated via direct questions. The mapping quality (i.e., the explained variance between the mapped value and the gold-standard structured response) is a critical design parameter. Validation tiles (randomized sets of structured items assigned for validation) must be designed to support planned analyses, with known joint and marginal selection probabilities for each item or item set needed in means, subgroups, or regression analyses. These constraints impose joint design requirements not only on validation-item randomization but also on which combinations of variables are validated together for regression support.

Estimation in AMV proceeds via two core steps: (1) Calibration: Using cross-fold validation, the relationship between mapped AI responses and direct validation answers is estimated to determine how much trust to place in the mapped value for each item or statistic (a shrinkage parameter $\lambda$). (2) Augmented Correction: The error that remains—estimated using the subset of cases with validation questions—is incorporated using an augmented inverse-probability weighting scheme that combines all mapped responses with validation-based correction. At both item and regression levels, the estimation is designed to guarantee unbiasedness for the finite-sample means or regression effects that would be obtained if full-structure data were available.

Sample Size and Precision/Cost Tradeoffs

The paper derives explicit formulas for the variance inflation and effective sample size requirements in AMV as a function of (a) mapping quality (unexplained variance ratio, $\rho$), (b) the number of validation questions ($B$) per respondent, and (c) the universe size of possible items $p$. The formula clarifies that a high-quality mapping ($\rho \ll 1$) enables substantial reductions in validation burden; conversely, if the mapping quality is low, the reduction in respondent burden vanishes.

After explicitly deriving sample size inflation:

Figure 1: Tradeoff between mapping quality, number of validation questions, and required sample size under realistic planning values for a structured-item universe.

the figure illustrates that, for a fixed margin of error and item universe, the required sample size can be substantially reduced if the mapping from open-ended conversation explains a large share of the structured-item variation, and/or if the number of validation items per respondent is increased. Critically, the framework enables ex ante quantification of the gains (or lack thereof) from introducing an AI mapping protocol, alerting the practitioner when further validation is needed.

Empirical Evaluation and Simulation

The statistical properties of AMV are demonstrated via simulation and two empirical studies:

Design Calibration Simulation

A simulation with $n=5,000$ respondents and 800 repeats contrasts mapping-only, validation-only, and AMV approaches across a range of validation probabilities. The results show that mapping-only estimation can be biased—sometimes severely so—if the mapping has systematic error; AMV and validation-only approaches converge as validation probability increases, but AMV exhibits lower RMSE than validation-only baseline when the mapped value is informative.

Figure 2: Root mean squared error comparison in simulation across varying validation probabilities for item means and regression-score estimation.

American Time Use Survey (ATUS) Emulation

Using a synthetic setting based on the ATUS, with a 32-variable structured diary recode and a set of 250 possible validation items, the authors present item mean and regression estimation under strong, moderate, and weak mapping-error scenarios. The moderate-scenario results demonstrate striking reductions in RMSE and elimination of bias for AMV over mapping-only and validation-tile-only baselines for both item means and regression coefficients. For example, mapping-only estimation for sleep and work minutes produces biases up to 18 minutes; AMV correction brings biases near zero, with standard error approximation consistently lower than validation-only estimates.

Figure 3: Mean RMSE (relative to reference) for seven structured time-use variables under varying validation burdens and mapping accuracies. AMV achieves consistently lower error when mapping is informative.

For regressions on sleep minutes and childcare participation, mapping-only estimation results in coefficient biases up to 5.7 minutes per hour of predictor change or 14 percentage points for children-in-household effect; AMV corrects these to within 0.8 minutes and 0.5 percentage points, respectively.

CHAMPS Verbal Autopsy Study

AMV is further illustrated on verbal autopsy data from the CHAMPS network. Here, narrative-only mapping exhibits large errors for a number of critical constructs (e.g., care-seeking, transport, treatment received). AMV provides tighter confidence intervals and brings the estimates back to the structured-item scales, even with only 9% of records validated per item or regression block.

Two regression analyses, one for traditional medicine use and one for treatment received during illness, show that AMV reduces the across-sample spread of coefficients by up to 10% compared to validation-only baselines for variables with narrative support. For constructs poorly reflected in the narrative, AMV prevents bias by relying more on the validated items.

Practical and Theoretical Implications

Practical Implications

• AMV enables conversational AI interview protocols with substantial respondent burden reduction only when the mapping from narrative to structure is highly accurate for the items of analytic interest.
• The design, estimation, and planning framework allows for principled evaluation prior to and during fielding of an AI-assisted survey, linking mapping performance directly to the number of structured questions that must still be asked.
• The approach generalizes to subgroups and regression models, with explicit support and calibration for subgroup-specific validation and regression blocks, critical for complex analytic use cases.
• The framework highlights that efficiency gains are not automatic and may dissipate when mapping is weak, when validation is insufficiently dense, or when analytic targets are not embedded in the validation design.

Theoretical Implications and Outlook

• AMV synthesizes double sampling, model-assisted survey inference, planned missingness, and prediction-powered inference, now adapted for respondent-by-variable matrix designs where AI mappings act as proxies, but the survey estimands are anchored to structured, validated responses.
• The methodology suggests a path forward for integrating ML-based measurement tools into official statistics and large-scale surveys, while maintaining inferential validity and quantifiable uncertainty, acknowledging measurement error and group differences.
• The paper also identifies significant next steps: extending the protocol to multi-modal data (images, speech, video), handling dialectal variation and ASR error, and integrating respondent-device interaction characteristics as part of the measurement error model.
• A critical caveat is that AMV, as a measurement-error correction framework, does not address potential nonresponse or mode effects induced by AI interviewing, trust or privacy perception changes, or accessibility barriers, which require auxiliary study.

Conclusion

Adaptive Matrix Validation provides a rigorous, flexible, and adaptable statistical architecture for AI-assisted interviewing, addressing the dual imperatives of burden reduction and inferential validity. It demonstrates, both in simulation and applied contexts, that substantial efficiency and accuracy gains are achievable—but strictly contingent on mapping quality and validation design. The framework operationalizes the margin at which conversational AI interviewing actually delivers on its promise and transparently quantifies the validation required to maintain statistical integrity. This approach is essential groundwork for the robust incorporation of AI-interviewing in public data collection and measurement research (2606.24244).