AI Conversational Interviewing: Scaling Up Semi-Structured and In-depth Interviews

Abstract: Public opinion research has long faced a trade-off between depth and scale: standardized surveys enable large-scale measurement but restrict respondents to researcher-defined categories, obscuring the diversity of unexpected considerations that underlie public sentiment. More conversational interviews provide richer insights through open-ended probing, but their reliance on trained human interviewers has kept them difficult to scale. This study introduces AI Conversational Interviewing as a method for collecting open-ended public opinion data at scale, pursuing three objectives: to demonstrate the analytical value of conversational text data for questions beyond the reach of closed-ended items; to assess the method's practical viability through participants' own evaluations; and to inform implementation by experimentally comparing voice-based, chat-based, and free-choice interview modes. We conducted a study combining an AI-led interview with a standardized survey on migration policy among 571 respondents recruited via Prolific and Payback Panel. The findings establish AI Conversational Interviewing as a viable and valuable addition to the social-science toolkit. The conversational transcripts surface considerations and reasoning that a comprehensive standardized battery does not capture such as markedly different mental models of migration among subgroups with similar attitudes levels. Among respondents who completed the interview, evaluations of the AI interview were at or above those of the standardized survey across modes, although completion itself varied by condition. By releasing open data and open-source pipeline materials, the study contributes to a growing literature on harnessing artificial intelligence to expand the methods of public opinion measurement.

• The paper introduces AI Conversational Interviewing, using LLMs as adaptive interviewers to blend qualitative depth with survey-scale data collection.
• It details a robust system architecture integrating chat and voice modalities, achieving high reliability and capturing distinct linguistic response patterns.
• Quantitative findings show enhanced respondent engagement and nuanced capture of latent mental models compared to traditional survey instruments.

AI Conversational Interviewing: Advancing Scalable, Open-Ended Public Opinion Measurement

Introduction

This study introduces and systematically analyzes "AI Conversational Interviewing" as a scalable methodology for semi-structured and in-depth public opinion research. The traditional dichotomy between structured survey instruments and qualitative, interviewer-led conversations has imposed a trade-off between analytical scale and depth. The paper challenges this dichotomy by operationalizing LLMs as adaptive interviewers capable of conducting open-ended interviews at web-scale. The system’s architecture is designed to collect nuanced, unstructured text data surpassing the scope of closed survey batteries by surfacing respondents’ mental models, latent considerations, and argument structures. This essay reviews the core conceptual, empirical, and methodological contributions, critically engages with quantitative results, and identifies implications and trajectories for computational social science.

Theoretical Foundations and Methodological Context

The paper situates AI Conversational Interviewing within the lineage of attitude measurement approaches (Figure 1), highlighting the qualitative inferential blind spots of standardized surveys, particularly their inability to capture the diversity of respondent reasoning, idiosyncratic associations, and domain-specific interpretive schemas.

Figure 1: Comparative strengths of standardized surveys, human-led conversational interviews, and AI-led conversational interviewing, emphasizing the dimensions of scalability, expressive depth, and analytic richness.

By using LLMs as fully-fledged, protocol-constrained interviewers, the study seeks to operationalize core methodological desiderata from the qualitative tradition (e.g., non-directiveness, active listening, adaptive probing) while maintaining the benefits of high-throughput, standardized data collection. The system enables mode assignment (voice, chat, or user choice) to systematically probe the interaction of medium and response behavior.

System Architecture and Experimental Design

The infrastructure integrates an LLM-based interviewing pipeline with the Qualtrics survey suite, supporting both chat and voice modalities. For the chat modality, OpenAI's GPT-4o is used via Chainlit; for voice, Vapi.ai orchestrates full-duplex interaction with OpenAI’s GPT-Realtime model. Respondents (N = 1,039 assigned; N = 571 complete and linked) were sampled from two digital access panels, and randomized to modality conditions. The system's workflow and participant flow are visualized in Figure 2.

Figure 2: High-level schematic of participant allocation, randomization, conversational intervention, and data linkage across survey and interview components.

The voice interface, supporting dynamic turn-taking and real-time feedback, was a particular focus of engineering (Figure 3).

Figure 3: States of the voice interface illustrating ID entry, pre-interview readiness, and real-time indication of speaker/listener transitions.

Quantitative Results: System Viability and Human Evaluation

Technical Implementation

The system achieved high reliability (over 80% of respondents reported no difficulty starting or ending the interviews). Most reported technical barriers centered on input procedures and device-specific interface quirks, particularly in the voice condition. Less than 1% reported inappropriate or harmful interviewer responses, with prompt engineering and real-time flagging mitigating LLM behavioral risk.

Respondent Evaluation

Respondents evaluated the AI interviewer as polite, motivating, clear, and impartial, with all attributes exceeding the scale midpoint. Compassionate behavior rated lower but still above center. AI-mediated conversations were generally characterized as relaxed and pleasant. When directly compared to standardized surveys, AI interviews scored higher on motivation and engagement, with participants indicating that the open format better captured individual perspectives.

A notable exception is that, despite favorable ratings, a majority in the chat condition expressed a preference to repeat standardized surveys in future data collections—primarily due to the higher cognitive burden imposed by text-based, open-ended input. In contrast, voluntary selection into the voice modality yielded a strong preference for the conversational format.

Mode Effects

Voice interaction elicited longer, faster, and lexically simpler responses (mean: 609 words per respondent, 52.5 wpm) than chat (mean: 300 words, 21.1 wpm). Voice responses tended to be shorter in sentence length and more paratactic, indicating alignment with naturalistic spoken registers. Text interaction, despite its cognitive tax, produced higher syntactic density and stylistic heterogeneity.

Analytical Advantages: Capturing Latent Mental Models

Traditional surveys, even with well-calibrated batteries, are constrained to anticipated dimensions and may exhibit framing effects that both obscure and shape respondent cognition. The open-ended, scalable conversational interviewing method surfaces previously ‘dark’ considerations—including spontaneous, subgroup-distinct frames (e.g., housing shortages versus worldview internalization versus hybrid or contradictory schemas among similarly-positioned respondents on Likert scales). The annotative pipeline—LLM-assisted, human-validated—supports fine-grained extraction of both issue repertoires and rhetorical frames, enabling computational sequence and topic analyses stratified by political identification.

Empirical Illustration: Party-Stratified Migration Attitudes

The study applies topic analysis and rhetorical frame classification to issue segments generated during conversational interviews. For example, while party means on a migration attitude scale stratify predictably (AfD most restrictive, Linke most permissive), the open-ended responses expose sharp differences in underlying mental models beyond scale location:

• Die Linke supporters prioritize labor issues, cohesion, and humanitarian frames.
• AfD supporters disproportionately invoke border security; humanitarian considerations are virtually absent.
• SPD and FDP supporters diverge significantly in topical emphasis despite adjacent scale scores.

Sequential analysis further demonstrates that the salience and ordering of considerations (e.g., immediate invocation of labor market, governing critique, humanitarian concerns) have substantial subgroup heterogeneity. Rhetorically, assertion types dominate (claim/eval >50%), but causal reasoning varies systematically: Die Linke supporters provide more causal chains, AfD supports evaluative assertion-heavy justification, while FDP opens more frequently with policy claims.

Practical and Theoretical Implications

Methodological Integration

AI Conversational Interviewing enables blending inductive mapping and structured measurement. Sequential deployments (interview to inform batteries), adaptive mixed-mode implementations, and hybrid item-level integration are all feasible, with each approach leveraging the expressive richness of conversational data and the comparability and coverage of structured survey modules.

Scaling and Reproducibility

Critical, but addressable, obstacles to operationalization in probability-based, national probability samples are identified: seamless data linkage, device-agnostic interface engineering, and respondent control over modality to mitigate attrition and burden.

Future Research Directions

Open questions include:

• Enumeration of best practices for prompt engineering, probe frequency, session length
• LLM interviewer model selection biases and interviewer effect estimation
• Cross-cultural validity and suitability of the approach in low-salience or sensitive domains
• Scaling of human validation for topic/argument annotation and advancement of computational grounded theory toolchains
• Technical and ethical guarantees for privacy-preserving, reproducible, and open science workflows

Conclusion

AI Conversational Interviewing demonstrates that LLMs can emulate the adaptive behavioral protocols of qualitative interviewers at survey scale. The method elicits and captures latent considerations and reasoning structures, enabling both richer analytical description and new forms of computational social scientific inference. While implementation bottlenecks persist—primarily technical and operational—these are not intrinsic barriers. The primary risk concerns overburden, especially in text modality; user control and voice-first designs are promising mitigations. In sum, this study provides strong evidence for the analytical and practical value of AI-led conversational interviewing as a complementary component in the public opinion research toolkit, and anticipates significant further innovation and experimentation at the intersection of NLP, computational social science, and survey methodology.