Synthetic Opinion Polling

• Synthetic opinion polling is a computational approach that estimates public opinion by synthesizing survey responses using LLMs, digital trace mining, and network-based methods.
• Methods involve LLM-simulated respondent generation, direct distribution modeling, and social media content mining, calibrated using techniques like multilevel regression and post-stratification.
• Challenges include bias, prompt sensitivity, and underdispersion, while best practices emphasize ensemble approaches, rigorous post-calibration, and benchmarking against traditional surveys.

Synthetic opinion polling refers to a broad set of computational techniques for estimating public opinion, voting intention, policy preferences, and related attitudes by synthesizing responses through automated means rather than querying individuals directly. The field has undergone rapid expansion due to advances in LLMs, network-based estimation schemes, and statistical calibration methods. Synthetic opinion polling encompasses approaches ranging from LLM-simulated survey respondents to opinion mining from digital traces and network-augmented collection in social graphs. These methods aim to achieve scalability, timeliness, and, contingent on careful calibration, accuracy rivaling or complementing traditional survey paradigms.

1. Core Methodological Paradigms

Synthetic opinion polling methods can be classified along three primary paradigms:

A. LLM-Simulated Respondents and Direct Distribution Modeling

LLMs are prompted to simulate individual survey responses by adopting specified demographic, ideological, or personality “personas.” Classic approaches generate thousands of synthetic “panelists” by prompting the LLM for each demographic cell, typically with a survey-question-specific template and optionally with free-form justifications for each answer (Sanders et al., 2023, Jiang et al., 2024, Heyde et al., 2024, González-Bustamante et al., 11 Sep 2025). Recent innovations replace the many-individual-queries protocol with Direct Distribution (DD) prompting: the LLM is directly asked to provide a full probability distribution over discrete response bins for a particular demographic cell, reducing both simulation cost and output variance (Gong et al., 6 Mar 2026).

B. Social Media–Based Inference and Content Mining

A related stream leverages large volumes of text data from social media, transcribed video interviews, or user digital feeds to produce aggregate opinion estimates. Texts are processed via opinion mining (e.g., mass LLM annotation of YouTube street-interview transcripts (Elmas et al., 2023)), sentiment aggregation over preclassified tweets (Saleiro et al., 2016), or multimodal LLM annotation of social media user “moulds” (Cerina, 7 Mar 2025). Outputs are then post-stratified or subjected to regression-based adjustment.

C. Statistical Calibration and Partial Pooling

Unrepresentative or “synthetic” samples require statistical adjustment for demographic and sampling biases. Methods include multilevel regression and post-stratification (MRP) (Rothschild et al., 2014, Cerina et al., 2023, Cerina, 7 Mar 2025), synthetic weighting for small-area estimation (Kuriwaki et al., 2021), or, in network settings, partial pooling via network-informed estimators (Nettasinghe et al., 2018). In some LLM settings, role creation using external knowledge bases via retrieval-augmented generation further mitigates representation gaps (Karanjai et al., 31 Mar 2025).

2. Key Algorithmic and Statistical Components

Prompt Engineering and Persona Synthesis

• Template Construction: Prompts encode demographic, ideological, or psychological features; for example, “Write a letter to the editor from the perspective of a [age] [gender] [race] American on the issue: [policy]…” (Sanders et al., 2023).
• Role Creation: Retrieval-augmented pipelines dynamically inject personality and socio-demographic context, as in the HEXACO-based “role creation” framework (Karanjai et al., 31 Mar 2025).
• Distributional Queries: DD prompting elicits an opinion distribution in a single call, accelerating estimation and often improving accuracy (Gong et al., 6 Mar 2026).

Quantitative Evaluation and Distributional Metrics

Synthetic polling performance is benchmarked by direct comparison to high-quality benchmark surveys (e.g. CES, GLES, CEP):

Metric	Formula	Interpretation
Mean Difference (MD)	$|\mu(p) - \mu(\hat{p})|$	Gap between means of empirical vs. synthetics
Stdev. Difference (SDD)	$\sigma(p) - \sigma(\hat{p})$	Difference in opinion spread
NEMD (Earth-Mover)	Normalized 1D Wasserstein distance	Overall distributional discrepancy
Correlation (Pearson/Spearman)	$\rho$	Alignment of cell-level means
KL/JS Divergence	$D_{KL}, D_{JS}$	Information-theoretic divergence
Classification/F1-Score	precision, recall, F1 over binary/multiclass tasks	Item-level synthetic-human agreement

LLM-based polling typically achieves high accuracy on means (Pearson’s $r > 0.9$ on many items), but underestimates variance at the demographic-cell level (Jiang et al., 2024, Gong et al., 6 Mar 2026, Ball et al., 22 Feb 2025). Trust and macro-attitude items yield higher agreement (F1 > 0.90 on Chilean trust items (González-Bustamante et al., 11 Sep 2025)) than behavioral or nuanced ideological choices.

3. Applications: Political, Social, and Behavioral Domains

Synthetic opinion polling has been deployed in a variety of substantive settings:

• Political/Election Forecasting: LLMs simulate vote choice or aggregate issue preference; synthetic estimates are post-stratified using demographic margins and sometimes combined with historical calibrators (Jiang et al., 2024, Cerina et al., 2023, Cerina, 7 Mar 2025). High state-level accuracy (up to 98%, RMSE ≈ 7.3 pp) has been demonstrated, but subgroup calibration (e.g., Hispanic, young voters) can be challenging (Cerina, 7 Mar 2025).
• Public Policy & Infrastructure: Agent-based LLM polling frameworks are used to estimate community sentiment towards proposed infrastructure (e.g., data centers), integrating ACS-based demographic context with project-specific attributes to generate synthetic regional samples (Wu et al., 27 Nov 2025).
• Media and Microblog Analysis: Aggregation of sentiment-labeled microblog posts (Twitter) can predict month-to-month shifts in party support during economic crises, with MAE for monthly polls reduced to ≈0.6% when predicting first differences (Saleiro et al., 2016).
• Computational Sociolinguistics: Prompt-based synthetic text generation enables the study of linguistic subgroups and sentiment-carrying sub-topics even in data-sparse domains (Feldman et al., 2022).

4. Biases, Failure Modes, and Robustness

Synthetic opinion polling faces several systematic challenges:

• Demographic Pattern Homogenization: Off-the-shelf LLMs often fail to capture nuanced subgroup variance; distributions are “over-smooth,” and underdispersion leads to underestimation of population heterogeneity (Ball et al., 22 Feb 2025, Gong et al., 6 Mar 2026, Jiang et al., 2024).
• Prompt Sensitivity and Temporal Generalization: Response distributions can be unstable under trivial paraphrasing (Ball et al., 22 Feb 2025), and knowledge cutoffs or lack of event-specific training (e.g., post-2021 issues) can yield substantial misestimation (Sanders et al., 2023, Heyde et al., 2024).
• Coverage and Language Bias: LLMs perform less well in poorly represented languages (e.g., Turkish ASR, German political context), multi-party systems, or on complex minority stratifications (Elmas et al., 2023, Heyde et al., 2024).
• Input/Annotation Noise: ASR transcript quality, absence of speaker diarization, and manual preprocessing can drastically reduce recall (Elmas et al., 2023).
• Sampling and Post-Stratification Deficiencies: Without explicit probabilistic sampling, synthetic samples require aggressive post-hoc reweighting (e.g., raking, iterative proportional fitting) to correct for over/under-representation of key groups (Wu et al., 27 Nov 2025, González-Bustamante et al., 11 Sep 2025, Kuriwaki et al., 2021).

Robustness strategies include multi-prompt ensembles, post-stratification, prompt and temperature tuning, and benchmarking multiple LLMs on the same synthetic sample (Karanjai et al., 31 Mar 2025, Wu et al., 27 Nov 2025, Ball et al., 22 Feb 2025).

5. Integration with Statistical Estimation, Weighting, and Calibration

Statistical methodologies play a central role in ensuring inferential validity:

• Multilevel Regression and Post-Stratification (MRP): Adjustment for demographic bias across stratified cells; both classical (Rothschild et al., 2014) and LLM-extracted/annotated synthetic datasets (Cerina et al., 2023, Cerina, 7 Mar 2025).
• Synthetic Area Weighting: Two-step weighting (across-area and post-stratification) enables individual-level partial pooling in small-area estimation, especially when some covariates are only available in the survey (Kuriwaki et al., 2021).
• Bias Correction for Dependent Selection: Offset terms in regression (following King & Zeng) enable correction for dependent-sample selection, as in social media or online selection samples (Cerina et al., 2023).
• Conformal Prediction: Calibration intervals for synthetic estimates, though not always implemented, are suggested for quantifying prediction uncertainty (Wu et al., 27 Nov 2025).

These strategies allow synthetic polling pipelines to address both observable and structured unobservable selection biases, yielding state- and subgroup-level estimates with competitive accuracy.

6. Limitations, Best Practices, and Future Directions

Limitations include inability to fully recover fine-grained or minority group opinions, risks of stereotype amplification, lack of real sampling variance in purely synthetic respondent pools, and non-repeatability due to proprietary model updates (González-Bustamante et al., 11 Sep 2025, Cerina, 7 Mar 2025, Ball et al., 22 Feb 2025). Best practices dictate:

• Prompt Iteration and Ensemble Approaches: Vary template, order, and presentation to assess prompt sensitivity (Ball et al., 22 Feb 2025, Heyde et al., 2024).
• Post-Collection Calibration: Apply raking, iterative proportional fitting, and demographic reweighting to align synthetic sample with true marginals (Wu et al., 27 Nov 2025, Kuriwaki et al., 2021).
• Benchmarking and Cross-Validation: Always benchmark against high-quality gold-standard surveys, report multiple error metrics (NEMD, KL/JS, correlation, F1), and release code/prompts for reproducibility (Jiang et al., 2024, Sanders et al., 2023, Gong et al., 6 Mar 2026).
• Transparent Reporting and Subgroup Diagnostics: Examine subgroup and cross-tab performance, and inspect error directionality (e.g., consistent overestimation of progressive attitudes in U.S. samples) (Jiang et al., 2024, Heyde et al., 2024).
• Hybrid Deployment: Combine small, focused human surveys with large-scale synthetic estimates for agile, cost-efficient opinion tracking and to inform model recalibration (Wu et al., 27 Nov 2025).

Future research directions include multilingual and cultural extension, dynamic updating of role-knowledge bases, entropy matching for variance regularization, and deployment within ethical and governance frameworks for sensitive decision environments (Karanjai et al., 31 Mar 2025, Gong et al., 6 Mar 2026).

---

References

• "Opinion Mining from YouTube Captions Using ChatGPT: A Case Study of Street Interviews Polling the 2023 Turkish Elections" (Elmas et al., 2023)
• "Demonstrations of the Potential of AI-based Political Issue Polling" (Sanders et al., 2023)
• "Polling Latent Opinions: A Method for Computational Sociolinguistics Using Transformer LLMs" (Feldman et al., 2022)
• "Synthesizing Public Opinions with LLMs: Role Creation, Impacts, and the Future to eDemocracy" (Karanjai et al., 31 Mar 2025)
• "Donald Trumps in the Virtual Polls: Simulating and Predicting Public Opinions in Surveys Using LLMs" (Jiang et al., 2024)
• "Vox Populi, Vox AI? Using LLMs to Estimate German Public Opinion" (Heyde et al., 2024)
• "Synthetic Area Weighting for Measuring Public Opinion in Small Areas" (Kuriwaki et al., 2021)
• "Emulating Public Opinion: A Proof-of-Concept of AI-Generated Synthetic Survey Responses for the Chilean Case" (González-Bustamante et al., 11 Sep 2025)
• "Artificially Intelligent Opinion Polling" (Cerina et al., 2023)
• "Human Preferences in LLM Latent Space: A Technical Analysis on the Reliability of Synthetic Data in Voting Outcome Prediction" (Ball et al., 22 Feb 2025)
• "What Do Your Friends Think? Efficient Polling Methods for Networks Using Friendship Paradox" (Nettasinghe et al., 2018)
• "The Mythical Swing Voter" (Rothschild et al., 2014)
• "Characterizing the ability of LLMs to recapitulate Americans' distributional responses to public opinion polling questions across political issues" (Gong et al., 6 Mar 2026)
• "What AI Speaks for Your Community: Polling AI Agents for Public Opinion on Data Center Projects" (Wu et al., 27 Nov 2025)
• "Sentiment Aggregate Functions for Political Opinion Polling using Microblog Streams" (Saleiro et al., 2016)
• "PoSSUM: A Protocol for Surveying Social-media Users with Multimodal LLMs" (Cerina, 7 Mar 2025)