Language Models Trained on Media Diets Can Predict Public Opinion

Abstract: Public opinion reflects and shapes societal behavior, but the traditional survey-based tools to measure it are limited. We introduce a novel approach to probe media diet models -- LLMs adapted to online news, TV broadcast, or radio show content -- that can emulate the opinions of subpopulations that have consumed a set of media. To validate this method, we use as ground truth the opinions expressed in U.S. nationally representative surveys on COVID-19 and consumer confidence. Our studies indicate that this approach is (1) predictive of human judgements found in survey response distributions and robust to phrasing and channels of media exposure, (2) more accurate at modeling people who follow media more closely, and (3) aligned with literature on which types of opinions are affected by media consumption. Probing LLMs provides a powerful new method for investigating media effects, has practical applications in supplementing polls and forecasting public opinion, and suggests a need for further study of the surprising fidelity with which neural LLMs can predict human responses.

• The paper presents a novel method for predicting public opinion by fine-tuning BERT on media diets from various media sources.
• Results show significant improvement with a correlation of r=0.458 for enhanced models versus r=0.274 for the baseline, highlighting media impact.
• The study demonstrates that incorporating news attention boosts predictive accuracy, suggesting AI can effectively supplement traditional survey methods.

LLMs Trained on Media Diets Can Predict Public Opinion

Introduction

The paper presents a novel approach to predicting public opinion by leveraging LLMs trained on media diets. The methodology involves adapting LLMs, particularly BERT, to specific datasets representing various forms of media (e.g., online news, TV, radio) over defined time periods. The central hypothesis is that LLMs can emulate public opinion by capturing the influence of media exposure, akin to the responses found in actual survey distributions.

Figure 1: Overview of media diet modeling approach.

Media Diet Modeling Approach

The implementation involves three primary steps. Firstly, a base LLM like BERT is chosen for its capacity to predict masked words, providing a foundation for semantic understanding. Secondly, the model is fine-tuned on datasets detailing media diets, which include specific media content the target audience consumes. Finally, the fine-tuned model is queried with prompts, akin to survey questions, to predict public responses using the derived media diet scores.

The score calculation uses a normalization approach that compares media diet model predictions to baseline BERT scores. Additionally, a synonym-grouping method is employed to enhance accuracy by aggregating probabilities across synonyms, mitigating issues related to surface form competition in linguistic processing.

Results and Analysis

Attitudes Towards COVID-19

The study initially focuses on attitudes toward COVID-19, using datasets from various media sources to create corresponding media diet models. Correlations between model scores and actual survey responses indicate significant predictive power, with a correlation score of $r=0.458$ for BERT-enhanced models compared to a baseline of $r=0.274$ for unadapted BERT.

Figure 2: Attitudes towards COVID-19: correlations and regressions on media diet scores and survey response proportions.

The media diet models demonstrate robustness across different media formats and show sensitivity to paraphrasing, confirming their generalizability. Importantly, including "attention to news" as a feature in regression models significantly improves the prediction accuracy, indicating that the level of media engagement among respondents is a crucial factor.

Consumer Confidence

The paper extends the analysis to consumer confidence, revealing that media diet models are more predictive for sociocentric and prospective survey questions. For instance, sociocentric-retrospective questions yield stronger correlations ($r=0.376$), highlighting the nuanced effect of media on public opinion, particularly when it involves nationwide economic perspectives or future expectations.

Figure 3: Consumer confidence: correlations and regressions on media diet scores and survey response proportions.

Discussion

The proposed method introduces a robust framework for augmenting traditional survey methods with AI models, offering a scalable solution to frequent and cost-intensive public opinion polling. However, reliance on AI models should not replace human-based surveys but rather serve as a supplement to efficiently track and analyze public opinion trends over time.

The ability of media diet models to predict survey responses raises compelling questions about media’s role in shaping public opinion and the potential biases introduced by selective exposure and echo chambers. These models open up avenues for exploring messaging effects in various contexts, such as the differential impact of specific phraseology in media content.

Conclusion

This research underscores the value of integrating AI with traditional public opinion research, facilitating real-time insights into societal trends. The innovative use of LLMs to forecast public opinion presents opportunities and challenges, emphasizing the importance of further refining these models to ensure accuracy and alignment with human intent. As this field progresses, such tools could significantly enhance research in political science and social psychology, supporting evidence-based decision-making processes.