Large Language Models (LLMs) as Agents for Augmented Democracy (2405.03452v3)

Abstract: We explore an augmented democracy system built on off-the-shelf LLMs fine-tuned to augment data on citizen's preferences elicited over policies extracted from the government programs of the two main candidates of Brazil's 2022 presidential election. We use a train-test cross-validation setup to estimate the accuracy with which the LLMs predict both: a subject's individual political choices and the aggregate preferences of the full sample of participants. At the individual level, we find that LLMs predict out of sample preferences more accurately than a "bundle rule", which would assume that citizens always vote for the proposals of the candidate aligned with their self-reported political orientation. At the population level, we show that a probabilistic sample augmented by an LLM provides a more accurate estimate of the aggregate preferences of a population than the non-augmented probabilistic sample alone. Together, these results indicates that policy preference data augmented using LLMs can capture nuances that transcend party lines and represents a promising avenue of research for data augmentation.

• The paper demonstrates that LLMs can model individual political preferences with a prediction accuracy of 69%-76%.
• The paper reveals that supplementing small random samples with LLM predictions improves aggregate preference estimation using a Borda-like method.
• The paper highlights ethical concerns and demographic biases, underscoring the need for diverse data in AI-augmented democratic systems.

Exploring the Role of LLMs in Augmented Democracy Systems

Introduction to Augmented Democracy

The concept of augmented democracy isn't new, but it's increasingly relevant in a world where digital democracy is taking center stage, shaping how decisions are made and how representatives interact with the public. Augmented democracy systems propose using software agents to enhance individual participation in political processes beyond just voting in elections. These systems could theoretically enable citizens to weigh in on multiple policy decisions directly, leveraging artificial intelligence to manage and reflect their views.

Core Insights from the Study

The paper discussed details an engaging experiment that applies various LLMs — specifically LLaMA-2 7B, Chat GPT 3.5. Turbo, Mistral 7B, and Falcon 7B — to the task of predicting individual and collective political preferences based on data from a collaborative government program experiment during a recent Brazilian presidential election.

Individual Preference Modeling

• The models achieved a prediction accuracy of 69%-76% for individual preferences. This implies that LLMs can robustly model the political preferences of individuals to a reasonable degree of accuracy.
• Interestingly, these models performed better when predicting preferences of specific demographic groups, particularly those who are liberal, college-educated, and younger. This highlights potential biases in how LLMs are trained and the populations they best represent.

Aggregate Preference Estimation

• Beyond individual predictions, the use of LLMs in estimating the collective or aggregate preferences was a significant aspect of the paper. The researchers used a Borda-like score to measure the popularity of policy proposals.
• When enhancing small random samples with model predictions, the accuracy of reflecting the broader community’s preferences improved markedly, especially in samples that represented less than 30%-40% of the total population.

Implications and Future Directions

The exploration of LLMs within the context of augmented democracy systems reveals both promising utilizations and notable challenges:

Practical Applications

• For practical purposes, these findings suggest that deploying LLMs could make it feasible to conduct more frequent, detailed, and inclusive consultations with the public on policy matters.

Ethical and Bias Considerations

• The better performance of LLMs with certain demographic profiles raises ethical questions about inclusivity and bias. It’s crucial for future models to be trained on more diverse datasets to mitigate these biases.

Future Developments in AI and Democracy

• The paper touches on the potential of multi-agent systems and more comprehensive modeling of preferences that could offer more nuanced insights into collective decision-making.
• Another conceivable advancement could involve more dynamically adaptive models that learn and evolve based on ongoing public input.

Final Thoughts

While the concept of AI-augmented democracy excites imaginations about enhanced participation and representation, the actual implementation of such systems must be approached with caution. Factors like the digital divide, the need for transparency in how these systems operate, and the potential for manipulation must all be carefully managed.

As AI continues to advance, its integration into our democratic processes seems increasingly inevitable. This paper provides a valuable foundation for understanding how such integration might work, but also underscores the significant work still required to ensure it benefits all sections of society. AI may not be the silver bullet for the challenges facing contemporary democracies, but it certainly opens up intriguing possibilities for the future of civic engagement.