2000 character limit reached
Crowdsourced Adaptive Surveys (2401.12986v2)
Published 16 Jan 2024 in cs.CL, cs.AI, cs.HC, and stat.AP
Abstract: Public opinion surveys are vital for informing democratic decision-making, but responding to rapidly evolving information environments and measuring beliefs within niche communities can be challenging for traditional survey methods. This paper introduces a crowdsourced adaptive survey methodology (CSAS) that unites advances in natural language processing and adaptive algorithms to generate question banks that evolve with user input. The CSAS method converts open-ended text provided by participants into survey items and applies a multi-armed bandit algorithm to determine which questions should be prioritized in the survey. The method's adaptive nature allows for the exploration of new survey questions, while imposing minimal costs in survey length. Applications in the domains of Latino information environments, national issue importance, and local politics showcase CSAS's ability to identify topics that might otherwise escape the notice of survey researchers. I conclude by highlighting CSAS's potential to bridge conceptual gaps between researchers and participants in survey research.
- Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
- Near-optimal regret bounds for thompson sampling. Journal of the ACM (JACM), 64(5):1–24.
- Guide to the 2008 cooperative congressional election survey. Harvard University, draft of February, 9.
- Open llm leaderboard.
- Carrasco-Farré, C. (2022). The fingerprints of misinformation: how deceptive content differs from reliable sources in terms of cognitive effort and appeal to emotions. Humanities and Social Sciences Communications, 9(1):1–18.
- Conspiratorial thinking in the latino community on the 2020 election. Research & Politics, 9(1):20531680221083535.
- Survey mode as a source of instability in responses across surveys. Field methods, 17(1):30–52.
- Internet, phone, mail, and mixed-mode surveys: The tailored design method. John Wiley & Sons.
- Agenda setting, public opinion, and the issue of immigration reform. Social Science Quarterly, 91(2):359–378.
- How to make causal inferences using texts. arXiv preprint arXiv:1802.02163.
- Elliott, K. J. (2020). Democracy’s pin factory: Issue specialization, the division of cognitive labor, and epistemic performance. American Journal of Political Science, 64(2):385–397.
- Confidence intervals for policy evaluation in adaptive experiments. Proceedings of the national academy of sciences, 118(15):e2014602118.
- Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674.
- Distinguishing between most important problems and issues? Public Opinion Quarterly, 75(3):545–555.
- Mixtral of experts.
- Key, V. O. (1961). Public opinion and the decay of democracy. The Virginia Quarterly Review, 37(4):481–494.
- Comparisons of party identification and policy preferences: The impact of survey question format. American Journal of Political Science, pages 941–964.
- Designing misinformation interventions for all: Perspectives from aapi, black, latino, and native american community leaders on misinformation educational efforts. Harvard Kennedy School Misinformation Review.
- Leech, B. L. (2002). Interview methods in political science. PS-WASHINGTON-, 35(4):663–664.
- Why so serious?: Survey trolls and misinformation. Available at SSRN: https://ssrn.com/abstract=3131087 or http://dx.doi.org/10.2139/ssrn.3131087.
- Capacity, diversity, and volatility of the public agenda: Trends from 1954 to 1994. Public Opinion Quarterly, 59(4):495–525.
- Does gpt-3 know what the most important issue is? using large language models to code open-text social survey responses at scale. Using Large Language Models to Code Open-Text Social Survey Responses At Scale (December 22, 2022).
- Mena, P. (2019). Principles and boundaries of fact-checking: Journalists’ perceptions. Journalism practice, 13(6):657–672.
- Computerized adaptive testing for public opinion surveys. Political Analysis, 21(2):172–192.
- So many questions, so little time: Integrating adaptive inventories into public opinion research. Journal of Survey Statistics and Methodology, 8(4):667–690.
- Adaptive Inventories: A Practical Guide for Applied Researchers. Cambridge University Press.
- Adaptive experimental design: Prospects and applications in political science. American Journal of Political Science, 65(4):826–844.
- Battling the coronavirusinfodemic’among social media users in africa. arXiv preprint arXiv:2212.13638.
- Polls and elections: Terror, war, and the economy in george w. bush’s approval ratings: The importance of salience in presidential approval. Presidential Studies Quarterly, 48(2):318–341.
- The rational public: Fifty years of trends in Americans’ policy preferences. University of Chicago Press.
- Using proprietary language models in academic research requires explicit justification. Nature Computational Science, pages 1–2.
- Perens, B. et al. (1999). The open source definition. Open sources: voices from the open source revolution, 1:171–188.
- Improving language understanding by generative pre-training.
- Word embeddings for the analysis of ideological placement in parliamentary corpora. Political Analysis, 28(1):112–133.
- Issue publics: How electoral constituencies hide in plain sight. Elements in Political Psychology.
- Wiki surveys: Open and quantifiable social data collection. PloS one, 10(5):e0123483.
- Twitter flagged donald trump’s tweets with election misinformation: They continued to spread both on and off the platform. Harvard Kennedy School Misinformation Review.
- Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
- Attention is all you need. Advances in neural information processing systems, 30.
- Confronting core issues: A critical test of attitude polarization.
- Velez, Y. R. (2023). Trade-offs in latino politics: Exploring the role of deeply-held issue positions using a dynamic tailored conjoint method. Aletheia.
- Latino-targeted misinformation and the power of factual corrections. The Journal of Politics, 85(2):789–794.
- The spread of true and false news online. science, 359(6380):1146–1151.
- Weller, S. C. (1998). Structured interviewing and questionnaire construction. Handbook of methods in cultural anthropology, pages 365–409.
- Wlezien, C. (2005). On the salience of political issues: The problem with ‘most important problem’. Electoral studies, 24(4):555–579.
- Judging llm-as-a-judge with mt-bench and chatbot arena.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.