OATH-Frames: Characterizing Online Attitudes Towards Homelessness with LLM Assistants (2406.14883v2)

Abstract: Warning: Contents of this paper may be upsetting. Public attitudes towards key societal issues, expressed on online media, are of immense value in policy and reform efforts, yet challenging to understand at scale. We study one such social issue: homelessness in the U.S., by leveraging the remarkable capabilities of LLMs to assist social work experts in analyzing millions of posts from Twitter. We introduce a framing typology: Online Attitudes Towards Homelessness (OATH) Frames: nine hierarchical frames capturing critiques, responses and perceptions. We release annotations with varying degrees of assistance from LLMs, with immense benefits in scaling: 6.5x speedup in annotation time while only incurring a 3 point F1 reduction in performance with respect to the domain experts. Our experiments demonstrate the value of modeling OATH-Frames over existing sentiment and toxicity classifiers. Our large-scale analysis with predicted OATH-Frames on 2.4M posts on homelessness reveal key trends in attitudes across states, time periods and vulnerable populations, enabling new insights on the issue. Our work provides a general framework to understand nuanced public attitudes at scale, on issues beyond homelessness.

• The paper presents OATH-Frames, a typology that leverages LLM-assisted expert annotations to reduce annotation time by 6.5× with minimal accuracy loss.
• It demonstrates the framework’s superiority over traditional classifiers by effectively capturing nuanced online sentiments and harmful rhetoric.
• The large-scale analysis reveals temporal and regional dynamics in U.S. public attitudes towards homelessness, informing policy and intervention strategies.

Characterizing Online Attitudes Towards Homelessness Using OATH-Frames

This paper introduces OATH-Frames, a novel framing typology designed to characterize online attitudes towards homelessness in the United States. The researchers leverage the capabilities of LLMs to assist social work experts in annotating large-scale online discourse from Twitter. The paper presents a comprehensive approach consisting of frame discovery, data annotation, and large-scale analysis.

Framework and Methodology

The framework for studying online attitudes consists of three primary components: frame discovery, data annotation, and analysis of annotated data.

1. Frame Discovery: The researchers created a typology called Online Attitudes Towards Homelessness (OATH-Frames), utilizing grounded theory and framing theory from social science. This typology includes three broad themes and nine fine-grained Issue-specific frames, capturing various critiques, responses, and perceptions related to homelessness.
2. Data Annotation: The team's annotation strategy comprised expert-only annotations, LLM-assisted expert annotations, and multilabel classification model predictions. By incorporating expert insights and chain-of-thought (CoT) explanations, the collaboration between experts and GPT-4 resulted in a six-fold increase in annotation speed with minimal reduction in accuracy.
3. Analysis: The paper examined 2.4 million posts annotated with OATH-Frames to uncover trends and attitudes across different states, time periods, and vulnerable populations. This large-scale analysis revealed the dynamic public sentiment towards homelessness and the efficacy of OATH-Frames in portraying nuanced societal attitudes.

Major Findings

1. Annotation Efficacy: The collaborative framework combining human expertise with GPT-4 reduced annotation time by approximately 6.5 times while only reducing accuracy by a 3-point F$_1$ score, demonstrating the viability of using LLMs to scale socially sensitive annotation tasks.
2. Comparison with Existing Models: OATH-Frames outperformed traditional sentiment and toxicity classifiers in capturing harmful rhetoric and nuanced public opinions. For instance, posts labeled with harmful generalizations about PEH were often missed by popular classifiers but captured effectively by OATH-Frames, thus revealing the limitations of existing sentiment analysis tools.
3. State and Temporal Dynamics: The analysis revealed the influence of socio-political factors on public attitudes towards homelessness. High cost of living and larger unsheltered PEH populations significantly shaped public discourse, with states like California seeing predominantly critical attitudes. Temporal analysis indicated periodic spikes in certain attitudes correlating with political events and policy changes.
4. Attitudes Towards Vulnerable Populations: When comparing PEH with other marginalized groups such as immigrants and Ukrainian refugees, the findings showed significant differences in public sentiment, ranging from nationalistic preferences to calls for government accountability and resource allocation critiques.

Theoretical and Practical Implications

Theoretically, this paper contributes to the understanding of public discourse on sensitive social issues by providing a scalable framework that incorporates insights from social science into LLM-assisted annotation processes. The introduction of OATH-Frames offers a refined tool for capturing complex societal attitudes, going beyond the capabilities of existing sentiment and toxicity measures.

Practically, the findings have significant implications for policymakers, social work practitioners, and advocacy groups. By understanding the nuances in public attitudes towards homelessness, stakeholders can design more effective communication strategies and policy interventions to address the issue. The scalability of the annotation process also provides a template for applying similar methodologies to other socially sensitive topics, enabling the analysis of large-scale online discourse in various domains.

Future Directions

Future research could expand the OATH-Frames framework to include additional contextual factors such as the geographic and demographic information of social media users or the specific targets of discourse (e.g., government policies vs. individual behaviors). Moreover, integrating more sophisticated LLMs and fine-tuning them with expert-defined guidelines may further enhance annotation quality and efficiency. Exploring these avenues could provide deeper insights into the dynamics of public opinion and contribute to more informed and effective policy-making in social welfare contexts.

In summary, OATH-Frames represents a significant advancement in the field of social work and computational linguistics by offering a robust, scalable framework to understand public attitudes towards homelessness. This approach not only highlights the current state of public sentiment but also paves the way for future studies to explore and address complex societal issues through advanced computational methods.