Decoding Concerns: Multi-label Classification of Vaccine Sentiments in Social Media (2312.10626v1)

Abstract: In the realm of public health, vaccination stands as the cornerstone for mitigating disease risks and controlling their proliferation. The recent COVID-19 pandemic has highlighted how vaccines play a crucial role in keeping us safe. However the situation involves a mix of perspectives, with skepticism towards vaccines prevailing for various reasons such as political dynamics, apprehensions about side effects, and more. The paper addresses the challenge of comprehensively understanding and categorizing these diverse concerns expressed in the context of vaccination. Our focus is on developing a robust multi-label classifier capable of assigning specific concern labels to tweets based on the articulated apprehensions towards vaccines. To achieve this, we delve into the application of a diverse set of advanced natural language processing techniques and machine learning algorithms including transformer models like BERT, state of the art GPT 3.5, Classifier Chains & traditional methods like SVM, Random Forest, Naive Bayes. We see that the cutting-edge LLM outperforms all other methods in this context.

• The paper introduces a novel multi-label classifier that leverages transformer models and GPT-3.5 to decode diverse vaccine sentiments on social media.
• It demonstrates that advanced prompting strategies with LLMs achieved a macro F1 score of 0.55, outperforming traditional and transformer-based methods.
• The findings underscore the role of rigorous data preprocessing and real-time social media monitoring in combating vaccine misinformation and informing public health strategies.

Multi-label Classification of Vaccine Sentiments in Social Media Using Advanced NLP Techniques

Introduction

The proliferation of vaccine-related misinformation on social media platforms, particularly reinforced by the COVID-19 pandemic, poses significant public health challenges. Misinformation can fuel vaccine hesitancy, undermining efforts to combat infectious diseases. This paper explores the development of a multi-label classifier designed to accurately categorize anti-vaccine sentiments articulated in tweets. The classifier leverages advanced NLP and machine learning techniques, including transformer models and LLMs like GPT 3.5, demonstrating superior performance in identifying nuanced vaccine concerns.

The Classification Task and Model Selection

The task involved the classification of tweets into one or more of twelve categories reflecting common anti-vaccine concerns. These categories include skepticism about vaccine necessity, opposition to mandatory vaccination policies, distrust in pharmaceutical companies, and fears over potential side effects, among others. Methodologically, the paper compared the efficacy of various models, including traditional methods (SVM, Random Forest, Naive Bayes with TF-IDF vectorization), transformer-based models (DistilBERT, BERT), and LLMs. Notably, the LLM approach, with customized prompting strategies for multi-label classification, outperformed other models.

Methodology

Data Pre-processing

To optimize model performance, extensive data preprocessing efforts were undertaken. These efforts included removing stopwords, lower casing, handling emojis, expanding contractions, and applying tokenization and lemmatization. The preprocessing steps ensured that the input data to the models was of high quality, reducing noise and enhancing the models' ability to understand the context of the tweets.

Model Training and Evaluation

Various models were trained and evaluated on a dataset comprising anti-vaccine tweets posted during 2020-21. The LLM approach, specifically using GPT 3.5, was highlighted for its effectiveness, showcasing the potential of LLMs in complex classification tasks. The paper details the use of a novel prompt template for LLMs, emphasizing logical reasoning in label assignment. This approach not only improved classification accuracy but also provided insights into the model's decision-making process.

Results and Discussion

The paper reports that the LLM approach achieved a macro F1 score of 0.55, signaling a modest but notable improvement over transformer models such as DistilBERT. The analysis also revealed the challenges in classifying tweets associated with less common concerns, such as conspiracy theories and religious objections to vaccines. These findings underscore the complexity of the sentiment classification task and the nuanced understanding required to effectively categorize diverse anti-vaccine sentiments.

Limitations and Future Directions

The paper acknowledges several limitations, including the restricted size of the dataset used for training the LLM due to resource constraints. Additionally, it points out the need for further exploration of the model’s behavior, particularly regarding its tendency to generate plausible but false content (hallucinations). Future research may focus on expanding the dataset, exploring a wider range of models and configurations, and investigating the incorporation of sentiment analysis into the classification process.

Conclusion

This paper contributes to the evolving field of NLP and machine learning by addressing the challenging problem of multi-label classification of vaccine-related sentiments on social media. Through the development and evaluation of a robust classifier, the research offers valuable insights for public health communication strategies, emphasizing the importance of real-time social media monitoring to inform evidence-based interventions. As the landscape of digital communication and public health continues to evolve, the methodologies and findings presented here provide a foundation for future research aimed at combating misinformation and fostering informed public discourse on critical health issues.