TwiBot-22: Towards Graph-Based Twitter Bot Detection (2206.04564v6)

Abstract: Twitter bot detection has become an increasingly important task to combat misinformation, facilitate social media moderation, and preserve the integrity of the online discourse. State-of-the-art bot detection methods generally leverage the graph structure of the Twitter network, and they exhibit promising performance when confronting novel Twitter bots that traditional methods fail to detect. However, very few of the existing Twitter bot detection datasets are graph-based, and even these few graph-based datasets suffer from limited dataset scale, incomplete graph structure, as well as low annotation quality. In fact, the lack of a large-scale graph-based Twitter bot detection benchmark that addresses these issues has seriously hindered the development and evaluation of novel graph-based bot detection approaches. In this paper, we propose TwiBot-22, a comprehensive graph-based Twitter bot detection benchmark that presents the largest dataset to date, provides diversified entities and relations on the Twitter network, and has considerably better annotation quality than existing datasets. In addition, we re-implement 35 representative Twitter bot detection baselines and evaluate them on 9 datasets, including TwiBot-22, to promote a fair comparison of model performance and a holistic understanding of research progress. To facilitate further research, we consolidate all implemented codes and datasets into the TwiBot-22 evaluation framework, where researchers could consistently evaluate new models and datasets. The TwiBot-22 Twitter bot detection benchmark and evaluation framework are publicly available at https://twibot22.github.io/

TwiBot-22: A Comprehensive Graph-Based Benchmark for Twitter Bot Detection

The paper "TwiBot-22: Towards Graph-Based Twitter Bot Detection" addresses the critical task of detecting Twitter bots by leveraging the graph structure of the Twitter network. As the prevalence of malicious bots on social platforms poses severe challenges such as misinformation dissemination and social manipulation, efficient detection methodologies are necessary. This paper presents TwiBot-22, a graph-based benchmark designed to support the development and evaluation of such detection methods.

Contributions and Data Design

TwiBot-22 constitutes a significant advancement over existing datasets primarily by its scale, heterogeneity, and annotation accuracy. Comprising one million users in the dataset, TwiBot-22 is roughly five times larger than TwiBot-20, the preceding largest dataset. This size is critical for evaluating models at the Twitter network scale and for training models capable of distinguishing subtle bot behaviors across diverse contexts.

Additionally, TwiBot-22 captures the complex heterogeneity present in social networks by including four types of entities (users, tweets, lists, and hashtags) and 14 types of relations such as follow, retweet, and mention, forming a rich heterogeneous graph. Such granularity allows researchers to explore advanced graph-based models that can uniquely identify bots based on nuanced interactions that simpler datasets might miss.

To ensure annotation quality in such a large dataset, the authors employ weak supervision strategies guided by 1,000 expert-verified annotations. This approach enhances the label accuracy compared to crowdsourced data, which often introduces noise and bias.

Evaluation and Results

The authors benchmark a broad array of 35 Twitter bot detection models on TwiBot-22, ranging from feature-based, text-based, to graph-based approaches. The empirical results highlight the superior performance of graph-based models, which leverage the network structure to capture relational patterns among users that signal bot activity. Models like R-GCN and RGT, which utilize relational graph convolutional and transformer architectures, respectively, demonstrate particular effectiveness.

Moreover, when compared to TwiBot-20 and other available datasets, TwiBot-22 consistently challenges models to perform better, as indicated by a general decrease in average performance. This highlights the complexity and varied nature of Twitter bot detection as bots become increasingly sophisticated in evading detection.

Implications and Future Research

The creation of TwiBot-22 suggests several avenues for future research. Firstly, the integration of multi-modal data—such as images and videos from tweets—could advance model capabilities to detect bots that mimic human activity using diverse modalities. Furthermore, as grounded in the results, ensuring that detection methods generalize well to unseen data remains a critical area. This necessitates the exploration of architectures that can adapt to the dynamic, evolving nature of social network interactions.

Practically, TwiBot-22 facilitates a standardized evaluation protocol for Twitter bot detection models, enabling better comparison and understanding of state-of-the-art methods. Its availability through an open framework further supports collaborative advancements and replication studies.

Conclusion

TwiBot-22 serves as a comprehensive benchmark for modern Twitter bot detection practices, offering an unprecedented scale and heterogeneity essential for deploying robust models capable of operating effectively on real-world data. Through this work, the authors not only set a new standard for dataset quality and evaluation but also pave the way for addressing the ongoing and escalating challenges posed by malicious online entities.