RumourEval 2019: Determining Rumour Veracity and Support for Rumours (1809.06683v1)

Abstract: This is the proposal for RumourEval-2019, which will run in early 2019 as part of that year's SemEval event. Since the first RumourEval shared task in 2017, interest in automated claim validation has greatly increased, as the dangers of "fake news" have become a mainstream concern. Yet automated support for rumour checking remains in its infancy. For this reason, it is important that a shared task in this area continues to provide a focus for effort, which is likely to increase. We therefore propose a continuation in which the veracity of further rumours is determined, and as previously, supportive of this goal, tweets discussing them are classified according to the stance they take regarding the rumour. Scope is extended compared with the first RumourEval, in that the dataset is substantially expanded to include Reddit as well as Twitter data, and additional languages are also included.

• The paper outlines the RumourEval 2019 shared task which expands automated rumour veracity and stance detection efforts to include diverse platforms like Reddit and languages such as Russian and Danish, building upon the 2017 task.
• RumourEval 2019 features two subtasks: SDQC classification for response stance (support, deny, query, comment) and veracity prediction (true, false, unverified with confidence).
• The research contributes to developing scalable automated misinformation detection systems by providing diverse datasets and benchmarks, crucial for cross-platform and multi-lingual applications.

RumourEval 2019: Advancements in Veracity and Stance Detection

The paper "RumourEval 2019: Determining Rumour Veracity and Support for Rumours" outlines the continuation of efforts to address automated rumour verification, focusing on evaluating the veracity and gathering supportive data regarding rumours. Building upon the initial RumourEval task in 2017, this iteration significantly expands the scope by incorporating diverse data sources such as Reddit and multiple languages, including Russian and Danish, alongside Twitter, which has been the primary focus.

Objectives and Task Structure

The RumourEval 2019 task is designed with two primary subtasks, both aimed at advancing the capability of NLP systems in the context of rumour checking:

1. Subtask A - SDQC Support Classification: This task involves analyzing the stance of tweets in response to a source claim. The task's objective is to classify these responses into categories: support, deny, query, or comment (SDQC). This classification captures the stance taken by individuals toward the rumour in question.
2. Subtask B - Veracity Prediction: Here, the task is to predict the truthfulness of the rumour based on the source material and any ensuing discussions. Participants are required to classify the rumour as true, false, or unverified, along with providing a confidence score.

The paper notes that the incorporation of data from platforms such as Reddit, as well as the inclusion of multiple languages, differentiate RumourEval 2019 from its predecessor. These additions aim to foster cross-platform and cross-lingual rumour detection research, which is crucial given the varied nature of misinformation across different online communities.

Methodological Insights and Results

RumourEval 2017 attracted a variety of computational approaches, including traditional machine learning methods like support vector machines and gradient boosting classifiers, as well as deep learning techniques such as LSTMs and CNNs. Accuracy rates achieved in RumourEval 2017 ranged markedly, emphasizing the challenging nature of rumour detection. Notably, the system achieving the highest accuracy integrated detailed analysis of discussion threads, highlighting the significance of conversational context in verification tasks.

For 2019, the paper suggests that stimulating innovation in the exploitation of information, especially that obtained from Subtask A, could yield better results in Subtask B. The combination of open and closed veracity prediction tasks into a single task in RumourEval 2019 is intended to promote varied methodological advancements and enrich the pool of publicly available benchmarks, facilitating rigorous comparisons of system performance.

Implications and Future Directions

RumourEval 2019's efforts have notable implications in both practical and theoretical domains. Practically, it addresses the urgent need for scalable, automated approaches to combat misinformation on social media, a task that is becoming increasingly complex as platforms grow in size and influence. The extension to new platforms and languages suggests practical applications in diverse social contexts, enabling the deployment of more robust multi-lingual and cross-platform verification systems.

Theoretically, RumourEval contributes to ongoing efforts to refine stance detection and rumour verification, essential components in automated fact-checking pipelines. The task's emphasis on richer, varied datasets will likely spur research that embraces the complexities of human communication as exhibited in social media discourse.

In the future, we can anticipate continued enhancements in automated rumour and stance detection methodologies. Technological advancements combined with increasing computational resources will likely enable more sophisticated models capable of delivering real-time, accurate verification of rumours across multiple contexts and platforms. Furthermore, collaborations between academia and industry, as encouraged by initiatives like RumourEval, will be critical to ensure these tools are adaptable and meet real-world demands.

RumourEval 2019 thus serves as a pivotal step towards advancing the field of rumour analysis, emphasizing the need for innovation and collaboration in addressing the challenges posed by misinformation.