Robust Evaluation Metrics for Competing Narrative Extraction

Develop robust metrics to assess the quality and coherence of extracted narratives and the accuracy of framing analysis in computational approaches to competing political narratives on social media.

Background

The paper presents an unsupervised pipeline that identifies topics, detects events, links events into stories, and separates competing narratives according to user communities. Because existing narrative-extraction datasets generally lack annotations for competing narratives, supervised training and systematic evaluation are difficult.

The authors state that their own evaluation is primarily qualitative and identify the development of robust metrics as an unresolved challenge. Such metrics would need to assess both the structural quality and coherence of automatically extracted narratives and the accuracy of the framing analysis used to characterize divergent political perspectives.