- The paper presents a novel task and dataset that operationalizes narrative similarity through both binary classification and embedding-based approaches.
- The study employs a rigorously annotated set of over 1,000 story triples with dynamic sampling and contrastive human judgments, achieving up to 78% accuracy in Track A.
- Findings reveal strong genre clustering in narrative embeddings, highlighting the challenges for fine-grained abstraction and setting directions for future research.
SemEval-2026 Task 4: Narrative Story Similarity and Narrative Representation Learning
Task Overview and Motivation
SemEval-2026 Task 4 addresses computational modeling of narrative similarity, defined as the subjective perception of story relatedness driven by abstract causality and progression rather than surface-level details such as names or settings (2604.21782). The task is operationalized as a binary classification: given a story triple (anchor, candidate A, candidate B), systems must determine which candidate is more narratively similar to the anchor. Two tracks are established:
- Track A (Comparative Narrative Similarity): Direct classification for similarity in triple context.
- Track B (Narrative Representation Learning): Production and evaluation of narrative embeddings, validated via cosine distance alignment with partial human similarity orderings.
Central to the task is a novel dataset comprised of over 1,000 annotated story summary triples, sourced from Wikipedia, processed via LLM-based filtering, and subjected to rigorous human annotation. The annotation guidelines explicitly delineate three narrative aspects—Course of Action, Outcomes, and Abstract Theme—without imposing weighting, thereby reflecting both narrative theory and intuitive judgment.
Data Collection, Annotation, and Quality
Story summaries are filtered to four to eight sentences for computational tractability and annotation resource efficiency. Candidate triples are sampled using the narrative embedding model story-emb to prioritize instances with meaningful abstract similarity, rejecting trivial surface correlations. Rejection sampling via LLMs ensures annotation focuses on challenging cases where models disagree.
Human annotation is performed by twelve experienced native or near-native English speakers, with each triple labeled by at least two annotators and ambiguous cases adjudicated with a third judgement. The annotation process incorporates contrastive setup, not scalar ratings, to circumvent scale-based inconsistency and to increase discriminative value. The dataset achieves Krippendorff's Alpha of 0.33, consistent with highly subjective similarity tasks. Under pooled annotator assumptions, theoretical accuracy relative to oracle labels is estimated at 89%, though this is acknowledged as an optimistic upper bound.
Synthetic data generation augments training resources, utilizing commercial LLMs prompted to produce similar and dissimilar narratives from seeded topics. All released data includes metadata and canary strings to monitor potential label contamination.
System Approaches and Quantitative Results
A total of 71 submissions from 46 teams were evaluated across both tracks, revealing diverse methodological variants:
- Ensembles and Dynamic Routing: Top systems (COGNAC, FactUEP, AI-Monitors) utilized majority voting across generations, aspect decomposition, and dynamic routing based on difficulty signals. This achieved a maximum accuracy of 78.00% in Track A, aligning closely with estimated human annotator accuracy.
- LLM Majority and Prompting: Multiple teams demonstrated efficacy with chain-of-thought prompting, iterative optimization, and structured in-context learning, though detailed ablations suggest mixed effects of extended reasoning.
- Symbolic Representations: While some teams (IITBoys, CascadeMind, CophiWue) employed symbolic narrative modeling (e.g., narrative graphs, event taxonomies), symbolic-only approaches yielded lower performance, consistent with findings about the inadequacy of structural functions for abstract summary matching.
- Embedding-based Representation: In Track B, pretrained embedding models (gemini-embedding-001, Qwen3-Embedding-8B) complemented by post-processing or fine-tuning attained up to 72.00% accuracy, with complex narrative core distillation strategies and aspect extraction prevalent among competitive submissions.
Performance drops approximately 10 points from Track A to Track B, paralleling trends observed in retrieval tasks where cross-encoder architectures outperform bi-encoders. Cross-team majority vote ensembling marginally outperforms best single systems in both tracks.
Strong numerical results include:
- Track A best system: 78.00% accuracy, matching pooled human annotator accuracy estimates.
- Track B best system: 72.00% accuracy, with competitive embeddings and explicit aspect extraction.
One bold finding is a strong Pearson correlation (ρ=0.67) between narrative similarity accuracy and genre clustering in embedding space, indicating that current narrative representations are largely genre-oriented. However, there is minimal correlation with chronological features, demonstrating embedding insensitivity to story release year.
Theoretical and Practical Implications
This task establishes an authoritative benchmark for narrative similarity in computational literary studies, the first large-scale public dataset evaluating abstract narrative similarity in fiction summaries based on human contrastive judgments. The approach improves over prior datasets relying on external similarity lists or scalar ratings by offering discriminative, interpretable, and theory-compatible annotations.
Representational learning for narrative similarity is theoretically significant for tasks in literary studies, story retrieval, and automated narrative analysis. The strong alignment between embedding-based similarity and genre underscores the potential for genre-driven retrieval and automated genre classification. However, this also suggests limitations in current models’ capacity for finer-grained narrative abstraction, which may require future work focusing on disentangled representations or more robust symbolic integration.
Practically, the task’s design—anchored in Wikipedia summary abstraction—facilitates computational scalability and generalizability. The enduring subjectivity in similarity judgments points to the necessity of models and evaluation protocols that incorporate human label variation, ideally stratified by socio-cultural and linguistic background.
Limitations and Future Directions
While human annotation quality approaches 90% accuracy relative to oracle labels under certain assumptions, inherent ambiguity remains, especially without weighting of similarity aspects. The dataset is biased towards narratives salient to current embedding models and restricted to English language and Western storytelling conventions.
Future research should aim for:
- Human label variation modeling: Incorporating diverse annotator backgrounds for richer similarity judgments.
- Cross-linguistic and cross-cultural extension: Addressing narrative conventions beyond English.
- Disentangled and aspect-specific embeddings: Refining representation learning to move beyond genre predominance.
- Improved symbolic integration: Hybrid neuro-symbolic models for deep narrative abstraction.
Conclusion
SemEval-2026 Task 4 provides a comprehensive benchmark for narrative similarity and representation learning, demonstrating wide methodological variance and robust performance across both classification and embedding tracks. The dataset and annotation protocol set new standards, emphasizing discriminative, human-compatible similarity definitions and operationalizing intuitive judgments within formal narrative theory. Remaining annotation ambiguity and model bias highlight critical directions for future research in computational narratology and automated literary analysis.