Skeleton-based Coherence Modeling in Narratives

Abstract: Modeling coherence in text has been a task that has excited NLP researchers since a long time. It has applications in detecting incoherent structures and helping the author fix them. There has been recent work in using neural networks to extract a skeleton from one sentence, and then use that skeleton to generate the next sentence for coherent narrative story generation. In this project, we aim to study if the consistency of skeletons across subsequent sentences is a good metric to characterize the coherence of a given body of text. We propose a new Sentence/Skeleton Similarity Network (SSN) for modeling coherence across pairs of sentences, and show that this network performs much better than baseline similarity techniques like cosine similarity and Euclidean distance. Although skeletons appear to be promising candidates for modeling coherence, our results show that sentence-level models outperform those on skeletons for evaluating textual coherence, thus indicating that the current state-of-the-art coherence modeling techniques are going in the right direction by dealing with sentences rather than their sub-parts.

• The paper introduces the SSN, a Siamese LSTM architecture that compares sentence and skeleton embeddings using a contrastive loss for coherence modeling.
• It demonstrates that full sentence models achieve around 92.9% accuracy in sentence order detection, outperforming skeleton-based models by about 9 percentage points.
• The study highlights limitations of skeleton extraction due to noise and context loss, reaffirming the effectiveness of full-text models for narrative coherence.

Skeleton-based Coherence Modeling in Narratives: A Technical Analysis

Introduction

The paper "Skeleton-based Coherence Modeling in Narratives" (2604.02451) investigates the efficacy of skeleton-based representations for coherence modeling in narrative text. Motivated by prior work deploying skeletons—key-phrases extracted via neural models—for narrative generation, the authors hypothesize that consistency between sentence skeletons could serve as a robust metric for local coherence. They propose and evaluate a new neural architecture, the Sentence/Skeleton Similarity Network (SSN), and systematically compare its utility when operating on skeletons as opposed to full sentences, yielding several significant empirical findings that challenge the expected superiority of skeletons for this task.

Background and Motivation

Coherence modeling remains central to advanced natural language understanding and generation. Traditional approaches, such as entity grids [Barzilay and Lapata, 2008], model local coherence via patterns in entity transitions, whereas neural models extend this by capturing broader semantic and discourse features [li2016neural]. Skeleton-based models, as introduced by Xu et al., provide an abstraction mechanism by extracting core entities, relations, and events—skeletons—from sentences, and have shown promise in text generation tasks. The present work inverts this paradigm, probing whether skeleton repetition and similarity is a meaningful criterion for textual coherence in a discriminative setting.

Figure 1: The skeleton-based model architecture for generating narratives, clearly illustrating separation between test and training phases.

Sentence/Skeleton Similarity Network (SSN) Architecture

The core contribution is the SSN, a neural Siamese architecture designed to embed and compare either sentences or their skeletons for coherence estimation. The workflow encodes two input sequences using LSTMs, optionally enhanced with a self-attention layer, then computes their normalized pairwise similarity—either cosine-based or Euclidean—and applies a contrastive loss.

The model incorporates FastText word embeddings at input for effective representation of sparse, unordered skeleton keywords, aiming to improve alignment despite the lack of local context or order. The contrastive training objective is formulated to maximize the similarity of truly consecutive sentences (label 1), and to minimize similarity for incoherent or random pairings (label 0).

Figure 2: The architecture of the Skeleton Similarity Network, including the optional attention layer and Siamese LSTM embedding pipeline.

Experimental Framework

Data

The experiments utilize the Visual Storytelling Dataset [huang2016visual], encompassing over 50,000 stories, each with up to six sentences. Skeletons for each sentence are extracted using the neural model of Xu et al., yielding datasets at both sentence and paragraph granularity for both skeleton and sentence-based evaluation.

Evaluation Metrics

Three complementary metrics are utilized:

1. Incoherent sentence pair detection: Can the model distinguish between genuine consecutive sentences versus unrelated pairs?
2. Incoherent story pair detection: Can it distinguish between ordered and jumbled sentence sequences within stories?
3. Pairwise classification accuracy: Agreement with the binary ground truth of sentence adjacency.

Model predictions are evaluated via accuracy, reporting the proportion of correct identifications across these metrics.

Baselines and Ablations

The SSN’s performance is compared not only between skeleton and sentence input modes, but also with non-parametric similarity measures (cosine, Euclidean) over BERT-derived embeddings. Ablation of the attention mechanism elucidates its contribution.

Results

Neural methods significantly outperformed non-parametric baselines, even those using contextual BERT embeddings. For instance, sentence order accuracy using BERT+cosine was 71.9%, while SSN on sentences achieved 92.9%. The top-line results are as follows:

Model	Sentence Order (%)	Story Order (%)	Pair Classification (%)
SSN-3 on sentences	92.9	69.6	82.2
SSN-3 on skeletons	84.2	62.9	73.8
SSNA-2 on sentences	92.3	68.0	81.4
SSNA-2 on skeletons	84.5	62.3	74.5

Contrary to initial hypothesis, full sentence-based SSN models consistently outperformed their skeleton-based counterparts across all metrics, with sentence order accuracy exceeding skeleton-based models by approximately 9 percentage points. Sentence-based input also yielded higher story coherence scores.

Notably, the addition of a self-attention layer yielded only marginal improvements, suggesting that, under the tested configuration and data constraints, stacked LSTM encoders are adequate for embedding coherence-relevant sentence features.

Analysis and Discussion

Skeletons vs. Sentences

The empirical findings contradict the initial assumption that skeletons, by distilling key content, would provide better discriminative power for narrative coherence. This shortfall is attributed to two main factors:

• Skeleton extraction quality: The upstream neural skeletonizer, if imperfect, introduces noise and information loss not recoverable downstream.
• Loss of order and context: Skeletons, lacking both syntactic structure and full lexical context, limit the Siamese SSN’s ability to model the fine-grained coherence features that sentence-level encoders naturally capture.

Sentence-based coherence modeling thus remains superior given available neural architectures and data, substantiating the dominant trend in recent state-of-the-art research.

Model Architecture

The lack of significant improvement from introducing self-attention highlights either limitations in the applied attention design or possible sufficiency of bi-directional LSTM encoders for short narrative coherence. Opportunities exist for incorporating more expressive attention mechanisms inspired by Transformer models.

Dataset Limitations and Generalizability

The relatively short document lengths in the Visual Storytelling Dataset likely constrain the model's ability to generalize to paragraph-level or long-range coherence. Existing literature [li2016neural] underscores the need for longer sequence analysis, e.g., essays of 500+ words, for robust evaluation of story-level coherence models.

Implications and Future Directions

The results confirm that, for existing architectures and moderate-length narrative datasets, full-sentence models remain optimal for coherence discrimination. Their signal-to-noise advantage, retention of order, and richer contextual encoding outweigh the hypothesized benefits of skeleton abstraction under current extraction and modeling techniques.

Nevertheless, improvements in skeleton extraction quality, more advanced attention mechanisms, and extension to longer documents remain promising avenues. Additionally, there is strong practical relevance for systems that can detect incoherent sentences within otherwise coherent text—a scenario underexplored in this work.

The broader implication is a reaffirmation of sentence-based neural coherence models as a technical baseline for both evaluation and system integration in NLG pipelines, with skeleton-based approaches requiring further maturation before supplanting full-text models.

Conclusion

The "Skeleton-based Coherence Modeling in Narratives" paper (2604.02451) rigorously evaluates the utility of skeleton-based representations for textual coherence modeling, introducing the SSN as a discriminative Siamese neural model. The study demonstrates that sentence-embedding models systematically outperform skeleton-based counterparts on coherence detection tasks. These findings indicate the continued relevance and efficacy of sentence-based approaches and provide a clear benchmark for future work on abstraction-based coherence modeling, especially as larger and more complex datasets become available and more advanced extraction and attention techniques are developed.