Do BERT Embeddings Encode Narrative Dimensions? A Token-Level Probing Analysis of Time, Space, Causality, and Character in Fiction

Abstract: Narrative understanding requires multidimensional semantic structures. This study investigates whether BERT embeddings encode dimensions of fictional narrative semantics -- time, space, causality, and character. Using an LLM to accelerate annotation, we construct a token-level dataset labeled with these four narrative categories plus "others." A linear probe on BERT embeddings (94% accuracy) significantly outperforms a control probe on variance-matched random embeddings (47%), confirming that BERT encodes meaningful narrative information. With balanced class weighting, the probe achieves a macro-average recall of 0.83, with moderate success on rare categories such as causality (recall = 0.75) and space (recall = 0.66). However, confusion matrix analysis reveals "Boundary Leakage," where rare dimensions are systematically misclassified as "others." Clustering analysis shows that unsupervised clustering aligns near-randomly with predefined categories (ARI = 0.081), suggesting that narrative dimensions are encoded but not as discretely separable clusters. Future work includes a POS-only baseline to disentangle syntactic patterns from narrative encoding, expanded datasets, and layer-wise probing.

• The paper demonstrates that BERT token embeddings encode narrative dimensions with 94% test accuracy using a linear probing classifier on annotated Pride and Prejudice data.
• Utilizing a robust annotation pipeline with LLM-assisted verification, the study mitigates class imbalance via weighted multinomial logistic regression and control experiments.
• The analysis reveals that while strong signals like 'character' are well captured, rare dimensions such as causality suffer from boundary leakage and semantic ambiguity.

Probing BERT for Narrative Structure: Evidence from Token-Level Fictional Annotations

Introduction

This study investigates whether BERT representations encode core narrative dimensions—time, space, causality, and character—at the token level in literary fiction, specifically in Pride and Prejudice (Ch. 1–5). Through a probing classifier analysis, the work operationalizes narrative semantics as a supervised classification task over token-level contextual embeddings obtained from BERT, drawing on meticulously annotated data. A rigorous methodology employing linear probing, class balancing, human-verified LLM-assisted annotation, control experiments with random embeddings, and comprehensive clustering and POS/semantic analyses enables the quantification of BERT’s capacity for encoding complex, discourse-level phenomena within localized token contexts.

Data Construction and Annotation Pipeline

The experimental corpus is comprised of 5,088 tokens from the opening chapters of Pride and Prejudice. Annotation involved a hybrid strategy: ChatGPT produced initial token-level scores for time, space, causality, character, or others under an iteratively refined, example-intensive codebook, with exhaustive manual verification ensuring label integrity. This pipeline leverages LLM-enabled annotation efficiency, balancing it with expert human oversight—a methodology increasingly favored in computational literary analysis.

The final label distribution displays extreme skew: "others" constitutes 70.1% of labels, "character" 23.1%, with "time," "space," and "causality" collectively under 7%. This heavy-tailed distribution supports subsequent probe results interpretation, particularly regarding class imbalance and rare-category error modes.

Figure 1: Label distribution in the annotated dataset, noting the high frequency of "others" relative to rare narrative categories.

Embedding Extraction, Alignment, and Probing

Token alignment is nontrivial given mismatches between annotation units and BERT subwords. The study aligns JSON-annotated tokens to BERT tokenizations, averaging subword embeddings where necessary. The representation for each annotated token thus reflects BERT's contextual semantics at its final layer (bert-base-uncased, 12-layer, 768-dim).

A multinomial logistic regression probe (scikit-learn, L-BFGS optimizer, class_weight=‘balanced’) is trained on frozen embeddings to predict the five-way narrative label. Stratified splits preserve rare-category presence in train/test. Class weighting is essential to mitigate data imbalance, with causality tokens weighted >100x the majority class.

A critical control: an identically structured probe trained on Gaussian random vectors (matched in size and variance to real embeddings) tests for genuine encoding versus spurious patterns or probe memorization.

Classification Results and Confusion Structure

The BERT-based probe achieves 94% test accuracy (macro recall 0.83), strongly outperforming the random baseline (47% accuracy; macro F1 ≤ 0.20 for rare classes), which validates that high scores reflect model representations rather than probe artifacts.

Figure 2: Confusion matrix for the real probe; high diagonal dominance with off-diagonal “Boundary Leakage” into the “others” category for rare dimensions.

The "character" and "others" labels yield F1 near 0.97; performance for "time" (F1 = 0.77), "space" (F1 = 0.59), and especially "causality" (F1 = 0.60) is substantially lower, a direct function of lexical/semantic ambiguity and sample size. A robust finding is that errors for rare dimensions are almost exclusively misclassifications as "others" rather than cross-confusion among minority categories—a phenomenon termed “Boundary Leakage.”

Detailed error analysis shows that causality tokens are dominated by function words (e.g., "for," "therefore"), known for semantic ambiguity and annotation inconsistency. The probe’s modest recall/precision for these pivots on the high functional polysemy of causal language in English literary prose.

Clustering, Dimensionality Reduction, and Narrative Structure

Dimensionality reduction via UMAP reveals that narrative tokens do not organize into discrete, separable clusters in BERT’s representational space. K-Means clustering in 768-dimensional space is misaligned with gold labels (ARI = 0.081), with the "character" class fragmented across several centroids and “others” dominating cluster density. Linear projections (PCA) account for minimal variance; non-linear projections (UMAP, Isomap) confirm entanglement of the minority narrative dimensions and separation “bursts” primarily associated with “character.”

Figure 3: UMAP projection showing color-coded narrative dimensions, highlighting peripheral dispersion of "character" and central concentration of others.

These observations are consistent with prevailing findings regarding the anisotropy and overlap of contextual embedding spaces in BERT, where task-specific semantic features are densely interleaved and rarely form natural clusters without supervision [see also (Tenney et al., 2019), D19-1006].

Syntactic Analysis and Functional Distinctions

Analysis of POS distributions across narrative dimensions demonstrates strong, annotation-induced correlations: "character" aligns with pronouns and proper nouns, "space" with adpositions/nouns, and "time"/"causality" with adverbs and subordinators. However, exceptions (e.g., numeric tokens denoting agents) reveal the fraught boundaries between syntactic and narrative-function categories.

Figure 4: POS tag distribution per narrative dimension, evidencing category-specific syntactic signals.

Multi-word token spans are more prevalent in "space" and "time," introducing inherent ambiguity for token-level probes due to inter-word phrase segmentation and phrase–token misalignment, which likely compounds the observed "Boundary Leakage."

Figure 5: Distribution of token spans per narrative dimension; "space" and "time" feature more multiword annotations compared to "character" and "causality."

Discussion and Limitations

The core evidence is that BERT final-layer token embeddings encode substantial, recoverable information about core narrative dimensions via linear separation. However, several caveats temper the interpretation:

• Annotation bias: By linking narrative dimensions to syntactic prototypes (e.g., pronouns → characters), the probe’s success could overindex on surface cues, not abstract narrative understanding. Absence of a carefully controlled POS-only baseline precludes full disambiguation.
• Rare class fragility: Labels like "causality" are lexically narrow and annotation-inconsistent, reflecting the challenge of operationalizing event-level semantics at token granularity.
• Boundary ambiguity: Narrative semantics are contextually diffuse; phrase-level or discourse-level context is often needed, yet the probe operates on single token embeddings. This granular mismatch limits capture of genuinely global narrative structure.
• Representational entanglement: Narrative dimensions are encoded, but they are highly intermixed and non-separable without supervision, as shown by unsupervised projections and cluster ARI.

Implications and Future Directions

Practically, this study demonstrates that BERT embeddings can be leveraged to recover narrative dimensions with high accuracy under supervised conditions but highlights that discourse-level semantics are not discretely encoded at the token embedding level. Theoretically, the results suggest that transformer models internalize substantial information about high-order narrative categories as distributed signals. However, robust category separation remains elusive, confirming that current contextual embeddings reflect a highly entangled, anisotropic representational geometry for complex semantic phenomena.

Future research should:

• Implement explicit POS-only baselines to isolate BERT's contribution beyond syntactic heuristics.
• Extend annotation to additional works and more balanced class distributions, including modern text and genres.
• Employ layer-wise probing to localize whether narrative signals accumulate preferentially in deeper BERT layers.
• Explore non-linear probing architectures (e.g., MLPs, contrastive learners) to model the anisotropic, complex boundaries found for narrative categories.
• Refine annotation schemes for better operationalization of causality and event structure, potentially leveraging discourse relation annotation paradigms.

Conclusion

This probing study supplies rigorous empirical evidence that BERT’s contextualized token representations encode information about narrative dimensions in fiction, recoverable by a linear probe far above random controls. However, the structure of this encoding is diffuse rather than cluster-like, shaped both by surface features and annotation scheme biases. Disentangling syntactic, lexical, and truly narrative–semantic signals will require significantly larger and more diverse datasets, enhanced annotation granularity, and advanced probing methodologies. These findings inform ongoing efforts in computational narratology and model interpretability to parse what deep pretrained models “understand” about complex, high-level aspects of discourse structure.

---

Reference:

"Do BERT Embeddings Encode Narrative Dimensions? A Token-Level Probing Analysis of Time, Space, Causality, and Character in Fiction" (2604.10786)