CSI Drama Dataset for Multimodal Analysis

• The CSI Drama Dataset is a multimodal resource comprising annotated text, visual, and audio data from 39 CSI episodes collected over five seasons.
• It aligns data at word, sentence, scene, and shot levels to support tasks such as perpetrator identification, narrative inference, and computational video memorability.
• It employs fine-grained annotation protocols and advanced feature extraction methods to ensure high-quality, reliable multimodal data for narrative analysis.

The CSI Drama Dataset is a multimodal, fine-grained resource enabling research on language understanding, visual perception, multimodal inference, and memorability within long-form television narratives. Built from 39 annotated episodes of CSI: Crime Scene Investigation spanning Seasons 1–5 (CBS, 2001–2005), it provides aligned data at the word, sentence, scene, and video shot levels. The dataset underpins research in perpetrator identification, multi-level narrative structure, and computational video memorability, combining time-stamped screenplays, video and audio features, and dense annotations for both text and visual modalities (Frermann et al., 2017, Cummins et al., 2022).

1. Dataset Composition and Modality Coverage

The dataset comprises 39 episodes, selected across five seasons, equating to approximately 27 hours of video. Each episode is annotated into either one or two independent cases, yielding 59 narrative cases in total. Episodes are further segmented by scene, sentence (dialogue and scene description), and, via computer vision, into about 280 shots per episode (total ≃10,920 shots).

Modalities are represented as follows:

• Textual: User-generated aligned screenplays with sentence-level time-stamps, incorporating dialogue and scene descriptions.
• Visual: Video frames sampled per sentence (for sequence labeling tasks) and at 3 fps per shot (for memorability modeling).
• Audio: Full sound track excluding dialogue, with features extracted for acoustic modeling.
• Annotation Tiers:
  • Word-level (perpetrator, suspect, other character/none flags; binary and categorical).
  • Scene-level (categorical labels including Crime Scene, Victim, Evidence, Perpetrator, etc.).
  • Shot-level (boundaries, memorability scores).

2. Annotation Schemes and Quality Control

Annotation is performed with high granularity and reliability, enabling sequence labeling tasks and the study of narrative inference.

• Two-pass perpetrator identification protocol:

1. Behavioral guess phase: At three-minute increments, annotators label each sentence for “perpetrator mentioned” and assign case affiliation. Annotators also indicate their first moment of confidence in perpetrator identity (“red-button” marking).
2. Gold mention phase: Post-viewing, annotators tag each token as perpetrator, suspect, or other/none, with strict guidelines (including coreference and nominal/pronoun assignment).

Inter-annotator agreement is high: Cohen’s κ=0.74 (perpetrator mentioned overall), 0.62 for the minority class, 0.96 for case assignment, and 0.90 (perpetrator) at token level, with percent agreement ≃0.82 (Frermann et al., 2017).

Scene-level and shot-level annotations are derived from screenplay structure and neural shot segmentation (TransNet V2). Cummins et al. (Cummins et al., 2022) further aggregate word-level flags into categorical columns and propagate aspect labels for scene-level summarization.

3. Preprocessing and Feature Extraction Methodologies

Multimodal feature extraction enables the dataset’s use in sequence labeling and memorability tasks.

• Text: Whitespace tokenization; sentences encoded via 1D convolutional neural networks (CNNs) over pre-trained 50-dimensional GloVe embeddings (fine-tuned during supervised learning). Output: 225-dimensional vector per sentence.
• Vision: For sequence models, one video frame is sampled for each sentence’s central time span; features are extracted using an Inception-v4 CNN (pre-trained on ImageNet), yielding 1536-dimensional embeddings per sentence. For memorability modeling, CLIP ViT-B/32 image embeddings (512-dimensional) are extracted from 3 fps samples per shot and average-pooled.
• Audio: 13 Mel-frequency cepstral coefficient (MFCC) vectors are computed every 5 ms; for each sentence, 5 MFCC vectors are concatenated into a 65-dimensional vector.
• Fusion: Sentence-level features are concatenated to yield 1826-dimensional multimodal vectors, further projected to 300-dimensional representations via ReLU-activated linear transformation.

4. Task Formalizations and Model Architectures

Perpetrator Identification

Perpetrator identification is formalized as a sequence labeling problem. Given sequence $X=\{x_1,\ldots,x_T\}$ of multimodal feature vectors, the model predicts $Y = \{y_1,\ldots,y_T\}$ with $y_t \in \{0,1\}$ (“perpetrator mentioned” indicator for sentence $t$), making only incremental, forward-in-time predictions:

• Model architecture: Single-layer, unidirectional LSTM (hidden size 128), with modality-specific encoders, feature fusion, and softmax output. Training uses Adam optimizer (learning rate 0.001), mini-batches of 6 episodes, and 100 epochs with dropout 0.5 on LSTM input.
• Loss: Cross-entropy plus $\ell_2$-regularization.

Model	Precision	Recall	F₁
PRO rule	19.5%	77.2%	31.1%
CRF	30.2%	16.1%	21.0%
MLP (T+V+A)	38.5%	42.3%	40.2%
LSTM (T+V+A)	42.8%	51.2%	46.6%
Human upper bound	76.3%	60.2%	67.3%

All methods evaluated on held-out test set, minority “perpetrator mentioned” class (Frermann et al., 2017).

Shot-Level Memorability

CSI episodes are shot-segmented via TransNet V2 at 30 fps, with ≈280 shots per episode (mean shot length ≈2.5s, variance ≈1.2s²). Memorability per shot is estimated using:

• Vision encoder: CLIP pretrained on 400 million (image, text) pairs (ViT-B/32).
• Regression head: Bayesian Ridge Regression trained on the Memento10k human memorability dataset.
• Pipeline: Frozen CLIP → BRR → inference on new CSI shots.

Memorability scores ($M_i\in[0,1]$ for shot $s_i$) are assigned by average-pooling frame-level CLIP embeddings followed by BRR inference. Distribution peaks in [0.7, 0.95], mean ≈0.82, std ≈0.05, with minor upward drift in later seasons. No human recall studies specific to CSI are included; memorability predictions depend on transfer from Memento10k (Cummins et al., 2022).

5. Accessibility, Data Formats, and Licensing

The dataset is publicly available under research-use terms, with the following directory structure:

• /csi-corpus/ (scene-level TSVs, perpetrator ID CSVs; no video redistribution)
• /CSI-Memorability/ (shot_memorability.csv, shot_boundaries.csv)

Video files are excluded for copyright reasons. Annotation files use standardized CSV/TSV schemas. Shot boundaries and memorability scores are distributed at https://github.com/scummins00/CSI-Memorability/blob/main/data/shot_memorability.csv, and aligned annotations at https://github.com/EdinburghNLP/csi-corpus (Frermann et al., 2017, Cummins et al., 2022).

6. Applications and Limitations

The CSI Drama Dataset supports diverse research directions:

• Natural language understanding: Testing incremental inference, sequence labeling, coreference, and joint multimodal reasoning in complex narrative settings.
• Memorability prediction: Relating visual properties (shot length, composition) and high-level structure to automatic recall estimates.
• Video indexing and retrieval: Weighting key moments by predicted memorability, generating automatic highlights.
• Film and television analysis: Studying links between narrative structure, cinematography, and inferred viewer attention.

Known limitations include domain specificity (all examples from CSI), restricted coverage (39 out of 337 episodes), lack of new human memorability studies for CSI (all memorability scores stem from model transfer from Memento10k), and potential noise from shot segmentation. A plausible implication is that generalization claims should be constrained to contexts similar to procedural crime dramas; further, annotation and sampling biases may affect topic and entity distributions.

7. Significance and Research Impact

The CSI Drama Dataset provides an authentic testbed for computational models that seek to approximate real-world natural language understanding, especially in settings demanding long-range, multimodal, and incremental reasoning. By combining rich annotations across text, audio, and video, it fosters progress in grounding textual inference in perceptual evidence and supports benchmarking of neural sequence labeling and computational memorability systems on non-synthetic, broadcast-scale data. Fundamental studies such as Frermann et al. (Frermann et al., 2017) and Cummins et al. (Cummins et al., 2022) have demonstrated its utility for both model development and empirical analysis of narrative media.