Critical Role D&D Dataset (CRD3)

• CRD3 is a comprehensive dataset derived from 159 Critical Role episodes, featuring 398,682 dialogue turns segmented into coherent narrative chunks.
• It employs methods like TF–IDF, K-means clustering, and VADER sentiment analysis to annotate character relationships and narrative context.
• The dataset supports multi-task learning for story continuation and character prediction using transformer-based bi-encoder architectures.

The Critical Role Dungeons & Dragons Dataset (CRD3) is a comprehensive resource designed for the study of character-driven, story-centric multi-user dialogue, with a particular focus on the modeling of character personas and inter-character relationships. Originating from the collaborative storytelling sessions of Critical Role, the dataset facilitates research into narrative generation, dialogue modeling, and multi-agent interaction in role-playing settings (Si et al., 2021).

1. Dataset Scope, Scale, and Composition

CRD3 comprises transcripts from 159 Critical Role episodes spanning two complete "seasons," resulting in 398,682 dialogue turns (lines). The dataset is segmented into 34,243 "chunks," each corresponding to a coherent mini-subplot, with grouping determined by ground-truth plot or location boundaries as annotated by the original dataset creators.

Each chunk includes:

• An abstractive human-written summary
• Speaker-attributed dialogue (players or Dungeon Master)
• First- and second-person narration

Chunk-level statistics post-extension:

• Train set: 11,400 chunks, average 38.4 turns and 4.06 distinct characters per chunk
• Dev set: 815 chunks, average 61.2 turns and 4.07 distinct characters per chunk
• Test set: 761 chunks, average 62.2 turns and 4.36 distinct characters per chunk

Roles include approximately six main player characters per episode (“Vox Machina” PCs such as Scanlan, Vex’ahlia, Grog, Keyleth, Percy, Vaxildan) and a Dungeon Master (DM), who is responsible for both narration and NPC dialogue, contributing roughly 10–15% of turns. The data encompasses an estimated 4–5 million tokens.

2. Preprocessing and Annotation Methodologies

2.1 Transcript Segmentation

Transcripts are divided into chunks at plot or location boundaries, guided by human annotators. Each chunk is furnished with a human-authored summary, first-/second-person narration, and full dialogue, all speaker-attributed.

2.2 Character Persona and Relationship Annotation

CRD3 does not provide explicit persona files; consistency of persona is inferred through the modeling of speaker identity within context.

Relationships are extracted using a multi-step automated process:

1. Pairwise free-form relationship descriptions were crawled from the Critical Role Wiki (e.g., "Scanlan and Percy had an amicable relationship").
2. Descriptions were vectorized using TF–IDF.
3. K-means clustering ($k=3$) was applied.
4. Each cluster was assigned a sentiment label (positive, neutral, negative) via the VADER sentiment analyzer.
5. Each chunk statically labels every character pair $(C_i, C_j)$ with the result from clustering.

Three relationship categories are defined:

• positive (friendly/allies)
• neutral (acquaintances/no strong affect)
• negative (rivals/enemies)

Heuristics leverage TF–IDF, scikit-learn’s K-means, and VADER sentiment polarity mapping.

3. Data Schema, Format, and Splitting

3.1 Data Files

• Dialogues and summaries are provided in JSONL format, with each record representing a chunk.
• Relationship annotations are represented as JSON lists of character triples: [“Character A”, “relation_label”, “Character B”].
• Train/dev/test splits are maintained as CSV files listing chunk identifiers and file paths.

Representative JSON record structure:

{ "chunk_id": "S1E05_023", "summary": "...", "dialogue": [ {"speaker":"Vex’ahlia","text":"Bundle up!"}, ... {"speaker":"DM","text":"No, you do not see Larkin around."} ], "narration": [ {"type":"first_person","text":"I attempt to see if Tooma is coming."}, {"type":"second_person","text":"Do we see Larkin around?"} ], "relations": [ ["Scanlan","neutral","Vex’ahlia"], ["Keyleth","positive","Scanlan"], ["Grog","negative","Vex’ahlia"] ] }

3.2 Data Split Rationale and Access

Splits are contiguous and chronological (i.e., train uses early episodes, dev uses mid-series, and test uses recent episodes), designed to prevent temporal leakage.

Split	Chunks	Avg. Turns/Chunk	Avg. Distinct Characters/Chunk
Train	11,400	38.4	4.06
Dev	815	61.2	4.07
Test	761	62.2	4.36

Licensing: CC BY-NC 4.0, including all extended annotations. Repository: https://github.com/georgiatech-crd3-extended.

4. Modeling Paradigms and Loss Definitions

Research leveraging CRD3 centers on multi-task learning for story-driven dialogue systems. The principal joint objective incorporates:

• Story continuation (ranking): Margin-based ranking loss $L_\text{rank}$ over possible responses.
• Next-character prediction (classification): Cross-entropy loss $L_\text{cls}$.

Total loss:

$$L_\text{total} = \lambda_1 \cdot L_\text{rank} + \lambda_2 \cdot L_\text{cls}$$

with hyperparameters $\lambda_1 = \lambda_2 = 0.5$.

Transformer-based bi-encoder architectures are used, inspired by Urbanek et al. (2019, LIGHT), employing shared encoders for conversational context and candidate utterances with dot-product scoring for retrieval and a parallel softmax layer for next-speaker classification.

5. Evaluation Metrics and Experimental Results

CRD3 supports dual evaluation protocols:

• Story continuation: Hits@N (e.g., Hits@1/10, Hits@5/10), measuring frequency of gold response in the top-N retrieved utterances.
• Character prediction: Weighted F1 over the full next-speaker set.

Ablational results demonstrate the utility of chunk summaries and relation annotations:

Model	Character Prediction (F1)	Story Continuation (Hits@1/10)
Base (single-task)	47.3	18.0
Base + Summary (multi-task)	49.0	20.4
Base + Relations (multi-task)	48.8	20.2
Full (summary + relations, multi-task)	48.8	21.3

Joint modeling of character relations and chunk summary context yields the highest story-continuation accuracy, substantiating the annotation extension’s benefit (Si et al., 2021).

6. Significance, Use Cases, and Accessibility

The CRD3 dataset is a benchmark for computational modeling of complex, interactive storytelling involving recurring characters, explicit relationships, and evolving narrative structure. It supports the development and evaluation of story-continuation models with sensitivity to conversational context, persona consistency, and multi-party dynamics. Models trained on CRD3 can be directly applied to research in open-domain dialogue, story understanding, and narrative generation.

The dataset’s transparent annotation pipeline (pairwise relation extraction, semi-automatic sentiment labeling, coherent splitting) makes it suitable both for model benchmarking and for the study of automated relationship and role extraction in naturalistic, multi-agent dialogic environments.

CRD3 and the accompanying codebase are available under a CC BY-NC 4.0 license, ensuring broad accessibility for academic research purposes.

---

For further technical details and empirical findings, see (Si et al., 2021).