DramaSR-532K: TV Drama Speaker Recognition

• DramaSR-532K is a large-scale benchmark designed for detailed speaker and character attribution in long-form TV dramas.
• It integrates audio, visual, and textual features to differentiate among over 900 characters using multimodal data.
• The benchmark tackles the open-set challenge and narrative continuity by combining dialogue semantics, face recognition, and acoustic embeddings.

Searching arXiv for the core paper and cited supporting works to ground the article with current identifiers. arxiv_search: query: "(Li et al., 2 Jul 2026) Reasoning LLM Improves Speaker Recognition in Long-form TV Dramas" max_results: 5 arxiv_search({"query":"(Li et al., 2 Jul 2026) Reasoning LLM Improves Speaker Recognition in Long-form TV Dramas","max_results":5}) DramaSR-532K is a large-scale benchmark for speaker recognition in long-form TV dramas, introduced in "Reasoning LLM Improves Speaker Recognition in Long-form TV Dramas" (Li et al., 2 Jul 2026). It frames speaker recognition not as generic diarization from raw audio, but as utterance-level character attribution in serialized narrative video with large casts, persistent cross-episode relationships, and open-set ancillary speakers. The benchmark comprises 532,456 dialogue utterances drawn from 13 TV series, 31 seasons, and 806 episodes, totaling 525h31min of video and covering 903 known characters and 7,512 total characters; the abstract summarizes this as 532K annotated dialogue lines across more than 900 unique characters (Li et al., 2 Jul 2026).

1. Task formulation and problem setting

DramaSR-532K defines speaker recognition in long-form dramas over a continuous video stream $\mathsf{V}$ of total duration $T$, a set of pre-segmented and time-aligned subtitle utterances,

$$\mathcal{S} = \{(\mathbf{t}_n, a_n, b_n)\}_{n=1}^N,$$

and a character library derived from facial recognition results over the entire video,

$\mathcal{F} = \{(\mathrm{id}_m, c_m)\}_{m=1}^M.$

Here, $\mathbf{t}_n$ is the transcript text of utterance $\mathbf{u}_n$, $a_n < b_n \in (0,T)$ are start and end timestamps, and $\mathrm{id}_m \in \{1,\dots,P,\dots,P+U\}$ indicates detected identities, with IDs $\leq P$ denoting known characters and IDs $>P$ denoting ancillary or background people. The recognition target is

$T$0

where $T$1 identifies a known character and $T$2 is a null class for unnamed or transient speakers (Li et al., 2 Jul 2026).

This formulation is explicitly an open-set character attribution problem. It assumes that utterances are already segmented and synchronized with dialogue text, and that a candidate character list is known. The resulting task differs from conventional speaker diarization: the central question is not primarily "who spoke when?" but "which story character said this line?" under multimodal and narrative constraints. A recurrent misconception is that drama speaker recognition is merely diarization on entertainment video. DramaSR-532K instead relocates difficulty from segmentation to attribution, where acoustic evidence, face presence, dialogue semantics, and long-range relational context must be combined.

2. Corpus composition and benchmark scale

The benchmark is built from 13 popular long-form TV dramas, including 3 English series and 10 Chinese series, spanning genres that include drama, comedy, thriller, war, palace drama, rural, romance, historical, teen, suspense, crime, and science fiction (Li et al., 2 Jul 2026). The English subset consists of Downton Abbey, Friends, and Lost. The Chinese subset consists of A Lifelong Journey (人世间), Battle of Changsha (战长沙), Empresses in the Palace (甄嬛传), Minning Town (山海情), Ode to Joy 1 (欢乐颂1), Qin Empire 2 (大秦帝国之纵横), Stand by Me 1 (一起同过窗1), The Long Night (沉默的真相), The Knockout (狂飙), and Three-Body (三体).

At aggregate level, the English subset contains 21 seasons, 409 episodes, 228h06min of video, 291 known characters, 1,888 total characters, and 218,835 utterances. The Chinese subset contains 397 episodes, 297h25min of video, 612 known characters, 5,624 total characters, and 313,621 utterances. Combined totals are 31 seasons, 806 episodes, 525h31min, 903 known characters, 7,512 total characters, and 532,456 utterances. Individual dramas range from 30 known speakers in Stand by Me 1 to 144 known speakers in Downton Abbey; total character counts reach 744 in Friends and Qin Empire 2, and 883 in The Knockout.

The benchmark’s scale is best understood relative to prior diarization or audio-visual speaker datasets.

Dataset	Domain	Reported scale
AMI	meeting	684 clips, ~100h total, 3–5 speakers
VoxConverse	TV show	448 videos, 63h50min, up to 21 speakers
AVA-AVD	movies	351 videos, 29h15min, up to 24 speakers
MSDWild	v-log	3,143 videos, 80h03min, max 10 speakers, 112K utterances
DramaSR	long-form TV drama	806 episodes, 525h31min, 30 / 69.5 / 144 known speakers per drama, 532K utterances

The salient distinction is cast size under narrative continuity. A single drama in DramaSR routinely contains 50–100+ recurring characters plus hundreds of ancillary roles, while existing SD or SV datasets rarely exceed 20–25 speakers per video. This suggests that difficulty arises not only from signal quality but also from combinatorial expansion of the candidate space in serialized story worlds.

3. Data construction and annotation pipeline

DramaSR-532K was assembled through a semi-automatic pipeline designed to preserve narrative continuity across episodes (Li et al., 2 Jul 2026). For each drama, opening and closing sequences were trimmed, and the remaining footage was concatenated into one continuous video per series.

Utterance extraction is based on hard-rendered subtitles rather than ASR. A subtitle-region bounding box is defined, PaddleOCR-v4 is run frame by frame to obtain a raw sequence of text snippets, and adjacent frames are grouped via normalized Levenshtein similarity,

$T$3

Frames with $T$4 are merged into a single subtitle line, producing preliminary utterances and timestamps. Each segment is then refined by passing its median frame to Qwen3-VL-32B, which corrects OCR errors, resolves overlapping subtitles, and enforces linguistic consistency. Approximately 1% of the corpus is manually checked where duration-to-text-length ratios are anomalous or context appears broken.

The character library is constructed from cast and role information extracted from ending credits via OCR and manually verified, then cross-checked with IMDB or TMDB for English series and Douban or BaiduBaike for Chinese series. Professional portraits are collected as an initial face library. The full drama is scanned at 1 FPS with InsightFace using ArcFace for face detection and recognition; frames with multiple faces are dropped, duplicates are removed at cosine similarity threshold 0.8, and low-quality or occluded faces with quality $T$5 are discarded. Fifteen representative images per character are sampled, after which all frames are scanned to produce time-stamped face appearances $T$6.

Speaker labeling proceeds in two stages. Automatic initialization uses label propagation from acoustic embeddings and visual anchors, reaching about 90% estimated accuracy. Human annotators then refine labels in a dedicated GUI that presents video playback with overlaid face boxes, initial speaker hypotheses, acoustic similarity statistics, and candidate character profiles. The annotation taxonomy distinguishes primary characters, ancillary roles with descriptive labels such as Man in Black or Reporter, special handling for off-screen sources, indexed anonymous roles such as Worker #1, an $T$7 tag for unresolved cases, explicit treatment of overlapping multi-speaker utterances, and splitting of sequential composite subtitle lines when required.

Quality control includes automatic flagging of high-entropy labels, primary-character lines with low acoustic similarity, off-screen primary-character speech, newly introduced ancillary characters, and all $T$8 instances. Main annotation requires 1.5–2.5 hours per hour of video, and QC validation requires approximately 10–15 minutes per hour. On 10% of the data double-annotated by two annotators, agreement is approximately 99.6% of subtitle lines, with final label noise estimated at approximately 0.5%.

4. Multimodal design and provided features

DramaSR-532K is multimodal at both dataset and tool level (Li et al., 2 Jul 2026). Audio is extracted with ffmpeg at 16kHz. For each utterance $T$9, the audio segment is cut with $$\mathcal{S} = \{(\mathbf{t}_n, a_n, b_n)\}_{n=1}^N,$$0ms padding, truncated at midpoints if overlapping neighboring utterances, and embedded as $$\mathcal{S} = \{(\mathbf{t}_n, a_n, b_n)\}_{n=1}^N,$$1 using ERes2Net in the 3D-Speaker toolkit. Embeddings are $$\mathcal{S} = \{(\mathbf{t}_n, a_n, b_n)\}_{n=1}^N,$$2-normalized, and cosine similarity in $$\mathcal{S} = \{(\mathbf{t}_n, a_n, b_n)\}_{n=1}^N,$$3 is the principal similarity metric.

Visual information is provided through frame-wise face detection and recognition with InsightFace, producing time-stamped face instances $$\mathcal{S} = \{(\mathbf{t}_n, a_n, b_n)\}_{n=1}^N,$$4. These identify known-character appearances as well as ancillary or unrecognized identities for background individuals. For captioning-oriented tool use, selected frames are rendered with red bounding boxes around faces and labeled with character names.

Text consists of subtitle transcripts $$\mathcal{S} = \{(\mathbf{t}_n, a_n, b_n)\}_{n=1}^N,$$5 and is used not only as linguistic input to the reasoning model but also as the substrate for character relationship extraction and for video clip and segment summarization. Clip-level and segment-level visual context are produced through a hierarchical pipeline: PySceneDetect generates shots of roughly 2–3 seconds; neighboring shots are adaptively merged into 8–15 second clips using CLIP ViT-L encodings of the first, middle, and last frames and a per-drama threshold defined as the top $$\mathcal{S} = \{(\mathbf{t}_n, a_n, b_n)\}_{n=1}^N,$$6 quantile of similarity values. Qwen3-VL-32B then produces approximately 300-word clip descriptions from 1 FPS sampled frames, dialogue transcripts, pseudo speaker labels, and face overlays. Clips are grouped into semantic sections by caption similarity using bge-large-zh-v1.5, merging neighboring clips if similarity exceeds 0.8; typical section size is $$\mathcal{S} = \{(\mathbf{t}_n, a_n, b_n)\}_{n=1}^N,$$7 clips. Qwen3-32B summarizes each section into a detailed description and a brief description with title.

Character relationships are extracted from transcripts and speaker labels by prompting Qwen3-32B to produce triplets $$\mathcal{S} = \{(\mathbf{t}_n, a_n, b_n)\}_{n=1}^N,$$8, where the relation is a free-form description such as "father", "colleague", "lover", or "enemy". These episode-level outputs are merged into a global relational ontology $$\mathcal{S} = \{(\mathbf{t}_n, a_n, b_n)\}_{n=1}^N,$$9 with temporal scope, enabling later inference from address terms and social roles. A plausible implication is that the benchmark encodes not merely co-occurrence statistics but a structured approximation to evolving narrative state.

5. Evaluation protocol and challenge structure

The primary benchmark task is utterance-level speaker or character identification: given dialogue context and, depending on the method, acoustic features, face evidence, clip captions, and character relations, predict $\mathcal{F} = \{(\mathrm{id}_m, c_m)\}_{m=1}^M.$0 for each utterance (Li et al., 2 Jul 2026). Utterance-wise accuracy is the main metric. For single-speaker utterances, a prediction is correct if it matches the primary-character ID, or outputs null if the ground truth is an ancillary role. For multi-speaker utterances, a prediction is counted correct if it is one of the ground-truth speaker IDs or null, reflecting that current methods treat the line as an indivisible unit. If ground truth is $\mathcal{F} = \{(\mathrm{id}_m, c_m)\}_{m=1}^M.$1, any prediction is accepted so that intrinsically ambiguous lines do not penalize models.

Evaluation is further stratified by utterance duration, character density, off-screen status, language, and individual series. Duration subsets are long $\mathcal{F} = \{(\mathrm{id}_m, c_m)\}_{m=1}^M.$2s, medium $\mathcal{F} = \{(\mathrm{id}_m, c_m)\}_{m=1}^M.$3–$\mathcal{F} = \{(\mathrm{id}_m, c_m)\}_{m=1}^M.$4s, short $\mathcal{F} = \{(\mathrm{id}_m, c_m)\}_{m=1}^M.$5–$\mathcal{F} = \{(\mathrm{id}_m, c_m)\}_{m=1}^M.$6s, and very short $\mathcal{F} = \{(\mathrm{id}_m, c_m)\}_{m=1}^M.$7–$\mathcal{F} = \{(\mathrm{id}_m, c_m)\}_{m=1}^M.$8s. Candidate-density subsets are defined by the number of candidate characters in a $\mathcal{F} = \{(\mathrm{id}_m, c_m)\}_{m=1}^M.$9s temporal neighborhood: easy for 1–2, medium for 3–4, and hard for 5+. Off-screen utterances are those where the ground-truth speaker’s face does not appear within $\mathbf{t}_n$0s.

The benchmark is explicitly designed to stress multimodal reasoning. Many utterances are shorter than 1s, with a significant portion under 0.5s, where ERes2Net embeddings become unstable. The label propagation baseline drops to 67.45% on very short utterances, compared with 85.34% on long utterances, 87.12% on medium utterances, and 82.37% on short utterances. Approximately 9.6K utterances, or 1.8% of the corpus, are off-screen. The benchmark also incorporates overlapping speech, background music and noise, cross-episode character reappearance, evolving relationships, and large candidate sets.

A key operational device is the neighborhood assumption used during initialization: $\mathbf{t}_n$1 A character is considered a candidate for utterance $\mathbf{t}_n$2 if the character’s face appears within this temporal window. This reduces but does not eliminate candidate-space explosion; 3–5+ candidate speakers remain common in crowd scenes and group conversations.

6. Baselines, DramaSR-LRM, and empirical findings

DramaSR-532K serves as the evaluation substrate for DramaSR-LRM, a large reasoning model for multimodal, tool-augmented speaker recognition (Li et al., 2 Jul 2026). The benchmark is used in a cross-drama setting: supervised fine-tuning uses approximately 10K chain-of-thought trajectories from A Lifelong Journey; reinforcement learning uses 10K labeled utterances from Empresses in the Palace; and the remaining 11 dramas, totaling 428K utterances, form the test suite.

The strongest non-LRM baseline is label propagation. For each character $\mathbf{t}_n$3, a set $\mathbf{t}_n$4 of candidate utterances is constructed from temporal overlap with face appearances, seed clusters $\mathbf{t}_n$5 are generated in acoustic space by greedy clustering with high similarity thresholds and manual verification, and an affinity-propagation-style assignment uses $\mathbf{t}_n$6, the mean of top-$\mathbf{t}_n$7 cosine similarities between utterance $\mathbf{t}_n$8 and the character’s seed set, where $\mathbf{t}_n$9. Labels are assigned above a decreasing similarity threshold $\mathbf{u}_n$0, and remaining cases are set to $\mathbf{u}_n$1. On the 428K test utterances, this baseline reaches 85.49% overall accuracy, with 82.41% on English dramas and 88.58% on Chinese dramas.

The pyannote.audio 2.1 diarization baseline reaches 79.82% overall after Hungarian matching in sliding windows of 100 dialogue lines. LLM-only baselines are markedly weaker without specialized adaptation: Qwen3-8B in direct zero-shot mode reaches 27.40%, Qwen3-8B plus SFT reaches 75.22%, and Qwen3-8B plus SFT with confidence sampling reaches 82.70%.

DramaSR-LRM uses three tools: $\mathbf{u}_n$2, a similarity matrix $\mathbf{u}_n$3; $\mathbf{u}_n$4, which provides local clip captions and global segment summaries; and $\mathbf{u}_n$5, which exposes the dynamic relationship graph. Training combines SFT on approximately 10K curated chain-of-thought trajectories generated by Gemini-3-Pro and GRPO reinforcement learning with group size $\mathbf{u}_n$6, KL penalty $\mathbf{u}_n$7, and a reward that combines accuracy and output-format correctness. Inference uses confidence sampling: the model is selectively invoked when the margin between top-1 and top-2 acoustic similarity scores is below $\mathbf{u}_n$8, with $\mathbf{u}_n$9 in the main experiments, leaving approximately 80% of easy utterances to label propagation.

Main results show 86.93% for DramaSR-LRM without confidence sampling and 87.79% with confidence sampling. The latter yields 85.22% on English dramas, 90.37% on Chinese dramas, 87.62% on long utterances, 88.92% on medium utterances, 85.70% on short utterances, and 76.65% on very short utterances. Relative to label propagation, this is a +2.30 absolute improvement, from 85.49 to 87.79, with relative error reduction of approximately 16%. Gains are especially pronounced for very short utterances, where accuracy rises from 67.45% to 76.65%, and for off-screen utterances, where accuracy rises from 13.4% to 52.4%. Tool ablations report 72.61% without $a_n < b_n \in (0,T)$0, 86.76% without $a_n < b_n \in (0,T)$1, and 87.55% without $a_n < b_n \in (0,T)$2, versus 87.79% with the full toolset. This suggests that visual captioning and relational context matter most when acoustic biometrics are weak, while the acoustic tool remains foundational.

7. Access, downstream relevance, and ethical considerations

The benchmark is intended for non-commercial research use, and the accompanying project page is identified as https://www.github.com/198808xc/DramaSR-LRM (Li et al., 2 Jul 2026). It is derived from publicly available television dramas, but the exact license terms are not fully specified in the paper text. A plausible implication is that users may need to obtain original video content separately and rely on released annotations, scripts, and derived features rather than raw copyrighted video.

DramaSR-532K is positioned as infrastructure for multimodal, character-centric video understanding. The benchmark’s relevance extends to long-form video question answering, story reasoning, video captioning and summarization, dialogue systems, narrative agents, semantic video editing, and speech-driven video generation. The paper reports a 20K+ QA benchmark over the same 13 dramas and shows that Qwen3-VL-32B reaches 21.6% QA accuracy without speaker labels, 70.3% with label-propagation labels, 72.0% with DramaSR-LRM labels, and 80.8% with ground-truth labels. This supports the claim that speaker attribution is a central intermediate variable for long-video understanding.

The ethical profile of the benchmark is mixed and explicitly acknowledged. On the positive side, the system exposes verifiable chain-of-thought trajectories and tool-use logs, creating a concrete testbed for interpretability in multimodal reasoning, and improved speaker recognition can support accessibility through better automated closed-captioning and descriptive audio. On the risk side, the underlying techniques—face recognition, voiceprint analysis, and persistent identity tracking—could generalize beyond public fictional media to surveillance or personal media contexts. The dataset also reflects the socio-demographic and cultural biases of popular English and Chinese television dramas, so models trained on it may inherit skewed representations of gender, ethnicity, and social roles. In this sense, DramaSR-532K is both a benchmark for narrative video understanding and a case study in the governance challenges of multimodal identity technologies.