DramaSR-LRM: Multimodal Speaker Recognition

• The paper introduces DramaSR-LRM, which refines speaker attributions in long-form TV dramas using a combination of acoustic, visual, linguistic, and relational evidence.
• It leverages a large reasoning model with multimodal tool-use and label propagation to tackle challenges like short utterances, off-screen speech, and crowded scenes.
• Experimental results on the DramaSR-532K benchmark show an improvement of up to 2.30% in accuracy, highlighting the benefits of selective reasoning over traditional methods.

Searching arXiv for DramaSR-LRM and closely related work on speaker recognition in long-form TV dramas. DramaSR-LRM is a speaker recognition system for long-form TV dramas that is built on a Large Reasoning Model and designed to refine initial attribution hypotheses through multimodal tool-use and contextual reasoning. Introduced together with the DramaSR-532K benchmark, the method addresses the task of assigning each pre-segmented dialogue line to a speaker drawn from a candidate character list, under conditions involving many characters, long temporal range, off-screen speech, crowded scenes, short utterances, visual ambiguity, and narrative-social dependencies. Its central premise is that speaker recognition in narrative video cannot be reduced to acoustic biometrics alone, especially when utterance duration drops below $1$ s and particularly below $0.5$ s; instead, attribution must integrate auditory, linguistic, visual, and relational evidence (Li et al., 2 Jul 2026).

1. Problem formulation and task definition

DramaSR-LRM is situated in the setting of speaker recognition in long-form TV dramas: given a pre-segmented dialogue line and a candidate character list, the system predicts which character spoke that line. The formulation is explicitly an open-set attribution problem in narrative video, not merely speech verification or diarization (Li et al., 2 Jul 2026).

The benchmark task is defined as

$$\mathcal{R}=\{r_n\}_{n=1}^N,\qquad r_n \in \{0,1,\ldots,P\},$$

where $r_n>0$ denotes a specific character ID from the library and $r_n=0$ denotes the null / ancillary / unknown-type class for non-canonical speakers. The setting is simplified in two ways that preserve the core attribution challenge: utterances are already segmented and aligned with subtitles, and the candidate character list is given in advance. The difficulty therefore lies not in segmentation, but in attribution.

The paper identifies several factors that make long-form dramas particularly difficult. A single drama can include dozens to over a hundred named characters; characters recur across episodes and seasons, making long-range narrative memory relevant; and attribution may depend on off-screen speech, crowded scenes and overlap, background noise, visual occlusion, or indirect visual evidence. Dialogue content itself is also informative, since kinship terms, honorifics, roles, and relationships can function as decisive cues. This suggests that the task is structurally multimodal and temporally extended rather than a local acoustic matching problem.

A common misconception is that LLMs can solve the task directly from transcript context. The reported baseline behavior contradicts that interpretation: direct use of Qwen3-8B is substantially weaker than the full system, indicating that reasoning alone is insufficient without specialized evidence sources, initialization, and confidence-aware invocation (Li et al., 2 Jul 2026).

2. DramaSR-532K benchmark

DramaSR-LRM was introduced together with DramaSR-532K, a large-scale benchmark intended to operationalize speaker recognition in long-form TV dramas (Li et al., 2 Jul 2026). The dataset scale and composition are as follows:

Aspect	Value
Long-form TV dramas	13
Seasons	31
Episodes	806
Total duration	525h 31min
Dialogue utterances	532,456
Known characters	903

The corpus contains 3 English dramas and 10 Chinese dramas. The appendix lists 291 / 1888 characters for English dramas, 612 / 5624 characters for Chinese dramas, and 903 / 7512 known/all characters overall. Each utterance is annotated with the dialogue text, start and end timestamps, and the speaker label.

The annotation pipeline is semi-automatic. It proceeds through OCR subtitle extraction from hard-rendered subtitles, character library construction from credits, web data, and facial samples, label propagation to obtain pseudo-labels, and human revision to produce final labels. Special taxonomy rules include assigning descriptive names to unknown but identifiable off-library characters, marking ambiguous identities as $[UNKNOWN]$, and splitting or labeling multi-speaker lines with $[MULTIPLE]$. Inter-annotator agreement is reported as about 99.6% agreement on subtitle lines in a 10% subset, with estimated final label noise about 0.5%.

The benchmark is designed to require multimodal integration. Acoustic cues include voiceprints and speaker embeddings; visual cues include character faces, on-screen presence, occlusion, clothing, and scene context; linguistic cues derive from transcript content and dialogue context; and relational/social cues include family ties, roles, honorifics, and social hierarchy. In this sense, DramaSR-532K formalizes speaker recognition as a broader narrative grounding problem rather than a narrowly phonetic one (Li et al., 2 Jul 2026).

3. System architecture and multimodal tool-use

DramaSR-LRM is described as a robust approach built upon a large reasoning model, specifically Qwen3-8B, and designed to autonomously aggregate contextual evidence via multimodal tool-use. Its overall logic is to begin with a strong acoustic/visual initialization via label propagation, query specialized tools for evidence, let the LRM synthesize that evidence and revise speaker labels, and optionally iterate (Li et al., 2 Jul 2026).

The pipeline is given as

$$\mathcal{R} \gets \mathtt{label\_prop}(\mathsf{V},\mathcal{S},\mathcal{F}),$$

followed by

$$\mathbf{K} \gets \mathtt{voice\_sim}(\mathsf{V},\mathcal{S},\mathcal{R}),$$

$$\mathcal{C} \gets \mathtt{video\_cap}(\mathsf{V},\mathcal{S},\mathcal{R}),$$

$0.5$0

$0.5$1

The inputs are raw video $0.5$2, transcript corpus $0.5$3, face set $0.5$4, current pseudo-labels $0.5$5, and a prompt/context window $0.5$6. The output is final attribution labels $0.5$7. The LRM does not operate on raw video directly; rather, it reasons over current attribution hypotheses, voice similarity scores, captions of local and global visual context, character relationship graphs, and dialogue context.

Three specialized tools organize the evidence space. The first, $0.5$8, computes an utterance-by-character similarity matrix

$0.5$9

using the label-propagation voiceprint metric based on mean top-$$\mathcal{R}=\{r_n\}_{n=1}^N,\qquad r_n \in \{0,1,\ldots,P\},$$0 cosine similarity between an utterance embedding and a character’s seed voiceprints. The second, $$\mathcal{R}=\{r_n\}_{n=1}^N,\qquad r_n \in \{0,1,\ldots,P\},$$1, provides hierarchical visual context through a local clip caption and a global segment summary. The procedure segments video into shots, merges shots into clips of about 10–15 seconds, captions each clip with a multimodal model, and summarizes groups of clips into a segment summary. The third, $$\mathcal{R}=\{r_n\}_{n=1}^N,\qquad r_n \in \{0,1,\ldots,P\},$$2, constructs a dynamic relational ontology,

$$\mathcal{R}=\{r_n\}_{n=1}^N,\qquad r_n \in \{0,1,\ldots,P\},$$3

covering stable relationships such as father / son, mother / daughter, friend, colleague, superior / subordinate, lover, and enemy. The relational memory is time-aware, since relationships can vary across episodes.

This modular decomposition is significant because it separates evidence extraction from attribution reasoning. A plausible implication is that the system treats speaker recognition as evidence aggregation under uncertainty rather than as direct end-to-end classification.

4. Initialization, training, and inference

The initialization stage is a strong label propagation baseline on which the full system is built (Li et al., 2 Jul 2026). For each utterance, the method extracts a 192D speaker embedding using ERes2Net with $$\mathcal{R}=\{r_n\}_{n=1}^N,\qquad r_n \in \{0,1,\ldots,P\},$$4-normalization. It adopts a neighborhood assumption stating that a character can only speak within a temporal window $$\mathcal{R}=\{r_n\}_{n=1}^N,\qquad r_n \in \{0,1,\ldots,P\},$$5 seconds of when their face appears: $$\mathcal{R}=\{r_n\}_{n=1}^N,\qquad r_n \in \{0,1,\ldots,P\},$$6

The baseline then builds high-confidence seed clusters from likely character-specific utterances, computes utterance-to-character affinity,

$$\mathcal{R}=\{r_n\}_{n=1}^N,\qquad r_n \in \{0,1,\ldots,P\},$$7

iteratively propagates labels using a decreasing threshold $$\mathcal{R}=\{r_n\}_{n=1}^N,\qquad r_n \in \{0,1,\ldots,P\},$$8, and assigns remaining utterances to $$\mathcal{R}=\{r_n\}_{n=1}^N,\qquad r_n \in \{0,1,\ldots,P\},$$9. This baseline is not merely preliminary; it is already strong, and the full LRM is designed to improve selectively on uncertain cases.

Training proceeds in two stages: supervised fine-tuning and reinforcement learning with GRPO. For SFT, a teacher model, Gemini-3-Pro, generates tool-use trajectories and rationales. Data curation focuses deliberately on hard cases, including utterances misattributed by label propagation, borderline samples where the top-1/top-2 acoustic margin is less than 0.03, and some easy samples for balance. About 20% of total utterances are selected for this curated set. The model is fine-tuned to invoke tools appropriately, produce structured reasoning, and output the correct speaker attribution.

RL post-training uses GRPO with an accuracy reward and a format reward, together with a KL penalty with coefficient

$r_n>0$0

Training details are specified as backbone Qwen3-8B, SFT for 3 epochs, RL for 2 epochs, and GRPO group size $r_n>0$1.

At inference time, label propagation first generates initial labels. The LRM is then invoked selectively through confidence sampling: only utterances satisfying

$r_n>0$2

are passed to the LRM, with default $r_n>0$3. This skips roughly 80% of utterances. If the LRM revises labels in a meaningful way, the updated labels can be fed back for another iteration. The main experiments use one iteration for efficiency, while a second pass yields only modest additional gain (Li et al., 2 Jul 2026).

5. Empirical performance and ablations

The experimental protocol is designed as a cross-drama generalization test. SFT uses about 10K CoT trajectories from A Lifelong Journey, RL uses 10K labeled utterances from Empresses in the Palace, and evaluation is conducted on the remaining 11 dramas, totaling 428K utterances. The implementation uses an 8-node NVIDIA H800 cluster and approximately 40 hours of training time (Li et al., 2 Jul 2026).

The primary metric is utterance-wise speaker recognition accuracy, with additional breakdowns by language, utterance duration, character density, and visual occlusion / off-screen speech. The main overall results are:

Method	Accuracy
Label Propagation	85.49%
DramaSR-LRM	86.93%
DramaSR-LRM w/ confidence sampling	87.79%

The best model therefore improves over Label Propagation by $r_n>0$4, corresponding to about a 16% relative error reduction. Other baselines include pyannote at 79.82%, Qwen3-8B direct use at 27.40%, Qwen3-8B + SFT at 75.22%, and Qwen3-8B + SFT + confidence sampling at 82.70%. These comparisons indicate that raw LLM usage is poor for speaker recognition, SFT helps substantially, and RL plus selective reasoning pushes performance beyond the strong label-propagation baseline.

The method is particularly effective on short utterances, which the paper identifies as the regime where acoustic biometrics are inherently unreliable. On Short utterances, performance improves from 82.37% to 85.70%, a gain of 3.33%. On Very Short utterances, it improves from 67.45% to 76.65%, a gain of 9.20%. By language, English total improves from 82.41% to 85.22%–85.49% depending on thresholding, while Chinese total improves from 88.58% to 90.37%.

By drama, notable gains are reported on Lost (+5.16%), Qin Empire 2 (+4.06%), The Knockout (+2.50%), and Friends (+2.30%). For utterances where the speaker’s face does not appear in the local neighborhood, DramaSR-LRM succeeds in 52.4% of cases, compared with 13.4% for the Label Propagation baseline. This is one of the clearest indications that the system benefits from relational and caption-based reasoning rather than relying solely on face cues.

Tool ablations further clarify the architecture. The full set achieves 87.79%; removing $r_n>0$5 drops accuracy to 72.61%; removing $r_n>0$6 yields 86.76%; removing $r_n>0$7 yields 87.55%; and the Label Propagation baseline remains at 85.49%. The paper interprets this as showing that voice similarity is the most important tool, video captioning is helpful especially for scene/action cues, and character relations provide a smaller average gain but help in hard relational cases such as “Dad” or “Mom.” Iterative refinement on The Long Night increases accuracy from 88.18% to 90.15% after the first LRM pass and to 90.40% after the second, indicating diminishing returns. Context-window robustness is also reported: overall accuracy varies by less than 0.2% across different context windows (Li et al., 2 Jul 2026).

6. Limitations, misconceptions, and significance

Several limitations delimit the current scope of DramaSR-LRM (Li et al., 2 Jul 2026). Multi-speaker utterances remain difficult because the current methods treat each line as an inseparable unit, making exact matching hard for choral or overlapping speech. The $r_n>0$8 and $r_n>0$9 cases are rare and do not materially affect reported results, but they indicate that the benchmark retains open-world and composite-utterance challenges. Off-screen speech, although substantially improved, is not solved. The method also depends on initial label propagation and confidence sampling, so very low-confidence cases still require robust upstream cues. In addition, the benchmark and method assume candidate speakers are provided, which simplifies the task relative to full real-world attribution.

These limitations help resolve two possible misconceptions. First, DramaSR-LRM is not an end-to-end raw audio/video speaker recognition system; it assumes pre-segmented utterances and an available candidate list. Second, it is not a replacement for strong acoustic initialization. The architecture is explicitly hybrid: label propagation supplies the initial hypothesis space, and the LRM performs evidence aggregation and revision on uncertain samples.

The paper argues that the method works because it combines complementary evidence sources. Audio helps when voice is clear, vision helps when the speaker is visible or actions are informative, relationship graphs help interpret address terms and social roles, dialogue context helps with narrative consistency and turn-taking, and reasoning helps override misleading acoustic cues in short or noisy lines. The practical implications cited include better closed captioning, better character-centric video understanding, better plot reasoning / dialogue grounding, and utility for long video QA, story understanding, multimedia indexing, video editing, and accessibility tools. The paper also reports downstream validation that better speaker recognition improves video captioning and video QA.

Within long-form multimodal understanding, DramaSR-LRM is best understood as a reasoning-centric attribution framework rather than a conventional speaker classifier. Its contribution lies not only in the reported gains over strong baselines, but in the explicit demonstration that narrative speaker recognition in long-form dramas requires an overview of voice similarity, hierarchical visual context, social relations, and selective reasoning under computational control (Li et al., 2 Jul 2026).