Reasoning LLM Improves Speaker Recognition in Long-form TV Dramas

Abstract: Long-form TV dramas present a formidable challenge for comprehensive video understanding, where deciphering complex storyline often relies on \textbf{speaker recognition}, the task of accurately attributing each spoken utterance to its respective character. In this paper, we advance this field through two primary contributions. (1) We introduce \textbf{DramaSR-532K}, a large-scale benchmark comprising 532K annotated dialogue lines across more than 900 unique characters, necessitating the integration of auditory, linguistic, and visual cues for speaker recognition. (2) We propose \textbf{DramaSR-LRM}, a robust approach built upon a large reasoning model (LRM). DramaSR-LRM is designed to autonomously aggregate contextual evidence via multimodal tool-use, synthesizing diverse inputs to achieve high-fidelity attribution. Experimental results demonstrate that DramaSR-LRM significantly outperforms existing baselines, particularly on short utterances where acoustic biometrics are inherently unreliable. \textit{All the data and code will be made publicly available at the project page: https://www.github.com/198808xc/DramaSR-LRM.}

• The paper introduces a novel LLM-based reasoning pipeline (DramaSR-LRM) that enhances speaker recognition in long-form TV dramas.
• It leverages multimodal tools like acoustic similarity, video captioning, and character relation analysis, refined via supervised fine-tuning and reinforcement learning.
• Empirical results show significant gains, with improvements up to 9.2% in accuracy for short utterances and high candidate density scenarios.

Large Reasoning LLMs for Speaker Recognition in Long-form TV Dramas

Introduction and Motivation

Speaker recognition (SR) in long-form narrative media such as TV dramas is a critical precondition for fine-grained video understanding, enabling accurate parsing of complex dialogues, character interactions, and downstream tasks like video QA and captioning. Prior studies in speaker diarization, verification, and multimodal integration have been largely confined to domains with fewer speakers, minimal role overlap, or short conversation clips. "Reasoning LLM Improves Speaker Recognition in Long-form TV Dramas" (2607.02504) systematically addresses the underexplored challenge of speaker recognition at scale in long-form, high-character-density drama series, introducing both a realistic benchmark (DramaSR) and an LLM-based reasoning pipeline (DramaSR-LRM).

The DramaSR Benchmark

To fill the gap in robust, large-scale SR datasets for complex narrative media, the paper establishes DramaSR, spanning over 532,000 manually verified utterances sourced from 13 TV dramas (English and Chinese) covering more than 900 unique characters and an even larger set of minor roles. Compellingly, each utterance is mapped to a ground-truth speaker via a semi-automatic pipeline: hard-rendered transcripts are extracted using OCR, character libraries are formed from credits and web data, and label propagation is combined with meticulous human annotation to produce high-fidelity attributions.

Figure 1: The DramaSR construction pipeline combines OCR transcript extraction, cast metadata mining, label propagation, and human annotation for ground truth.

DramaSR differentiates itself from prior datasets by order of magnitude in character count, narrative length, and utterance volume, directly targeting the open-set classification nature of SR in dramas, where speakers may be unlisted, visually occluded, or interact through oblique linguistic context.

Methodology: The DramaSR-LRM Pipeline

DramaSR-LRM leverages a large reasoning model (LRM; Qwen3-8B backbone) orchestrated via supervised fine-tuning (SFT) and reinforcement learning (RL), iteratively refining initial labels from an acoustic-visual label propagation baseline. The LRM is empowered with multimodal tool-use primitives:

• $\mathtt{voice\_sim}$: acoustic similarity calculation between utterance voiceprints and reference character embeddings.
• $\mathtt{video\_cap}$: hierarchical video captioning, providing clip- and segment-level visual scene information.
• $\mathtt{char\_relation}$: relational graph construction parsing dialogue for explicit or implicit social links.

Chain-of-thought (CoT) reasoning allows the LRM to dynamically route hard instances (acoustic ambiguity, off-screen speakers, pronouns, short utterances) to the appropriate mixture-of-experts toolchain, integrating evidence before updating speaker attributions.

Figure 2: Chain-of-thought reasoning illustrates the LRM’s tool-routing procedure for speaker disambiguation.

The pipeline iteratively alternates between label propagation (updating seeds and affinity clusters via acoustic and visual anchors) and reasoning-based corrections. RL (with GRPO) further stabilizes policy execution across high-entropy or ambiguous utterances, maximizing end-task accuracy and format compliance.

Experimental Evaluation

Recognition Accuracy

In held-out evaluation over 11 unseen dramas (428k utterances), DramaSR-LRM achieves 87.79% average accuracy, outperforming the acoustic-visual label propagation baseline by 2.3% absolute (relative error reduction of ~16%). Results are consistent across both English and Chinese series and in genres with high character turnover or rapid dialogue. Performance is most pronounced in adversarial conditions:

• Short/Very Short Utterances (<1s, <0.5s): DramaSR-LRM yields up to 9.20% absolute improvements, as acoustic information becomes intrinsically unreliable.
• High Candidate Density (≥5 characters per window): Outperforms by disambiguating visually and relationally similar co-occurring speakers.
• Visual Absence (off-screen speakers): LRM solves 52.4% of cases, up from 13.4% for the baseline.

  Figure 3: Speaker recognition accuracy rises with increased LLM engagement ('confidence sampling') in hard cases and shorter utterances.

Performance is robust to the confidence-sampling threshold, with LRM selectively activated for ambiguous samples (margin below 0.10 between top two acoustic candidates). For easy cases, the cost-effective LP baseline is reused, optimizing compute without accuracy sacrifice.

Tool and Iterative Ablations

Ablation studies show $\mathtt{voice\_sim}$ dominates in overall contribution, but $\mathtt{video\_cap}$ and $\mathtt{char\_relation}$ are disproportionately valuable for short utterances and pronoun/relationship-driven context. Iterative passes further improve recognition up to 0.25% in second runs by self-consistency, but the computational cost is non-negligible and thus typically omitted from primary results.

Downstream Video Understanding

Speaker recognition is shown to be foundational for higher-level video understanding. Substituting speaker tags from DramaSR-LRM for the LP baseline in multimodal LLMs (video QA and captioning) yields substantive gains in grounding: QA accuracy climbs from 70.3% (baseline) to 72.0% (DramaSR-LRM), with a remaining gap to the upper bound (80.8%, using ground-truth speakers). The qualitative shift mainly manifests in "who-said-what" type reasoning and attribution of dialogic intent in ambiguous contexts.

Figure 4: Improved speaker recognition directly impacts the quality of video captioning and question-answering in downstream MLLMs.

Practical and Theoretical Implications

This research concretely demonstrates that high-quality SR—especially when integrating multimodal evidence via structured tool-use and CoT reasoning—serves as the linchpin for complex narrative video understanding workflows. The pipeline is scalable, robust to language/domain shift, and modular toward further tool incorporation (e.g., end-to-end audio segmentation or open-world speaker description). The explicit logging of tool-use and reasoning steps also enhances model interpretability, a recurring concern in multimodal LLMs. With both data and code publicly released, DramaSR provides a critical testbed for the next generation of multimodal AI targeting open-ended narrative analysis, cross-modal retrieval, and dialog-centric video understanding.

From a theoretical perspective, DramaSR-LRM validates RL-finetuned LLMs’ capacity to optimize for cross-modal logical consistency, overcoming unimodal ambiguities. The generalization to new languages and genres also suggests potential for universal character-centric video comprehension frameworks aligned with story understanding.

Conclusion

Reasoning-augmented LLMs with multimodal integration can significantly advance speaker recognition in complex, realistic long-form video domains. As demonstrated in DramaSR-LRM, structured tool use, CoT, and reinforcement post-training yield robust empirical gains, especially in low-signal or ambiguous contexts. The DramaSR benchmark stands as a resource for continued research at the intersection of language, vision, and long-range sequential reasoning, with promising future extensions in open-set identification and end-to-end scene parsing.