Speaker-Role Diarization (RD)

• Speaker-Role Diarization (RD) is the process of segmenting audio streams into time intervals labeled with semantically meaningful participant roles instead of generic speaker IDs.
• The paper highlights that integrating lexical, audiovisual, and ASR-derived side information improves role classification accuracy and overall diarization performance.
• RD methods vary from supervised role classification to constraint-guided clustering and joint ASR+RD models, each tailored to specific application domains such as legal, medical, or multimedia settings.

Speaker-Role Diarization (RD) is the role-aware variant of speaker diarization. Traditional speaker diarization answers “who spoke when?” using generic labels such as speaker-1 and speaker-2, whereas RD answers “which role spoke when?” for labels such as doctor versus patient, host versus guest, therapist versus patient, teacher versus student, judge versus unreferenced speaker, or child versus adult (Ghosh et al., 14 Jul 2025). In application settings, role labels are often more useful than anonymous speaker identities because downstream systems frequently depend on functional participant categories rather than arbitrary cluster indices (Ghosh et al., 14 Jul 2025). Existing RD work spans linguistically aided dyadic diarization, multimodal constrained clustering, reference-dependent judicial speech attribution, classroom role separation, joint ASR+RD modeling, and open-world audiovisual character attribution (Flemotomos et al., 2019, Flemotomos et al., 2022, Tumminia et al., 2021, Khan et al., 16 May 2025, Xu et al., 25 Jan 2026, Huang et al., 17 Mar 2026).

1. Conceptual scope and relation to conventional diarization

Speaker diarization is conventionally defined as partitioning audio into segments labeled by speaker identity over time. RD preserves that temporal labeling objective but replaces generic speaker IDs with semantically meaningful role labels. In the doctor-patient setting, for example, the target is not speaker-1 versus speaker-2, but DOC versus PAT; in child-adult interactions, the serialized output explicitly emits <child> or <adult> together with timestamps and transcript content (Ghosh et al., 14 Jul 2025, Xu et al., 25 Jan 2026).

The literature distinguishes several regimes. In some domains, roles are assumed to have a one-to-one mapping with speakers, as in dyadic psychotherapy with therapist and patient (Flemotomos et al., 2019). In others, roles are informative but not identical to speaker identity: a single speaker may play multiple roles, or multiple speakers may share the same role, so role information is used as a constraint rather than a direct label (Flemotomos et al., 2022). In legal proceedings, the task is reference-dependent rather than fully open diarization: judicial speech is identified against a predefined reference library of judges, and non-library speakers are rejected by thresholds (Tumminia et al., 2021). In visual media, the setting becomes open-world speaker registration plus diarization, where the number of speakers is not fixed and off-screen characters must be registered dynamically (Huang et al., 17 Mar 2026).

A recurring theme is that RD is not reducible to generic clustering. Role information may come from lexical regularities, transcript structure, enrollment metadata, audiovisual anchors, or explicit supervision in a joint ASR model. This suggests that RD is best understood as a family of task formulations rather than a single algorithmic template.

2. Problem formulations and supervision regimes

One formulation converts diarization from unsupervised clustering into supervised role classification. In dyadic psychotherapy, the pipeline is audio segmentation → text segmentation → role recognition → role-profile estimation → audio segment classification. The method assumes a fixed known set of roles and a one-to-one mapping between roles and speakers. Role-specific LLMs assign each text segment to the role whose model yields minimum perplexity, then x-vectors from role-assigned segments are pooled into role profiles and final audio segments are classified by PLDA scoring against those profiles (Flemotomos et al., 2019).

A second formulation treats roles as side information for constrained clustering. In multimodal clustering with role-induced constraints, a BERT-based text classifier predicts segment roles such as therapist versus patient or host versus non-host. High-confidence role predictions are converted into pairwise must-link and cannot-link constraints over segments, and those constraints modify the affinity matrix used by spectral clustering (Flemotomos et al., 2022). The key construct is the constraint matrix

$$Z_{ij} = \begin{cases} +1, & \text{if there exists an ML constraint between } i \text{ and } j \ -1, & \text{if there exists a CL constraint between } i \text{ and } j \ 0, & \text{if no constraint exists} \end{cases}$$

which is then propagated and injected into the clustering graph (Flemotomos et al., 2022).

A third formulation is ASR-synchronized RD. In doctor-patient conversations, a frozen ASR transducer provides a forced alignment path, and an auxiliary RD transducer is trained only at non-blank ASR emission positions with cross-entropy rather than RNNT loss (Ghosh et al., 14 Jul 2025). The corresponding RD loss is

$$\mathcal{L}_{RD} = \sum_{r_{t,u} \in \mathbf{a_r^*},\, r_{t,u}\neq -} -\ln \mathbb{P}(r_{t,u}\mid f^{RD}_{1:T}, g^{RD}_{1:u-1})$$

where supervision is attached only to role-labeled alignment positions (Ghosh et al., 14 Jul 2025). The same work argues that word prediction and role prediction require different predictor contexts, motivating separate task-specific predictors rather than a shared predictor (Ghosh et al., 14 Jul 2025).

A fourth formulation jointly models ASR, role labels, and timestamps in a single sequence generator. For child-adult interactions, a Whisper-based encoder-decoder emits segments of the form

$$<|t\_start|>\;<speaker>\;\text{transcript}\;<|t\_end|>$$

while a frame-level diarization head predicts child, adult, and silence. The system is trained by

$$\mathcal{L}_{total} = \mathcal{L}_{ASR} + \lambda_{diar}\mathcal{L}_{diar},$$

with $\lambda_{diar}=1$ (Xu et al., 25 Jan 2026). Forced decoding enforces a valid output state sequence and diarization-guided silence suppression constrains timestamp placement (Xu et al., 25 Jan 2026).

A fifth formulation is reference-dependent verification. In SCOTUS oral arguments, Reference-Dependent Speaker Verification builds a Reference Audio Library of judge d-vectors and assigns each target frame or window to the reference speaker with highest cosine similarity, unless score_thresh and sim_thresh indicate an unreferenced speaker (Tumminia et al., 2021). This is role-focused in the sense that the operational target is judicial speech rather than generic speaker clusters.

These formulations differ in whether roles are closed-set or open-world, whether roles are labels or constraints, and whether supervision is lexical, acoustic, audiovisual, or transcript-synchronous.

3. Methodological families

RD methods in the literature fall into several technical families.

Role-conditioned profile classification replaces speaker clustering with role-profile estimation. Role-specific n-gram LLMs classify text segments by perplexity, confidence filtering keeps only the most reliable segments, and x-vector means define role profiles that are scored by PLDA against uniformly segmented audio (Flemotomos et al., 2019). This family is effective when roles are known in advance and linguistic style differs by role.

Constraint-guided spectral clustering keeps an audio-based clustering backbone but alters the affinity graph using text-predicted roles (Flemotomos et al., 2022). The method uses role-induced must-link and cannot-link constraints, E$^2$CP constraint propagation, $p$-thresholding of the affinity matrix, and normalized spectral clustering. It is specifically designed for settings where role and speaker are related but not identical.

Lexically informed speaker-turn modeling is adjacent to RD rather than role-labeling proper. A GRU-based model estimates word-level speaker turn probabilities from ASR words, word boundaries, and speaker embeddings, and those probabilities define lexical adjacency matrices that are fused with x-vector-based acoustic graphs for spectral clustering (Park et al., 2020). This does not predict roles, but it supplies turn-structure cues that can support role-sensitive segmentation.

ASR-guided role prediction synchronizes role inference with token emission. In doctor-patient audio, the recommended architecture uses ASR (CNN-2) + RD (RNN) because ASR tolerates limited predictor context whereas role prediction benefits from longer context (Ghosh et al., 14 Jul 2025). The same framework also proposes RD-guided blank suppression: if the top ASR non-blank token is in a deletion-prone set such as yeah or okay, the ASR posterior is sufficiently confident, and the RD posterior is highly confident, then blank probability is suppressed during beam search (Ghosh et al., 14 Jul 2025).

Joint end-to-end ASR+RD integrates transcript generation and role diarization into a single model. In child-adult interactions, the diarization head predicts frame-level child/adult/silence labels, a serialized decoder emits structured role-attributed segments, and a state-machine masks invalid next-token choices (Xu et al., 25 Jan 2026). This family addresses the error propagation inherent in cascaded diarization-first or ASR-first systems.

Reference-dependent verification uses external role inventories instead of clustering. Judicial diarization constructs a judge-specific reference library from RTTM-aligned audio intervals, averages d-vectors within intervals, and compares target embeddings to reference embeddings by cosine similarity (Tumminia et al., 2021). This is transparent and domain-specific rather than open-set.

Multi-stage role separation in noisy classrooms uses the NeMo diarization pipeline with explicit teacher-student versus all-speaker evaluation. The pipeline decomposes into voice activity detection, speaker embedding extraction, and clustering. The work studies segment-wise VAD, frame-wise VAD, wav2vec 2.0 VAD, Whisper timestamps, and a hybrid VAD

$$Y_i = \alpha \cdot \text{frame-vad}_i + (1-\alpha)\cdot \text{whisper}_i$$

to balance low miss against low false alarm (Khan et al., 16 May 2025).

Open-world multimodal registration and diarization extends RD to films and TV. CineSRD first clusters visual anchors, registers initial speakers by voting from visual clusters to audio clusters, refines adjacent subtitle-line speaker turns using an audio LLM plus timbre similarity, and supplements off-screen speakers by group confidence scoring (Huang et al., 17 Mar 2026). The same-speaker decision score is

$P_{std} = w\cdot P_{alm} + (1-w)\cdot S_{tim},$

and groups with confidence below $\eta$ are registered as new speakers (Huang et al., 17 Mar 2026).

Together, these families show that RD has evolved from role-conditioned post hoc classification toward joint, multimodal, and open-world formulations.

4. Domains, datasets, and benchmark settings

RD has been studied in a set of highly domain-specific corpora rather than in a single universal benchmark. The operational meaning of “role” changes across domains, which is central to experimental design.

Domain	Role formulation	Representative work
Psychotherapy	therapist vs patient	(Flemotomos et al., 2019, Flemotomos et al., 2022)
Podcasts	host vs non-host	(Flemotomos et al., 2022)
Legal proceedings	judges vs unreferenced speakers	(Tumminia et al., 2021)
Classrooms	teacher vs students; all speakers	(Khan et al., 16 May 2025)
Medical conversations	doctor vs patient; other roles	(Ghosh et al., 14 Jul 2025)
Child-adult interactions	child vs adult	(Xu et al., 25 Jan 2026)
Visual media	character-level open-world annotation	(Huang et al., 17 Mar 2026)

In psychotherapy, the PSYCH corpus is split into train, dev, and test with 74, 44, and 25 sessions respectively, and no speaker overlap across splits (Flemotomos et al., 2019). The multimodal constrained-clustering study also uses UCC psychotherapy sessions with 50 train, 26 dev, and 20 eval sessions (Flemotomos et al., 2022). In podcast RD, the TAL dataset contains 663 episodes split 593/34/36, with average 17.7 speakers per episode, and role labels are operationalized as Ira Glass versus non-host (Flemotomos et al., 2022).

In legal diarization, the Oyez-derived SCOTUS corpus uses six cases for reference/development and 25 docket-17 cases for test, with about 58 minutes per case on average and about 51 minutes after VAD (Tumminia et al., 2021). In classrooms, the role-aware experiments use ClassBank with 44 hours and M-Powering Teachers with 6 hours, and evaluate both teacher-student diarization and all-speaker diarization with oracle speaker counts (Khan et al., 16 May 2025).

In medical RD, DoPaCo is an internal doctor-patient corpus with 1750 training hours, 26.5 validation hours, and 35.0 evaluation hours (Ghosh et al., 14 Jul 2025). In child-adult interaction modeling, Playlogue yields 16.20 h train, 5.07 h dev, and 6.75 h test after preprocessing, while ADOS-Mod3 yields 5.36 h train, 1.66 h dev, and 7.05 h test with strict speaker disjointness between train/dev and test (Xu et al., 25 Jan 2026).

For open-world audiovisual RD, SubtitleSD contains Chinese, Chinese-Hard, and English subsets with 130 videos, 92h35m total duration, and 1,054 distinct speakers (Huang et al., 17 Mar 2026). The Chinese-Hard subset is explicitly constructed to include multiple Chinese dialects and very large speaker diversity (Huang et al., 17 Mar 2026).

This domain fragmentation implies that benchmark comparability across RD papers is limited. A plausible implication is that progress is often measured within a role ecology—therapy, legal, classroom, clinical, or cinematic—rather than across a standardized task definition.

5. Evaluation metrics and empirical findings

DER remains the canonical diarization metric. In several papers it is defined as

$$\mathcal{L}_{RD} = \sum_{r_{t,u} \in \mathbf{a_r^*},\, r_{t,u}\neq -} -\ln \mathbb{P}(r_{t,u}\mid f^{RD}_{1:T}, g^{RD}_{1:u-1})$$0

or equivalently as the sum of false alarm, missed speech, and speaker confusion normalized by total reference speech (Khan et al., 16 May 2025, Dubey et al., 2016). RD work also uses role-aware transcript metrics. The doctor-patient ASR-guided system reports R-WDER, a role-based word diarization error rate in which doctor-hypothesized words can only match doctor reference words, patient-hypothesized words can only match patient reference words, and “other” can match one other reference speaker (Ghosh et al., 14 Jul 2025). The child-adult joint model reports mtWER, WER, AER, and DER, where

$$\mathcal{L}_{RD} = \sum_{r_{t,u} \in \mathbf{a_r^*},\, r_{t,u}\neq -} -\ln \mathbb{P}(r_{t,u}\mid f^{RD}_{1:T}, g^{RD}_{1:u-1})$$1

and

$$\mathcal{L}_{RD} = \sum_{r_{t,u} \in \mathbf{a_r^*},\, r_{t,u}\neq -} -\ln \mathbb{P}(r_{t,u}\mid f^{RD}_{1:T}, g^{RD}_{1:u-1})$$2

for role $$\mathcal{L}_{RD} = \sum_{r_{t,u} \in \mathbf{a_r^*},\, r_{t,u}\neq -} -\ln \mathbb{P}(r_{t,u}\mid f^{RD}_{1:T}, g^{RD}_{1:u-1})$$3 (Xu et al., 25 Jan 2026).

Empirical findings vary by formulation. In linguistically aided psychotherapy diarization, the audio-only baseline is 11.05% DER, the linguistically aided system with tagger segmentation reaches 7.71% DER with reference transcripts, and 8.37% DER with ASR transcripts; confidence-based profile selection further improves the ASR-transcript setting to 7.84% DER (Flemotomos et al., 2019). In multimodal constrained clustering, UCC improves from 1.38% DER to 1.31% DER, while TAL improves from 42.22% DER to 23.86% DER, indicating that role constraints are much more valuable in the harder multi-speaker podcast setting than in the already easy dyadic clinical setting (Flemotomos et al., 2022).

In judicial speech attribution, Reference-Dependent Speaker Verification achieves 13.8% DER for speakers covered by the reference audio library on a held-out test set, with $$\mathcal{L}_{RD} = \sum_{r_{t,u} \in \mathbf{a_r^*},\, r_{t,u}\neq -} -\ln \mathbb{P}(r_{t,u}\mid f^{RD}_{1:T}, g^{RD}_{1:u-1})$$4, $$\mathcal{L}_{RD} = \sum_{r_{t,u} \in \mathbf{a_r^*},\, r_{t,u}\neq -} -\ln \mathbb{P}(r_{t,u}\mid f^{RD}_{1:T}, g^{RD}_{1:u-1})$$5, and maximum DER $$\mathcal{L}_{RD} = \sum_{r_{t,u} \in \mathbf{a_r^*},\, r_{t,u}\neq -} -\ln \mathbb{P}(r_{t,u}\mid f^{RD}_{1:T}, g^{RD}_{1:u-1})$$6 (Tumminia et al., 2021). In classrooms, the best teacher-student result is 17.4% DER on MPT and the best all-speaker result is 44.4% DER, both with Robust-Large wav2vec 2.0 VAD + Whisper hybrid VAD (Khan et al., 16 May 2025).

In doctor-patient ASR-guided RD, the best proposed configuration ASR (CNN-2) + RD (RNN) gives validation R-WDER 6.7 and evaluation R-WDER 7.1, and RD-guided beam search slightly improves validation WER to 14.29 while preserving evaluation R-WDER 7.1 (Ghosh et al., 14 Jul 2025). In child-adult joint modeling, the proposed system achieves the best mtWER on both Playlogue and ADOS for both Whisper-small and Whisper-large. On ADOS with Whisper-large, it reports mtWER 21.7, WER 20.6, AER 1.1, and DER 18.4; on Playlogue with Whisper-large, it reports mtWER 34.3, WER 32.2, AER 2.1, and DER 42.6 (Xu et al., 25 Jan 2026).

Related streaming diarization work, although not explicitly role-labeled, is relevant to the temporal core of RD. Chronological self-training for real-time diarization reports that 1 second of enrollment is sufficient to reach over 95% accuracy and that average diarization error rates can be as low as 10% on 700 CALLHOME files across 6 languages (Padfield et al., 2022). This suggests that in role-aware streaming systems, label-efficient online adaptation may substantially reduce the initial enrollment burden.

Across studies, the principal lesson is consistent: improvements in role attribution depend not only on role modeling, but also on segmentation fidelity, VAD quality, transcript synchronization, and confidence calibration.

6. Limitations, misconceptions, and unresolved issues

A central limitation is that role and speaker are not universally interchangeable. The multimodal constraint paper explicitly states that it does not assume a strict one-to-one mapping between roles and speakers in general, because one speaker may play multiple roles or multiple speakers may share the same role (Flemotomos et al., 2022). By contrast, the linguistically aided psychotherapy system depends on a fixed known set of roles and a one-to-one mapping between roles and speakers (Flemotomos et al., 2019). Results obtained under the latter assumption do not automatically transfer to meetings, classrooms, or visual media.

Another limitation is dependence on side information. Some RD systems require transcripts or ASR output (Flemotomos et al., 2019, Flemotomos et al., 2022, Ghosh et al., 14 Jul 2025, Xu et al., 25 Jan 2026); some require a reference library of known judges (Tumminia et al., 2021); some require audiovisual anchors and subtitle lines (Huang et al., 17 Mar 2026). This side information can be highly effective, but it narrows generality and changes the task from generic open-set diarization to a more structured inference problem.

The literature also repeatedly cautions that several diarization papers are only indirectly related to RD. The two-channel RNN paper reformulates diarization as per-channel binary speech activity classification and “does not solve the harder diarization problem of discovering unknown speakers or assigning roles in a multi-speaker audio stream” (Sharma et al., 2020). The PLTL system provides diarization infrastructure for participation and turn-taking analysis but is “still not a direct speaker-role labeling system” (Dubey et al., 2016). USED is a unified speaker extraction plus generic speaker diarization model and “is not a speaker-role diarization model” because it does not predict semantic roles (Ao et al., 2023). Benchmarking diarization models without role labels is useful for the speaker-segmentation substrate of RD, but “good diarization is a prerequisite, but not sufficient” for role labeling (Lanzendörfer et al., 30 Sep 2025).

Several unresolved technical issues recur across domains. In classrooms, DER is strongly driven by VAD quality, and short student segments are particularly difficult because they are brief and quiet (Khan et al., 16 May 2025). In child-adult modeling, child speech is consistently harder than adult speech, and Playlogue remains noisy and boundary-sensitive (Xu et al., 25 Jan 2026). In doctor-patient ASR-guided RD, small-word deletions motivate explicit RD-guided blank suppression (Ghosh et al., 14 Jul 2025). In visual media, off-screen speech, cross-modal asynchrony, and very large speaker sets make the task fundamentally open-world (Huang et al., 17 Mar 2026).

A plausible synthesis is that RD is hardest when four factors coincide: non-unique role-to-speaker mappings, weak or noisy lexical evidence, short or overlapping turns, and absence of stable external anchors. Current work addresses these factors piecemeal—through constraints, synchronization, auxiliary heads, reference libraries, or multimodal registration—but no single formulation eliminates all of them simultaneously.