Audio-Visual Diarization

• Audio-visual diarization is a technique that fuses audio and video cues to accurately determine who spoke when in multiparty scenarios.
• It leverages multi-modal features such as vocal dynamics, lip movements, face recognition, and spatial cues, employing advanced fusion methods like cross-attention.
• Performance is measured using metrics like DER and cpCER across varied datasets, demonstrating substantial improvements over unimodal approaches.

Audio-visual diarization addresses the “who spoke when” problem by leveraging both auditory and visual signals to associate speech segments with speaker identities in multiparty conversational or media scenarios. Unlike unimodal diarization, which is limited by either audio ambiguity (noise, overlap, indistinguishable timbres) or visual artifacts (occlusion, low resolution, off-screen speakers), audio-visual diarization fuses complementary cues such as vocal tract dynamics and lip movement, face identity, spatial localization, and multi-modal synchronization. This approach is foundational to robust meeting transcription, speaker-attributed ASR, and in-the-wild video analytics, spanning diverse domains such as meetings, broadcast media, movies, daily-life recordings, and egocentric perspectives.

1. Problem Definition and Evaluation Metrics

Audio-visual speaker diarization (AVSD) formulates the diarization task as jointly inferring speaker boundaries and identities using synchronized audio and visual streams. The core objective is to partition audiovisual recordings into non-overlapping (or, in more advanced systems, possibly overlapping) segments, each labeled by a speaker index, answering “who spoke when.”

The principal metric is the Diarization Error Rate (DER):

$$\mathrm{DER} = \frac{\mathrm{FA} + \mathrm{MISS} + \mathrm{SPKERR}}{\mathrm{TOTAL}}$$

where FA is the total false alarm duration, MISS is missed speech, SPKERR is speaker attribution error, and TOTAL is the total annotated speech time. DER is typically computed without any forgiveness collar (i.e., all errors at segment boundaries are fully penalized), and fully accounting for overlapping speech, which is critical for realistic evaluation in multiparty scenarios (Wang et al., 2023, Gao et al., 20 May 2025, Xu et al., 2021, Cheng et al., 2024, Yin et al., 2023).

For diarization-and-recognition tasks (AVDR, “who spoke what when”), concatenated minimum-permutation character error rate (cpCER) is used, as in the CHiME-6 protocol, to jointly measure segmentation and recognition accuracy (Wang et al., 2023).

2. Datasets and Benchmark Scenarios

AVSD has been benchmarked across controlled environments (meeting rooms), broadcast/TV content, and unconstrained “in-the-wild” data:

• Meetings: Corpora such as AMI, AVDIAR, and MISP-Meeting (Gao et al., 20 May 2025, Mingote et al., 2024) provide multi-speaker conversational data with far-field audio and multi-camera streams, high speaker overlap (up to 57%), and reference headset tracks.
• Broadcast and TV: Datasets such as REPERE, ETAPE, RTVE, and MVAD represent TV shows with studio-quality audio/video and enrollment media for celebrity identification (Mingote et al., 2024, Bost et al., 2018).
• In-the-wild/Movies: AVA-AVD, MSDWild, VoxConverse, and Ego4D challenge systems with real background noise, diverse acoustic conditions, frequent off-screen speech, and rapid scene changes (Xu et al., 2021, Mingote et al., 2024, Min, 2023).
• Daily/egocentric recordings: Ego4D characterizes daily-life scenarios with egomotion, occlusions, and pervasive off-screen speakers (Min, 2022, Min, 2023).

Table: Characteristic Datasets for Audio-Visual Diarization

Corpus	Domain	# Speakers/clip	Overlap	Off-screen	Language	AV alignment	Notable Challenges
MISP-Meeting	Meetings	4–8	>50%	Few	Mandarin	Far-field	Noise, reverberation, lighting, overlap
AVA-AVD	Movies, in-the-wild	2–24	Any	~2.4/clip	Multilingual	Varied	Occlusion, shot changes, background music
Ego4D	Egocentric daily	2–10	Frequent	Yes	Multilingual	Mobile	Egomotion, off-screen speech
RTVE/REPERE	TV broadcasts	2–12	Moderate	Occasional	Spanish/French	Studio	Named-entity identification

In all evaluated settings, DER < 20% is achievable in controlled meetings, but can exceed 30% in unconstrained conditions due to occlusions, domain gaps, and reference errors (Mingote et al., 2024, Gao et al., 20 May 2025).

3. Model Architectures and Fusion Strategies

3.1 Audio-Visual Encoders

• Audio: 1D/2D CNNs (ResNet-34, ECAPA-TDNN), TCNs, or Transformers process log-Mel FBANKs or MFCCs, sometimes supplemented by dereverberation (e.g., NARA-WPE) or multi-channel beamforming (Gao et al., 20 May 2025, He et al., 2024, Wang et al., 2023).
• Visual: Lip/face encoders (ResNet, ArcFace, RetinaFace) extract per-frame or segment-level features; lip ROIs are common, as they probe vocal activity (Yin et al., 2023, Wang et al., 2023).
• Speaker Embeddings: i-vectors, x-vectors, or ECAPA-TDNNs learned on speaker-ID datasets, providing identity-aware features for fusion (Zhao et al., 2023, Zhang et al., 2023).
• Synchronization Branches: Some networks incorporate explicit synchronization modules (e.g., contrastive audio-video nets) to measure temporal correlation between modalities (He et al., 2024, Ding et al., 2020).

3.2 Fusion Mechanisms

• Concatenation: Early architectures concatenate audio and visual embeddings, followed by BLSTM/Transformer/Conformer back-ends (Zhang et al., 2023, Gao et al., 20 May 2025).
• Cross-Attention and Self-Attention: Modern systems employ cross-modal multi-head attention (audio↔visual) to dynamically align streams, followed by self-attention to capture conversational context (Li et al., 3 Jun 2025, Yin et al., 2023, He et al., 2024).
• Quality-Aware Fusion: Some frameworks compute frame-level quality scores for each modality; fusion weights are then adaptively recalibrated based on reliability, e.g., down-weighting the video stream during occlusions (He et al., 2024).
• Pairwise/Graph-Based Similarity: Segment-level or graph neural network (GNN) architectures define nodes as persons/segments and fuse spatial, temporal, and cross-modal edges for nodewise speech activity inference (Min, 2023, Yin et al., 2023).
• Two-Step and Masked Fusion: AFL-Net demonstrates performance gains via sequential cross-attention (audio+face, then lips), and random masking of modalities during training to model missing faces/lips (Yin et al., 2023).

Ablation studies consistently show that cross-attention and joint training of encoders/decoders yield substantial DER reductions (e.g., >3–4% absolute improvement), and masking improves robustness to dropped or occluded visual input (Yin et al., 2023, Zhao et al., 2023).

4. Pipeline Variants and End-to-End Modeling

Modular Pipelines

• Early pipelines decouple VAD, ASD, embedding extraction, and speaker clustering (e.g., SyncNet+lips, clustering of audio/visual partitions, or combinatorial matching) (Ding et al., 2020, Bost et al., 2018, Chung et al., 2020). Such systems often require pipelined fusion and post-processing (AHC, median filtering, or DOVER-Lap).

End-to-End Models

• Modern systems train sequence-to-sequence or binary classification networks to directly predict multi-speaker speech activity via permutation-invariant training or multi-output sigmoid heads (He et al., 2024, Li et al., 3 Jun 2025, Cheng et al., 2024). End-to-end models natively handle overlapping speech, propagate cross-modal uncertainty, and allow attention modules to align or recalibrate streams on degraded input.

AVDR (Diarization + Recognition)

• AVDR extends AVSD by requiring speaker-attributed fully transcribed output. Baseline systems segment audio/video with diarization hypotheses, feed aligned clips into AVSR modules (e.g., DNN-HMM with joint audio-visual MS-TCN features), and concatenate hypotheses for character error rate evaluation using permutation-invariant alignment (Wang et al., 2023).

5. Error Analysis and Robustness

Experimental results across multiple benchmarks reveal diagnostic error patterns:

• Visual Degradation: Far-field, dim, or occluded lip ROIs degrade visual VAD and AVSD performance (e.g., DER increases from 13.09% to >18% on poor-quality video (Wang et al., 2023, Yin et al., 2023)). Masking strategies and self-supervised pretraining can mitigate these effects (He et al., 2024, Yin et al., 2023).
• Acoustic Interference: TV noise, high reverberation, or noise bursts can cause up to 20% missed speech or speaker-attribution error in audio-centric branches. Beamforming and supervised separation are proposed solutions (Wang et al., 2023, Gao et al., 20 May 2025).
• Indistinguishable Speakers and Lip Peristalsis: Similar-timbre speakers confound audio clustering; non-phonatory lip motion (peristalsis) increases FA and SPKERR. Only multi-modal fusion can resolve most such ambiguities (Wang et al., 2023).
• Off-Screen Speakers: Robust off-screen handling is achieved by random masking, explicit “wearer” nodes (egocentric graphs), or off-screen embedding alignment (Yin et al., 2023, Min, 2023, Cheng et al., 2024).
• Overlaps and Turn-Taking: DER remains high (>18%) on high-overlap (>50%) meeting data. End-to-end diarization and TS-VAD variants natively support multi-label outputs and speaker alignment (He et al., 2024, Gao et al., 20 May 2025, Cheng et al., 2024).

6. Advanced and Emerging Methods

• Permutation-Invariant Training: Losses and evaluation protocols that minimize over all possible speaker-output assignments are essential for end-to-end systems (Li et al., 3 Jun 2025, Gao et al., 20 May 2025, Wang et al., 2023).
• Constraint-Based and Semantic Fusion: Methods now propagate both visual constraints (face tracks, talking-head alignment) and semantic (text/ASR) constraints using graph or joint propagation algorithms to refine affinity matrices before clustering, achieving further DER/JER reductions (Cheng et al., 2024).
• Self-Supervised and Pretrained Backbones: Leveraging models such as HuBERT and WavLM for audio and large-scale face/lip encoders for visual has been shown to improve both robustness and cross-domain transfer, particularly in limited-label scenarios (Zhao et al., 2023, Yin et al., 2023, He et al., 2024).

7. Challenges and Future Directions

Main technical frontiers include:

• Fully Exploiting Video: Visual cues are under-utilized in top-performing audio-dominant models, especially under clean audio. Cross-modal attention gating, multiple-camera or per-speaker view fusion, and dynamic modality weighting remain active research areas (Gao et al., 20 May 2025, Yin et al., 2023).
• Handling Severe Overlap and Off-Screen Speech: Advances in sequence-to-sequence TS-VAD, cross-modal graph nets, and speaker alignment modules are critical for highly overlapped or egocentric settings (Cheng et al., 2024, Min, 2023).
• Scalability Across Domains: Meta-learning, domain adaptation, and self-supervised pretraining are required for robust deployment across diverse settings (meetings, TV, in-the-wild, egocentric) (Mingote et al., 2024, He et al., 2024).
• Efficiency and Real-Time Constraints: Model compression, streaming variants of attention/Conformers, and lightweight graph nets must be addressed for online and embedded applications (Gao et al., 20 May 2025, Zhang et al., 2023).
• Integration With Speaker Naming: For TV archives, frameworks now assign celebrity identities by fusing face galleries and speaker embeddings, indicating a trend toward diarization with explicit name assignment (Mingote et al., 2024, Bost et al., 2018).
• Multi-Modal Clustering and Constraint Propagation: Joint propagation of audio, visual, and text/semantic constraints in affinity matrices further sharpens speaker partition boundaries (Cheng et al., 2024).

Audio-visual diarization continues to evolve from modular pipelines and late fusion systems to unified, end-to-end, context-sensitive networks capable of exploiting all available modalities—even under unconstrained, noisy, and ambiguous real-world conditions. Empirically, when both audio and visual streams are exploited in a principled, jointly-optimized fashion, DER reductions exceeding 50% over single-modality baselines have been routinely demonstrated in high-overlap and complex scenarios (Wang et al., 2023, Gao et al., 20 May 2025, Xu et al., 2021, Yin et al., 2023, Zhao et al., 2023, Cheng et al., 2024, He et al., 2024).