Valid and Comparable Evaluation Protocols Across Target Speaker Extraction Cues

Establish evaluation protocols that reveal target-selection failures and enable controlled comparisons across audio, visual, spatial, textual, and neural target speaker extraction cues without eliminating modality-specific requirements.

Background

Target speaker extraction systems are evaluated using heterogeneous metrics and datasets. Signal-level measures such as SI-SDR require isolated target references, which are often unavailable in natural recordings, while perceptual, speaker-identity, ASR, and non-intrusive metrics measure different aspects of performance.

Comparisons across cue types are confounded by differences in datasets, acoustic conditions, sensing requirements, backbone capacity, and training data. The paper explicitly identifies the unresolved issue as developing protocols that expose target confusion and permit fair, controlled cross-cue comparison while preserving the requirements specific to each modality.