Position-Aware Target Speaker Extraction for Long-Form Multi-Party Conversations: A Diarization-Free Framework for ASR

Abstract: In long-form multi-party conversations, highly imbalanced speaker activity and frequent overlap make it difficult to identify "who spoke when and what". Sliding-window continuous speech separation (CSS) mitigates sparse supervision, but often suffers from cross-window speaker inconsistency and residual crosstalk, which in practice requires diarization for reliable speaker attribution. Motivated by the stability of speakers' directions of arrival (DOAs) in meetings, we propose PATSE, a multi-channel Position-Aware Target Speaker Extraction front-end that uses DOA as a spatial prior to directly extract the speech of each target speaker. PATSE combines a DOA-guided spatial encoder and conditioner to generate speaker-attributed streams, from which speaker activity can be inferred via simple post-processing (e.g., VAD) without explicit diarization. Experiments on both replayed and real conversations show consistent ASR gains outperforming CSS and diarization-based pipelines.

• The paper introduces PATSE, a DOA-conditioned extraction framework that directly outputs target speaker streams, bypassing conventional diarization.
• It combines multi-channel encoding, spatial encoding, and FiLM-based conditioning to manage overlapping speech and imbalanced speaker activity.
• Experiments on LibriReplay-DOA and TEIDAN demonstrate significant WER and DER reductions compared to traditional diarization and separation methods.

Position-Aware Target Speaker Extraction for Long-Form Multi-Party Conversations

Introduction and Context

Speaker-attributed transcription in multi-party, long-form conversational scenarios is a well-established challenge, especially given the prevalence of overlapping speech, highly imbalanced speaker activity, and the demand to robustly determine "who spoke when and what." Conventional continuous speech separation (CSS) approaches, typically based on sliding-window source separation and post-hoc identity linking, are hampered by cross-window speaker identity drift and residual crosstalk, making accurate diarization a prerequisite for high-fidelity speaker-attributed automatic speech recognition (ASR). Existing diarization pipelines and embedding-based target speaker extraction (TSE) methods are limited by unreliable boundary detection in overlap, the need for enrollment audio, and embedding non-stationarity under real-world conversational dynamics.

To overcome these limitations, the paper introduces the Position-Aware Target Speaker Extraction (PATSE) framework, which leverages direction of arrival (DOA) as a stable spatial prior. PATSE directly produces target-conditioned speaker streams, eliminating both the diarization bottleneck and the cross-output crosstalk inherent to slot-based separation paradigms. The significance of PATSE is underscored through rigorous experimentation on both a newly released real-room dataset (LibriReplay-DOA) and the real-world TEIDAN conversation corpus, showing substantial and consistent ASR and diarization gains.

PATSE Framework and Methodological Advances

PATSE is a multi-channel target speaker extraction front-end that conditions separation directly on target DOA priors. The framework decomposes into several interacting components: a multi-channel audio encoder with feature fusion (MCFF), a spatial encoder incorporating both observed and theoretical phase cues, a FiLM-based spatial conditioner for target-sensitive modulation, and a separation backbone (TIGER). This architecture is depicted schematically in Figure 1.

Figure 1: Overall Architecture of the PATSE Framework.

Multi-Channel Audio Encoding

For $M$-channel audio input, each channel is independently encoded into band-partitioned feature maps. The MCFF module integrates per-channel features into a global single-stream representation using a transform–average–concatenate pipeline. This ensures permutation invariance and robustness to microphone channel count.

Target-Specific Spatial Encoding

PATSE’s spatial encoder computes, for each microphone pair, the Interaural Phase Difference (IPD) and the Theoretical Phase Difference (TPD) according to the hypothesized target DOA. The Phase Similarity Feature (PSF) quantifies the alignment between the observed mixture and the theoretical expectation. Stacking all microphone pairs yields a comprehensive spatial profile that is refined by self-attention layers within frequency bands and aggregated into final target-specific features.

FiLM-Based Spatial Conditioning

A feature-wise linear modulation (FiLM) mechanism modulates the encoded audio features using parameters generated from the target’s spatial representation. This enables dynamic conditioning of the entire separation pipeline on the current spatial query, maximizing target-specific extraction without relying on static enrollment or unreliable embeddings.

Efficient Activity-Aware Supervision

Training employs an activity-aware loss, applying a residual log-energy objective on silent regions and an SNR-based objective on active regions, facilitating balanced learning under extreme activity imbalance and silence.

Corpus Construction: LibriReplay-DOA Dataset

Recognizing the lack of real-room, DOA-annotated datasets, the authors introduce LibriReplay-DOA, where multi-party LibriSpeech segments are replayed through loudspeakers at fixed physical azimuths and recorded with a circular microphone array. The dataset systematically varies the target–interferer angular separation (15°, 45°, 90°, 120°) and overlap ratio (four bins from 0% up to 100%) to probe challenging spatial and conversational edge cases. This spatial design is summarized in Figure 2.

Figure 2: Four Spatial Configurations for LibriReplay-DOA.

Experimental Evaluation

Baseline Systems and Configuration

Comprehensive baselines include traditional DOA-guided beamforming (DSB+Gate), FastMNMF blind separation, diarization-then-separation (Sortformer+GSS), and slot-based CSS frameworks (FasNet-TAC and TIGER backbones). All CSS baselines utilize oracle speaker assignment for evaluation, providing an inflated performance upper bound. Downstream recognition is performed using Whisper Large-v3, and segmentation employs Silero-VAD for non-diarization pipelines.

LibriReplay-DOA Results

Across speaker–interferer angular separations and all overlap ratios, PATSE delivers consistent and marked reductions in WER relative to all baselines. For example, under the most challenging configuration (15° target–interferer angle and 75–100% overlap), PATSE (PT+FT) attains a WER of 22.8%, outperforming CSS (TIGER: 49.1%) and Sortformer+GSS (56.5%). Notably, even with oracle speaker tracks, CSS performance is significantly lower than PATSE, underscoring the inherent limitations of slot-based separation and the structural advantage of direct target conditioning. PATSE demonstrates a strong benefit in avoiding crosstalk and permutation errors by extracting one clean stream per spatial target. FastMNMF only becomes weakly competitive in scenarios with extreme overlap, which are rare in practice.

TEIDAN (Real-World Conversation) Results

On the TEIDAN corpus—capturing real triadic English conversation with high spontaneous overlap—PATSE achieves the lowest WER (20.5%) and substantially outperforms all baselines in diarization error rate (DER: 13.8% vs. >28% for others). Critically, this demonstrates that PATSE's spatial prior conditioning remains robust under real conversational variability and does not degrade under naturalistic activity patterns and speaker spatial layouts.

Implications and Future Directions

The PATSE framework ushers in a shift from identity-based or diarization-dependent pipelines to direct, spatially-informed target extraction for multi-party ASR. The practical implication is a reduction in system complexity, since speaker activity and attribution can be inferred via lightweight post-processing (e.g., voice activity detection) rather than explicit diarization or identity tracking modules. The theoretical implication is the decoupling of speaker and source models: spatial priors reduce the burden on learned speaker representations and naturally align with the stationary behavior common in conversational meetings.

Looking forward, possible extensions include: (1) leveraging multimodal spatial priors, such as fusing vision-based DOA estimation when microphone geometry is imperfect; (2) investigating robustness to varying array geometry and reverberant environments; (3) adapting the conditioning scheme for moving or dynamic speakers; and (4) synthesizing spatial- and content-based priors for enhanced target speaker modeling in highly dynamic and unconstrained scenarios.

Conclusion

PATSE represents an effective, fully diarization-free front-end for multi-party conversation transcription. By introducing DOA-conditioned target speaker extraction and releasing a real-room DOA-benchmarked corpus, the study demonstrates both theoretical and empirical superiority over diarization, slot-based CSS, and separation pipelines—yielding strong gains in ASR and diarization under both controlled and real-world conditions. The approach is poised for practical adoption in meeting transcription, real-time dialogue systems, and robust multi-party ASR deployments.