Breaking the Quality--Intelligibility Trade-off in Streaming Target Speaker Extraction via Deep-Feature-Anchored Preference Optimization

Abstract: Generative streaming models for Target Speaker Extraction (TSE) commonly exhibit a quality--intelligibility trade-off, wherein naive optimization for perceptual audio quality tends to degrade speech intelligibility, and conversely. We reveal that this trade-off arises not from the constraints of streaming architectures, but from an inappropriate choice of optimization anchor. Directly optimizing against audio quality metrics induces catastrophic reward hacking, where content critical to pronunciation and intelligibility is systematically erased to maximize a proxy score. To break this bottleneck, we propose two complementary improvements: an enlarged Conformer convolution kernel for richer local spectro-temporal modeling, and WavLM-anchored Direct Preference Optimization (DPO) fine-tuning strategy. DPO preference pairs are ranked by WavLM cosine similarity, a deep acoustic feature encoding both phonetic structure and speaker identity, providing an optimization anchor that resists hacking. Under a 560 ms streaming chunk size, the proposed method achieves a 10.9% relative intelligibility improvement (word error rate: 0.138 to 0.123), with marginal simultaneous gains in audio quality and speaker similarity.

• The paper introduces a deep-feature-anchored DPO strategy that mitigates the quality-intelligibility trade-off in streaming target speaker extraction.
• It employs an enlarged Conformer encoder and WavLM similarity to optimize both perceptual quality (DNSMOS) and transcription clarity (WER).
• Experimental results on the Libri2Mix corpus show significant improvements in WER, perceptual quality, and speaker similarity compared to baseline models.

Breaking the Quality-Intelligibility Trade-off in Streaming Target Speaker Extraction with Deep-Feature-Anchored DPO

Introduction

Target Speaker Extraction (TSE) focuses on isolating the speech of a specific target speaker from multi-speaker mixtures using a reference utterance. Recent advances leverage generative architectures, yielding higher perceptual quality than traditional discriminative mask-based models. However, these generative methods, when adapted for streaming, exhibit a quality-intelligibility trade-off: improving perceptual audio quality (e.g., via DNSMOS) degrades intelligibility (WER), and vice versa. This work identifies the root cause not as a limitation of streaming or autoregressive inference, but as reward hacking induced by misaligned optimization anchors for Direct Preference Optimization (DPO). The paper proposes two complementary improvements: (1) enlarging the Conformer encoder’s convolution kernel for richer spectro-temporal modeling, and (2) introducing a DPO strategy anchored on deep speech features (WavLM similarity), which proves robust to proxy metric exploitation and maintains high intelligibility without sacrificing perceptual quality.

Figure 1: Overview of the StarTSE architecture with the proposed DPO alignment strategy. The semantic pathway (Shared Conformer Encoder and SELM, highlighted in blue) receives gradient updates during DPO, while the acoustic pathway (ARLM and Codec Decoder, shaded in gray) remains frozen.

Technical Approach

The core architecture builds on StarTSE, leveraging a strictly causal streaming encoder-decoder framework with an explicit division into semantic (discrete token generation) and acoustic (codec embedding and waveform reconstruction) pathways. The semantic pathway comprises a shared Conformer encoder (with a variable convolution kernel) and an autoregressive SELM transformer for chunk-wise token prediction. The acoustic refinement module (ARLM) enhances the SELM outputs, followed by a codec decoder with historical context refinement for streaming smoothness.

Key innovations:

• Conformer kernel enlargement: Setting the convolution kernel $k=15$ enables modeling of longer local contexts, directly reducing WER and improving spectral representation without loss of real-time responsiveness (RTF $\ll$ 1.0 on modern GPUs).
• DPO preference pair construction: For each input mixture, 16 stochastic candidates are generated and independently ranked by DNSMOS (quality), WER (intelligibility), and WavLM cosine similarity (deep-feature alignment).

  Figure 2: Preference pair construction pipeline: 16 candidates per mixture are evaluated on DNSMOS, WER, and WavLM. Each metric defines its own preferred (y_w) and non-preferred (y_l) outputs for DPO training.

• Semantic pathway alignment: During DPO, only the encoder and SELM weights are updated, preventing semantic drift and isolating the effect of preference optimization on intelligibility and quality.
• DPO loss: The standard DPO contrastive objective is used, with experiments varying the KL constraint $\beta$ to control policy deviation from the pretrained reference.

Experimental Evaluation and Results

The framework is evaluated on the Libri2Mix corpus, with DNSMOS, WER (via Whisper LargeV3), speaker similarity (WavLM and WeSpeaker), and ISR (inference success rate) as metrics. Baselines include discriminative systems (SpEx+, WeSep) and generative models (TSELM-L, LauraTSE) in both offline and streaming/casual modes.

Baseline and DPO Results

• StarTSE ($k=15$ Conformer) serves as the reference streaming generative model, achieving WER 0.129 and OVL 3.121 with 100% ISR.
• DPO$_{\text{DNSMOS}}$: Optimizing for perceptual quality leads to catastrophic reward hacking—WER collapses (up to 0.316), as the network smooths out transient phonetic features necessary for intelligibility.
• DPO$_{\text{WER}}$: Optimizing for transcription improves neither WER nor perceptual quality further, revealing a narrow/saturated optimization signal.
• DPO$_{\text{WavLM}}$: WavLM-anchored optimization breaks the trade-off, yielding simultaneous improvement in both WER (0.123, a 10.9% relative reduction) and perceptual quality (OVL 3.122). Speaker similarity also improves.

Figure 3: Mel spectrograms of the same utterance. Ground truth (a), DPO$_{\text{WavLM}}$ (b) matches both low and mid-frequency cues; DPO$_{\text{DNSMOS}}$ (c) suppresses essential content, yielding smoothed but unintelligible speech (reward hacking).

Mechanism Analysis

The comparative analysis across DPO preference anchors highlights that only deep speech feature alignment (WavLM) avoids the pitfalls of perceptual smoothing and semantic drift. When DPO is anchored to DNSMOS, the model exploits the metric’s blind spots, producing audio that is smooth and artifact-free but lacks necessary linguistic information. WER-based anchors, being non-differentiable and narrow, saturate quickly and do not yield further improvement while also degrading subjective quality. Deep speech representations like WavLM encode both phonetic and speaker information and serve as an effective regularizer, naturally balancing the competing objectives.

Architectural Ablation

Systematic ablation reveals that naive Conformer integration (small kernel $k=3$) is suboptimal, degrading both WER and DNSMOS relative to the original Transformer encoder. However, increasing the kernel size to $\ll$0 enables the network to capture syllabic and segmental structures critical in challenging mixtures, leading to a significant WER reduction without quality compromise.

Implications and Future Directions

The results decisively show that the quality-intelligibility trade-off in streaming generative TSE is not an architectural limitation, but a byproduct of reward design and optimization anchor selection. Deep-feature-anchored DPO (specifically, WavLM cosine similarity) yields Pareto improvements—simultaneously advancing perceptual, intelligibility, and speaker-similarity metrics under real-time streaming constraints.

Practical implications include more reliable real-time ASR enhancement, improved user experience in voice-activated systems, and robustness to unseen acoustic conditions. On the theoretical side, this establishes deep-feature alignment as a robust anchor for preference-based generative modeling in speech processing, providing a principled countermeasure against reward hacking.

Future work should address:

• Extending these techniques to lower-latency streaming regimes (e.g., <200ms chunk size)
• Generalization to more diverse and realistic multi-speaker, noisy, or reverberant environments
• Multi-objective DPO with dynamic weighting to explicitly control the balance between perceptual quality, intelligibility, and identity, rather than relying on a single deep-feature metric

Conclusion

This study demonstrates that deep-feature-anchored DPO—leveraging self-supervised representations such as WavLM—enables the construction of streaming TSE systems that avoid the long-standing quality-intelligibility trade-off. By directly aligning the semantic pathway to salient deep speech features while freezing the acoustic module, the approach yields strong and simultaneous gains in WER, perceptual quality, and speaker similarity, as evidenced by both quantitative metrics and spectrogram analysis. This establishes both a practical recipe for robust streaming TSE and a theoretical foundation for future research in model alignment using rich neural feature spaces.