Cross-Talk Speech Reduction, by Separation, for Separation

Abstract: In conversational speech separation and recognition tasks, close-talk microphones are typically attached to each speaker during training data collection to capture near-field, close-talk mixture signals, in addition to using far-field microphones to record far-field mixture signals. Each such close-talk mixture exhibits a reasonably high energy level for the wearer and could intuitively serve as weak supervision for training far-field speech separation models directly on real-recorded far-field signals. However, they are not sufficiently clean for this purpose, as they often contain strong cross-talk speech from other speakers in addition to background noise. To address this, we propose cross-talk reduction (CTR), a task aiming to isolate the wearer's speech from each close-talk mixture, and a novel method called CTRnet, which can be trained directly on real-recorded pairs of close-talk and far-field mixtures to accomplish CTR. Building on CTRnet, we further propose pseudo-label based far-field speech separation (PuLSS), which uses CTRnet's estimated clean speech as pseudo-labels to train models for separating far-field mixtures. A key advantage of the proposed framework is that both CTRnet and PuLSS can be trained on real-recorded data from the target domain, addressing the generalization gap commonly observed when models are trained exclusively on simulated data. On the CHiME-6 dataset, our framework achieves state-of-the-art ASR performance under both oracle and estimated speaker diarization, surpassing all CHiME-{7,8} challenge submissions. To our knowledge, it is the first neural speech separation method that substantially outperforms guided source separation on real conversational "speech-in-the-wild" data.

• The paper introduces CTRnet, which uses blind deconvolution to reduce cross-talk without relying on clean sources, enabling robust separation for far-field ASR.
• It combines unsupervised, weakly-supervised, and pseudo-label-guided training to tackle domain mismatch and speaker permutation challenges.
• Experimental results on the CHiME-6 dataset show significant performance gains over GSS, setting new benchmarks in real conversational settings.

Cross-Talk Speech Reduction, by Separation, for Separation: A Technical Essay

Motivation and Problem Formulation

Traditional speech separation systems for ASR rely heavily on supervised deep learning trained with large-scale synthetic datasets, where clean speech sources are available by simulation. However, synthetic training concocts a persistent domain mismatch due to the inability to accurately model real acoustic environments, leading to substantial generalization gaps in real far-field conversational settings. A widely adopted workaround during training data collection involves attaching close-talk microphones to each speaker, yielding mixtures with a dominant energy level from the wearer but with significant cross-talk and ambient noise contamination. The paper addresses the unsuitability of these close-talk mixtures as direct supervision for far-field separation models and introduces cross-talk reduction (CTR) via separation—a task that aims to isolate the wearer's speech from its close-talk mixture without relying on clean sources.

The approach is formalized through blind deconvolution, treating the observed close-talk and far-field mixtures as convolutive sums of multiple reverberant sources, where both the true close-talk signals and their relative transfer functions (RTFs) are unknown. Estimation under unsupervised or weakly-supervised losses is performed by enforcing reconstructive consistency with the observed mixtures via mixture constraints.

CTRnet: Cross-Talk Reduction by Blind Deconvolution

CTRnet is a neural network architecture that, instead of being trained in a supervised manner on simulated clean/mixture pairs, is trained unsupervised or weakly-supervised on actual pairs of close-talk and far-field mixtures. The objective exploits the physical model: estimates of close-talk speech and corresponding RTFs are penalized if, when convolved, they fail to reconstruct the observed mixtures. This is achieved with a mixture-constraint loss that embodies the blind deconvolution formulation.

A weakly-supervised variant incorporates speaker activity timestamps, routinely annotated or auto-derived in real datasets, to mitigate over-/under-separation inherent in unsupervised clustering when the active speaker count in a segment is variable. Frame muting propagates activity masks onto neural outputs prior to loss calculation, and speaker-activity loss further regularizes the estimates.

The semi-supervised CTRnet extends training to both real-recorded mixtures (weakly-supervised loss) and simulated mixtures (supervised loss), combining benefits of domain coverage and realism. Additional architectural enhancements include explicit modeling of ambient noise and dereverberation mechanisms in the loss, leveraging neural outputs for both speech and noise, and applying reverberation filters to promote dereverberated estimates.

PuLSS: Pseudo-Label Based Far-Field Speech Separation

Leveraging close-talk speech estimates from CTRnet, the PuLSS model is trained to perform supervised separation on far-field mixtures using pseudo-labels derived by linearly filtering CTRnet outputs with RTFs estimated from the far-field microphones. Speaker permutation ambiguity—a major challenge in long-form speech separation—is resolved by conditioning PuLSS input features on speaker activity timestamps, achieving output channel consistency without the need for PIT or block-level reconciliation.

The loss functions for PuLSS comprise a direct pseudo-label matching term and an additional filtered output loss that can approximate the original close-talk estimates through convolutive prediction, penalizing both gain and content errors. Simulated mixtures with access to clean sources are also incorporated in a semi-supervised joint loss.

Inference with PuLSS can utilize either oracle or diarization-estimated speaker activity timestamps for conditioning, and experiments demonstrate robust performance with both.

Experimental Validation and Results

Experiments are executed on the CHiME-6 dataset, which presents unconstrained, highly challenging dinner-party conversational speech with severe cross-talk, signal degradation, and non-trivial environmental noise. Block-wise training and inference strategies ensure tractable processing on long-form audio.

Strong numerical results are reported:

• CTRnet achieves cpWER of 19.5% on close-talk mixtures, outperforming the unprocessed baseline and GSS signal processing separation methods.
• PuLSS reaches 19.5% cpWER under oracle diarization with a fine-tuned Parakeet ASR backend, surpassing prior challenge-best systems and GSS by a substantial margin.
• With estimated diarization, PuLSS attains a tcpWER of 28.5% versus 33.5% for GSS, demonstrating robustness to diarization quality and generalization beyond oracle conditions.

The framework systematically outperforms GSS—which has been the dominant frontend for distant ASR in benchmarks such as CHiME-{7,8}—establishing the effectiveness of domain-adapted pseudo-label training with CTRnet outputs.

Practical and Theoretical Implications

This research sets a new precedent for neural speech separation in real conversational scenarios. The technical innovations—in particular, blind deconvolution-based cross-talk reduction, domain-adapted pseudo-label generation, and the conditioning of separation models on timestamp features—address both the lack of annotated clean sources for real data and the permutation challenges in continuous separation.

Practically, the system enables training on true acoustic mixtures from the deployment domain, directly circumventing generalization failures observed with synthetic training. Theoretical advancements include scalable unsupervised and semi-supervised separation in non-ideal domains, improved modeling of noise and reverberant signal components, and novel use of data annotation artifacts for input conditioning.

Future research directions identified include end-to-end joint training with downstream ASR models, deployment in broader conversational domains beyond CHiME-6, and adaptation to unknown- or variable-number speaker settings. Exploring more flexible diarization integration and direct training under estimated activity timestamps remains open.

Conclusion

The paper presents a two-stage framework for robust, domain-matched conversational speech separation: CTRnet for cross-talk reduction by blind deconvolution, and PuLSS for supervised far-field separation using pseudo-labels derived from CTRnet estimates. Empirical evaluation demonstrates substantial improvements over state-of-the-art signal processing and neural baselines on real conversational speech, establishing new benchmarks for ASR accuracy and separation fidelity in unconstrained environments. The methodology robustly generalizes across diarization quality and is extensible to varied domain scenarios.