Pseudo-Label Far-Field Speech Separation (PuLSS)

• The paper introduces a two-stage real-data training strategy that uses close-talk estimates transformed into pseudo-labels for supervised far-field separation.
• It employs CTRnet for cross-talk reduction and RTF projection to generate acoustically accurate pseudo-labels, achieving notable cpWER improvements on CHiME-6.
• The method addresses domain mismatch by training on real mixtures, effectively outperforming traditional guided source separation techniques.

Searching arXiv for the cited PuLSS and closely related papers to ground the article in current sources. Pseudo-Label Based Far-Field Speech Separation (PuLSS) is a two-stage real-data training strategy for far-field speech separation in which estimated close-talk speech is converted into pseudo-labels for supervised training on real-recorded far-field mixtures. In the formulation introduced in "Cross-Talk Speech Reduction, by Separation, for Separation" (Wang et al., 19 May 2026), PuLSS is the second stage of a framework whose first stage, CTRnet, performs cross-talk reduction on each speaker’s close-talk microphone recording. The resulting close-talk estimates are projected into the acoustics of a reference far-field microphone and then used as training targets for a far-field separator. The framework is motivated by the supervision gap in real conversational recordings: close-talk microphones provide higher-SNR views of the wearer than far-field arrays, but the close-talk recordings are still mixtures rather than clean labels. On CHiME-6, the framework achieves state-of-the-art ASR performance under both oracle and estimated speaker diarization, and is presented as the first neural speech separation method that substantially outperforms guided source separation on real conversational "speech-in-the-wild" data (Wang et al., 19 May 2026).

1. Problem setting and rationale

PuLSS addresses a specific failure mode of conventional far-field separation pipelines: models trained on simulated mixtures with clean sources often do not generalize well to real recordings because of domain mismatch. The motivating observation is that conversational training corpora frequently contain both close-talk microphones attached to speakers and far-field microphone arrays in the room. The close-talk channels already contain the wearer’s speech at a much higher SNR than the far-field mixture, but they also contain cross-talk speech from other speakers and background noise, so they cannot be used directly as supervised targets (Wang et al., 19 May 2026).

The acoustic setting is described in the STFT domain by

$$Y_{a}(t, f) = \sum_{c=1}^{C} X_{a}(c,t,f) + V_{a}(t,f),$$

$$Y_{p}(t, f) = \sum_{c=1}^{C} X_{p}(c,t,f) + V_{p}(t,f),$$

where $Y_a$ and $Y_p$ are the observed close-talk and far-field mixtures, $X_a(c)$ and $X_p(c)$ are speaker $c$’s reverberant contributions at the corresponding microphones, and $V_a$, $V_p$ are noise terms (Wang et al., 19 May 2026). The critical point is that the close-talk recordings are themselves mixtures. PuLSS therefore does not treat close-talk audio as ground truth; instead, it treats it as a source of weak supervision that must first be cleaned.

This design distinguishes PuLSS from purely simulated supervised separation and from weakly supervised methods that operate only at the mixture level. A plausible implication is that PuLSS is best understood as a supervised separator trained with learned, acoustically projected pseudo-targets rather than with direct clean labels.

2. CTRnet as the prerequisite pseudo-label generator

PuLSS depends on CTRnet, which is introduced as a cross-talk reduction task aiming to isolate the wearer’s speech from each close-talk mixture. The paper reformulates the close-talk and far-field mixtures using linear filtering with a close-talk speech variable $Z(c)$:

$$Y_{p}(t, f) = \sum_{c=1}^{C} X_{p}(c,t,f) + V_{p}(t,f),$$0

$$Y_{p}(t, f) = \sum_{c=1}^{C} X_{p}(c,t,f) + V_{p}(t,f),$$1

CTR is then posed as a blind deconvolution problem in which both the speech estimates $$Y_{p}(t, f) = \sum_{c=1}^{C} X_{p}(c,t,f) + V_{p}(t,f),$$2 and the filters $$Y_{p}(t, f) = \sum_{c=1}^{C} X_{p}(c,t,f) + V_{p}(t,f),$$3 are chosen so that reconstructed close-talk and far-field mixtures match the observations (Wang et al., 19 May 2026).

CTRnet takes all close-talk mixtures as input and outputs estimated close-talk speech signals $$Y_{p}(t, f) = \sum_{c=1}^{C} X_{p}(c,t,f) + V_{p}(t,f),$$4. Its unsupervised training signal is a mixture-consistency objective,

$$Y_{p}(t, f) = \sum_{c=1}^{C} X_{p}(c,t,f) + V_{p}(t,f),$$5

with close-talk and far-field consistency terms computed after estimating linear filters by forward convolutive prediction (FCP). The paper also introduces a more robust weighting term based on a 90th percentile statistic rather than a maximum, to reduce sensitivity to impulsive clicks (Wang et al., 19 May 2026).

A practical complication is that the number of active speakers varies over time. To mitigate over-separation and under-separation, the framework uses speaker-activity timestamps $$Y_{p}(t, f) = \sum_{c=1}^{C} X_{p}(c,t,f) + V_{p}(t,f),$$6 as weak supervision. The outputs are muted according to

$$Y_{p}(t, f) = \sum_{c=1}^{C} X_{p}(c,t,f) + V_{p}(t,f),$$7

and a speaker-activity penalty is added:

$$Y_{p}(t, f) = \sum_{c=1}^{C} X_{p}(c,t,f) + V_{p}(t,f),$$8

The combined weakly supervised loss is

$$Y_{p}(t, f) = \sum_{c=1}^{C} X_{p}(c,t,f) + V_{p}(t,f),$$9

CTRnet can also be trained semi-supervisedly on both simulated and real-recorded mixtures, and the paper further extends it with explicit noise prediction and a delayed self-filter term for reverberation modeling (Wang et al., 19 May 2026).

Empirically, CTRnet establishes whether the pseudo-label pipeline is viable. On CHiME-6, the best close-talk estimation reported is 15.0% cpWER with CTRnet + oracle diarization on the test set, compared with 19.5% for unprocessed close-talk mixtures. The paper also reports that supervised CTRnet trained only on simulation performs badly on real data, whereas weakly supervised and semi-supervised variants perform better, especially when using averaged binaural close-talk input and optional noise or reverberation modeling (Wang et al., 19 May 2026).

3. Pseudo-label construction and PuLSS supervision

Once CTRnet produces $Y_a$0, PuLSS converts these close-talk estimates into pseudo-labels at a reference far-field microphone rather than using the close-talk estimates directly. For speaker $Y_a$1 and reference far-field microphone $Y_a$2, a short RTF filter is estimated by

$Y_a$3

where $Y_a$4 is an estimated time delay between close-talk and far-field recordings. The delay is found by enumeration over $Y_a$5 (Wang et al., 19 May 2026).

The pseudo-label is then defined as

$Y_a$6

This construction is central: the target is the close-talk estimate projected into the acoustics of the far-field reference microphone, not the raw close-talk estimate itself (Wang et al., 19 May 2026).

PuLSS trains a supervised far-field separator on real-recorded far-field mixtures using these pseudo-labels. Its primary loss is

$Y_a$7

where $Y_a$8 is the separated output for speaker $Y_a$9. Because the RTF-projected pseudo-labels can be imperfect, the paper adds a close-talk-estimation auxiliary loss,

$Y_p$0

with $Y_p$1 estimated by another FCP-style regression. The combined criterion is

$Y_p$2

The paper explicitly states that this auxiliary term is important because pseudo-labels created by an RTF filter can be lower quality than the CTRnet estimate itself (Wang et al., 19 May 2026).

Permutation is not handled with PIT. Instead, speaker-activity timestamps are used to create masked magnitude spectrogram features, $Y_p$3, which are concatenated with the real and imaginary components of the mixture and fed to the network. Each output channel is thereby conditioned to correspond to a particular speaker (Wang et al., 19 May 2026).

4. Training regimes, architecture, and operational pipeline

The complete training pipeline is staged. First, real conversational data are recorded with close-talk microphones on each speaker and far-field microphone arrays. Second, CTRnet is trained on paired close-talk and far-field mixtures in unsupervised, weakly supervised, or semi-supervised form. Third, CTRnet estimates close-talk speech for every training segment. Fourth, an RTF is estimated from each close-talk estimate to a reference far-field microphone. Fifth, pseudo-labels are generated at the far-field microphone. Sixth, PuLSS is trained on far-field mixtures using these pseudo-labels, optionally with the CTE loss and optionally with simulated-data supervision (Wang et al., 19 May 2026).

At inference, CTRnet is not used. The separator takes a far-field mixture block together with speaker-activity timestamps, which may come from oracle diarization or from an external diarization system, and outputs separated speech in one forward pass for downstream ASR (Wang et al., 19 May 2026). This is a common point of confusion: PuLSS uses CTRnet to generate training pseudo-labels, but CTRnet itself is not required during deployment.

PuLSS also has a semi-supervised form in which real-recorded mixtures use pseudo-labels while simulated mixtures use clean direct-path speech:

$Y_p$4

The overall objective is

$Y_p$5

This mixed-data regime is intended to combine the realism of pseudo-labeled real mixtures with the availability of clean supervision in simulation (Wang et al., 19 May 2026).

For both CTRnet and PuLSS, the paper uses TF-GridNet with complex spectral mapping. The reported STFT settings are 16 ms window and 8 ms hop for CTRnet, and 32 ms window and 16 ms hop for PuLSS. Optimization uses Adam, batch size 2, learning rate $Y_p$6 halved on plateau, and gradient clipping with L2 norm 1.0 (Wang et al., 19 May 2026).

5. Evaluation on CHiME-6 and reported performance

The principal evaluation is on CHiME-6, described as a hard real-recorded dinner-party corpus with 4 speakers per session, each speaker wearing a binaural close-talk microphone, and far-field recordings from 6 Kinect devices, each with 4 microphones. Sessions last 120–150 minutes and include reverberation, noise, clipping, frame drops, sync errors, and moving speakers. For training, sessions are split into 12-second blocks with 11-second overlap, yielding 123,339 blocks (Wang et al., 19 May 2026).

For semi-supervised training, the paper also constructs simulated data that follow real overlap patterns, with clean speech sampled from LibriSpeech and EARS, noise from FSD50K, CHiME-6 noise, and REVERB air-conditioning noise, room acoustics simulated with Pyroomacoustics, close-talk microphones placed at 0.2–0.5 m from speakers, up to 4 noise sources, and reverberation time sampled from 0.2–0.7 s (Wang et al., 19 May 2026).

Under oracle diarization, the reported far-field ASR results are:

• Unprocessed mixture: 62.6% cpWER
• GSS (24-channel): 38.5% cpWER
• Supervised far-field separation trained on simulation: 49.0% cpWER
• PuLSS with real-recorded pseudo-labels: 35.4% cpWER
• PuLSS + CTE: 32.2% cpWER
• PuLSS + simulated-data training + CTE: 31.3% cpWER
• Best PuLSS variant with stronger DNN / ASR adaptation: 30.0% cpWER with default ASR and 19.5% cpWER with fine-tuned Parakeet ASR (Wang et al., 19 May 2026)

The strongest oracle-diarization result, 19.5% cpWER, slightly improves over the prior best challenge system, USTC: 19.8% cpWER, and substantially beats GSS under the same fine-tuned ASR setup, reported as PuLSS: 19.5% vs GSS: 29.7% cpWER (Wang et al., 19 May 2026). Under estimated diarization, PuLSS also remains competitive, with 31.7% tcpWER using STCON diarization and 28.5% tcpWER using USTC diarization (Wang et al., 19 May 2026).

These results are the basis for the paper’s principal claim that PuLSS is the first neural speech separation method to substantially outperform guided source separation on real conversational speech-in-the-wild. A plausible interpretation is that the decisive factor is not only the separator architecture, but the ability to train the separator directly on real-recorded target-domain data.

6. Relation to adjacent methods, assumptions, and limitations

PuLSS belongs to a broader family of methods that replace unavailable clean labels with weak or pseudo supervision, but its mechanism is distinct. In "Mixture to Mixture: Leveraging Close-talk Mixtures as Weak-supervision for Speech Separation" (Wang, 2024), M2M uses close-talk mixtures as weak supervision for far-field separation through a mixture-constraint loss after FCP-based filtering. M2M does not generate source pseudo-labels; instead, it enforces that filtered speaker estimates reconstruct both close-talk and far-field mixtures. In "ctPuLSE: Close-Talk, and Pseudo-Label Based Far-Field, Speech Enhancement" (Wang, 2024), the pseudo-label source is a close-talk enhancement model, CTSEnet, whose outputs supervise far-field enhancement on paired real recordings. By contrast, PuLSS uses a dedicated cross-talk reduction stage, then projects the estimated close-talk speech into the far-field acoustic domain before training a separator. In "An Adapter based Multi-label Pre-training for Speech Separation and Enhancement" (Wang et al., 2022), pseudo-labeling is moved earlier in the pipeline into HuBERT pre-training via multi-pseudo-label masked speech prediction; that approach is complementary rather than equivalent, because it improves upstream representations rather than directly training a separator.

Several misconceptions are explicitly contradicted by the PuLSS formulation. Close-talk recordings are not treated as clean labels; they are mixtures that often contain strong cross-talk speech and background noise (Wang et al., 19 May 2026). PuLSS is not unsupervised at the separation stage; it is a supervised far-field separator trained on pseudo-labels generated by CTRnet (Wang et al., 19 May 2026). CTRnet is not needed at inference for PuLSS; it is a training-time pseudo-label generator (Wang et al., 19 May 2026). The framework also does not rely on PIT for output assignment; it uses speaker-activity timestamps to resolve permutation (Wang et al., 19 May 2026).

The paper notes several limitations. Non-verbal wearer sounds such as breathing, chewing, and laughing may be preserved by CTRnet but may not be captured by far-field microphones or annotated by diarization, causing mismatch. The framework assumes a fixed maximum number of speakers $Y_p$7 in the processing block. The experiments are focused on CHiME-6, although the method is described as extendable to other corpora with paired close-talk and far-field recordings. The work also does not explore training PuLSS with estimated diarization during training or end-to-end joint fine-tuning of PuLSS with ASR (Wang et al., 19 May 2026).

Taken together, these properties place PuLSS at the intersection of real-data weak supervision, diarization-guided separation, and pseudo-label learning. This suggests that its broader significance lies in demonstrating a concrete path from noisy close-talk mixtures to effective real-domain supervision for far-field neural separation.