Guided Source Separation Techniques

• Guided source separation is a technique that uses auxiliary information to constrain the inversion of audio mixtures for targeted source extraction.
• It integrates modalities such as diarization, synchronized video, textual queries, and generative priors to enhance separation accuracy and performance.
• Recent advances include real-time block-online processing, GPU acceleration, and innovative guidance strategies that improve standard metrics like SDR and WER.

Guided source separation comprises source-separation methods in which auxiliary information is used to constrain estimation of individual components from a mixture. In the speech literature, Guided Source Separation (GSS) is specifically described as “a speech separation method that uses diarization information to update parameters of the generative model of observation signals” (Horiguchi et al., 2020). More broadly, the same guiding principle appears in methods conditioned on synchronized video, natural-language queries, user-provided mimicry and masks, weak labels, pretrained generative priors, physical forward models, and external knowledge such as musical scores or voice-activity labels (Zhu et al., 2020, Tan et al., 2023, Wen et al., 2 Jul 2025, Ni et al., 7 Jul 2025, Ho et al., 25 Feb 2026). Across these settings, the underdetermined inverse problem is not solved from the mixture alone; it is solved under additional constraints that determine which component should be extracted, how it should be reconstructed, or which decompositions are admissible.

1. Common formulations and sources of guidance

A recurring formulation starts from a mixture written either in the time domain as $x(t)=\sum_i s_i(t)$ or in the time–frequency domain as $X_{\text{mix}}(t,f)=\sum_n X_n(t,f)$. Many systems then estimate a source-specific mask and form $\hat X_n=\hat B_n\odot X_{\text{mix}}$ or $\hat A^S=\hat M\odot A^S$, with the guidance signal entering the mask predictor through conditioning features, gating variables, or explicit constraints (Zhu et al., 2021, Tan et al., 2023). In visually guided models, the auxiliary input is a synchronized frame sequence or motion representation; in language-queried systems, it is a text embedding; in user-guided music separation, it is a waveform mimicry condition and mel-spectrogram domain masks; in score- or physics-guided systems, it is a structured prior supplied by an external model of the signal (Song et al., 2023, Wen et al., 2 Jul 2025, Ni et al., 7 Jul 2025, Ho et al., 25 Feb 2026).

The point of intervention differs by paradigm. In cACGMM-based speech GSS, diarization information enters the E-step by forcing inactive speakers to have zero posterior mass, so the guidance modifies latent assignment directly (Horiguchi et al., 2020). In conditional separators, the guidance is fused with audio representations in a U-Net, predictive-coding network, transformer, or other separator backbone (Song et al., 2023). In prior-guided inference, the guidance becomes part of the objective itself, as in flow-based maximum-likelihood or MAP separation, score-based $\alpha$-posterior optimization, and regularized NMF under GMM priors (Zhu et al., 2022, Jayashankar et al., 2023, Grais et al., 2013).

This suggests a useful unifying distinction between three mechanisms. Guidance may constrain latent source assignment, condition the separator network, or regularize the solution space through an explicit prior. The literature includes all three, and several systems combine them.

2. Diarization-guided multichannel speech separation

In meeting transcription and distant ASR, GSS denotes a multichannel front end that couples diarization with blind source separation. The standard pipeline combines WPE dereverberation, mask estimation with a complex Angular Central Gaussian Mixture Model (cACGMM), and MVDR beamforming. In the original offline formulation, the observation is the $M$-channel STFT vector $x_{t,f}\in\mathbb C^M$, the per-frequency mixture model is a cACGMM, and oracle diarization $d_t^{(k)}\in\{0,1\}$ is incorporated by setting $\gamma_{t,f}^{(k)}=0$ whenever $d_t^{(k)}=0$ in the E-step (Horiguchi et al., 2020). GPU-oriented descriptions of the same family of methods emphasize the decomposition of the observed meeting signal into early speech, late reverberation, and additive noise, followed by WPE, CACGMM EM iterations, and mask-based MVDR beamforming (Raj et al., 2022).

The principal engineering challenge is runtime and latency. Block-online GSS addresses this by processing a session in blocks of length $X_{\text{mix}}(t,f)=\sum_n X_n(t,f)$0 with pre-context $X_{\text{mix}}(t,f)=\sum_n X_n(t,f)$1, concatenating the current block with preceding context, updating only active speakers, and performing exactly one EM iteration per block. The result is fixed latency equal to the block size rather than the utterance length, with complexity $X_{\text{mix}}(t,f)=\sum_n X_n(t,f)$2 per block. On CHiME-6 and a meeting corpus, the method achieved almost the same performance as the conventional offline GSS algorithm but with 32x faster calculation, which the paper describes as sufficient for real-time applications (Horiguchi et al., 2020).

A complementary line of work accelerates the same inference pattern on modern hardware. A GPU implementation based on CuPy arrays, segment batching into “super-segments,” frequency batching, cached contraction paths for einsum calls, and threaded data loading reports an effective speed-up of approximately 292× over the original CPU baseline while preserving ASR performance, and provides reproducible pipelines for LibriCSS, AMI, and AliMeeting (Raj et al., 2022). This makes large-scale ablation of context duration, number of channels, WPE usage, noise classes, and CACGMM iterations practical.

Later refinements focus on downstream beamforming details. Reference microphone selection for GSS-based speech enhancement has traditionally used SNR, but recent work argues that this neglects differences in early-to-late-reverberant ratio across microphones. Two alternatives based on the normalized $X_{\text{mix}}(t,f)=\sum_n X_n(t,f)$3-norm, either alone or combined with SNR through min–max normalized costs, are reported to reduce macro-average tcpWER on a CHiME-8 distant ASR system (Lohmann et al., 31 Oct 2025). A related but distinct guided beamforming formulation integrates DNN-estimated source power spectra into convolutional beamformer optimization through an inverse-Gamma prior over source variances, improving WER on REVERB-2MIX and approaching an oracle-mask system (Nakatani et al., 2021).

3. Visual, motion, and trimodal guidance

Visually guided sound source separation assumes a mixture audio stream together with one or more synchronized video streams that depict the underlying sources. A representative early formulation takes a single-channel mixture audio signal, forms a time–frequency spectrogram, and uses synchronized videos to recover clean component spectrograms while also producing a localization mask in the image plane (Zhu et al., 2020). The same paradigm was later generalized to appearance-only, motion-aware, and audio-visual-language settings.

Several models organize the problem around mask prediction with increasingly structured fusion. “Sep-Stereo” unifies stereo generation and source separation by treating source separation as a particular type of audio spatialization. Its associative pyramid network (APNet) uses a coarse-to-fine side branch that fuses decoder audio features with ResNet-18 visual features and predicts complex masks for left/right channels or source A/B. The framework improves stereophonic audio generation while performing accurate sound separation with a shared backbone, and reports separation SDR of approximately $X_{\text{mix}}(t,f)=\sum_n X_n(t,f)$4 dB versus $X_{\text{mix}}(t,f)=\sum_n X_n(t,f)$5 dB for PixelPlayer (Zhou et al., 2020). The “Cascaded Opponent Filter” framework instead performs iterative mask refinement: after an initial stage, later stages compute residual filters $X_{\text{mix}}(t,f)=\sum_n X_n(t,f)$6 that identify components to be reassigned between sources, while SSLM provides a pixel-level source-location mask. On MUSIC, a 3-stage COF reaches SDR $X_{\text{mix}}(t,f)=\sum_n X_n(t,f)$7, SIR $X_{\text{mix}}(t,f)=\sum_n X_n(t,f)$8, and SAR $X_{\text{mix}}(t,f)=\sum_n X_n(t,f)$9 (Zhu et al., 2020).

A second cluster of methods emphasizes motion. Zhu and Rahtu’s AMnet decomposes the task into an audio–appearance stage and an audio–motion stage. The motion pathway is trained through Audio–Motion Embedding (AME), which aligns motion and synchronized audio with a triplet loss, and the fusion module is an audio–motion transformer with self-attention and cross-attention blocks. AMnet reports SDR $\hat X_n=\hat B_n\odot X_{\text{mix}}$0 dB on MUSIC-21 and $\hat X_n=\hat B_n\odot X_{\text{mix}}$1 dB on AVE, while also improving localization measured by cIoU and AUC of CAM overlap with object masks (Zhu et al., 2021). AVPC replaces explicit fusion blocks by predictive coding: visual features define a target at the bottom layer, audio features recursively minimize prediction error against that target, and a self-supervised Representation Co-Prediction pretraining stage improves audiovisual features. On MUSIC-21, AVPC-RCoP reaches SDR $\hat X_n=\hat B_n\odot X_{\text{mix}}$2, SIR $\hat X_n=\hat B_n\odot X_{\text{mix}}$3, and SAR $\hat X_n=\hat B_n\odot X_{\text{mix}}$4 with $\hat X_n=\hat B_n\odot X_{\text{mix}}$5M parameters (Song et al., 2023).

Trimodal systems add language. VAST adapts CLIP-based vision-LLMs to audio-visual source separation through latent-caption extraction, an audio–language consistency loss, and a trimodal consistency loss based on KL divergence between caption-driven and audio-driven visual attention maps. It supports both audio–visual and audio–language inference, and on SOLOS reports SDR $\hat X_n=\hat B_n\odot X_{\text{mix}}$6, exceeding the cited Co-Separation baseline of $\hat X_n=\hat B_n\odot X_{\text{mix}}$7; on MUSIC duets it reports SDR of approximately $\hat X_n=\hat B_n\odot X_{\text{mix}}$8 dB (Tan et al., 2023). A plausible implication is that, once visual and textual embeddings share a common space, the same separator can be queried either by regions in a video or by free-form text.

4. Generative priors, latent steering, and test-time optimization

A different tradition treats guidance as a prior over plausible sources rather than as a modality paired with the mixture. In “Music Source Separation with Generative Flow,” one unconditional Glow model is trained per instrument family, exact likelihoods are computed by the change-of-variable formula, and separation is performed by latent-space optimization under a mixture likelihood based on generalized KL divergence. The paper states that the source-only supervised method is competitive with a fully supervised approach and can flexibly add new types of sources without retraining the entire model (Zhu et al., 2022). This modularity follows directly from the fact that each source has its own independently trained prior.

Score-based methods make the same move with diffusion-style priors. “Score-based Source Separation with Applications to Digital Communication Signals” defines an $\hat X_n=\hat B_n\odot X_{\text{mix}}$9-posterior

$\hat A^S=\hat M\odot A^S$0

uses separately trained score networks for the smoothed source priors, and performs stochastic gradient updates that combine the prior score with a data-consistency term. In RF mixtures, the paper reports a BER reduction of 95% over classical and existing learning-based methods, and analyzes the resulting estimates as approaching the modes of an underlying discrete distribution (Jayashankar et al., 2023).

Large pretrained generative models can also be steered without retraining. “Unsupervised Source Separation By Steering Pretrained Music Models” couples OpenAI’s Jukebox VQ-VAE with a frozen music tagger and optimizes only the latent code of the generative model so that the decoded-and-masked audio matches a desired tag distribution. The method performs no weight updates, uses a cross-entropy objective on tag probabilities, and reports SDRi of approximately $\hat A^S=\hat M\odot A^S$1–$\hat A^S=\hat M\odot A^S$2 dB across a wider variety of instruments than any tested supervised or unsupervised baseline (Manilow et al., 2021).

Recent diffusion-based work has moved from class labels toward direct user or query guidance. DGMO explores whether pretrained text-to-audio diffusion models can perform zero-shot language-queried separation without task-specific training, and introduces a two-stage procedure in which the diffusion model first generates query-conditioned reference signals and a spectrogram mask is then optimized to match their mel representations (Lee et al., 3 Jun 2025). GuideSep makes the guidance explicitly interactive: a diffusion-based music source separator is conditioned on a waveform mimicry input, such as humming or playing the target melody, together with positive and negative mel-spectrogram masks. On Slakh2100, the full model reports an overall SDR of $\hat A^S=\hat M\odot A^S$3, and the subjective evaluation reports separation-quality MUSHRA $\hat A^S=\hat M\odot A^S$4 versus $\hat A^S=\hat M\odot A^S$5 for the mask-prediction baseline (Wen et al., 2 Jul 2025). The paper also notes that mel-spectral masks alone are very strong cues, mimicry alone works moderately well, and combining both gives the best results.

5. Weak labels, statistical priors, physics, and external knowledge

Guidance need not come from synchronized modalities or learned generative models. One line of work uses weakly labelled data to create pseudo-supervision. “Source separation with weakly labelled data” first trains a sound event detection system on weakly labelled AudioSet, then uses its time-wise predictions to select anchor segments likely to contain a target event, and finally trains a conditional U-Net to regress from mixtures of anchor segments back to a target segment conditioned on the target’s tag-prediction vector. The system is reported to separate 527 kinds of sound classes within a single model and to achieve an average SDR of $\hat A^S=\hat M\odot A^S$6 dB over 527 classes in AudioSet (Kong et al., 2020).

Earlier model-based source separation used statistical priors directly on latent coefficients. “Source Separation using Regularized NMF with MMSE Estimates under GMM Priors with Online Learning for the Uncertainties” treats NMF gains as distorted observations of valid log-domain patterns, models the clean patterns with a GMM prior, computes the MMSE estimate under the learned distortion covariance $\hat A^S=\hat M\odot A^S$7, and embeds that estimate in a regularized IS-NMF objective. The paper reports that the regularized NMF algorithm improves source separation performance compared with NMF without prior or with other prior models (Grais et al., 2013). This is a prior-guided formulation in the strict sense: the source model does not merely initialize optimization but changes the multiplicative updates themselves.

A more recent model-based variant is physics-guided rather than statistics-guided. “Physics-Guided Dual Implicit Neural Representations for Source Separation” assumes that the measured signal consists of a physically meaningful component, a structured background, and noise, and jointly trains two coordinate-based INRs: a Kernel-Net that learns a spatially varying distortion kernel over a local neighborhood and a Background-Net that learns a smooth background. The self-supervised objective combines reconstruction with an $\hat A^S=\hat M\odot A^S$8 penalty on the background network. In a four-dimensional inelastic neutron scattering case study, the method reports that optimal hyperparameters $\hat A^S=\hat M\odot A^S$9 yield near-perfect separation, reveal previously hidden magnon rings, and achieve compression ratios $\alpha$0 (Ni et al., 7 Jul 2025).

External knowledge can also be symbolic or categorical. “A Knowledge-Driven Approach to Music Segmentation, Music Source Separation and Cinematic Audio Source Separation” uses MIDI scores to build score-constrained HMM topologies for music segmentation and uses frame-level VAD labels as conditioning information for cinematic separation. In the music case, score-guided HMM alignment produces single-instrument segments that are remixed into pseudo-mixtures for separator training; in the cinematic case, projected VAD vectors are concatenated with spectrogram features and fed to a separator. On Slakh2100 piano–bass mixtures, knowledge-driven fine-tuning with oracle boundaries raises BSRNN from $\alpha$1 to $\alpha$2 SDR, and on DNR-nonverbal the VAD-conditioned SepReformer reaches Speech $\alpha$3, Music $\alpha$4, and SFX $\alpha$5 SDR (Ho et al., 25 Feb 2026).

6. Evaluation criteria, recurring limitations, and scope

The field is evaluated with task-specific metrics rather than a single universal criterion. Audio separation papers commonly report SDR, SIR, and SAR; open-vocabulary and zero-shot methods also report SI-SDR, SDRi, and CLAP Score; meeting front ends report WER, CER, or tcpWER; spatialization-oriented systems use STFT Distance and Envelope Distance; localization-oriented audiovisual work uses cIoU and AUC; RF systems use BER; and perceptual studies use DNSMOS, NISQA, NI-PESQ, NI-STOI, or MUSHRA depending on the application (Jayashankar et al., 2023, Song et al., 2023, Lee et al., 3 Jun 2025, Lohmann et al., 31 Oct 2025, Wen et al., 2 Jul 2025).

A recurring misconception is that guidance implies full supervision with mixture–source pairs. Multiple papers explicitly reject that assumption. COF is trained end-to-end using a large set of unlabelled videos, AMnet is trained in a self-supervised manner, VAST uses only unlabeled video and audio pairs, TagBox performs no weight updates, DGMO is training-free, and the knowledge-driven framework does not depend on any pre-segmented training data (Zhu et al., 2020, Zhu et al., 2021, Tan et al., 2023, Manilow et al., 2021, Lee et al., 3 Jun 2025, Ho et al., 25 Feb 2026). Guided source separation is therefore not a single supervision regime; it is a family of methods in which side information changes the feasible decomposition.

The limitations reported in the literature are equally heterogeneous. Sep-Stereo only handles left/right horizontal spatial cues and notes that complex backgrounds can confuse the side-placement strategy (Zhou et al., 2020). COF relies on good audio–video synchronization and visible motion, while its runtime and memory grow linearly with $\alpha$6 source pairs (Zhu et al., 2020). AVPC identifies homo-musical separation and off-screen or intermittent sounds as difficult cases (Song et al., 2023). In speech GSS, context improves mask estimation but increases latency, and block-online performance degrades sharply if $\alpha$7 frames (Horiguchi et al., 2020). DGMO requires approximately $\alpha$8 gradient steps across $\alpha$9 refinement rounds plus DDIM calls, amounting to minutes per $M$0 s clip on a single GPU, and reuses the mixture phase without refinement (Lee et al., 3 Jun 2025). GuideSep notes occasional timbre inexactness and reduced performance for highly polyphonic targets when the mimicry input is strictly monophonic (Wen et al., 2 Jul 2025). Knowledge-driven systems remain dependent on the accuracy of score or VAD side information (Ho et al., 25 Feb 2026).

Taken together, these results indicate that guided source separation is best understood as a design principle rather than a single architecture. The mixture is separated not by unconstrained inversion, but by coupling the inverse problem to auxiliary structure: diarization for speaker activity, video for visible causation, language for semantic targeting, user sketches for interactive extraction, priors for admissible source manifolds, physics for forward-model consistency, or knowledge sources for segmentation and conditioning. This suggests that future progress will continue to come from better forms of guidance as much as from larger separator backbones.