SURF: Separation via Unsupervised Remixing Flow

Abstract: The goal of single-channel source separation is to reconstruct $K$ sources given their mixture. In supervised settings where vast amounts of clean source data are available, this challenging, ill-posed problem has been addressed successfully by generative diffusion and flow-based prior models. However, access to such clean source samples is often limited, and even when available, supervised models are vulnerable to domain shifts. To bridge this gap, we present Separation via Unsupervised Remixing Flow (SURF), an unsupervised flow matching approach for source separation that learns directly from observed mixtures. This method relies on a novel combination of state-of-the-art supervised flow matching and regression-based self-supervised techniques. At a high level, starting from a teacher model, we utilize a "remixing" step to bootstrap the learning of a student flow model from the teacher's estimates. We provide insights into the objectives optimized by this approach and draw a novel connection to the Wake-Sleep algorithm. Empirical evaluations on image and audio benchmarks demonstrate that SURF establishes a new state-of-the-art, significantly outperforming existing unsupervised methods. See our demo page for examples. https://google.github.io/df-conformer/surf/

• The paper introduces SURF, an unsupervised remixing flow model that unifies generative flow matching with self-supervised remixing for effective source separation.
• It employs a teacher-student loop with permutation-invariant training and dual loss functions (ReMixIT and Self-Remixing) to stabilize learning and reduce artifacts.
• Empirical results on image and audio tasks demonstrate state-of-the-art performance, achieving high fidelity and improved perceptual quality over regression-based methods.

SURF: Separation via Unsupervised Remixing Flow

Motivation and Problem Setting

Single-channel source separation remains a paradigmatic ill-posed inverse problem in signal processing and generative modeling, where one seeks to recover $K$ independent sources $\{\vx^{(k)}\}_{k=1}^K$ from only their sum $\vm = \sum_k \vx^{(k)}$. Contemporary supervised approaches achieve strong results via conditional generative models—especially flow matching, diffusion models, and regression with mixture-consistency constraints—but these require large quantities of clean, isolated source data and are structurally vulnerable to domain shift. When clean data is scarce or non-existent (e.g., bioacoustics, real-world sound scenes), unsupervised methods such as MixIT, ReMixIT, and Self-Remixing are predominant. Their paradigm involves a teacher/student model bootstrapping pipeline, distilling pseudo-labels from mixtures themselves through remixing and permutation-invariant training, but these methods are restricted to regression-style estimators and cannot leverage the sample diversity and artifact reduction of generative flows.

The paper introduces SURF (Separation via Unsupervised Remixing Flow), which unifies the generative, permutation-equivariant regularization of conditional flow matching with the data efficiency and empirical stability of regression-based self-supervised remixing. The key technical development involves marrying flow-matched ODE learning with population-level remixing pipelines, extending self-distillation to conditional generative velocity models trained purely on mixture data, with no requirement for pre-trained diffusion or clean sources.

Figure 1: Schematic of the SURF pipeline—mixtures are passed through a teacher model, remixed, then used as pseudo-supervision for a student flow-matching model via either ReMixIT (estimate-matching) or Self-Remixing (mixture-matching).

From Flow Matching to Unsupervised Generative Separation

Flow Matching Source Separation

The conditional flow matching framework for source separation (~FLOSS) aims to model the velocity field $\vv_\theta$ of an ODE which, when integrated from a structured noise initialization constrained to preserve the mixture $\vm$, maps to samples from $p(\vx | \vm)$. This regime is inherently permutation invariant. The supervised loss is derived from matching the learned drift to differences between ground-truth sources and initialization, with a PIT (permutation-invariant training) wrapper to mitigate source order ambiguity:

$\mathcal{L}_{\text{MFM}} = \mathbb{E}\left[ \|\vv_{\theta}(\vx_t, t, \vm) - (\sigma \vx_1 - \vx_0)\|^2 \right]$

where $\sigma$ is the PIT-minimizing permutation for each example. This flow-matching method achieves strong artifact suppression, stability, and high fidelity—contingent, however, on clean sources for supervision.

Self-Supervised Remixing Frameworks

Unsupervised methods MixIT, ReMixIT, and Self-Remixing address the absence of ground truth by generating pseudo-training data: a teacher model separates (possibly noisy) sources, which are randomly remixed across batches, synthetic mixtures are constructed, and a student model is trained to reconstruct either the remixed sources (ReMixIT) or the mixture itself (Self-Remixing), using a regression loss. These pipelines are robust under large teacher error due to their cross-mixture recombination and PIT alignment, but are structurally tied to regression models, and fail to regularize the output distribution directly.

SURF transcends this limitation by enabling velocity-field (generative) student models within the remixing self-supervised skeleton, using conditional flow matching as the underlying student. Regarded through a new analysis, the remixing operation can be seen as instantiating a form of Wake-Sleep variational inference, with the teacher inducing an implicit source prior.

Relaxing Regression: Generative Remixing in SURF

At the algorithmic level, SURF operates in an iterative teacher-student loop:

• A teacher model generates source estimates for a batch of mixtures.
• These estimates are globally permuted and remixed across the batch, yielding synthetic mixtures.
• The flow-matching student receives noisy initializations (mixture-consistent Gaussian noise) and is trained on the residual between predicted and target velocities, with targets corresponding to either the remixed teacher sources (ReMixIT loss) or mixture consistency (Self-Remixing loss).
• PIT is applied at the flow-matching velocity level to align outputs/permutations.
• The teacher is updated via EMA from the student.

Both ReMixIT and Self-Remixing variants are realized for the generative flow model, with respective loss functions: $\mathcal{L}_{\text{RM-FM}} = \mathbb{E}\left[ \|\vv_\theta(\cdot) - (\text{remixed source} - \text{init})\|^2 \right]$

$$\mathcal{L}_{\text{SR-FM}} = \mathbb{E}\left[ \|(\text{mixture consistency residual})\|^2 \right]$$

with analytic connections established to the population-level stochastic process, permutation symmetry, and alignment objectives.

Notably, the drift/velocity for the flow model can be interpreted via a Tweedie-style regressor, i.e., the clean source estimate at an intermediate ODE step is $\{\vx^{(k)}\}_{k=1}^K$0. This enables the flow-matching model to serve as a denoising generator within the remixing pipeline.

Theoretical Contributions

A rigorous analysis demonstrates:

• The remixing objective converges to a variant of the Wake-Sleep algorithm, wherein the pseudo-source prior is induced implicitly by the teacher's marginal distribution over remixed estimates.
• In the population limit ($\{\vx^{(k)}\}_{k=1}^K$1), unsupervised remixing with flow matching optimizes a “pseudo-supervised” conditional generative objective penalized by background-averaged student errors and, for ReMixIT, a term proportional to teacher error alignment.
• Explicit decomposition of the loss gradient shows Self-Remixing’s correction terms vanish as student systematic error decorrelates between independently remixed anchors—enabling effective pseudo-supervised learning even under substantial initial teacher error.

Empirical Results

Extensive experiments are conducted on both image (MNIST, CIFAR10) and audio (Libri2Mix, AudioSet) benchmarks, measuring separation quality in both supervised and unsupervised regimes, and against both regression and generative baselines.

On image separation:

Figure 2: Qualitative separation comparisons (MNIST top, CIFAR10 bottom) showing visual artifacts under regression and substantial improvement with generative SURF methods.

Figure 3: Further qualitative separation results on CIFAR10, displaying faithfulness and fewer artifacts for SURF.

Figure 4: Further qualitative separation results on MNIST.

SURF achieves state-of-the-art among unsupervised methods, with PSNR/SSIM/LPIPS/FID competitive with or surpassing several supervised generative baselines. Inception and FID scores indicate sample diversity and perceptual realism exceeding non-generative approaches. For instance, on CIFAR10, SURF (Self-Remixing) achieves FID $\{\vx^{(k)}\}_{k=1}^K$2— $\{\vx^{(k)}\}_{k=1}^K$3 lower than BASIS or MixIT.

On audio tasks, including both speech-centric (Libri2Mix) and universal sound mixtures (AudioSet), SURF again establishes new SOTA across unsupervised separation metrics, closing the gap to supervised flow matching (SI-SDR within $\{\vx^{(k)}\}_{k=1}^K$4–$\{\vx^{(k)}\}_{k=1}^K$5dB); notably, perceptual audio quality (DNSMOS, PESQ) is also significantly improved, due to flow-matched generative priors suppressing the artifacts typical of regression.

Implications and Future Developments

By demonstrating that velocity-field generative models can be trained in a completely unsupervised, mixture-only regime, SURF substantially expands the applicability of ODE-driven prior-based source separation to real-world, label-scarce domains and complex signals. The theoretical connection to Wake-Sleep suggests that further advances may be possible by refining the self-distillation prior, leveraging more sophisticated aggregation distributions, or plugging in advanced generative architectures (e.g., EDM flows or high-dimensional score models).

SURF’s permutation-invariant population analysis and explicit loss decomposition provide a blueprint for extending self-supervised generative learning beyond source separation, e.g., to permutation-invariant multi-object inverse problems in vision, molecular modeling, or scientific imaging. Future research can explore hierarchically structured priors, robustification to heavy-tailed errors, or self-adaptive remixing schedules, as well as the extension of remixing flows to nonlinear mixing processes.

Conclusion

SURF represents a significant advancement in unconditional generative source separation, showing robust empirical convergence, improved perceptual quality, and strong theoretical underpinnings in permutation-invariant unsupervised learning. It generalizes remixing self-supervision to the generative modeling domain, achieving high-fidelity separation from mixture-only data across diverse modalities. This positions generative flow-based models as viable candidates for in-the-wild source separation without reliance on labeled or clean sources, and establishes a well-analyzed baseline for follow-up work in self-supervised conditional generative modeling.