Scalable Discriminative Source Separation (TISDiSS)

• The paper presents TISDiSS, a unified framework that jointly optimizes architecture, loss functions, and dynamic inference iterations for scalable source separation.
• It employs early-split multi-loss supervision and shared-parameter modules to enable robust performance with reduced computational and memory costs.
• Experiments on benchmarks like WSJ0-2mix show state-of-the-art SI-SNR gains and parameter efficiency, making TISDiSS ideal for adaptive real-time and embedded applications.

Training-Time and Inference-Time Scalable Discriminative Source Separation (TISDiSS) is a unified framework for source separation that jointly optimizes architecture, loss functions, and deployment strategies to achieve both high performance and operational flexibility. Designed to address the escalating computational and memory costs of deep discriminative models as separation requirements scale, TISDiSS integrates early-split multi-loss supervision, shared-parameter iterative modules, and dynamic inference repetitions. The fundamental premise is to permit trade-offs between speed, accuracy, and resource use by making both training and inference scalable, without necessitating retraining for different operational constraints (Feng et al., 19 Sep 2025).

1. Framework Organization and Key Principles

TISDiSS structures the source separation network around three main modules:

• Separator: Responsible for the primary discrimination and extraction of sources within the TF (time–frequency) representation.
• Splitter: Acts as an intermediate module ("early-split" point) inserted prior to the Decoder, to facilitate per-source stream segregation.
• Reconstructor: Refines individual source estimates post-splitting, implementing triple-path processing across time, frequency, and speaker dimensions.

A distinctive aspect is the early-split multi-loss supervision, with auxiliary loss terms applied at various intermediate outputs:

• The Separator and Reconstructor both emit intermediate separated signals.
• Each such output is directly supervised using a permutation-invariant SI-SNR loss.
• The Splitter additionally provides loss feedback at its output, which is used to supervise the tail end of the Separator.

The shared-parameter design is essential for parameter efficiency and practical deployment; the same network weights are reused across multiple iterations within modular blocks. This is balanced against potential degradation in separation quality by incorporating residual (skip) connections and placing the Splitter before the Decoder to minimize negative impact.

Dynamic inference repetitions allow the number of iterations (depth) of Separator and Reconstructor modules during inference to be varied at runtime, thereby enabling performance scaling without retraining.

2. Dynamic Inference Scaling

A haLLMark of TISDiSS is its ability to flexibly modulate inference depth:

• Adjustable Iterations ($N_\text{sep}$, $N_\text{re}$): The user may select the number of times the Separator ($N_\text{sep}$) or Reconstructor ($N_\text{re}$) modules are applied, trading latency and resource use against separation quality.
• No Retraining Required: The architecture is constructed and trained so that any valid number of separator/reconstructor iterations (within the predetermined range) yields a meaningful output. This is enforced via multi-loss supervision at all relevant depths.
• Fine-Tuning Scalability: The paper demonstrates that, even when a model is trained with limited iteration depth, a short additional fine-tuning phase with increased $N_\text{re}$ is sufficient to match performance at higher depths, suggesting that the learned representations generalize well to deeper inference.

This inference-time scaling supports deployment scenarios with heterogeneous constraints: low-latency applications can use shallow inference with minimal degradation, while high-quality offline processing can exploit greater depth.

3. Multi-Loss Supervision and Early-Split Architecture

TISDiSS’s training objective is a weighted sum of SI-SNR losses applied at multiple stages: $$\mathcal{L} = \frac{1}{K}\left[\lambda_\mathrm{last}\mathcal{L}_\mathrm{last} + \lambda_\mathrm{sep}\overline{\mathcal{L}}_\mathrm{sep} + \lambda_\mathrm{split}\mathcal{L}_\mathrm{split} + \lambda_\mathrm{re}\overline{\mathcal{L}}_\mathrm{re}\right]$$ where $K$ is the number of active loss terms. The notation $\overline{\mathcal{L}}_\mathrm{sep}$ and $\overline{\mathcal{L}}_\mathrm{re}$ denotes the mean loss across $(N_\text{sep}-1)$ or $(N_\text{re}-1)$ intermediate outputs, respectively. $\mathcal{L}_\mathrm{split}$ is the loss at the Splitter output, and $\mathcal{L}_\mathrm{last}$ supervises the final Decoder output. All loss computations use permutation-invariant versions to resolve source labeling ambiguities.

Early-split supervision is crucial for shallower inference scenarios—models trained with this strategy do not suffer catastrophic performance loss when executed with just one or two iterations during deployment. The Splitter boosts expressive power, particularly when parameters are shared, by signaling source-specific information before the final reconstruction stage.

4. Architectural and Training Systematics

TISDiSS's design leverages modern dual-path and triple-path transformer modules for the Separator and Reconstructor, inspired by TF-locoformer and SepReformer backbones. This consistency in architectural design ensures fairness in benchmarking and supports ablation studies.

Parameter Sharing: All (or some) separator or reconstructor blocks share their network weights. Empirical results demonstrate that, with residual connections and early splitting, the resultant performance drop is minimal—down to a fraction of a dB in SI-SNR relative to full-weight, non-shared variants—while the memory and compute footprint is reduced by factors of 2-4 depending on depth (Feng et al., 19 Sep 2025).

Fine-Tuning and Scalability: Experiments show that models trained with lower $N_\text{re}$ can be efficiently fine-tuned with a higher $N_\text{re}$, recouping much of the performance gain from increased iterative depth without additional full retraining.

5. Performance Evaluation

On standard speech separation benchmarks including WSJ0-2mix, Libri2Mix, and WHAMR!, TISDiSS achieves:

• State-of-the-art SI-SNRi/SDRi: Comparable or superior to previous best models; e.g., SI-SNRi ≈ 25 dB with configurations using multiple separator and reconstructor repetitions.
• Parameter Efficiency: Achieves these metrics with just 8 million parameters, versus 21–35 million in SepReformer or TF-Locoformer, owing to shared-parameter iterative design.
• Consistent Generalization: Maintains ≈1 dB SI-SNRi gain over strong baselines across clean and reverberant (WHAMR!) conditions, indicating robustness.
• Shallow-Inference Advantages: When run with only a few inference steps, performance drop is limited—due to early-split supervision and shared design—making the approach attractive for resource-constrained and low-latency deployment.

6. Applications and Deployment

TISDiSS is well-suited for:

• Adaptive Real-Time Applications: Dynamic inference scaling supports real-time systems that must balance latency and quality.
• Deployment on Embedded/Edge Devices: Reduced model size and memory requirements enable practical use in resource-constrained environments such as mobile phones, smart speakers, or hearing aids.
• Unified Model for Multiple Scenarios: A single TISDiSS model instance can be used for both offline batch processing (maximal depth) and low-latency interactive tasks (minimal depth), facilitating maintainability and deployment logistics.
• Upstream Supply for Generative Models: By producing cleaner separated signals at scale, TISDiSS enables the preparation of high-quality training sets for conditional generative audio models (Feng et al., 19 Sep 2025).

7. Comparative Position and Future Implications

TISDiSS directly addresses the brittleness and cost scaling bottleneck present in deep discriminative source separation. Unlike classical approaches (e.g., NMF, DNN-NMF hybrids, or fully end-to-end discriminative pipelines without explicit scaling mechanisms), TISDiSS supplies systematic training and inference designs for scalability without performance collapse at shallower depths.

The multi-loss, early-split, and shared-parameter strategems in TISDiSS have potential implications for other structured signal-processing domains, particularly wherever deep pipelines must trade off resource footprint for high-fidelity performance. Systematic analyses presented in (Feng et al., 19 Sep 2025) may inspire follow-on work targeting more general classes of discriminative or even generative separation models. The paradigm shift from "one inference depth fits all" to a dynamically scalable architecture–loss–inference co-design is likely to persist as the scale and diversity of audio separation applications continue to expand.