SuDoRM-RF++: Efficient Universal Audio Separation

• The paper introduces a refined design that replaces mask-based estimation with direct latent target estimation, enhancing convergence and efficiency.
• SuDoRM-RF++ is a family of architectures that employs U-ConvBlocks for successive downsampling and resampling of multi-resolution features to facilitate universal sound separation.
• The model demonstrates competitive SI-SDR improvements and reduced computational cost, with extensions addressing causal streaming and sampling-frequency-independent processing.

Searching arXiv for the core SuDoRM-RF and SuDoRM-RF++ papers to ground the article in the relevant literature. SuDoRM-RF++ is a family of end-to-end, time-domain neural architectures for universal sound source separation built around the SuDoRM-RF separator backbone, where SuDoRM-RF denotes SUccessive DOwnsampling and Resampling of Multi-Resolution Features. In the literature, the “++” designation refers specifically to a variant that retains the multiresolution U-ConvBlock separator philosophy of the original SuDoRM-RF model while altering the estimation head, normalization, decoder arrangement, and activation parameterization (Tzinis et al., 2021). It is designed for single-channel mixtures and targets both source-domain universality—speech, environmental sounds, and open-domain mixtures—and, in later extensions, broader robustness to recording conditions such as sampling frequency (Tzinis et al., 2021, Nakamura et al., 2023).

1. Nomenclature and lineage

The architectural lineage begins with the model introduced under the stylized title “Sudo rm -rf: Efficient Networks for Universal Audio Source Separation”, whose body defines the architecture as SuDoRM-RF, not SuDoRM-RF++ (Tzinis et al., 2020). That 2020 work establishes the essential waveform separation pipeline: a learned encoder, a separator composed of repeated U-ConvBlocks, and a learned decoder. It also frames the main efficiency rationale: high-quality separation with limited floating point operations, memory requirements, number of parameters, and latency (Tzinis et al., 2020).

The explicit introduction of SuDoRM-RF++ appears in the 2021 paper “Compute and memory efficient universal sound source separation” (Tzinis et al., 2021). That paper presents a small family of related models: SuDoRM-RF, SuDoRM-RF++, SuDoRM-RF++ GC, and C-SuDoRM-RF++. The relationship is architectural continuity rather than replacement by an unrelated design. SuDoRM-RF++ preserves the same core separator idea—successive downsampling and resampling of multi-resolution features—but changes the output formulation and several implementation choices (Tzinis et al., 2021).

A common misconception is that the 2020 paper introduces “SuDoRM-RF++.” It does not. The 2020 paper is the architectural precursor; the 2021 paper is the source that explicitly defines the “++” variants (Tzinis et al., 2020, Tzinis et al., 2021).

2. Separation setting and model family

SuDoRM-RF++ addresses single-channel end-to-end audio source separation in a deliberately broad sense. In the 2021 formulation, “universal” has two distinct dimensions: universality with respect to source domain and universality with respect to source cardinality (Tzinis et al., 2021). The evaluated tasks include:

• WSJ0-2mix for two-speaker speech separation.
• ESC50-based mixtures for two-source environmental or non-speech separation.
• FUSS for open-domain universal sound separation, where mixtures may contain speech, music, engines, wind, rain, and many other real-world sounds, with 1 to 4 active sources (Tzinis et al., 2021).

The fixed-maximum-output formulation used for variable source count assumes a known maximum number of output slots $N$, while allowing only $N' < N$ active sources in a given mixture. For FUSS, the model uses $N=4$ output slots and handles $N' \in \{1,2,3,4\}$ by learning to emit near-zero estimates in inactive slots under a permutation-invariant objective (Tzinis et al., 2021).

The family structure is summarized below.

Variant	Defining change	Intended use
SuDoRM-RF	Mask-based latent estimation with separate decoders	Original non-causal model
SuDoRM-RF++	Direct latent target estimation, single decoder, GLN	Refined non-causal variant
SuDoRM-RF++ GC	SuDoRM-RF++ with group communication	Parameter-reduced variant
C-SuDoRM-RF++	Causal convolutions, no normalization	Streamable real-time use

This organization suggests that “SuDoRM-RF++” is best understood not as a single frozen architecture but as a refined branch within a broader SuDoRM-RF design family.

3. Core architecture: encoder, U-ConvBlocks, and multiresolution processing

Like the original SuDoRM-RF, SuDoRM-RF++ is a time-domain learned encoder–separator–decoder system rather than an STFT-based separator (Tzinis et al., 2021). For an input waveform mixture $x \in \mathbb{R}^T$, the encoder is a 1D convolution followed by ReLU: $$v = \mathcal{E}(x) = \operatorname{ReLU}\!\left( \operatorname{Conv1D}_{N,K,K/2}(x) \right) \in \mathbb{R}^{N \times L}.$$ The 2021 paper specifies $N=512$, $K=21$ at 8 kHz, $K=41$ at 16 kHz, and stride $K/2$ (Tzinis et al., 2021). As in the earlier paper, this learned front-end produces a nonnegative latent mixture representation and reduces temporal resolution immediately through the encoder stride (Tzinis et al., 2020).

The separator first projects the latent representation into an internal channel space: $N' < N$0 with default $N' < N$1 for non-causal models and $N' < N$2 for the causal model (Tzinis et al., 2021). The main separator then applies $N' < N$3 repeated U-ConvBlocks: $N' < N$4 The scale naming convention is explicit: 2.0x uses $N' < N$5, 1.0x uses $N' < N$6, 0.5x uses $N' < N$7, and 0.25x uses $N' < N$8 (Tzinis et al., 2021).

Within each U-ConvBlock, the input is first expanded from $N' < N$9 channels to $N=4$0, processed by a depthwise convolution at full temporal resolution, then passed through repeated stride-2 depthwise 1D convolutions to create a hierarchy of lower-resolution feature maps. In the default non-causal configuration, $N=4$1, $N=4$2, $N=4$3, $N=4$4, and $N=4$5 (Tzinis et al., 2021). The coarser features are then returned to the original temporal resolution by nearest-neighbor interpolation and fused by additive top-down aggregation: $N=4$6 Finally, the block projects back to $N=4$7 channels and applies a residual connection: $N=4$8

Operationally, this is the meaning of SUccessive DOwnsampling and Resampling of Multi-Resolution Features. The separator grows effective receptive field through multiscale processing inside each residual block rather than through very deep stacks of dilated convolutions or recurrent dual-path modules (Tzinis et al., 2020, Tzinis et al., 2021). The U-ConvBlock therefore occupies the same conceptual role across the entire family, including later GC and causal variants.

4. What distinguishes SuDoRM-RF++ from SuDoRM-RF

The principal architectural change is the removal of mask estimation. In the original SuDoRM-RF, the separator predicts source-specific latent pre-mask codes, converts them to masks through a source-wise softmax, and multiplies those masks with the encoded mixture representation $N=4$9 to obtain latent source estimates (Tzinis et al., 2020). In SuDoRM-RF++, by contrast, the model directly estimates the latent target signals after the final separator output (Tzinis et al., 2021).

The 2021 paper identifies four defining changes for SuDoRM-RF++ (Tzinis et al., 2021):

Component	SuDoRM-RF	SuDoRM-RF++
Estimation head	Mask estimation	Direct latent estimation
Decoder arrangement	Different decoders per source	One trainable decoder module
Normalization	LN	GLN
PReLU parameterization	One learnable parameter per channel	One learnable parameter per activation layer

The decoder distinction is explicit. Original SuDoRM-RF uses different decoders for different sources, whereas SuDoRM-RF++ uses a single shared decoder: $N' \in \{1,2,3,4\}$0 The paper reports that replacing LN with GLN significantly improves convergence, and that simplifying PReLU to one learnable slope parameter per activation layer reduces parameter count (Tzinis et al., 2021).

The paper does not claim that SuDoRM-RF++ is uniformly superior in every possible scenario. It explicitly notes that original SuDoRM-RF may still be preferable when mask estimation is easier than direct latent target estimation, when bounded masks in $N' \in \{1,2,3,4\}$1 provide a useful inductive bias, or when separate source-specific decoders are advantageous for heterogeneous source classes (Tzinis et al., 2021). This point matters because later work sometimes treats “++” as an unqualified replacement. The published account is more specific: the “++” design is a refined variant with a different inductive bias, not a categorical repudiation of mask-based separation.

5. Efficiency profile and empirical results

Efficiency is central to the SuDoRM-RF++ program. The 2021 paper reports forward-pass CPU results for 1 second at 8 kHz and backward-pass GPU profiling, comparing the SuDoRM-RF family against ConvTasNet, DPRNN, and Two-Step TDCN (Tzinis et al., 2021).

For forward CPU inference at 8 kHz, SuDoRM-RF++ 1.0x reports 17.0 dB SI-SDRi on speech and 8.6 dB on non-speech, with 2.11 GFLOPs, 0.79 GB, and 0.17 s runtime (Tzinis et al., 2021). The corresponding SuDoRM-RF 1.0x reports 17.0 / 8.4 dB, 2.45 GFLOPs, 0.79 GB, and 0.17 s; ConvTasNet reports 15.3 / 7.7 dB at 5.16 GFLOPs; DPRNN reports 18.8 / 7.2 dB at 48.81 GFLOPs; and Two-Step TDCN reports 16.1 / 8.2 dB at 7.11 GFLOPs (Tzinis et al., 2021).

For backward-pass GPU profiling, SuDoRM-RF++ 1.0x uses 2.72M parameters, 16.23 GFLOPs, 0.99 GB, and 0.27 s, while SuDoRM-RF++ 1.0x GC reduces parameters to 0.30M at 2.72 GFLOPs, but with 1.21 GB memory and 0.35 s wall-clock time (Tzinis et al., 2021). The paper emphasizes that parameter count alone does not determine deployment efficiency: group communication can reduce parameters substantially while not necessarily improving memory or runtime proportionally (Tzinis et al., 2021).

For FUSS, the strongest SuDoRM-RF-family result reported in the 2021 paper is SuDoRM-RF++ 2.0x, with 25.9 dB SI-SDR for $N' \in \{1,2,3,4\}$2, 10.9 dB SI-SDRi for $N' \in \{1,2,3,4\}$3, 10.6 dB for $N' \in \{1,2,3,4\}$4, 7.8 dB for $N' \in \{1,2,3,4\}$5, and average 9.8 dB across $N' \in \{1,2,3,4\}$6–$N' \in \{1,2,3,4\}$7 active sources (Tzinis et al., 2021). The paper states that it nearly matches TDCN++ on average and outperforms it for the hardest $N' \in \{1,2,3,4\}$8 case (Tzinis et al., 2021).

Ablations in the same paper clarify why the “++” formulation took the shape it did. With $N' \in \{1,2,3,4\}$9, replacing LN with GLN improves WSJ0-2mix performance from 15.9 dB to 16.8 dB. Larger separator width, deeper stacks, and larger multiresolution depth $x \in \mathbb{R}^T$0 also improve performance; for example, $x \in \mathbb{R}^T$1, GLN reaches 18.9 dB (Tzinis et al., 2021). This suggests that the SuDoRM-RF backbone is not only an efficiency-oriented design but also a scalable one.

6. Variants, causal streaming, and sampling-frequency-independent extension

The paper’s two principal extensions of SuDoRM-RF++ are group communication and causal streaming. In SuDoRM-RF++ GC, channels are split into 16 groups, processed independently with shared parameters, and then coupled through a self-attention module for communication across groups (Tzinis et al., 2021). The purpose is to reduce parameters, particularly in the bottleneck $x \in \mathbb{R}^T$2 channel-mixing layers.

In C-SuDoRM-RF++, all non-causal convolutions are replaced by causal convolutions, all normalization layers are removed, and the model is configured for real-time operation (Tzinis et al., 2021). On a laptop CPU for 1 second / 8000 samples, Table 5 reports: $x \in \mathbb{R}^T$3 achieves 9.1 dB SI-SDRi in 50.4 ms; $x \in \mathbb{R}^T$4 achieves 10.1 dB in 88.2 ms; and $x \in \mathbb{R}^T$5 achieves 10.3 dB in 165.9 ms. The paper summarizes this as about 10–20× faster than real time for some causal settings (Tzinis et al., 2021).

A later extension addresses sampling-frequency independence by adopting SuDoRM-RF++ 1.0x GC as the baseline architecture (Nakamura et al., 2023). The 2023 paper does not introduce a new separator core. Instead, it replaces only the encoder and decoder with sampling-frequency-independent (SFI) convolutional layers, while keeping the separator core architecturally the same as standard SuDoRM-RF++ (Nakamura et al., 2023). The key mechanism is to generate digital convolution kernels from a latent analog filter $x \in \mathbb{R}^T$6 as a function of the input sampling frequency $x \in \mathbb{R}^T$7, using a least-squares design: $x \in \mathbb{R}^T$8 The front-end kernel size and stride are then scaled with sampling frequency: $x \in \mathbb{R}^T$9

This extension is noteworthy because it preserves the central SuDoRM-RF++ assumption that most sample-rate dependence resides in the analysis/synthesis filterbanks and in latent time resolution, rather than in the U-ConvBlock separator itself (Nakamura et al., 2023). The paper reports that resampling-based handling of unseen sampling frequencies degrades performance, whereas the proposed SFI method works more consistently across 8, 12, \ldots, 44, 48 kHz test conditions after training only at 48 kHz (Nakamura et al., 2023).

The same paper also records an important practical limitation: attempts to train the original SuDoRM-RF on 48 kHz data with $$v = \mathcal{E}(x) = \operatorname{ReLU}\!\left( \operatorname{Conv1D}_{N,K,K/2}(x) \right) \in \mathbb{R}^{N \times L}.$$0, $$v = \mathcal{E}(x) = \operatorname{ReLU}\!\left( \operatorname{Conv1D}_{N,K,K/2}(x) \right) \in \mathbb{R}^{N \times L}.$$1 led to output collapse, with all output signals becoming zeros after the first epoch, and similar numerical instability across several $$v = \mathcal{E}(x) = \operatorname{ReLU}\!\left( \operatorname{Conv1D}_{N,K,K/2}(x) \right) \in \mathbb{R}^{N \times L}.$$2 choices (Nakamura et al., 2023). This makes the 2023 study especially relevant for understanding SuDoRM-RF++ as a practically viable high-sample-rate branch of the family.

7. Interpretation, trade-offs, and technical significance

SuDoRM-RF++ occupies a specific position in the design space of time-domain separation architectures. Its distinctive separator is neither a large dilated-TCN stack nor a dual-path recurrent module, but a stack of lightweight residual U-shaped blocks that exploit depthwise convolutions, stride-2 temporal downsampling, nearest-neighbor interpolation, and additive multiscale fusion to expand receptive field at comparatively low cost (Tzinis et al., 2020, Tzinis et al., 2021). The efficiency argument is therefore architectural rather than merely implementational.

Three trade-offs recur across the literature. First, direct latent estimation versus mask estimation: the “++” design removes masking, but later work built on SuDoRM-RF++ 1.0x GC reverts to mask estimation and notes that direct prediction had been observed to make training numerically unstable (Nakamura et al., 2023). Second, parameter count versus realized efficiency: the GC variant dramatically reduces parameters, but may increase memory and fail to reduce wall-clock time proportionally (Tzinis et al., 2021). Third, universality versus specialization: the family is explicitly framed for universal sound separation, yet the original mask-based SuDoRM-RF may still be preferable when strong bounded-mask inductive bias or source-specific decoders are desirable (Tzinis et al., 2021).

Taken together, these results suggest that SuDoRM-RF++ is best characterized as a refined and extensible architectural template for universal source separation rather than as a single canonical instantiation. Its stable identity across papers lies in the SuDoRM-RF backbone—successive downsampling and resampling of multi-resolution features via repeated U-ConvBlocks—while the “++” label denotes a particular head-level and normalization-level reformulation that proved competitive in both fixed-source and variable-source-count settings (Tzinis et al., 2021). Subsequent work extends this template toward parameter reduction, causality, and sampling-frequency robustness without discarding the underlying separator philosophy (Tzinis et al., 2021, Nakamura et al., 2023).