HT-Demucs: Hybrid Audio Separation

• HT-Demucs is a hybrid neural architecture combining time and frequency domain processing with a Transformer bottleneck for superior music source separation.
• It employs dual U-Net branches with convolutional encoding and cross-domain attention to capture both local and long-range dependencies effectively.
• Using extensive data augmentation and sparse attention strategies, HT-Demucs achieves state-of-the-art SDR gains on benchmarks like MUSDB18-HQ.

HT-Demucs is an advanced neural source separation architecture designed for music source separation that unifies convolutional U-Net processing in both time and frequency domains with a Transformer-based cross-domain bottleneck. Introduced to address the challenge of leveraging both local acoustic structure and long-range contextual dependencies, HT-Demucs augments the Hybrid Demucs bi-U-Net architecture by incorporating a “Hybrid Transformer Encoder,” significantly improving separation quality when provided with sufficient training data and augmentations (Rouard et al., 2022).

1. Architectural Overview

HT-Demucs is a hybrid bi-U-Net model, operating simultaneously in the time (waveform) and frequency (spectral) domains. The architecture consists of:

• Waveform U-Net Branch: 5-layer encoder-decoder acting directly on time-domain audio.
• Spectrogram U-Net Branch: 5-layer encoder-decoder acting on STFT magnitude, with phase reconstructed for synthesis.
• The outer four encoders and decoders of each branch are convolutional.
• The innermost encoder and the first decoder are replaced by a stack of interleaved Transformer layers—termed the Hybrid Transformer Encoder—which process tokens from both domains.

After separate domain processing, outputs from both branches are combined (summed) to generate the final source estimate. This architecture enables time–frequency interaction, with the Transformer bottleneck responsible for modeling long-range dependencies and cross-domain correlations (Rouard et al., 2022).

2. Hybrid Transformer Encoder Details

The Hybrid Transformer Encoder at the model's bottleneck comprises L layers of alternating self-attention and cross-attention:

• Self-Attention per Domain: Each domain's tokens (shape $\mathbb{R}^{n \times d}$) undergo multi-head self-attention. Each layer includes LayerNorm, multi-head attention, LayerScale with $\epsilon=10^{-4}$, and a feed-forward block with hidden size $4d$.
• Cross-Domain Attention: Every other Transformer layer replaces self-attention with cross-attention, allowing the time branch to attend to the frequency branch tokens, and vice versa. For instance, updates of the time-domain tokens follow:

$$X^t \leftarrow X^t + \mathrm{LayerScale} \cdot \mathrm{softmax}(Q^t (K^f)^\top / \sqrt{d_k}) V^f$$

where $Q^t = X^t W_Q^t$, $K^f = X^f W_K^f$, $V^f = X^f W_V^f$.

• Positional Encodings: Sinusoidal encodings are injected into both branches—1D for time, 2D for spectrogram.
• Sparse Attention Kernels: For long sequences, an LSH-based sparse attention mechanism decreases quadratic complexity, hashing each token into 32 rounds and 4 buckets, and retaining only query-key matches that collide sufficiently, enforcing 90% sparsity and $\mathcal{O}(N \log N)$ scaling.

Parameterization includes $d=384$ (or $512$ in larger variants), $h=8$ attention heads, Transformer depth $L=5$ (or $7$), and inner-dimension $4d$ for MLP blocks (Rouard et al., 2022).

3. Training Procedure and Fine-Tuning

HT-Demucs is trained on multi-source music datasets combining MUSDB18-HQ (117 songs) and an additional 800 internally curated multi-track songs. Core training strategies include:

• Data Augmentation: In-batch remixing of stems and pitch/tempo modification.
• Optimization: Adam optimizer with $lr=3 \times 10^{-4}$, batch size 32, no weight decay, and an exponential moving average. Training runs for $1200$ epochs with $800$ batches per epoch.
• Loss Function: $L_1$ loss is computed on the sum of time- and frequency-domain reconstructions versus the target waveform.
• Per-Source Fine-Tuning: After multi-target training, the model is fine-tuned separately for each target source (drums, bass, vocals, other) for $50$ epochs at $lr=10^{-4}$, disabling mix/repitch augmentation and applying gradient norm clipping and weight decay of $0.05$. At inference, overlapping windows (25% overlap) are processed and cross-faded (Rouard et al., 2022).

4. Quantitative Performance and Ablation Studies

HT-Demucs achieves state-of-the-art results on music source separation benchmarks:

Model	MUSDB18-HQ median SDR (dB)
Hybrid Demucs (no extra data)	7.64
HT-Demucs (no extra data)	8.49
HT-Demucs (+800 songs)	8.80
+ per-source FT	9.00
+ sparse attention + FT	9.20

The Transformer bottleneck alone yields gains of +0.85 dB over Hybrid Demucs with matched training data. Sparse attention enables context length extension and adds up to +0.14 dB on longer segments. Removing remix augmentation decreases SDR by 0.7 dB, whereas increased Transformer depth (L=5→7) and larger model width (d=512) provide modest improvements (Rouard et al., 2022).

Ablations confirm that both the cross-domain Transformer and sufficient data augmentation/training set size are required to realize consistent performance advantages.

5. Implementation and Model Flow

Key hyper-parameters of HT-Demucs:

• $d_{\text{model}}=384$ (or $512$)
• 8 attention heads, $d_k=48$
• Transformer depth $L=5$ or $7$
• LayerScale $\epsilon=10^{-4}$
• Sparse attention: 90% sparsity via LSH

The forward pass follows:

def HTDemucsForward(x_wave: [batch, 2, T]): # 1) Outer conv-encoders z_t = ConvEnc1_t(x_wave) ... z_f = ConvEnc1_f(STFT(x_wave)) ... # 2) Flatten + positional encodings tokens_t = FlattenTime(z_t) + PosEnc1D tokens_f = FlattenSpec(z_f) + PosEnc2D # 3) Cross-domain Transformer Encoder for layer in 1...L: if layer is odd: tokens_t = SelfAttnLayer(tokens_t) tokens_f = SelfAttnLayer(tokens_f) else: tokens_t = CrossAttnLayer(tokens_t, tokens_f) tokens_f = CrossAttnLayer(tokens_f, tokens_t) # 4) Unflatten, decode ... # time and frequency decoding branches # 5) Output summation return y_t + y_f

This pipeline enables the explicit modeling of intra- and inter-domain dependencies crucial for separating overlapping sources with both local and contextual cues (Rouard et al., 2022).

6. Contextualization and Applications

HT-Demucs is primarily intended for music source separation, notably for the MUSDB task comprising drums, bass, vocals, and "other." The cross-domain Transformer addresses critical failure modes of purely convolutional or single-domain systems, specifically the limited ability to model long-range temporal and spectral dependencies. When provided with enough diverse multi-track training material and augmentations, the model's bi-U-Net with Transformer bottleneck enables state-of-the-art separation quality.

A plausible implication is that similar hybrid architectures could be extended to other source separation domains (e.g., speech) wherever joint modeling of temporal and spectral phenomena is required—as demonstrated in successor works such as HD-DEMUCS for speech restoration (Kim et al., 2023).

7. Comparative Analysis and Limitations

HT-Demucs outperforms prior Demucs variants (single U-Net, Hybrid Demucs) and non-U-Net baselines (e.g., Band-Split RNN) when large, diverse training data is available. Its architectural advantage arises from combining local feature extraction with long-range Transformer modeling and explicit cross-domain attention. However, performance gains diminish with limited training data, and Transformer bottleneck brings significant compute/memory requirements, particularly for long audio clips, mitigated partially by sparse attention strategies. The model’s sensitivity to data scale and augmentation effectiveness is highlighted by ablation results: removal of training augmentations or insufficient dataset size causes notable degradation in signal-to-distortion ratio (Rouard et al., 2022).

---

HT-Demucs represents a culmination of U-Net-based local feature learning and Transformer-mediated contextual reasoning, achieving state-of-the-art results in source separation by optimally bridging time- and frequency-domain modeling through a cross-domain attention bottleneck (Rouard et al., 2022).