Real-Time Audio Separation Techniques

• Real-time audio separation involves decomposing a mixed audio stream into distinct signals with minimal delay, crucial for interactive applications.
• Advanced methods use deep learning architectures like TasNet and ConvTasNet, achieving state-of-the-art quality by optimizing latency and computational efficiency.
• Applications span hearing aids, AR/VR, real-time remixing, and voice interaction, with promising results using multichannel and multimodal setups.

Real-time audio separation is the task of decomposing an audio stream containing multiple, possibly overlapping sources into distinct component signals, fast enough for live or interactive use. This domain spans monaural, multichannel, and multimodal (e.g., audio-visual) settings, and addresses a wide spectrum of applications, including hearing prostheses, in-car voice interaction, AR/VR audio, real-time remixing, and industrial acoustic monitoring. Recent advances leverage efficient deep architectures, causal and streaming designs, and hardware-aware optimizations to achieve state-of-the-art separation quality with tight latency, memory, and compute constraints.

1. Fundamental Principles and Problem Definitions

The goal in real-time audio separation is to map a mixture signal $x(t)$, potentially comprising $N$ underlying sources $\{s_i(t)\}_{i=1}^N$, to a set of estimates $\{\hat s_i(t)\}_{i=1}^N$ with minimal delay such that each $\hat s_i(t)$ predominantly contains energy from $s_i(t)$ and suppresses interference from the other sources. For mono or multi-microphone inputs, the mixture model can be represented as $x(t) = \sum_{i=1}^N s_i(t) + n(t)$ (for single-channel) or $$\mathbf{x}(t) = \sum_{i=1}^N \mathbf{a}_i s_i(t) + \mathbf{n}(t)$$, with additive noise and spatial mixing. Separation performance is typically evaluated using metrics such as scale-invariant signal-to-distortion ratio improvement (SI-SDRi), source-to-distortion ratio (SDR), and perceptual scores (DNSMOS, PESQ).

Real-time systems must guarantee end-to-end latency (including algorithmic and computational delay) well below task or user thresholds (e.g., <20 ms for hearing aids, <50 ms for interactive streaming), match or approach offline upper-bound separation quality, and remain operational under limited compute, memory, and battery budgets (Luo et al., 2017, Itani et al., 5 Aug 2025, Tzinis et al., 2021, Wu et al., 17 Nov 2025).

2. Real-Time-Capable Architectures and Model Classes

A variety of neural architectures have enabled practical real-time separation. Temporal convolutional and time-domain encoders/decoders (e.g., TasNet and ConvTasNet) replace slow STFT/iSTFT pipelines, while operated with causal (unidirectional) RNNs or attention blocks for low latency and streaming operation.

Key classes:

• Time-domain masking networks: TasNet directly models source reconstruction in short waveform windows, with a learnable encoder/decoder set and a mask estimation network (LSTM or TCN-based). Inference latency is reduced to a few milliseconds per frame by eliminating explicit frequency transformation (Luo et al., 2017).
• U-Net style structures: UX-Net employs multi-resolution convolutional and recurrent processing in a compact U-shaped topology, incorporating cumulative normalization and efficient filtering, supporting single- and multi-channel separation at 3 ms latency with low compute overhead (Patel et al., 2022).
• Multi-resolution convolutional backbones: SuDoRM-RF and SuDoRM-RF++ leverage successive downsampling/resampling and U-ConvBlocks to capture long-range temporal dependencies efficiently; their causal versions run at up to 20× real time on CPUs with minimal separation metric loss (Tzinis et al., 2021).
• TF domain MLP-mixers and convolutional RNNs: TF-MLPNet mixes frequency and channel dimensions using stackable MLP blocks and substitutes time-sequential recurrences with fully parallel, frequency-batched LSTM surrogates using 1×1 convolutions. Quantized models run several times faster than real time on embedded NPUs, with marginal performance loss (Itani et al., 5 Aug 2025).
• Efficient music separation UNets: RT-STT is a single-path time–frequency convolution and time-domain filter UNet with channel-expansion fusion, achieving <1 ms GPU inference and >0.5 dB cSDR improvement over prior streaming baselines. Quantization to FP16/INT8 further reduces latency and model size (Wu et al., 17 Nov 2025).

These models balance latency, separation quality, and computational requirements, with explicit trade-off parameters such as frame length, kernel size, network depth, and quantization policy.

3. Latency, Causality, and Streaming Inference

Meeting real-time constraints necessitates causal processing (no future context), efficient buffering, and streaming inference. Principal design features include:

• Short analysis frames: Time-domain approaches use frames as short as 2–5 ms; frequency-domain models (e.g., HS-TasNet) adopt minimal STFT windows (e.g., 23 ms) to bound lookahead latency (Venkatesh et al., 2024, Patel et al., 2022).
• Strict causality and buffer reuse: Gated RNNs and attention, when made causal, operate strictly on current and past context, often supplemented by storing hidden states and attention keys across streaming calls (stateful streaming) to eliminate recomputation (Kabeli et al., 2021). Stateless streaming, which reprocesses history without internal state handover, degrades both efficiency and SI-SDR (Kabeli et al., 2021).
• Chunk-based and overlapped processing: Many architectures process overlapping frames or chunks to align separation outputs with precise timing, reduce boundary artifacts, and achieve constant throughput (e.g., SAGRNN's overlap-chunking and stateful streaming) (Kabeli et al., 2021).
• Quantized and tiny-model optimizations: TF-MLPNet, with mixed-precision QAT, fits in under 500 kB and runs at 3.6 ms per 6 ms chunk on sub-100 mW processors, suitable for always-on hearable devices (Itani et al., 5 Aug 2025).
• Hardware-optimized pipelining: In music separation tasks (RT-STT), STFT, separation, and iSTFT are pipelined in separate threads (GPU or CPU), and fused layer kernels exploit wide-channel convolutions for efficient inference (Wu et al., 17 Nov 2025).

Typical measured end-to-end latencies range from <5 ms (TasNet (Luo et al., 2017), UX-Net (Patel et al., 2022), MIMO-TasNet (Han et al., 2020)) to ~23 ms (HS-TasNet, RT-STT), or up to 138 ms for very deep chunked streaming models (SAGRNN (Kabeli et al., 2021)), with real-time factors (RTF) well below 1 on commodity hardware for all causal models referenced.

4. Spatial, Multichannel, and Multimodal Extensions

Separation with spatial awareness, binaural cues, or integrated visual input enables robust performance in complex scenes:

• MIMO/Binaural separation: Extending time-domain architectures (TasNet, Wavesplit) to multi-microphone or binaural input allows joint preservation of interaural time and level difference (ITD/ILD), critical for source externalization in AR, VR, or hearing aids. Mask-and-sum structures with HRIR training maintain spatial cues with <2 μs ITD, <0.2 dB ILD error and ~15.6 dB SNRi (Han et al., 2020, Han et al., 2023).
• Speaker tracking and permutation invariance: Online clustering and conditioning via speaker profiles (as in Wavesplit/FiLM) minimizes speaker-swapping in long, moving speech scenes, lowering swaps to <1 per 24 s with preserved localization (Han et al., 2023).
• Area-based and steerable neural separation: Neural Steering (CRUSE with inference-adaptive phase pre-warping) allows rapid redirection of spatial focus at inference by a simple complex weight applied to the input spectrogram, without retraining the network. This mechanism retains SI-SDR within the noise margin for angles up to ~45°, supporting flexible teleconferencing and smart AR headset applications (Strauss et al., 2024).
• Visual integration: RTFS-Net incorporates cross-dimensional attention fusion (CAF) blocks for audio-visual speech separation, learning efficient joint representations in the time–frequency domain and reducing MACs by >8× relative to SOTA time-domain AV models, while remaining operational at RTF < 0.06 (Pegg et al., 2023).

5. Music Source Separation and Low-Latency Demixing

Recent work brings real-time, low-latency separation to music demixing, targeting remix, live streaming, and assistive listening use cases:

• Hybrid architectures: HS-TasNet combines a spectral (STFT/LSTM) stream and a time-domain (TasNet) stream, summing outputs to exploit both harmonic (spectral) and transient (waveform) details, yielding 4.65 dB SDR at 23 ms latency on MusDB (Venkatesh et al., 2024).
• Lightweight and quantized UNets: RT-STT, a TFC-TDF causal UNet with channel-expansion feature fusion and post-training quantization, achieves 5.17 dB cSDR at <1 ms GPU inference and 23 ms latency, outperforming HS-TasNet and other real-time benchmarks with under 0.4 M parameters (Wu et al., 17 Nov 2025).
• Edge-oriented dense networks: MMDenseNet with cIRM, time-axis and frequency-axis self-attention, band-merge-split, and feature look-back is shown to trade off SDR and latency for chordal accompaniment separation: 14.9–15.0 dB SDR at ~4.7 s latency to 13.7 dB at 0.6 s, with chunk-wise streaming and context-preserving concatenation (Wang et al., 2024).
• Subtractive architectures in real environments: In monophonic, reverberant music and speech separation, cascaded causal CRUSE modules with subtractive mask estimation achieve SI-SDR and MOS metrics on par with noncausal state-of-the-art (SepFormer) at 1/10th computational cost (Neri et al., 2023).

6. Evaluation Methodologies, Limitations, and Practical Considerations

Critical evaluation for real-time systems requires balancing multiple metrics:

• Metrics: SI-SDR improvement, SNR improvement, SDR, DNSMOS, PESQ, cross-talk suppression (CSE), and real-time factor (RTF) (Tzinis et al., 2021, Neri et al., 2023, Han et al., 2020).
• Trade-offs: Causal/streaming conversion yields modest drops in SI-SDR/SDR (e.g., ≤0.8 dB), but often with order-of-magnitude compute or memory savings. Stateless streaming approaches, or aggressive quantization without knowledge distillation, can produce greater quality loss (Kabeli et al., 2021, Itani et al., 5 Aug 2025).
• Limitations: Spatial selectivity degrades at large steering angles or in strong reverberation; feature concatenation for spatial cues has diminishing returns for closely spaced sources. Channel expansion and band-merge-split techniques marginally increase memory/computation but yield proportional SDR gains (Wang et al., 2024, Wu et al., 17 Nov 2025).
• Deployment: SOTA architectures fit real-time constraints for mobile CPU, GPU, and low-power NPU hardware, and are amenable to further on-device optimizations (thread pipelining, kernel fusion, pruning, or group-communication weight sharing). Use-case specific tuning (e.g., for live remixing, AR, hearing augmentation) involves tuning latency-vs-quality and model-size-vs-throughput along Pareto-efficient trade-off curves (Wu et al., 17 Nov 2025, Tzinis et al., 2021).

7. Future Directions and Open Problems

Anticipated research and engineering challenges include:

• Extending causal and streaming models to general $N$-talker and variable-source settings while maintaining strict latency guarantees (Tzinis et al., 2021).
• Robustness under severe reverberation, variable array topology, and non-speech source classes.
• Synergistic integration of spatial, spectral, and cross-modal cues, including dynamic region tracking and multimodal DOA estimation (Strauss et al., 2024, Pegg et al., 2023).
• Minimizing quality loss for aggressive quantization, pruning, and TinyML deployment while maintaining adaptivity and cross-domain generalization (Itani et al., 5 Aug 2025, Wu et al., 17 Nov 2025).
• Theoretical bounds on optimal streaming/causal tradeoffs for arbitrary auditory scenes, and transfer of white-box model insights (e.g. closed-form SFS) to machine-learned systems (Ma et al., 2019).

These directions underline an active and multidisciplinary research frontier, driven by advances in deep causal architectures, hardware-aware design, and hybrid signal-modeling paradigms.