Interaural time difference loss for binaural target sound extraction (2408.00344v1)

Abstract: Binaural target sound extraction (TSE) aims to extract a desired sound from a binaural mixture of arbitrary sounds while preserving the spatial cues of the desired sound. Indeed, for many applications, the target sound signal and its spatial cues carry important information about the sound source. Binaural TSE can be realized with a neural network trained to output only the desired sound given a binaural mixture and an embedding characterizing the desired sound class as inputs. Conventional TSE systems are trained using signal-level losses, which measure the difference between the extracted and reference signals for the left and right channels. In this paper, we propose adding explicit spatial losses to better preserve the spatial cues of the target sound. In particular, we explore losses aiming at preserving the interaural level (ILD), phase (IPD), and time differences (ITD). We show experimentally that adding such spatial losses, particularly our newly proposed ITD loss, helps preserve better spatial cues while maintaining the signal-level metrics.

• The paper introduces a novel ITD loss that optimizes cross-correlation to effectively preserve critical spatial cues.
• It employs a multi-task loss combining signal-level and spatial metrics, maintaining robust SI-SNR and SNR performance.
• Experimental results show a 26.2μs reduction in ITD errors, demonstrating enhanced spatial fidelity across diverse acoustic conditions.

Interaural Time Difference Loss for Binaural Target Sound Extraction

The paper presents an investigation into binaural target sound extraction (TSE) with a focus on preserving spatial cues through the introduction of explicit spatial losses, particularly an interaural time difference (ITD) loss. The key contribution of this work is a novel ITD loss that aims to preserve spatial information more effectively than existing methods.

Background

Traditional target sound extraction (TSE) systems leverage neural networks to isolate a sound of interest from a mixture. While conventional single-channel TSE systems primarily rely on signal-level losses, these approaches tend to overlook spatial cues such as interaural level difference (ILD) and interaural time difference (ITD). Such spatial information is critical for applications like hearing aids and audio post-production where spatial localization of sound is crucial. The paper hypothesizes that explicitly incorporating spatial losses can improve the spatial fidelity of extracted sounds in a binaural context.

Proposed Methodology

The authors propose a multi-task loss combining signal-level losses with spatial losses to enhance the binaural TSE system. The key innovations include:

1. Interaural Level Difference (ILD) Loss: Measures the level difference between left and right channels to preserve spatial integrity.
2. Interaural Phase Difference (IPD) Loss: Measures the phase difference, addressing the circularity problem of phase measurements.
3. Novel Interaural Time Difference (ITD) Loss: Focuses on the errors between cross-correlation coefficients of estimated and reference signals, facilitating better ITD retention.

The ITD loss is particularly notable for its novel approach of being directly related to the cross-correlation of signals, ensuring differentiability and thereby suitability for neural network training.

Experimental Setup

Experiments were conducted using a dataset of simulated reverberant mixtures from various sound classes, including environmental sounds and urban noise. The dataset involved convolution of sound signals with head-related transfer functions (HRTFs) and room impulse responses (RIRs) to produce binaural recordings. The TSE model employed is an encoder-decoder architecture conditioned on a target sound class embedding.

Results

The evaluation metrics encompassed both signal-level (SI-SNR and SNR) and spatial metrics (ΔILD, ΔIPD, ΔITD). The results demonstrated:

1. Improvement in ILD and ITD Errors: Systems incorporating ILD and ITD losses showed reductions in ILD and ITD errors without degrading signal-level metrics.
2. Superiority of ITD Loss: The ITD loss led to the most substantial reduction in ITD errors (26.2μs), indicating the model’s enhanced ability to preserve spatial cues across different sound environments.
3. Minimal Impact on Signal-Level Performance: Despite adding spatial losses, the models maintained comparable SI-SNR, SNR, and failure rates to the baseline, confirming the robustness of the multi-task learning approach.

Implications and Future Work

The findings highlight the importance of integrating spatial losses in binaural TSE systems to optimize spatial cue preservation. The proposed ITD loss, in particular, offers a promising direction for enhancing binaural processing in various applications, including binaural speech enhancement and separation tasks. Future developments could investigate extending these techniques to other audio processing domains and further refine the ITD loss function, potentially incorporating perception-based filters to optimize performance.

This research underscores the potential for spatially-aware loss functions to elevate the fidelity of neural network-based sound extraction systems, thereby contributing to improved user experiences in real-world applications where spatial localization of sound sources is crucial.