SDR - half-baked or well done? (1811.02508v1)

Abstract: In speech enhancement and source separation, signal-to-noise ratio is a ubiquitous objective measure of denoising/separation quality. A decade ago, the BSS_eval toolkit was developed to give researchers worldwide a way to evaluate the quality of their algorithms in a simple, fair, and hopefully insightful way: it attempted to account for channel variations, and to not only evaluate the total distortion in the estimated signal but also split it in terms of various factors such as remaining interference, newly added artifacts, and channel errors. In recent years, hundreds of papers have been relying on this toolkit to evaluate their proposed methods and compare them to previous works, often arguing that differences on the order of 0.1 dB proved the effectiveness of a method over others. We argue here that the signal-to-distortion ratio (SDR) implemented in the BSS_eval toolkit has generally been improperly used and abused, especially in the case of single-channel separation, resulting in misleading results. We propose to use a slightly modified definition, resulting in a simpler, more robust measure, called scale-invariant SDR (SI-SDR). We present various examples of critical failure of the original SDR that SI-SDR overcomes.

• The paper critiques standard SDR implementations by revealing how BSS Eval can misrepresent performance in single-channel speech tasks and proposes SI-SDR for accurate assessment.
• SI-SDR enhances measurement reliability by rescaling signals to enforce orthogonality between target and residual noise, avoiding undue performance inflation.
• Empirical analyses using filtering optimization and spectral band deletion confirm SI-SDR's robustness, offering a more truthful reflection of signal quality than conventional SDR.

Evaluation and Adaptation of the Signal-to-Distortion Ratio (SDR) in Single-Channel Speech Enhancement and Source Separation

In their paper, "SDR -- half-baked or well done?", Jonathan Le Roux et al. engage in a critical analysis of the Signal-to-Distortion Ratio (SDR) measure utilized in the BSS Eval toolkit for assessing speech enhancement and source separation algorithms. SDR is traditionally employed to measure the effectiveness of separating a target signal from noise or interference. The authors argue that the existing implementations of SDR in the BSS Eval toolkit, particularly in single-channel scenarios, often yield misleading results due to inherent flaws in their design.

Critique of Existing SDR Implementations

The paper points out two primary versions of SDR implementation in BSS Eval:

1. bss_eval_sources: This version accommodates channel variations by allowing significant modifications to the signal through a time-invariant 512-tap filter. This flexibility can unrealistically increase the SDR by forgiving substantial frequency alterations.
2. bss_eval_images: This variant equates SDR to the Signal-to-Noise Ratio (SNR), lacking any rescaling, which can be exploited by algorithms to artificially boost SDR values without genuine enhancement.

The authors argue these approaches undermine the reliability of SDR as a measure, particularly in single-channel cases where distortions due to channel variations can be misinterpreted as improvements.

Proposed Scale-Invariant SDR (SI-SDR)

To address these shortcomings, the authors propose a modified version of SDR, termed Scale-Invariant SDR (SI-SDR). SI-SDR introduces a robust approach by rescaling either the target or estimated signals to ensure orthogonality between the target signal and the residual noise. The formula derived for SI-SDR ensures that it is invariant to linear scale changes, offering a fairer comparison between algorithms.

SI-SDR has the advantages of:

• Avoiding the overestimation of performance due to large permissible modifications in the reference.
• Streamlining computation, as it relies only on a scalar factor rather than a complex filter.
• Demonstrating resilience to artificial boosting of results by ensuring orthogonality in residuals.

Analysis of Failure Modes

The paper presents several critical analyses and empirical demonstrations where the conventional SDR fails but SI-SDR performs reliably:

1. Filtering Optimization: When a filter is optimized to minimize SI-SDR, conventional SDR scores remain high despite substantial modifications that deteriorate the signal perceptibly.
2. Frequency Bin Deletion: Progressive deletion of spectral bands shows that SDR can paradoxically increase despite clear signal degradation, while SI-SDR appropriately reflects the increasing distortion.
3. Varying Band-Stop Filter Gain: When varying the gain of a band-stop filter applied to a noisy signal, conventional SDR improves as the signal spectrum is suppressed, whereas SI-SDR and SNR metrics provide a more accurate degradation measurement.

Performance Comparison

The paper also includes a performance comparison on the wsj0-2mix dataset to investigate the practical implications of using SI-SDR versus conventional SDR. Results indicate consistent differences between SDR and SI-SDR values, typically around 0.5 dB, highlighting the shortcomings of SDR in reflecting true perceptual quality. Algorithms employing SI-SDR show a more nuanced performance assessment, beneficial in fields where signal quality is imperative.

Implications and Future Directions

The proposed SI-SDR has significant practical and theoretical implications:

• Practical Impact: Provides a reliable measure for comparing the effectiveness of algorithms in single-channel source separation and speech enhancement tasks. This ensures appropriated acknowledgment of genuine advancements without falling prey to illusory improvements.
• Theoretical Implications: Encourages more thorough consideration of scaling issues and spectral integrity in developing causal and non-causal filters for signal processing.

The authors also suggest a potential scale-dependent SDR (SD-SDR) for cases where rescaling needs to be penalized, offering a solution to account for scaling as a form of distortion. Future research could further refine these metrics and explore their application across varied contexts in speech and audio processing, including other non-stationary signals beyond speech.

In summary, Jonathan Le Roux et al. propose a more reliable and insightful alternative to traditional SDR, advocating for the adoption of SI-SDR in single-channel source separation and speech enhancement evaluation to ensure that performance metrics are both fair and reflective of true algorithmic advancement.