Polyphonic Sound Detection Score (PSDS)

• PSDS is a metric that quantifies polyphonic sound event detection accuracy by integrating over a range of decision thresholds.
• It evaluates both event localization and classification using intersection-based criteria with defined tolerances like DTC, GTC, and CTTC.
• Widely adopted in DCASE challenges, PSDS guides robust SED system development and benchmarks against conventional F1 and segment-based metrics.

The Polyphonic Sound Detection Score (PSDS) is a scalar performance metric developed for polyphonic sound event detection (SED), particularly in scenarios characterized by simultaneous, overlapping, and often ambiguously annotated sound events. PSDS quantifies system accuracy in event localization and classification by integrating over the entire trade-off curve between true and false positives, rather than evaluating performance at a single operating threshold. Its design explicitly addresses the limitations of conventional metrics—such as event-based and segment-based F-scores—by accommodating decision threshold variability, onset/offset uncertainty, polyphony, and application-specific tolerance requirements. The metric was introduced by Bilen et al. and is now standard in evaluations such as DCASE Task 4, having considerable influence on system development, benchmarking, and comparison across SED research (Bilen et al., 2019, Hu et al., 2022, Ronchini et al., 2022).

1. Conceptual Foundations and Motivation

SED systems historically relied on event-based or segment-based metrics that assess the temporal presence of target classes through frame- or event-wise comparison. Segment-based metrics (e.g., segment error rate) evaluate fixed-length audio windows, lacking fine temporal granularity. Conventional event-based F1-scores require precise correspondence (typically within fixed "collars" such as ±200 ms) and are highly sensitive to annotation subjectivity, which impacts reliability in benchmarking polyphonic outputs (Ferroni et al., 2020). PSDS unifies these perspectives by measuring detection performance as a function of system confidence (threshold), explicitly tolerating boundary imprecision, and more robustly modeling overlaps and cross-triggers (detections in the wrong class). By integrating over a range of detection thresholds, PSDS produces an operating-point-independent, single-value summary that more accurately reflects a system’s pragmatic utility and generalization capacity (Bilen et al., 2019, Ebbers et al., 2022).

2. Mathematical Formulation and Operating-Point Aggregation

Let $G = \{g_1, \dots, g_N\}$ represent ground-truth events, and $D = \{d_1, \dots, d_M\}$ represent detected events, each tagged with class, onset, offset, and confidence score. PSDS relies on intersection-based event matching with three core parameters:

• Detection Tolerance Criterion (DTC): fraction of a detection that must overlap with any ground-truth of the same class.
• Ground-Truth Intersection Criterion (GTC): fraction of a ground-truth event that must be overlapped by any detection of the same class.
• Cross-Trigger Tolerance Criterion (CTTC): quantification of false positives resulting from detections that overlap with ground truth events of incorrect classes.

For a grid of operating points (thresholds), detections are matched to references using these criteria to produce counts:

• TP (true positives): correct matches.
• FP (false positives): unmatched detections (including cross-triggers, scaled by a penalty $\alpha_{CT}$).
• FN (false negatives): unmatched ground-truth events. Class-wise true positive ratio and effective false positive rate are defined as: $$r^*_{TP,c}(\tau) = \frac{N^*_{TP,c}(\tau)}{|\mathcal{Y}_c|}, \quad e^*_c(\tau) = R^*_{FP,c}(\tau) + \alpha_{CT} \cdot \frac{1}{|C|-1} \sum_{\hat{c}\neq c} R^*_{CT,c,\hat{c}}(\tau)$$ where $R^*_{FP,c}$ and $R^*_{CT,c,\hat{c}}$ are normalized rates as per (Bilen et al., 2019, Ebbers et al., 2022).

The PSD-ROC curve is an upper-envelope, class-averaged plot of $r(e)$ across all operating points: $$r(e) = \mu_{TP}(e) - \alpha_{ST} \cdot \sigma_{TP}(e)$$ where $\mu_{TP}(e)$ is the mean true positive proportion across classes and $\sigma_{TP}(e)$ its standard deviation (penalizing class imbalance). PSDS is the normalized area under this curve up to a maximum acceptable false positive rate $e_{max}$: $$\mathrm{PSDS} = \frac{1}{e_{max}}\int_0^{e_{max}} r(e) \, de$$ In practice, this integral is evaluated by Riemann sum over thresholds. System outputs are typically post-processed (e.g., median filtering, minimum event duration), and all matching is performed with application-specified DTC, GTC, CTTC, and $\alpha_{CT}$ (Ebbers et al., 2023, Ebbers et al., 2022).

3. PSDS Variants, Application Scenarios, and Parameterization

The DCASE challenges standardized multiple PSDS variants to represent divergent application contexts (Ronchini et al., 2022):

• PSDS1 ("fine-grained localization, fast reaction"): Enforces stringent overlap criteria (DTC and GTC typically set to 0.7), does not penalize cross-triggers beyond counting as false alarms ($\alpha_{CT}=0$), favors rapid, temporally precise detection scenarios (e.g., alarms, robotics).
• PSDS2 ("coarse localization, class-confusion avoidance"): Relaxes overlaps (DTC and GTC typically 0.1), applies significant cross-trigger penalty ($\alpha_{CT}=0.5$), emphasizes label purity over boundary precision, suited for long-duration monitoring (Ronchini et al., 2022, Nam et al., 2021, Hai et al., 23 Sep 2025).

Parameter configurations—including threshold grid, false positive rate bounds, and post-processing—are fixed by challenge protocol to ensure reproducibility and fair system comparison. The choice of $e_{max}$ (FPR limit) strongly influences PSDS: lower values restrict to low-false-alarm operation; higher values permit broader FPR ranges (Ebbers et al., 2022, Nam et al., 2024).

4. Implementation, Post-Processing, and Threshold-Independence

PSDS is implemented via a sweep over decision thresholds (and, optionally, multiple post-processing strategies) on class-probability outputs. The most widely used software stacks are referenced in DCASE codebases and the sed_scores_eval package (Ebbers et al., 2023, Ebbers et al., 2022). The evaluation pipeline consists of:

• Extracting scored event hypotheses for each possible threshold.
• Matching events to ground truth using intersection-based tolerances for both boundaries and class correspondences.
• Aggregating the per-threshold (or per-(post-processing, threshold) tuple) TP/FP/FN/CT statistics to derive $r(e)$ for each $e$.
• Integrating $r(e)$ up to $e_{max}$ for final PSDS computation.

Extensions such as post-processing independent PSDS (piPSDS) generalize the metric by maximizing $r_c(e)$ across multiple post-processing schemes at each $e$, providing a best-case envelope free from hyperparameter selection bias (Ebbers et al., 2023). Median-filter independent PSDS (miPSDS) is a special case aggregating over different median filter lengths.

5. Comparative Analysis with Classical Metrics

Event-based F1-score metrics demand explicit matching within fixed time collars (e.g., ±200 ms) and treat all events equally, but their strictness varies with event duration—tolerant on short events, unforgiving on long—and are prone to annotation subjectivity. Segment-based metrics use co-occurrence within interval bins but cannot resolve fine event boundaries and their temporal granularity is arbitrary (Ferroni et al., 2020). PSDS, via intersection-based matching, attains invariance to event length and boundary imprecision, and—by integrating over the entire TPR–FPR operating regime—produces summary performance figures analogous to area-under-curve metrics in detection theory, but tuned for polyphonic, multi-class, temporally imprecise problems. Systems that maximize single-point F1 but behave poorly at other thresholds are strongly penalized by PSDS, making it a stricter and more holistic evaluation (Bilen et al., 2019, Ferroni et al., 2020).

Metric Type	Threshold Dependent	Tolerance Type	Robust to Boundary Subjectivity	Polyphony Integration
F1, event	Yes	Collar-based	No	Only split/merge
F1, segment	Yes	Windowed/interval	Partial	Yes
PSDS	No	Intersection-based	Yes	Full

6. Influence in SED System Development and Benchmarking

PSDS has become the principal system selection and benchmarking criterion in DCASE Task 4 and associated SED research, driving architectural and data-centric advances. For example, multi-branch and multi-dilated convolutional architectures are evaluated with PSDS1 to optimize for robust event separation across thresholds (Nam et al., 2024). Data augmentation strategies, such as generative sample synthesis, are now compared by gains in both PSDS1 and PSDS2, with results showing a ~5% PSDS1 improvement from targeted, temporally coherent augmentation (Hai et al., 23 Sep 2025). Comparative leaderboard gains are consistently reported as absolute increases in PSDS; e.g., MGA-Net surpasses prior SED baselines with +0.021 to +0.027 gains (Hu et al., 2022). PSDS thus not only ranks systems more reliably but also guides robust, generalizable SED design (Ronchini et al., 2022).

7. Discussion of Limitations, Extensions, and Current Research

PSDS’s main limitations are its sensitivity to chosen tolerances ($\rho_{DTC}$, $\rho_{GTC}$, $\alpha_{CT}$, $e_{max}$), threshold grid density, and post-processing parameters. Results from different works are strictly comparable only under identical settings. There is also nontrivial computational overhead, particularly for exact threshold-sweeping and multi-dimensional post-processing analyses (Ebbers et al., 2022). Current extensions—including piPSDS—address post-processing bias, enabling fairer competition without hyperparameter overfitting (Ebbers et al., 2023). Variants such as "true PSDS1," which use bounding-box post-processing, further enhance threshold-independence and temporal boundary robustness (Nam et al., 2024). PSDS’s continued adoption in new SED tasks and synthetic soundscape experiments affirms its value as a robust metric, capturing both event-wise temporal precision and class discrimination across a wide range of application requirements (Ronchini et al., 2022, Hai et al., 23 Sep 2025).