Sound Event Localization & Detection

Updated 14 October 2025

Sound Event Localization and Detection is the process of simultaneously identifying, classifying, and spatially localizing overlapping sound events in complex environments.
Modern SELD frameworks leverage convolutional recurrent neural networks and signal processing techniques to extract spatial cues and manage challenges like polyphony and reverberation.
SELD systems are evaluated with unified metrics combining detection accuracy, localization error, and trackwise association to ensure robust performance.

Sound Event Localization and Detection (SELD) is the integrated computational task of jointly identifying, classifying, and spatially localizing temporally overlapping sound events in a scene. SELD is a core capability for intelligent auditory systems, with applicability in autonomous robotics, surveillance, augmented reality, smart environments, and immersive multimedia. Modern SELD systems typically employ microphone arrays or binaural input to estimate the time–frequency activity of discrete events (sound event detection, SED) and their 3D direction-of-arrival (DOA), often under conditions of polyphony, reverberation, and low signal-to-noise ratio. Recent SELD frameworks span single-stage deep learning pipelines, cascaded two-stage approaches, parametric–data-driven hybrids, and event–trackwise architectures, all evaluated using unified metrics of detection, localization, and their joint association.

1. Fundamental Concepts and Input Representations

SELD unifies SED and spatial localization; the system inputs are raw or transformed multichannel audio—FOA, binaural, or array signals—and the output per time frame includes active event class probabilities and their spatial positions (azimuth, elevation, optionally distance). Key spatial cues include interaural time difference (ITD), interaural level difference (ILD), spatial harmonics (FOA), and head-related transfer function (HRTF)-induced spectral notches. Most SELD pipelines transform input signals using the short-time Fourier transform (STFT) to obtain complex spectral coefficients. Features split into:

Magnitude and phase spectrograms: Directly input to CNNs/CRNNs for generality across array formats (Adavanne et al., 2018).
Spatial features: Cross-spectral phase (GCC-PHAT), IPD/ILD, intensity vectors, and custom input matrices like BTFF for binaural cues (Lee et al., 28 Jul 2025, Lee, 6 Aug 2025).
Learned spatial features: Recent innovations include neural generalized cross-correlation (NGCC-PHAT), which applies learnable filterbanks for TDOA estimation in overlapping scenarios (Berg et al., 30 Aug 2024).

Data-driven systems often augment input data via spatial rotations, channel swapping, or additional simulated reverb/noise (Shimada et al., 2020, Ronchini et al., 2020, Roman et al., 29 Jan 2024).

2. Neural and Parametric Architectures

The prevailing model architecture for joint SELD is the convolutional recurrent neural network (CRNN), which combines convolutional layers (for local time–frequency pattern extraction) with recurrent layers (bidirectional GRUs or LSTMs for temporal context) and splits into parallel output heads:

SED Head: Multi-label frame-wise classification, typically via sigmoid activation, predicting active classes per frame.
DOA Head: Cartesian (x, y, z) regression on the unit sphere (SELDnet (Adavanne et al., 2018)), or classification on discrete DOA grids, or angle regression.
Trackwise/Permutation-invariant Outputs: To accommodate same-class overlaps and multiple concurrent events, trackwise outputs and permutation-invariant losses (e.g., PIT/ADPIT) are used (Hu et al., 2022, Berg et al., 30 Aug 2024).

Alternative strategies include:

Two-stage or modular approaches: Decoupling SED and DOA branches to prevent task interference, sometimes with explicit “masking” of the DOA training to active event frames (Cao et al., 2019, Nguyen et al., 2019, Yu, 30 Jul 2025).
Hybrid parametric/data-driven: Using parametric DOA estimation as a front-end to segment events, followed by deep classification of beamformed monophonic signals (1908.10133).
Self-supervised pre-training: Adapting wav2vec 2.0–style self-supervised learning—w2v-SELD—to leverage large unlabeled spatial audio corpora, yielding robust representations prior to fine-tuning for SELD (Santos et al., 2023).

3. Association and Output Mechanisms

A critical component is resolving the correspondence between detected event classes and spatial locations, especially with overlapping sources:

Dual-branch linking: SELDnet-style coupling of parallel SED and DOA outputs associates the two branches by masking DOA estimates with class activity (Adavanne et al., 2018, Shimada et al., 2020).
Spatial stream segregation: Probabilistic soft-masks generated from spatial cues (ITD/ILD) segregate the mixture into candidate streams per spatial region, where detection proceeds per stream (Trowitzsch et al., 2019).
Trackwise, permutation-invariant schemes: Outputs organized into fixed tracks per time frame, with assignment ambiguity resolved via permutation-invariant training, facilitate explicit modeling of polyphonic and same-class overlaps (Hu et al., 2022, Krause et al., 18 Mar 2024).

For 3D SELD, novel representations extend the output vector to include distance estimation alongside DOA (Krause et al., 18 Mar 2024), while for binaural/bio-inspired systems, feature design explicitly mimics human HRTF cues for vertical and front-back disambiguation (Lee et al., 28 Jul 2025, Lee, 6 Aug 2025).

4. Evaluation Protocols and Metrics

SELD systems are evaluated using metrics reflecting both detection and localization, and (when possible) their joint accurate association:

Detection (SED): Error rate (ER) and F-score, typically on one-second non-overlapping frames.
Localization (DOA): Localization error (LE or DOAE), computed as angular distance between reference and prediction, and localization recall (LR), the fraction of correctly localized events per time or per event.
Joint metrics: Event-based metrics using the Hungarian algorithm for optimal assignment (see formulas (1)-(4) in (Politis et al., 2020)) to produce class-dependent LE_CD and LR_CD, and location-dependent F-score F_LD (Politis et al., 2020, Berg et al., 30 Aug 2024).
Association-aware error rate: Event predictions are scored as correct only when both class and spatial proximity match the reference (typically within thresholds of 10° or 20°).

Error analysis reveals that strong frame-level recall, not just DOA regression accuracy, is essential for joint SELD performance—robustness to spatially shifted sources and ability to recover the correct number of events are critical (Adavanne et al., 2018, Trowitzsch et al., 2019, Politis et al., 2020).

5. Core Methodological Advances

Significant methodological developments include:

Activity-coupled output vectors: ACCDOA or multi-ACCDOA frameworks integrate activity and location in a single unified target, streamlining training and inference (Shimada et al., 2020, Krause et al., 18 Mar 2024).
Trackwise consistent outputs: By enforcing temporal consistency in event-to-track assignments, frameworks improve event tracking and facilitate subsequent processing like beamforming (Yu, 30 Jul 2025).
Feature learning for spatial cues: NGCC-PHAT uses learnable filterbanks and permutation-invariant losses to recover multiple TDOAs per frame for spatially entangled sources, outperforming fixed GCC-PHAT in polyphonic scenarios (Berg et al., 30 Aug 2024).
Binaural feature design: The BTFF incorporates mel-spectrogram, velocity, ITD, ILD, and SC maps to explicitly encode both spectral and HRTF-based localization cues, enabling accurate azimuth and elevation estimation with only two channels (Lee et al., 28 Jul 2025, Lee, 6 Aug 2025).

A plausible implication is that further improvements in SELD will likely arise from input representations that better disentangle spatial cues under polyphony, loss functions that directly optimize angular association, and architectures that allow explicit event–track modeling.

6. Error Analysis, Limitations, and Perspectives

Comprehensive error analysis highlights persistent challenges:

Polyphony: Systems struggle to detect and localize all overlapping events, with recall and localization recall dropping as polyphony rises. Most systems perform best at the polyphony levels dominant in training data (Nguyen et al., 2021).
Reverberation, interference: Reverberant scenes with unknown directional interferences significantly increase substitution and misassociation errors.
Ambiguity in track assignments: Unless models explicitly enforce trackwise consistency, event instances may drift across tracks, complicating association (Yu, 30 Jul 2025).
Impact of dataset design: Training set balance over polyphony, class, and spatial condition is a primary factor influencing observed error rates and system generalizability.

Future directions suggested include refinement of data augmentation—particularly targeting hard SNR, polyphony, and spatial scenarios—and the design of explicit association mechanisms or attention models to tackle event–location assignment under high polyphony (Nguyen et al., 2021, Hu et al., 2022).

7. Applications and Implications

SELD systems are central to a broad array of applications, including:

Surveillance and smart environments: Triggering alarms or retrieving relevant information based on spatio-temporally localized events (Adavanne et al., 2018).
Robotics and autonomous navigation: Spatial scene understanding for human–robot interaction or obstacle avoidance; binaural SELD specifically targets humanoid robot audition (Lee et al., 28 Jul 2025, Lee, 6 Aug 2025).
Assistive technology and AR/VR: Enhancing immersion or awareness by real-time alignment of audio streams with virtual or visual content (Roman et al., 29 Jan 2024).
Real-world audio–visual fusion: Merging audio and visual cues improves localization and event association, especially when audio cues are ambiguous or events are occluded (Roman et al., 29 Jan 2024).

A plausible implication is that as SELD architectures mature, real-world deployment will increasingly rely on robust, generalized systems resilient to domain shifts, polyphonic clutter, and mismatches in sensor geometry, supported by continuous advances in multi-modal data fusion, self-supervised pre-training, and unsupervised association learning.