BiSELDnet: Binaural Sound Event Localization

• BiSELDnet is a binaural neural network that integrates sound event detection and 3D localization using two-channel audio and HRTF cues.
• It uses a CRNN architecture with convolutional and recurrent layers to regress class-wise unit direction vectors based on an ACCDOA-style representation.
• The model leverages interaural time, level, and spectral cues to overcome front-back ambiguities and improve elevation estimation in both synthetic and noisy settings.

Searching arXiv for the cited BiSELDnet papers to ground the article. BiSELDnet is a neural network for Binaural Sound Event Localization and Detection (BiSELD), a task that jointly detects sound event classes and localizes each active event in 3D space from two-channel binaural audio using head-related transfer function (HRTF) cues. In the cited work, BiSELDnet is defined as a CRNN that operates on the Binaural Time-Frequency Feature (BTFF), an eight-channel representation designed to encode interaural time difference (ITD), interaural level difference (ILD), and spectral cues (SC) alongside spectro-temporal information useful for detection. The model outputs class-wise 3D direction vectors in an ACCDOA-style formulation, thereby coupling detection and localization in a single regression target (Lee et al., 28 Jul 2025, Lee, 6 Aug 2025).

1. Definition and task formulation

BiSELD is defined as the task of detecting which sound event classes are active over time and localizing each active event in 3D space—specifically azimuth and elevation—using only two-channel binaural audio. In this formulation, the binaural signals are treated as they would be recorded by human-like ears on a humanoid robot’s head, and the system is explicitly motivated by human spatial hearing and HRTF-based cues (Lee et al., 28 Jul 2025).

BiSELDnet addresses this task by producing, for each time frame and each sound class, a 3D direction vector whose direction encodes the direction of arrival and whose magnitude encodes activity. This follows the ACCDOA idea: active classes are represented by non-zero Cartesian direction vectors, while inactive classes are represented by $(0,0,0)$ (Lee et al., 28 Jul 2025, Lee, 6 Aug 2025).

For the Cartesian encoding, the source direction is represented as

$$x = \cos\phi \cos\theta,\quad y = \cos\phi \sin\theta,\quad z = \sin\phi$$

in one formulation, and as

$$x = \cos\theta \cos\phi,\quad y = \cos\theta \sin\phi,\quad z = \sin\theta$$

in the other, reflecting the papers’ respective angle conventions for azimuth and elevation (Lee et al., 28 Jul 2025, Lee, 6 Aug 2025). In both cases, the central point is that BiSELDnet regresses unit Cartesian direction vectors and derives activity from vector magnitude, using a threshold of $0.5$ for activation (Lee et al., 28 Jul 2025, Lee, 6 Aug 2025).

This formulation situates BiSELDnet as the binaural counterpart of SELD systems standardized in DCASE, but with an explicit emphasis on the difficulties of two-channel sensing: front-back ambiguity, elevation estimation, and dependence on learned HRTF localization cues rather than direct multichannel array geometry (Lee et al., 28 Jul 2025, Lee, 6 Aug 2025).

2. Binaural Set and data generation

The principal benchmark associated with BiSELDnet is the Binaural Set, a synthetic dataset generated by convolving mono sound events with measured HRIRs. In one description, the HRIRs are taken from the KAIST HRTF database and selected at 12 azimuths from $-180^\circ$ to $+180^\circ$ in $30^\circ$ steps and 4 elevations from $-30^\circ$ to $+60^\circ$ in $30^\circ$ steps, yielding 48 distinct spatial directions (Lee et al., 28 Jul 2025). In the other description, the HRTF analysis is based on a B&K HATS dummy head measured in an anechoic chamber over a denser grid, while the Binaural Set used for training and evaluation is still synthesized over 48 directions with 12 azimuths and 4 elevations (Lee, 6 Aug 2025).

Foreground events are drawn from NIGENS and DCASE 2016 Task 2, and all audio is resampled to 32 kHz in order to preserve high-frequency HRTF spectral cues below 16 kHz (Lee et al., 28 Jul 2025). The dataset uses 12 classes in BiSELDnet when constructing 60-s mixtures, with one event sample per class arranged temporally to form overlapping and non-overlapping scenes (Lee et al., 28 Jul 2025, Lee, 6 Aug 2025).

The reported dataset statistics include Train: 1,008 mixtures, Validation: 216 mixtures, and Test: 216 mixtures, together with Test-H and Test-V subsets for horizontal-plane and median-plane analysis. The total duration is reported as 60,480 s for the clean set, and the noisy setting extends this construction to SNRs of 30, 20, 10, 0 dB using urban background noise from DCASE2019 Task 1 (Lee et al., 28 Jul 2025, Lee, 6 Aug 2025).

Each synthesized sample is paired with a CSV annotation containing class label, onset time, offset time, azimuth, and elevation. These are converted to frame-wise labels in which each class at each frame is assigned either a unit Cartesian direction vector or $$x = \cos\phi \cos\theta,\quad y = \cos\phi \sin\theta,\quad z = \sin\phi$$0, again following the ACCDOA representation (Lee et al., 28 Jul 2025, Lee, 6 Aug 2025).

A plausible implication is that the Binaural Set was designed not merely as a generic binaural corpus, but as a controlled environment for isolating the contribution of ITD, ILD, and pinna-related spectral cues under both clean and noisy conditions.

3. Binaural Time-Frequency Feature

The Binaural Time-Frequency Feature (BTFF) is the input representation on which BiSELDnet operates. BTFF is an 8-channel tensor of shape $$x = \cos\phi \cos\theta,\quad y = \cos\phi \sin\theta,\quad z = \sin\phi$$1, where $$x = \cos\phi \cos\theta,\quad y = \cos\phi \sin\theta,\quad z = \sin\phi$$2 is the number of time frames and 64 is the number of mel-frequency bins (Lee et al., 28 Jul 2025, Lee, 6 Aug 2025).

The eight channels are:

1. Left mel-spectrogram
2. Right mel-spectrogram
3. Left velocity-map
4. Right velocity-map
5. ITD-map
6. ILD-map
7. Left spectral-cue map
8. Right spectral-cue map

These channels are constructed on a common time–mel grid and are explicitly intended to combine SED-oriented spectro-temporal information with psychoacoustically motivated spatial features (Lee et al., 28 Jul 2025, Lee, 6 Aug 2025).

The mel-spectrogram channels encode harmonic structure, modulations, onsets, and offsets. The V-map channels represent frame-wise temporal derivatives of the spectrogram and are intended to emphasize rising and decaying energy patterns, helping with event boundary detection (Lee et al., 28 Jul 2025, Lee, 6 Aug 2025).

The ITD-map is derived from the interaural phase relation and restricted to frequencies below 1.5 kHz, based on the rationale that ITD is robust and unambiguous in the low-frequency range. One formulation gives

$$x = \cos\phi \cos\theta,\quad y = \cos\phi \sin\theta,\quad z = \sin\phi$$3

with mel projection below $$x = \cos\phi \cos\theta,\quad y = \cos\phi \sin\theta,\quad z = \sin\phi$$4 kHz (Lee et al., 28 Jul 2025). The other expresses the same principle through a phase-derived delay

$$x = \cos\phi \cos\theta,\quad y = \cos\phi \sin\theta,\quad z = \sin\phi$$5

again used only for $$x = \cos\phi \cos\theta,\quad y = \cos\phi \sin\theta,\quad z = \sin\phi$$6 (Lee, 6 Aug 2025).

The ILD-map is computed from interaural level differences and restricted to frequencies above 5 kHz, where head shadow becomes strong and where the cited work reports front-back asymmetry as an informative cue. One expression is

$$x = \cos\phi \cos\theta,\quad y = \cos\phi \sin\theta,\quad z = \sin\phi$$7

with mel projection for $$x = \cos\phi \cos\theta,\quad y = \cos\phi \sin\theta,\quad z = \sin\phi$$8 (Lee et al., 28 Jul 2025). The second paper states the same channel as

$$x = \cos\phi \cos\theta,\quad y = \cos\phi \sin\theta,\quad z = \sin\phi$$9

used for $$x = \cos\theta \cos\phi,\quad y = \cos\theta \sin\phi,\quad z = \sin\theta$$0 (Lee, 6 Aug 2025).

The SC-maps are high-frequency mel-spectrograms above 5 kHz. They are intended to capture pinna-induced spectral structure, including the N1 and N2 notch behavior associated with elevation and front-back discrimination. In the later paper, the SC-map discussion is directly tied to PRTF analysis and to the observation that N1 around 8–10 kHz shifts upward in frequency with increasing elevation (Lee, 6 Aug 2025).

The BTFF design therefore mirrors a standard decomposition of human spatial hearing: ITD for low-frequency azimuth, ILD for high-frequency azimuth and front-back asymmetry, and spectral cues for elevation. This suggests that the representation is intended to embed domain knowledge before statistical learning, rather than leaving all cue extraction to the network.

4. Architecture and output mapping

BiSELDnet is consistently described as a CRNN that uses convolutional layers to extract local time-frequency patterns, recurrent layers to model temporal context, and dense layers to regress class-wise 3D direction vectors (Lee et al., 28 Jul 2025, Lee, 6 Aug 2025). Both descriptions use the same input tensor, $$x = \cos\theta \cos\phi,\quad y = \cos\theta \sin\phi,\quad z = \sin\theta$$1, and the same final output dimensionality, $$x = \cos\theta \cos\phi,\quad y = \cos\theta \sin\phi,\quad z = \sin\theta$$2, corresponding to 12 classes $$x = \cos\theta \cos\phi,\quad y = \cos\theta \sin\phi,\quad z = \sin\theta$$3 3 Cartesian components (Lee et al., 28 Jul 2025, Lee, 6 Aug 2025).

The two papers describe two architecture variants associated with the same overall BiSELDnet concept.

Source	Front-end	Temporal block	Output head
(Lee et al., 28 Jul 2025)	CNN stack with Conv(3×3, 64) blocks, BatchNorm, ReLU, MaxPool	[BiGRU](https://www.emergentmind.com/topics/bidirectional-gated-recurrent-unit-bigru)(128)	Dense(128) → Dense(72) → Dense(36), tanh
(Lee, 6 Aug 2025)	10 repeated Trinity modules with depthwise separable convolution	3 BiGRU layers: 512, 256, 128	Dense(128) → Dense(72) → Dense(36), tanh

In the earlier paper, the CNN front-end uses two convolutional blocks with pooling that reduce time by a factor of 5 and frequency to a small residual dimension before reshaping for the GRU (Lee et al., 28 Jul 2025). In the later paper, the selected architecture is BiSELDnet-v4, which replaces simpler convolutional blocks with Trinity modules composed of parallel depthwise separable convolution branches approximating $$x = \cos\theta \cos\phi,\quad y = \cos\theta \sin\phi,\quad z = \sin\theta$$4, $$x = \cos\theta \cos\phi,\quad y = \cos\theta \sin\phi,\quad z = \sin\theta$$5, and $$x = \cos\theta \cos\phi,\quad y = \cos\theta \sin\phi,\quad z = \sin\theta$$6 receptive fields, followed by concatenation, residual connection, batch normalization, and ReLU (Lee, 6 Aug 2025).

The use of depthwise separable convolution is justified in the later paper by the statement that BTFF has low cross-channel correlation, so channel-wise filtering followed by pointwise mixing is computationally appropriate (Lee, 6 Aug 2025). The later paper also reports parameter counts for several variants, including ~6.14 M parameters for the Trinity-based BiSELDnet-v4, ~6.45 M for an Xception-based version, and ~50 M for a hierarchical CRNN variant (Lee, 6 Aug 2025). The earlier paper reports a compact model size of 763,020 parameters for the CRNN configuration it describes (Lee et al., 28 Jul 2025).

In both descriptions, the final dense layer uses tanh activation so that outputs lie in $$x = \cos\theta \cos\phi,\quad y = \cos\theta \sin\phi,\quad z = \sin\theta$$7, consistent with Cartesian unit-vector targets (Lee et al., 28 Jul 2025, Lee, 6 Aug 2025). Detection is obtained by thresholding vector magnitude at 0.5, and localization is read off from vector direction (Lee et al., 28 Jul 2025, Lee, 6 Aug 2025).

This architectural continuity indicates that “BiSELDnet” names a family of binaural SELD models unified by the same task definition, BTFF input, and ACCDOA-style output, with the later Trinity-based variant extending the earlier CRNN design toward a larger and more explicitly multi-scale front-end.

5. Training objective, optimization, and interpretability

BiSELDnet is trained with a single MSE loss over the class-wise Cartesian vectors. In the earlier paper this is written as

$$x = \cos\theta \cos\phi,\quad y = \cos\theta \sin\phi,\quad z = \sin\theta$$8

while the later paper gives the same ACCDOA-style regression objective over samples and classes (Lee et al., 28 Jul 2025, Lee, 6 Aug 2025). The key point in both formulations is that inactive classes are pushed toward the zero vector and active classes are pushed toward the correct unit direction vector. No separate classification loss is used (Lee et al., 28 Jul 2025, Lee, 6 Aug 2025).

The optimization setup is also consistent across the descriptions: Adam, batch size 128, training for up to 1,000 epochs, with early stopping if validation performance does not improve for 50 epochs (Lee et al., 28 Jul 2025, Lee, 6 Aug 2025). The implementation environment reported in the earlier paper is Python + Keras + TensorFlow 2.5, running on 3× NVIDIA RTX 3090, 128 GB RAM, and Ubuntu 20.04 (Lee et al., 28 Jul 2025).

The later paper introduces Vector Activation Map (VAM), a Grad-CAM–style method adapted to vector regression. VAM uses the norm of the output direction vector as a scalar score, backpropagates it to a chosen pivot layer, computes channel-wise weights by global average pooling of gradients, forms a weighted feature map, applies ReLU, and resizes the resulting saliency map to the BTFF resolution (Lee, 6 Aug 2025).

VAM analysis is reported to show that, for baby crying in the median plane, the model focuses on the N1 notch region in the SC-map and on the time regions where the event is active (Lee, 6 Aug 2025). This supports the claim that BiSELDnet learns to exploit the same type of high-frequency pinna cues that underlie human elevation perception. A plausible implication is that the interpretability method is used not only to inspect performance but to validate the original psychoacoustic design of BTFF.

6. Evaluation and reported performance

The evaluation follows DCASE SELD metrics, including segment-level F-score and Error Rate (ER) for detection, Localization Error (LE) and Localization Recall (LR) for localization, and joint location-aware metrics such as $$x = \cos\theta \cos\phi,\quad y = \cos\theta \sin\phi,\quad z = \sin\theta$$9, $0.5$0, $0.5$1, $0.5$2, and the combined SELD error (Lee et al., 28 Jul 2025, Lee, 6 Aug 2025).

The earlier paper defines

$0.5$3

$0.5$4

and

$0.5$5

while the later paper presents the same metrics with the SED and DOA terms averaged by $0.5$6 before forming SELD error (Lee et al., 28 Jul 2025, Lee, 6 Aug 2025). Despite this presentation difference, both use the same DCASE-style joint evaluation framework.

BTFF ablation results

The earlier paper reports median results over 10 runs for the contribution of BTFF sub-features (Lee et al., 28 Jul 2025):

Input	Setting	Key reported effect
MS vs MS + V-map	Full test set	V-map improves detection
MS, MS + ITD, MS + ITD + ILD	Test-H	ITD and ILD improve horizontal localization
MS vs MS + SC-map	Test-V	SC-map improves vertical localization

For MS only versus MS + V-map, the paper reports improvement from SELD error = 0.214 to 0.189, with F$0.5$7 improving from 75.0% to 77.9% (Lee et al., 28 Jul 2025). For horizontal localization on Test-H, MS + ITD + ILD yields LE$0.5$8 = 4.2° and SELD error = 0.124, compared with 17.3° and 0.217 for MS only (Lee et al., 28 Jul 2025). For vertical localization on Test-V, MS + SC-map improves LE$0.5$9 from 25.2° to 12.2° and SELD error from 0.211 to 0.145 (Lee et al., 28 Jul 2025).

When all BTFF channels are used, the earlier paper reports ER$-180^\circ$0 = 0.210, F$-180^\circ$1 = 87.1%, LE$-180^\circ$2 = 4.4°, LR$-180^\circ$3 = 92.1%, DOA error = 0.052, and SELD error = 0.110 (Lee et al., 28 Jul 2025).

Architecture and noisy-condition comparisons

The later paper reports cleaner and more extensive comparisons across noisy conditions and baselines. For the clean architecture comparison, BiSELDnet-v4 reports ER$-180^\circ$4=0.114, F$-180^\circ$5=91.8%, LE$-180^\circ$6=2.5°, LR$-180^\circ$7=93.3%, SED error=0.098, DOA error=0.040, and SELD error=0.069 (Lee, 6 Aug 2025).

The same paper compares BiSELD against BL-SELD (ACCDOA) and SOTA-SELD (SALSA-Lite) under urban background noise. Examples reported include:

• SNR=30 dB (horizontal): BL-SELD ≈ 0.208, SOTA-SELD ≈ 0.193, BiSELD ≈ 0.118
• SNR=30 dB (median): BL-SELD 0.193, SOTA-SELD 0.146, BiSELD 0.069
• SNR=20 dB (horizontal): BL 0.227, SOTA 0.198, BiSELD 0.137
• SNR=10 dB (median): BL 0.238, SOTA 0.214, BiSELD 0.147
• SNR=0 dB (horizontal): BL 0.289, SOTA 0.254, BiSELD 0.214
• SNR=0 dB (median): BL 0.299, SOTA 0.252, BiSELD 0.219 (Lee, 6 Aug 2025)

The later paper therefore characterizes BiSELD as outperforming adapted baselines and the cited SALSA-Lite-based SELD baseline under all tested SNRs and on both horizontal and median-plane subsets (Lee, 6 Aug 2025).

7. Relation to human spatial hearing, limitations, and extensions

BiSELDnet is explicitly framed as a machine analogue of human-like auditory perception. The ITD-map is associated with low-frequency phase differences and the MSO, the ILD-map with high-frequency level differences and the LSO, and the SC-maps with pinna-related transfer functions and elevation-dependent notches such as N1 and N2 (Lee et al., 28 Jul 2025, Lee, 6 Aug 2025). The later paper states that VAM visualization confirms that the network focuses on the N1 notch frequency for elevation estimation (Lee, 6 Aug 2025).

The HRTF discussion in the later paper is comparatively detailed. It defines

$-180^\circ$8

discusses non-causality in ipsilateral HRIRs due to OTF referencing, and describes circular shifting by at least

$-180^\circ$9

to obtain causal HRIRs suitable for smooth ITD analysis (Lee, 6 Aug 2025). It also reports that low-frequency ITD is near zero at front and back and peaks at roughly $+180^\circ$0 near lateral directions, while high-frequency ILD exhibits strong directionality and front-back asymmetry (Lee, 6 Aug 2025). These observations are used to justify the frequency partition employed in BTFF.

Several limitations are stated. Both papers note reliance on synthetic data and on a single HRTF source, whether framed as the KAIST HRTF or the measured HATS HRTF, and both caution that generalization to different heads, microphone placements, and acoustic environments is not yet demonstrated (Lee et al., 28 Jul 2025, Lee, 6 Aug 2025). Both also note that the present formulation usually assumes at most one active source per class per frame and therefore does not directly solve same-class polyphony; multi-ACCDOA and permutation-invariant training are mentioned as possible extensions (Lee et al., 28 Jul 2025, Lee, 6 Aug 2025).

Further directions described in the data include real-world evaluation, moving sources and head rotations, HRTF personalization or transfer, and more efficient implementations through pruning, quantization, or lighter architectures (Lee et al., 28 Jul 2025, Lee, 6 Aug 2025). The later paper also suggests improving the SC-map through a linear frequency scale above 12 kHz to better resolve N2 notches for robust elevation estimation (Lee, 6 Aug 2025).

Taken together, these papers define BiSELDnet as a binaural SELD framework centered on a psychoacoustically informed representation, a CRNN family of regressors with ACCDOA-like outputs, and a research program aimed at enabling full 3D localization from two horizontal channels for humanoid robots and related binaural sensing platforms (Lee et al., 28 Jul 2025, Lee, 6 Aug 2025).