---
title: 'BiSELDnet: Binaural Sound Event Localization'
url: https://www.emergentmind.com/topics/biseldnet
type: topic
---

# BiSELDnet: Binaural Sound Event Localization

Searching arXiv for the cited BiSELDnet papers to ground the article.
BiSELDnet is a neural network for **Binaural Sound Event Localization and Detection (BiSELD)**, a task that jointly detects sound event classes and localizes each active event in **3D space** from **two-channel binaural audio** using **head-related transfer function (HRTF)** cues. In the cited work, BiSELDnet is defined as a **CRNN** that operates on the **Binaural Time-Frequency Feature (BTFF)**, an eight-channel representation designed to encode **interaural time difference (ITD)**, **interaural level difference (ILD)**, and **spectral cues (SC)** alongside spectro-temporal information useful for detection. The model outputs class-wise **3D direction vectors** in an **ACCDOA-style** formulation, thereby coupling detection and localization in a single regression target [2507.20530], [2508.04333].

## 1. Definition and task formulation

BiSELD is defined as the task of detecting which sound event classes are active over time and localizing each active event in **3D space**—specifically azimuth and elevation—using only **two-channel binaural audio**. In this formulation, the binaural signals are treated as they would be recorded by human-like ears on a humanoid robot’s head, and the system is explicitly motivated by **human spatial hearing** and **HRTF-based cues** [2507.20530].

BiSELDnet addresses this task by producing, for each time frame and each sound class, a **3D direction vector** whose **direction** encodes the direction of arrival and whose **magnitude** encodes activity. This follows the **ACCDOA** idea: active classes are represented by non-zero Cartesian direction vectors, while inactive classes are represented by \((0,0,0)\) [2507.20530], [2508.04333].

For the Cartesian encoding, the source direction is represented as

\[
x = \cos\phi \cos\theta,\quad
y = \cos\phi \sin\theta,\quad
z = \sin\phi
\]

in one formulation, and as

\[
x = \cos\theta \cos\phi,\quad
y = \cos\theta \sin\phi,\quad
z = \sin\theta
\]

in the other, reflecting the papers’ respective angle conventions for azimuth and elevation [2507.20530], [2508.04333]. In both cases, the central point is that BiSELDnet regresses **unit Cartesian direction vectors** and derives activity from vector magnitude, using a threshold of \(0.5\) for activation [2507.20530], [2508.04333].

This formulation situates BiSELDnet as the binaural counterpart of SELD systems standardized in DCASE, but with an explicit emphasis on the difficulties of **two-channel** sensing: **front-back ambiguity**, **elevation estimation**, and dependence on learned **HRTF localization cues** rather than direct multichannel array geometry [2507.20530], [2508.04333].

## 2. Binaural Set and data generation

The principal benchmark associated with BiSELDnet is the **Binaural Set**, a synthetic dataset generated by convolving mono sound events with measured **HRIRs**. In one description, the HRIRs are taken from the **KAIST HRTF database** and selected at **12 azimuths** from \(-180^\circ\) to \(+180^\circ\) in \(30^\circ\) steps and **4 elevations** from \(-30^\circ\) to \(+60^\circ\) in \(30^\circ\) steps, yielding **48 distinct spatial directions** [2507.20530]. In the other description, the HRTF analysis is based on a **B&K HATS dummy head** measured in an anechoic chamber over a denser grid, while the Binaural Set used for training and evaluation is still synthesized over **48 directions** with **12 azimuths** and **4 elevations** [2508.04333].

Foreground events are drawn from **NIGENS** and **DCASE 2016 Task 2**, and all audio is resampled to **32 kHz** in order to preserve high-frequency HRTF spectral cues below 16 kHz [2507.20530]. The dataset uses **12 classes** in BiSELDnet when constructing **60-s mixtures**, with one event sample per class arranged temporally to form overlapping and non-overlapping scenes [2507.20530], [2508.04333].

The reported dataset statistics include **Train: 1,008 mixtures**, **Validation: 216 mixtures**, and **Test: 216 mixtures**, together with **Test-H** and **Test-V** subsets for horizontal-plane and median-plane analysis. The total duration is reported as **60,480 s** for the clean set, and the noisy setting extends this construction to SNRs of **30, 20, 10, 0 dB** using urban background noise from **DCASE2019 Task 1** [2507.20530], [2508.04333].

Each synthesized sample is paired with a CSV annotation containing class label, onset time, offset time, azimuth, and elevation. These are converted to **frame-wise labels** in which each class at each frame is assigned either a unit Cartesian direction vector or \((0,0,0)\), again following the **ACCDOA** representation [2507.20530], [2508.04333].

A plausible implication is that the Binaural Set was designed not merely as a generic binaural corpus, but as a controlled environment for isolating the contribution of **ITD**, **ILD**, and **pinna-related spectral cues** under both clean and noisy conditions.

## 3. Binaural Time-Frequency Feature

The **Binaural Time-Frequency Feature (BTFF)** is the input representation on which BiSELDnet operates. BTFF is an **8-channel** tensor of shape \(T \times 64 \times 8\), where \(T\) is the number of time frames and **64** is the number of mel-frequency bins [2507.20530], [2508.04333].

The eight channels are:

1. Left mel-spectrogram  
2. Right mel-spectrogram  
3. Left velocity-map  
4. Right velocity-map  
5. ITD-map  
6. ILD-map  
7. Left spectral-cue map  
8. Right spectral-cue map

These channels are constructed on a common time–mel grid and are explicitly intended to combine **SED-oriented** spectro-temporal information with **psychoacoustically motivated** spatial features [2507.20530], [2508.04333].

The mel-spectrogram channels encode harmonic structure, modulations, onsets, and offsets. The **V-map** channels represent frame-wise temporal derivatives of the spectrogram and are intended to emphasize **rising** and **decaying** energy patterns, helping with event boundary detection [2507.20530], [2508.04333].

The **ITD-map** is derived from the interaural phase relation and restricted to frequencies **below 1.5 kHz**, based on the rationale that ITD is robust and unambiguous in the low-frequency range. One formulation gives

\[
A_t(m,k) = -\operatorname{Im}\ln\left(\frac{P_R(m,k)}{P_L(m,k)}\right)
\]

with mel projection below \(1.5\) kHz [2507.20530]. The other expresses the same principle through a phase-derived delay

\[
\Delta\tau(m,k) = - \frac{1}{\omega_k}\phi_Z(m,k),
\]

again used only for \(f \le 1.5\text{ kHz}\) [2508.04333].

The **ILD-map** is computed from interaural level differences and restricted to frequencies **above 5 kHz**, where head shadow becomes strong and where the cited work reports **front-back asymmetry** as an informative cue. One expression is

\[
\Delta S(m,k) = 10\log_{10}\left|\frac{P_R(m,k)}{P_L(m,k)}\right|^2,
\]

with mel projection for \(k > k_{5000}\) [2507.20530]. The second paper states the same channel as

\[
\Delta S(m,k) = 10 \log_{10} \frac{|P_R(m,k)|^2}{|P_L(m,k)|^2} = S_R(m,k) - S_L(m,k),
\]

used for \(f \ge 5\text{ kHz}\) [2508.04333].

The **SC-maps** are high-frequency mel-spectrograms above **5 kHz**. They are intended to capture pinna-induced spectral structure, including the **N1** and **N2** notch behavior associated with elevation and front-back discrimination. In the later paper, the SC-map discussion is directly tied to **PRTF** analysis and to the observation that **N1 around 8–10 kHz** shifts upward in frequency with increasing elevation [2508.04333].

The BTFF design therefore mirrors a standard decomposition of human spatial hearing: **ITD** for low-frequency azimuth, **ILD** for high-frequency azimuth and front-back asymmetry, and **spectral cues** for elevation. This suggests that the representation is intended to embed domain knowledge before statistical learning, rather than leaving all cue extraction to the network.

## 4. Architecture and output mapping

BiSELDnet is consistently described as a **CRNN** that uses convolutional layers to extract local time-frequency patterns, recurrent layers to model temporal context, and dense layers to regress class-wise 3D direction vectors [2507.20530], [2508.04333]. Both descriptions use the same input tensor, \(T \times 64 \times 8\), and the same final output dimensionality, \(T' \times 36\), corresponding to **12 classes \(\times\) 3 Cartesian components** [2507.20530], [2508.04333].

The two papers describe two architecture variants associated with the same overall BiSELDnet concept.

| Source | Front-end | Temporal block | Output head |
|---|---|---|---|
| [2507.20530] | CNN stack with `Conv(3×3, 64)` blocks, BatchNorm, ReLU, MaxPool | `BiGRU(128)` | Dense(128) → Dense(72) → Dense(36), tanh |
| [2508.04333] | 10 repeated **Trinity modules** with depthwise separable convolution | 3 BiGRU layers: 512, 256, 128 | Dense(128) → Dense(72) → Dense(36), tanh |

In the earlier paper, the CNN front-end uses two convolutional blocks with pooling that reduce time by a factor of **5** and frequency to a small residual dimension before reshaping for the GRU [2507.20530]. In the later paper, the selected architecture is **BiSELDnet-v4**, which replaces simpler convolutional blocks with **Trinity modules** composed of parallel depthwise separable convolution branches approximating **\(3\times3\)**, **\(5\times5\)**, and **\(7\times7\)** receptive fields, followed by concatenation, residual connection, batch normalization, and ReLU [2508.04333].

The use of depthwise separable convolution is justified in the later paper by the statement that BTFF has **low cross-channel correlation**, so channel-wise filtering followed by pointwise mixing is computationally appropriate [2508.04333]. The later paper also reports parameter counts for several variants, including **~6.14 M parameters** for the Trinity-based BiSELDnet-v4, **~6.45 M** for an Xception-based version, and **~50 M** for a hierarchical CRNN variant [2508.04333]. The earlier paper reports a compact model size of **763,020 parameters** for the CRNN configuration it describes [2507.20530].

In both descriptions, the final dense layer uses **tanh** activation so that outputs lie in \([-1,1]\), consistent with Cartesian unit-vector targets [2507.20530], [2508.04333]. Detection is obtained by thresholding vector magnitude at **0.5**, and localization is read off from vector direction [2507.20530], [2508.04333].

This architectural continuity indicates that “BiSELDnet” names a family of binaural SELD models unified by the same task definition, BTFF input, and ACCDOA-style output, with the later Trinity-based variant extending the earlier CRNN design toward a larger and more explicitly multi-scale front-end.

## 5. Training objective, optimization, and interpretability

BiSELDnet is trained with a **single MSE loss** over the class-wise Cartesian vectors. In the earlier paper this is written as

\[
\mathcal{L}_{\text{MSE}} = \frac{1}{T C} \sum_{t=1}^{T} \sum_{c=1}^{C} \|\hat{\mathbf{d}}_{t,c} - \mathbf{d}_{t,c}\|_2^2,
\]

while the later paper gives the same ACCDOA-style regression objective over samples and classes [2507.20530], [2508.04333]. The key point in both formulations is that inactive classes are pushed toward the zero vector and active classes are pushed toward the correct unit direction vector. No separate classification loss is used [2507.20530], [2508.04333].

The optimization setup is also consistent across the descriptions: **Adam**, **batch size 128**, training for up to **1,000 epochs**, with **early stopping** if validation performance does not improve for **50 epochs** [2507.20530], [2508.04333]. The implementation environment reported in the earlier paper is **Python + Keras + TensorFlow 2.5**, running on **3× NVIDIA RTX 3090**, **128 GB RAM**, and **Ubuntu 20.04** [2507.20530].

The later paper introduces **Vector Activation Map (VAM)**, a Grad-CAM–style method adapted to vector regression. VAM uses the norm of the output direction vector as a scalar score, backpropagates it to a chosen pivot layer, computes channel-wise weights by global average pooling of gradients, forms a weighted feature map, applies ReLU, and resizes the resulting saliency map to the BTFF resolution [2508.04333].

VAM analysis is reported to show that, for **baby crying** in the median plane, the model focuses on the **N1 notch region** in the SC-map and on the time regions where the event is active [2508.04333]. This supports the claim that BiSELDnet learns to exploit the same type of high-frequency pinna cues that underlie human elevation perception. A plausible implication is that the interpretability method is used not only to inspect performance but to validate the original psychoacoustic design of BTFF.

## 6. Evaluation and reported performance

The evaluation follows **DCASE SELD metrics**, including segment-level **F-score** and **Error Rate (ER)** for detection, **Localization Error (LE)** and **Localization Recall (LR)** for localization, and joint location-aware metrics such as \(ER_{20^\circ}\), \(F_{20^\circ}\), \(LE_{CD}\), \(LR_{CD}\), and the combined **SELD error** [2507.20530], [2508.04333].

The earlier paper defines

\[
\text{SED error} = \text{ER}_{20^\circ} + (1 - \text{F}_{20^\circ}),
\]

\[
\text{DOA error} = \frac{\text{LE}_{CD}}{180} + (1 - \text{LR}_{CD}),
\]

and

\[
\text{SELD error} = \frac{\text{SED error} + \text{DOA error}}{2},
\]

while the later paper presents the same metrics with the SED and DOA terms averaged by \(1/2\) before forming SELD error [2507.20530], [2508.04333]. Despite this presentation difference, both use the same DCASE-style joint evaluation framework.

### BTFF ablation results

The earlier paper reports median results over **10 runs** for the contribution of BTFF sub-features [2507.20530]:

| Input | Setting | Key reported effect |
|---|---|---|
| **MS** vs **MS + V-map** | Full test set | V-map improves detection |
| **MS**, **MS + ITD**, **MS + ITD + ILD** | **Test-H** | ITD and ILD improve horizontal localization |
| **MS** vs **MS + SC-map** | **Test-V** | SC-map improves vertical localization |

For **MS only** versus **MS + V-map**, the paper reports improvement from **SELD error = 0.214** to **0.189**, with **F\(_{20^\circ}\)** improving from **75.0%** to **77.9%** [2507.20530]. For horizontal localization on **Test-H**, **MS + ITD + ILD** yields **LE\(_{CD}\) = 4.2°** and **SELD error = 0.124**, compared with **17.3°** and **0.217** for **MS only** [2507.20530]. For vertical localization on **Test-V**, **MS + SC-map** improves **LE\(_{CD}\)** from **25.2°** to **12.2°** and **SELD error** from **0.211** to **0.145** [2507.20530].

When all BTFF channels are used, the earlier paper reports **ER\(_{20^\circ}\) = 0.210**, **F\(_{20^\circ}\) = 87.1%**, **LE\(_{CD}\) = 4.4°**, **LR\(_{CD}\) = 92.1%**, **DOA error = 0.052**, and **SELD error = 0.110** [2507.20530].

### Architecture and noisy-condition comparisons

The later paper reports cleaner and more extensive comparisons across noisy conditions and baselines. For the clean architecture comparison, **BiSELDnet-v4** reports **ER\(_{20^\circ}\)=0.114**, **F\(_{20^\circ}\)=91.8%**, **LE\(_{CD}\)=2.5°**, **LR\(_{CD}\)=93.3%**, **SED error=0.098**, **DOA error=0.040**, and **SELD error=0.069** [2508.04333].

The same paper compares BiSELD against **BL-SELD (ACCDOA)** and **SOTA-SELD (SALSA-Lite)** under urban background noise. Examples reported include:

- **SNR=30 dB (horizontal)**: BL-SELD **≈ 0.208**, SOTA-SELD **≈ 0.193**, **BiSELD ≈ 0.118**
- **SNR=30 dB (median)**: BL-SELD **0.193**, SOTA-SELD **0.146**, **BiSELD 0.069**
- **SNR=20 dB (horizontal)**: BL **0.227**, SOTA **0.198**, **BiSELD 0.137**
- **SNR=10 dB (median)**: BL **0.238**, SOTA **0.214**, **BiSELD 0.147**
- **SNR=0 dB (horizontal)**: BL **0.289**, SOTA **0.254**, **BiSELD 0.214**
- **SNR=0 dB (median)**: BL **0.299**, SOTA **0.252**, **BiSELD 0.219** [2508.04333]

The later paper therefore characterizes BiSELD as outperforming adapted baselines and the cited **SALSA-Lite**-based SELD baseline under all tested SNRs and on both horizontal and median-plane subsets [2508.04333].

## 7. Relation to human spatial hearing, limitations, and extensions

BiSELDnet is explicitly framed as a machine analogue of **human-like auditory perception**. The **ITD-map** is associated with low-frequency phase differences and the **MSO**, the **ILD-map** with high-frequency level differences and the **LSO**, and the **SC-maps** with **pinna-related transfer functions** and elevation-dependent notches such as **N1** and **N2** [2507.20530], [2508.04333]. The later paper states that **VAM visualization** confirms that the network focuses on the **N1 notch frequency** for elevation estimation [2508.04333].

The HRTF discussion in the later paper is comparatively detailed. It defines

\[
H_{L,R}(\phi,\theta,f) = \frac{G_{L,R}(\phi,\theta,f)}{G_0(\theta,f)},
\]

discusses **non-causality** in ipsilateral HRIRs due to OTF referencing, and describes circular shifting by at least

\[
T_{\max} = \frac{\ell}{c}
\]

to obtain causal HRIRs suitable for smooth ITD analysis [2508.04333]. It also reports that low-frequency ITD is near zero at front and back and peaks at roughly **\(\pm 600\mu s\)** near lateral directions, while high-frequency ILD exhibits strong directionality and front-back asymmetry [2508.04333]. These observations are used to justify the frequency partition employed in BTFF.

Several limitations are stated. Both papers note reliance on **synthetic data** and on a **single HRTF** source, whether framed as the **KAIST HRTF** or the measured **HATS HRTF**, and both caution that generalization to different heads, microphone placements, and acoustic environments is not yet demonstrated [2507.20530], [2508.04333]. Both also note that the present formulation usually assumes at most one active source per class per frame and therefore does not directly solve **same-class polyphony**; **multi-ACCDOA** and permutation-invariant training are mentioned as possible extensions [2507.20530], [2508.04333].

Further directions described in the data include **real-world evaluation**, **moving sources and head rotations**, **HRTF personalization or transfer**, and **more efficient implementations** through pruning, quantization, or lighter architectures [2507.20530], [2508.04333]. The later paper also suggests improving the SC-map through a **linear frequency scale above 12 kHz** to better resolve **N2** notches for robust elevation estimation [2508.04333].

Taken together, these papers define BiSELDnet as a binaural SELD framework centered on a psychoacoustically informed representation, a CRNN family of regressors with ACCDOA-like outputs, and a research program aimed at enabling **full 3D localization from two horizontal channels** for humanoid robots and related binaural sensing platforms [2507.20530], [2508.04333].

Source: https://www.emergentmind.com/topics/biseldnet