---
title: 'Binaural Set: Audio Cue & Dataset Framework'
url: https://www.emergentmind.com/topics/binaural-set
type: topic
---

# Binaural Set: Audio Cue & Dataset Framework

“Binaural Set” appears in recent binaural-audio literature as a designation for several closely related constructs: a canonical set of binaural spatial cues, a compact computational feature set, and a curated set of binaural recordings or impulse responses used to train and evaluate rendering, synthesis, enhancement, and localization systems. Across these usages, the invariant is a structured encoding of the relation between left- and right-ear signals—whether through inter-aural time, level, phase, and spectral differences; through fixed-channel representations such as BMFD and BTFF; or through datasets organized by source direction, listener pose, room acoustics, and HRTF or BRIR measurements [2312.16884] [2106.15659] [2505.01369] [2507.20530].

## 1. Terminological scope

In the cited literature, the term is not restricted to a single artifact. It denotes either a set of cues, a set of channels, or a set of recordings and annotations, depending on the problem formulation.

| Usage | Definition in the literature | Representative source |
|---|---|---|
| Spatial cue set | ITD, ILD, IPD, and monaural spectral cues | [2312.16884] |
| Auditory-model feature set | Five BMFD outputs: BEL, BIL, BIc, BIR, BER | [2106.15659] |
| Rendering or dataset framework | HRIR/BRIR collections and programmable binaural mixes | [2505.01369] |
| Synthetic benchmark dataset | HRTF-based 60 s mixtures for BiSELD | [2507.20530] |

Additional usages extend the term to paired corpora for supervised learning. These include a custom binaural speech dataset built by replaying the VCTK corpus in a real room and re-recording with three 3Dio binaural rigs, and a paired ambisonic–binaural dataset captured with a Zoom H3-VR and a Neumann KU100 dummy head [2207.03697] [2211.02301]. A related audio-visual usage is the omni-directional street-scene dataset recorded with eight microphones arranged as four binaural pairs around a 360° camera rig [2109.02763].

This multiplicity of meanings is not accidental. A plausible implication is that binaural research treats the “set” as a bridge concept linking physical acoustics, perceptual theory, signal representation, and machine-learning supervision.

## 2. Psychoacoustic cue structure

The most fundamental “binaural set” is the cue set described by Duplex Theory. Horizontal-plane localization exploits small differences between signals at the left and right ears. Low-frequency sounds, below approximately \(1500\,\mathrm{Hz}\), are predominantly localized by inter-aural time differences, whereas high-frequency sounds, above approximately \(2000\,\mathrm{Hz}\), rely on inter-aural level differences. Inter-aural phase differences and monaural spectral cues, including pinna-induced notches, contribute to fine localization, elevation, and front-back discrimination [2312.16884].

The classical Woodworth rigid-sphere model gives
\[
\mathrm{ITD}(\theta)=\frac{r\,(\theta+\sin\theta)}{c},
\]
with \(c \simeq 343\,\mathrm{m/s}\) at \(20\,^\circ\mathrm{C}\). For \(\theta=90^\circ\), \(r=0.089\,\mathrm m\), and \(T=18\,^\circ\mathrm C\), the reported value is approximately \(0.669\,\mathrm{ms}\). ILD is defined by
\[
\mathrm{ILD}(f,\theta)=20\log_{10}\left|\frac{P_L(f,\theta)}{P_R(f,\theta)}\right|
=20\log_{10}\left|\frac{H_L(f,\theta)}{H_R(f,\theta)}\right|,
\]
and IPD by
\[
\mathrm{IPD}(f,\theta)=2\pi f\,\mathrm{ITD}(\theta).
\]
Experimentally, ILDs are small, below \(3\,\mathrm{dB}\), below \(1\,\mathrm{kHz}\), but grow to \(10\)–\(20\,\mathrm{dB}\) above \(4\,\mathrm{kHz}\); monaural spectral cues arise from direction-dependent filtering by the pinnae, ear canal, and torso, with notches and peaks around \(4\)–\(16\,\mathrm{kHz}\) [2312.16884].

The same cue structure is explicitly operationalized in recent feature engineering. In BiSELD, ITD is extracted from phase-difference terms only for bins \(f \le 1.5\,\mathrm{kHz}\), ILD is computed only for bins \(f>5\,\mathrm{kHz}\), and SC-maps retain the high-frequency mel content above \(5\,\mathrm{kHz}\); the mapping directly follows the cue-frequency regimes described in psychoacoustics [2507.20530].

Recording-method comparisons further delimit what must be preserved for a complete binaural set. In the reported measurements, the individual-HRTF baseline yields measured \(\mathrm{ITD}\approx0.69\,\mathrm{ms}\), the full dummy head \(\approx0.67\,\mathrm{ms}\), the semi-dummy head \(\approx0.83\,\mathrm{ms}\), the Jecklin Disc \(\approx0.58\,\mathrm{ms}\), and ORTF \(\approx0.50\,\mathrm{ms}\). The full dummy head most closely matched individualized HRTF in ITD and better reproduced the ILD rise and slope up to about \(4\,\mathrm{kHz}\); quasi-binaural methods such as ORTF and Jecklin Disc produced minimal ILD until mid/high frequencies and were reported as less effective for precise \(3\mathrm D\) localization over headphones [2312.16884]. This directly counters the common misconception that binaural recording is equivalent to ordinary stereophonic spacing.

## 3. Feature representations and auditory-model decoders

A highly explicit computational usage of “Binaural Set” appears in the generalized monaural and binaural auditory model. There, the non-adaptive binaural stage produces five output channels per auditory band \(p\): BEL, BER, BIL, BIR, and BIc. With gain \(a=3\) and fixed interaural delay
\[
\tau(p)=\frac{1}{4}\frac{1}{f_p},
\]
the outputs are
\[
\mathrm{BEL}(p,t)=L(p,t), \qquad \mathrm{BER}(p,t)=R(p,t),
\]
\[
\mathrm{BIL}(p,t)=\max\{L(p,t)-a\,R(p,t-\tau(p)),0\},
\]
\[
\mathrm{BIR}(p,t)=\max\{R(p,t)-a\,L(p,t-\tau(p)),0\},
\]
\[
\mathrm{BIc}(p,t)=\sqrt{L(p,t)\,R(p,t)}.
\]
These five simultaneous streams form the “Binaural Set,” after which DC-power and envelope-power SNR features are extracted and combined in a unified decision stage for psychoacoustic detection or speech intelligibility prediction [2106.15659].

A more recent machine-learning representation is the Binaural Time-Frequency Feature, or BTFF, used for joint sound event localization and detection. BTFF is an eight-channel representation comprising left and right mel-spectrograms, left and right velocity maps, one ITD-map, one ILD-map, and left and right SC-maps. The rationale is explicitly divided by cue type: mel spectra encode spectral envelope for SED, velocity maps capture onset/offset dynamics, the ITD-map provides a low-frequency azimuth cue, the ILD-map provides a high-frequency azimuth cue and front-back asymmetry, and SC-maps capture pinna-induced elevation information [2507.20530].

Neural binaural rendering work on ambisonic input adopts a different but related decomposition. The reported system uses first-order B-format input \(X\in\mathbb C^{4\times T\times F}\), pairwise inter-channel phase differences re-encoded with the omnidirectional magnitude into a real tensor \(D\in\mathbb R^{6\times T\times F}\), and predicts ear-specific masks and phase-difference encodings rather than directly regressing complex spectrogram bins. Reconstruction is
\[
|\hat Y^b| = M^b \odot |O_B|,\qquad
\angle \hat Y^b = \angle O_B + \angle M^b,
\]
followed by inverse STFT. The paper states that this decoupling stabilizes phase learning compared to directly regressing complex spectrogram bins [2211.02301].

Mono-to-binaural generation conditioned on visual information extends the feature-set idea beyond audio-only cues. The reported architecture uses a U-Net-style audio encoder, ViT-Large image and depth encoders, and hierarchical cross-modal attention at each decoder layer. It predicts the complex spectrogram difference \(\hat O=\mathrm{STFT}(x_l-x_r)\), from which
\[
\hat x_l(t)=\frac{x_m(t)+x_o(t)}{2},\qquad
\hat x_r(t)=\frac{x_m(t)-x_o(t)}{2}.
\]
The data state that image attention progressively focuses on active sounding regions, whereas depth attention attends to large planar geometry and encodes relative source distance cues [2111.08046].

## 4. Corpora and measurement infrastructures

Binaural sets in the dataset sense span measured HRIR libraries, BRIR corpora, paired end-to-end corpora, and synthetic benchmarks. Binamix is built around the SADIE II Database, which provides HRIR and BRIR data for 20 subjects: 2 dummy heads with 8,802 measurement points on the sphere and 18 human subjects with up to 2,818 HRIR points and 50 BRIR points on a Lebedev quadrature. It supports both anechoic HRIRs and reverberant BRIRs, selectable reverb types from the AIR database, and custom loaders for user-supplied HRTF or BRIR files [2505.01369].

The Princeton 3D Audio and Applied Acoustics Laboratory BRIR corpus is a high-resolution moving-listener dataset. It contains \(11\) \(y\)-positions, \(21\) \(x\)-positions, \(37\) azimuths, and \(8\) loudspeakers, yielding \(68{,}376\) stereo BRIRs. The room is an irregular, near-shoebox listening room with carpet, partially treated walls and ceiling, a measured \(\mathrm{RT}_{60}=0.24\,\mathrm s\) averaged over \(1.3\)–\(6.3\,\mathrm{kHz}\), and BRIRs high-pass filtered at \(100\,\mathrm{Hz}\) to remove low-frequency hum and ambient noise. Data are stored as SOFA 2.1 files at \(48{,}000\,\mathrm{Hz}\), with translation steps of \(0.05\,\mathrm m\) in \(x\) and \(y\), and azimuth steps of \(5^\circ\) over \([-90^\circ,+90^\circ]\) [2403.12258].

The audio-visual “Binaural Set” introduced for Binaural SoundNet uses eight microphones arranged as four binaural pairs at azimuths \(0^\circ,90^\circ,180^\circ,270^\circ\), recorded by a Zoom F8 at \(96\,\mathrm{kHz}\), \(24\)-bit, together with a GoPro Fusion \(360^\circ\) camera. The dataset contains recordings from 165 distinct street and intersection sites within a \(5\,\mathrm{km}\times5\,\mathrm{km}\) area of Zürich, with total raw duration approximately \(15\,\mathrm h\), segmented into \(64{,}250\) non-overlapping \(2\,\mathrm s\) clips. Clip selection retained audio energy above a chosen threshold and visual change of at least \(5\%\) of pixels relative to background [2109.02763].

Paired training corpora for supervised binaural rendering occupy a different point in the design space. The ambisonic-to-binaural dataset consists of \(49\) minutes of live band music in a single \(20\,\mathrm m^2\) rehearsal room, segmented into \(49\) one-minute scenes, with \(31\) minutes for training and \(18\) minutes for evaluation. The devices—a Zoom H3-VR and a Neumann KU100—were co-located at the room center, and no artificial scene rotations were used [2211.02301]. The end-to-end binaural speech-synthesis corpus replays the VCTK corpus in a real room and re-records it with three 3Dio binaural rigs, covering 109 speakers and approximately \(42\) hours of binaural audio at \(48\,\mathrm{kHz}\); positions and orientations were tracked at \(240\,\mathrm{Hz}\) in a \(4.6\,\mathrm m\times2.4\,\mathrm m\) \(3\mathrm D\) volume, in a medium-sized non-anechoic lab with \(T_{60}\approx400\,\mathrm{ms}\) [2207.03697].

Synthetic benchmark construction appears in BiSELD. There, a subset of the measured KAIST HRTF database is used, with azimuths from \(-180^\circ\) to \(+180^\circ\) in \(30^\circ\) steps and elevations from \(-30^\circ\) to \(+60^\circ\) in \(30^\circ\) steps, for \(48\) total directions. Five-second event excerpts are convolved with the corresponding HRIR pairs, and \(12\) such segments are placed non-overlapping into each \(60\,\mathrm s\) clean-condition mixture. The published benchmark contains \(40{,}320\) training mixtures, \(8{,}640\) validation mixtures, \(8{,}640\) test mixtures, and two specialized test subsets of \(2{,}160\) mixtures each, for a total of \(60{,}480\) samples [2507.20530].

## 5. Rendering, mixing, and end-to-end synthesis

In programmable dataset generation, the binaural set is operationalized as a signal-processing pipeline. Binamix defines a `TrackObject` with audio signal \(x(t)\), azimuth \(\phi\), elevation \(\theta\), level, and reverb blend. Optional mono reverb is applied as
\[
x_{\mathrm{rev}}(t)=x*r,
\]
followed by retrieval or interpolation of \(\{h_L,h_R\}\), convolution
\[
y_L(t)=x_{\mathrm{rev}}*h_L,\qquad y_R(t)=x_{\mathrm{rev}}*h_R,
\]
track normalization, and summation across tracks
\[
Y_L=\sum_{k=1}^K y_{L,k},\qquad Y_R=\sum_{k=1}^K y_{R,k}.
\]
Interpolation uses a modified Delaunay triangulation: measured directions are projected to an equirectangular grid, the containing triangle is found, barycentric weights are computed in \(3\mathrm D\), and the interpolated IR is
\[
h_{\mathrm{interp}}(n)=\sum_{i=1}^3 w_i h_i(n),\qquad \sum_i w_i=1.
\]
Five interpolation modes are provided: “nearest,” “planar,” “two_point,” “three_point,” and “auto” [2505.01369].

Measured BRIR corpora support a closely related rendering formalism. With BRIRs \(h_{L,i}(t)\) and \(h_{R,i}(t)\) loaded from SOFA, anechoic sources \(x_i(t)\) are rendered by
\[
y_L(t)=\sum_{i=1}^N x_i(t)*h_{L,i}(t),\qquad
y_R(t)=\sum_{i=1}^N x_i(t)*h_{R,i}(t).
\]
For head orientation, the lookup may be driven by the rotation matrix
\[
R(\psi,\theta,\phi)=R_z(\psi)\cdot R_y(\theta)\cdot R_x(\phi),
\]
with spatial interpolation expressed as
\[
h_{\mathrm{interp}}=\sum_k w_k(x,y,\psi)\,h_{\mathrm{meas},k},\qquad \sum_k w_k=1,\quad w_k\ge 0.
\]
The published examples include head-tracked listening, personal sound-zone synthesis, and machine-learning data augmentation for moving-listener binaural tasks [2403.12258].

End-to-end synthesis folds compression, spatialization, and reconstruction into a single trainable model. The reported binaural speech system is an encoder–quantizer–decoder VQ-VAE combining a Conv1D encoder, an \(N\)-layer residual vector quantizer updated by exponential moving averages, a partially conditioned binaural decoder with FiLM conditioning on the last \(K\) decoder blocks, Gaussian Fourier feature embedding of the position–orientation feature \(c(t)\), and a differentiable WarpNet implementing monotonic time warping for interaural time differences. The generator
\[
f(x,c)=\mathrm{Dec}(\mathrm{Quant}(\mathrm{Enc}(x)),c)
\]
is trained with interaural-difference, phase, adversarial, feature-matching, and mel-spectrogram losses, with empirically chosen weights \(\lambda_{\mathrm{diff}}=1\), \(\lambda_{\mathrm{adv}}=1\), \(\lambda_{\mathrm{pha}}=0.01\), \(\lambda_{\mathrm{fm}}=2\), and \(\lambda_{\mathrm{mel}}=45\). Training proceeds in two stages: mono-pretraining with \(c=0\), then fine-tuning with full binaural conditioning. In ablation, adding the adversarial terms cuts the DPLM spatialization error by approximately \(10\%\) and restores natural-sounding background noise and reverberation in the spectrogram [2207.03697].

A separate end-to-end route is the ambisonic-to-binaural mapper trained on paired recordings rather than measured HRTFs. The system maps first-order ambisonic STFTs and duplex-theory-inspired phase-difference features directly to binaural output and is explicitly presented as a way to bypass extensive anechoic HRIR campaigns [2211.02301].

## 6. Evaluation regimes, applications, and limitations

Evaluation of binaural sets is task-specific, but the cited work converges on objective fidelity, spatial accuracy, and perceptual realism. In end-to-end binaural speech synthesis, the reported test metrics are waveform \(L_2\), mel-spectrogram \(L_1\), and DPLM. The “Proposed joint” system yields Wave-\(L_2=0.807\), Mel-\(L_1=0.631\), and DPLM \(=0.106\), while two-alternative forced-choice listening tests with \(N=24\) participants show the proposed system preferred more than \(70\%\) against the cascade baseline and more than \(80\%\) against the decoder-only system in naturalness-closeness tests; spatialization ties the decoder-only system at approximately \(50/50\), with no significant difference [2207.03697].

For ambisonic binaural rendering, objective metrics are SDR and LSD, and subjective metrics are MOS for quality, timbre, localization, and immersion. The reported GRU model attains SDR \(=7.32\) and MOS values \(3.83\), \(3.58\), \(3.87\), and \(3.58\), while the UNet attains SDR \(=8.03\) and LSD \(=0.93\). Feature ablations report that complex input plus relative phase outperforms magnitude alone for MOS, mask-plus-phase-decoupling substantially outperforms direct complex regression, and combined \(L_1^{\mathrm{wav}}+L_1^{\mathrm{sp}}\) loss is preferred to single-domain losses [2211.02301].

For joint sound event localization and detection, performance is summarized by DCASE-style SELD metrics. On the published benchmark, the full BTFF reaches \(ER_{20^\circ}=0.210\), \(F_{20^\circ}=87.1\%\), localization error \(=4.4^\circ\), localization recall \(=92.1\%\), and SELD error \(=0.110\). The paper attributes distinct roles to the sub-features: V-map improves detection, ITD/ILD improve horizontal localization, and SC-map captures vertical cues [2507.20530].

Applications extend beyond rendering and synthesis. Binamix lists codec evaluation, spatial audio quality metric development, machine learning for direction-of-arrival estimation or no-reference quality, and surround-to-binaural emulation [2505.01369]. The 3D3A BRIR dataset adds head-tracked listening, crosstalk-cancellation research, personal-sound-zone design, and moving-listener data augmentation [2403.12258]. ClearBuds demonstrates a deployed synchronized binaural set in wearable hardware: two earbuds form a synchronized binaural microphone array with synchronization error less than \(64\,\mu\mathrm s\), neural runtime \(21.4\,\mathrm{ms}\) on an iPhone 12 Pro, and approximately \(109\,\mathrm{ms}\) total one-way latency; in a user study with 37 participants over 1,041 in-the-wild clips, the full system improves noise intrusiveness from \(2.02\) to \(3.77\) and MOS from \(2.49\) to \(3.10\) [2206.13611].

Several limitations recur across the literature. Static heads cannot resolve cone-of-confusion ambiguities without head-tracking; generic HRTF playback may degrade elevation cues because pinnae shape and ear-canal resonance differ across subjects; and flat-response studio microphones do not reproduce human pinna notches around \(6\)–\(8\,\mathrm{kHz}\) [2312.16884]. Dataset diversity is often constrained: the ambisonic-binaural dataset is restricted to a single room and live band content, the internal speech-synthesis corpus uses a single medium-sized non-anechoic lab, and the BiSELD benchmark is published only in clean condition without added reverberation or noise [2211.02301] [2207.03697] [2507.20530]. These limitations indicate that, in current usage, a “Binaural Set” is best understood not as a single standardized object but as a structured research instrument whose adequacy depends on which subset of binaural hearing—cue fidelity, head geometry, room acoustics, motion, or task annotation—it is designed to preserve.

Source: https://www.emergentmind.com/topics/binaural-set