Binaural Set: Audio Cue & Dataset Framework

• Binaural Set is a structured collection of spatial cues, computational features, and recordings that encode interaural time, level, and phase differences.
• It operationalizes psychoacoustic principles and analytic models, enabling evaluation and synthesis of spatial audio through methods like BTFF and HRIR/BRIR datasets.
• The concept unifies diverse methodologies—from synthetic benchmarks to end-to-end neural rendering—to enhance spatial accuracy and perceptual realism in audio applications.

“Binaural Set” appears in recent binaural-audio literature as a designation for several closely related constructs: a canonical set of binaural spatial cues, a compact computational feature set, and a curated set of binaural recordings or impulse responses used to train and evaluate rendering, synthesis, enhancement, and localization systems. Across these usages, the invariant is a structured encoding of the relation between left- and right-ear signals—whether through inter-aural time, level, phase, and spectral differences; through fixed-channel representations such as BMFD and BTFF; or through datasets organized by source direction, listener pose, room acoustics, and HRTF or BRIR measurements (Tan, 2023, Biberger et al., 2021, Barry et al., 2 May 2025, Lee et al., 28 Jul 2025).

1. Terminological scope

In the cited literature, the term is not restricted to a single artifact. It denotes either a set of cues, a set of channels, or a set of recordings and annotations, depending on the problem formulation.

Usage	Definition in the literature	Representative source
Spatial cue set	ITD, ILD, IPD, and monaural spectral cues	(Tan, 2023)
Auditory-model feature set	Five BMFD outputs: BEL, BIL, BIc, BIR, BER	(Biberger et al., 2021)
Rendering or dataset framework	HRIR/BRIR collections and programmable binaural mixes	(Barry et al., 2 May 2025)
Synthetic benchmark dataset	HRTF-based 60 s mixtures for BiSELD	(Lee et al., 28 Jul 2025)

Additional usages extend the term to paired corpora for supervised learning. These include a custom binaural speech dataset built by replaying the VCTK corpus in a real room and re-recording with three 3Dio binaural rigs, and a paired ambisonic–binaural dataset captured with a Zoom H3-VR and a Neumann KU100 dummy head (Huang et al., 2022, Zhu et al., 2022). A related audio-visual usage is the omni-directional street-scene dataset recorded with eight microphones arranged as four binaural pairs around a 360° camera rig (Dai et al., 2021).

This multiplicity of meanings is not accidental. A plausible implication is that binaural research treats the “set” as a bridge concept linking physical acoustics, perceptual theory, signal representation, and machine-learning supervision.

2. Psychoacoustic cue structure

The most fundamental “binaural set” is the cue set described by Duplex Theory. Horizontal-plane localization exploits small differences between signals at the left and right ears. Low-frequency sounds, below approximately $1500\,\mathrm{Hz}$, are predominantly localized by inter-aural time differences, whereas high-frequency sounds, above approximately $2000\,\mathrm{Hz}$, rely on inter-aural level differences. Inter-aural phase differences and monaural spectral cues, including pinna-induced notches, contribute to fine localization, elevation, and front-back discrimination (Tan, 2023).

The classical Woodworth rigid-sphere model gives

$$\mathrm{ITD}(\theta)=\frac{r\,(\theta+\sin\theta)}{c},$$

with $c \simeq 343\,\mathrm{m/s}$ at $20\,^\circ\mathrm{C}$. For $\theta=90^\circ$, $r=0.089\,\mathrm m$, and $T=18\,^\circ\mathrm C$, the reported value is approximately $0.669\,\mathrm{ms}$. ILD is defined by

$$\mathrm{ILD}(f,\theta)=20\log_{10}\left|\frac{P_L(f,\theta)}{P_R(f,\theta)}\right| =20\log_{10}\left|\frac{H_L(f,\theta)}{H_R(f,\theta)}\right|,$$

and IPD by

$2000\,\mathrm{Hz}$0

Experimentally, ILDs are small, below $2000\,\mathrm{Hz}$1, below $2000\,\mathrm{Hz}$2, but grow to $2000\,\mathrm{Hz}$3–$2000\,\mathrm{Hz}$4 above $2000\,\mathrm{Hz}$5; monaural spectral cues arise from direction-dependent filtering by the pinnae, ear canal, and torso, with notches and peaks around $2000\,\mathrm{Hz}$6–$2000\,\mathrm{Hz}$7 (Tan, 2023).

The same cue structure is explicitly operationalized in recent feature engineering. In BiSELD, ITD is extracted from phase-difference terms only for bins $2000\,\mathrm{Hz}$8, ILD is computed only for bins $2000\,\mathrm{Hz}$9, and SC-maps retain the high-frequency mel content above $$\mathrm{ITD}(\theta)=\frac{r\,(\theta+\sin\theta)}{c},$$0; the mapping directly follows the cue-frequency regimes described in psychoacoustics (Lee et al., 28 Jul 2025).

Recording-method comparisons further delimit what must be preserved for a complete binaural set. In the reported measurements, the individual-HRTF baseline yields measured $$\mathrm{ITD}(\theta)=\frac{r\,(\theta+\sin\theta)}{c},$$1, the full dummy head $$\mathrm{ITD}(\theta)=\frac{r\,(\theta+\sin\theta)}{c},$$2, the semi-dummy head $$\mathrm{ITD}(\theta)=\frac{r\,(\theta+\sin\theta)}{c},$$3, the Jecklin Disc $$\mathrm{ITD}(\theta)=\frac{r\,(\theta+\sin\theta)}{c},$$4, and ORTF $$\mathrm{ITD}(\theta)=\frac{r\,(\theta+\sin\theta)}{c},$$5. The full dummy head most closely matched individualized HRTF in ITD and better reproduced the ILD rise and slope up to about $$\mathrm{ITD}(\theta)=\frac{r\,(\theta+\sin\theta)}{c},$$6; quasi-binaural methods such as ORTF and Jecklin Disc produced minimal ILD until mid/high frequencies and were reported as less effective for precise $$\mathrm{ITD}(\theta)=\frac{r\,(\theta+\sin\theta)}{c},$$7 localization over headphones (Tan, 2023). This directly counters the common misconception that binaural recording is equivalent to ordinary stereophonic spacing.

3. Feature representations and auditory-model decoders

A highly explicit computational usage of “Binaural Set” appears in the generalized monaural and binaural auditory model. There, the non-adaptive binaural stage produces five output channels per auditory band $$\mathrm{ITD}(\theta)=\frac{r\,(\theta+\sin\theta)}{c},$$8: BEL, BER, BIL, BIR, and BIc. With gain $$\mathrm{ITD}(\theta)=\frac{r\,(\theta+\sin\theta)}{c},$$9 and fixed interaural delay

$c \simeq 343\,\mathrm{m/s}$0

the outputs are

$c \simeq 343\,\mathrm{m/s}$1

$c \simeq 343\,\mathrm{m/s}$2

$c \simeq 343\,\mathrm{m/s}$3

$c \simeq 343\,\mathrm{m/s}$4

These five simultaneous streams form the “Binaural Set,” after which DC-power and envelope-power SNR features are extracted and combined in a unified decision stage for psychoacoustic detection or speech intelligibility prediction (Biberger et al., 2021).

A more recent machine-learning representation is the Binaural Time-Frequency Feature, or BTFF, used for joint sound event localization and detection. BTFF is an eight-channel representation comprising left and right mel-spectrograms, left and right velocity maps, one ITD-map, one ILD-map, and left and right SC-maps. The rationale is explicitly divided by cue type: mel spectra encode spectral envelope for SED, velocity maps capture onset/offset dynamics, the ITD-map provides a low-frequency azimuth cue, the ILD-map provides a high-frequency azimuth cue and front-back asymmetry, and SC-maps capture pinna-induced elevation information (Lee et al., 28 Jul 2025).

Neural binaural rendering work on ambisonic input adopts a different but related decomposition. The reported system uses first-order B-format input $c \simeq 343\,\mathrm{m/s}$5, pairwise inter-channel phase differences re-encoded with the omnidirectional magnitude into a real tensor $c \simeq 343\,\mathrm{m/s}$6, and predicts ear-specific masks and phase-difference encodings rather than directly regressing complex spectrogram bins. Reconstruction is

$c \simeq 343\,\mathrm{m/s}$7

followed by inverse STFT. The paper states that this decoupling stabilizes phase learning compared to directly regressing complex spectrogram bins (Zhu et al., 2022).

Mono-to-binaural generation conditioned on visual information extends the feature-set idea beyond audio-only cues. The reported architecture uses a U-Net-style audio encoder, ViT-Large image and depth encoders, and hierarchical cross-modal attention at each decoder layer. It predicts the complex spectrogram difference $c \simeq 343\,\mathrm{m/s}$8, from which

$c \simeq 343\,\mathrm{m/s}$9

The data state that image attention progressively focuses on active sounding regions, whereas depth attention attends to large planar geometry and encodes relative source distance cues (Parida et al., 2021).

4. Corpora and measurement infrastructures

Binaural sets in the dataset sense span measured HRIR libraries, BRIR corpora, paired end-to-end corpora, and synthetic benchmarks. Binamix is built around the SADIE II Database, which provides HRIR and BRIR data for 20 subjects: 2 dummy heads with 8,802 measurement points on the sphere and 18 human subjects with up to 2,818 HRIR points and 50 BRIR points on a Lebedev quadrature. It supports both anechoic HRIRs and reverberant BRIRs, selectable reverb types from the AIR database, and custom loaders for user-supplied HRTF or BRIR files (Barry et al., 2 May 2025).

The Princeton 3D Audio and Applied Acoustics Laboratory BRIR corpus is a high-resolution moving-listener dataset. It contains $20\,^\circ\mathrm{C}$0 $20\,^\circ\mathrm{C}$1-positions, $20\,^\circ\mathrm{C}$2 $20\,^\circ\mathrm{C}$3-positions, $20\,^\circ\mathrm{C}$4 azimuths, and $20\,^\circ\mathrm{C}$5 loudspeakers, yielding $20\,^\circ\mathrm{C}$6 stereo BRIRs. The room is an irregular, near-shoebox listening room with carpet, partially treated walls and ceiling, a measured $20\,^\circ\mathrm{C}$7 averaged over $20\,^\circ\mathrm{C}$8–$20\,^\circ\mathrm{C}$9, and BRIRs high-pass filtered at $\theta=90^\circ$0 to remove low-frequency hum and ambient noise. Data are stored as SOFA 2.1 files at $\theta=90^\circ$1, with translation steps of $\theta=90^\circ$2 in $\theta=90^\circ$3 and $\theta=90^\circ$4, and azimuth steps of $\theta=90^\circ$5 over $\theta=90^\circ$6 (Qiao et al., 2024).

The audio-visual “Binaural Set” introduced for Binaural SoundNet uses eight microphones arranged as four binaural pairs at azimuths $\theta=90^\circ$7, recorded by a Zoom F8 at $\theta=90^\circ$8, $\theta=90^\circ$9-bit, together with a GoPro Fusion $r=0.089\,\mathrm m$0 camera. The dataset contains recordings from 165 distinct street and intersection sites within a $r=0.089\,\mathrm m$1 area of Zürich, with total raw duration approximately $r=0.089\,\mathrm m$2, segmented into $r=0.089\,\mathrm m$3 non-overlapping $r=0.089\,\mathrm m$4 clips. Clip selection retained audio energy above a chosen threshold and visual change of at least $r=0.089\,\mathrm m$5 of pixels relative to background (Dai et al., 2021).

Paired training corpora for supervised binaural rendering occupy a different point in the design space. The ambisonic-to-binaural dataset consists of $r=0.089\,\mathrm m$6 minutes of live band music in a single $r=0.089\,\mathrm m$7 rehearsal room, segmented into $r=0.089\,\mathrm m$8 one-minute scenes, with $r=0.089\,\mathrm m$9 minutes for training and $T=18\,^\circ\mathrm C$0 minutes for evaluation. The devices—a Zoom H3-VR and a Neumann KU100—were co-located at the room center, and no artificial scene rotations were used (Zhu et al., 2022). The end-to-end binaural speech-synthesis corpus replays the VCTK corpus in a real room and re-records it with three 3Dio binaural rigs, covering 109 speakers and approximately $T=18\,^\circ\mathrm C$1 hours of binaural audio at $T=18\,^\circ\mathrm C$2; positions and orientations were tracked at $T=18\,^\circ\mathrm C$3 in a $T=18\,^\circ\mathrm C$4 $T=18\,^\circ\mathrm C$5 volume, in a medium-sized non-anechoic lab with $T=18\,^\circ\mathrm C$6 (Huang et al., 2022).

Synthetic benchmark construction appears in BiSELD. There, a subset of the measured KAIST HRTF database is used, with azimuths from $T=18\,^\circ\mathrm C$7 to $T=18\,^\circ\mathrm C$8 in $T=18\,^\circ\mathrm C$9 steps and elevations from $0.669\,\mathrm{ms}$0 to $0.669\,\mathrm{ms}$1 in $0.669\,\mathrm{ms}$2 steps, for $0.669\,\mathrm{ms}$3 total directions. Five-second event excerpts are convolved with the corresponding HRIR pairs, and $0.669\,\mathrm{ms}$4 such segments are placed non-overlapping into each $0.669\,\mathrm{ms}$5 clean-condition mixture. The published benchmark contains $0.669\,\mathrm{ms}$6 training mixtures, $0.669\,\mathrm{ms}$7 validation mixtures, $0.669\,\mathrm{ms}$8 test mixtures, and two specialized test subsets of $0.669\,\mathrm{ms}$9 mixtures each, for a total of $$\mathrm{ILD}(f,\theta)=20\log_{10}\left|\frac{P_L(f,\theta)}{P_R(f,\theta)}\right| =20\log_{10}\left|\frac{H_L(f,\theta)}{H_R(f,\theta)}\right|,$$0 samples (Lee et al., 28 Jul 2025).

5. Rendering, mixing, and end-to-end synthesis

In programmable dataset generation, the binaural set is operationalized as a signal-processing pipeline. Binamix defines a TrackObject with audio signal $$\mathrm{ILD}(f,\theta)=20\log_{10}\left|\frac{P_L(f,\theta)}{P_R(f,\theta)}\right| =20\log_{10}\left|\frac{H_L(f,\theta)}{H_R(f,\theta)}\right|,$$1, azimuth $$\mathrm{ILD}(f,\theta)=20\log_{10}\left|\frac{P_L(f,\theta)}{P_R(f,\theta)}\right| =20\log_{10}\left|\frac{H_L(f,\theta)}{H_R(f,\theta)}\right|,$$2, elevation $$\mathrm{ILD}(f,\theta)=20\log_{10}\left|\frac{P_L(f,\theta)}{P_R(f,\theta)}\right| =20\log_{10}\left|\frac{H_L(f,\theta)}{H_R(f,\theta)}\right|,$$3, level, and reverb blend. Optional mono reverb is applied as

$$\mathrm{ILD}(f,\theta)=20\log_{10}\left|\frac{P_L(f,\theta)}{P_R(f,\theta)}\right| =20\log_{10}\left|\frac{H_L(f,\theta)}{H_R(f,\theta)}\right|,$$4

followed by retrieval or interpolation of $$\mathrm{ILD}(f,\theta)=20\log_{10}\left|\frac{P_L(f,\theta)}{P_R(f,\theta)}\right| =20\log_{10}\left|\frac{H_L(f,\theta)}{H_R(f,\theta)}\right|,$$5, convolution

$$\mathrm{ILD}(f,\theta)=20\log_{10}\left|\frac{P_L(f,\theta)}{P_R(f,\theta)}\right| =20\log_{10}\left|\frac{H_L(f,\theta)}{H_R(f,\theta)}\right|,$$6

track normalization, and summation across tracks

$$\mathrm{ILD}(f,\theta)=20\log_{10}\left|\frac{P_L(f,\theta)}{P_R(f,\theta)}\right| =20\log_{10}\left|\frac{H_L(f,\theta)}{H_R(f,\theta)}\right|,$$7

Interpolation uses a modified Delaunay triangulation: measured directions are projected to an equirectangular grid, the containing triangle is found, barycentric weights are computed in $$\mathrm{ILD}(f,\theta)=20\log_{10}\left|\frac{P_L(f,\theta)}{P_R(f,\theta)}\right| =20\log_{10}\left|\frac{H_L(f,\theta)}{H_R(f,\theta)}\right|,$$8, and the interpolated IR is

$$\mathrm{ILD}(f,\theta)=20\log_{10}\left|\frac{P_L(f,\theta)}{P_R(f,\theta)}\right| =20\log_{10}\left|\frac{H_L(f,\theta)}{H_R(f,\theta)}\right|,$$9

Five interpolation modes are provided: “nearest,” “planar,” “two_point,” “three_point,” and “auto” (Barry et al., 2 May 2025).

Measured BRIR corpora support a closely related rendering formalism. With BRIRs $2000\,\mathrm{Hz}$00 and $2000\,\mathrm{Hz}$01 loaded from SOFA, anechoic sources $2000\,\mathrm{Hz}$02 are rendered by

$2000\,\mathrm{Hz}$03

For head orientation, the lookup may be driven by the rotation matrix

$2000\,\mathrm{Hz}$04

with spatial interpolation expressed as

$2000\,\mathrm{Hz}$05

The published examples include head-tracked listening, personal sound-zone synthesis, and machine-learning data augmentation for moving-listener binaural tasks (Qiao et al., 2024).

End-to-end synthesis folds compression, spatialization, and reconstruction into a single trainable model. The reported binaural speech system is an encoder–quantizer–decoder VQ-VAE combining a Conv1D encoder, an $2000\,\mathrm{Hz}$06-layer residual vector quantizer updated by exponential moving averages, a partially conditioned binaural decoder with FiLM conditioning on the last $2000\,\mathrm{Hz}$07 decoder blocks, Gaussian Fourier feature embedding of the position–orientation feature $2000\,\mathrm{Hz}$08, and a differentiable WarpNet implementing monotonic time warping for interaural time differences. The generator

$2000\,\mathrm{Hz}$09

is trained with interaural-difference, phase, adversarial, feature-matching, and mel-spectrogram losses, with empirically chosen weights $2000\,\mathrm{Hz}$10, $2000\,\mathrm{Hz}$11, $2000\,\mathrm{Hz}$12, $2000\,\mathrm{Hz}$13, and $2000\,\mathrm{Hz}$14. Training proceeds in two stages: mono-pretraining with $2000\,\mathrm{Hz}$15, then fine-tuning with full binaural conditioning. In ablation, adding the adversarial terms cuts the DPLM spatialization error by approximately $2000\,\mathrm{Hz}$16 and restores natural-sounding background noise and reverberation in the spectrogram (Huang et al., 2022).

A separate end-to-end route is the ambisonic-to-binaural mapper trained on paired recordings rather than measured HRTFs. The system maps first-order ambisonic STFTs and duplex-theory-inspired phase-difference features directly to binaural output and is explicitly presented as a way to bypass extensive anechoic HRIR campaigns (Zhu et al., 2022).

6. Evaluation regimes, applications, and limitations

Evaluation of binaural sets is task-specific, but the cited work converges on objective fidelity, spatial accuracy, and perceptual realism. In end-to-end binaural speech synthesis, the reported test metrics are waveform $2000\,\mathrm{Hz}$17, mel-spectrogram $2000\,\mathrm{Hz}$18, and DPLM. The “Proposed joint” system yields Wave-$2000\,\mathrm{Hz}$19, Mel-$2000\,\mathrm{Hz}$20, and DPLM $2000\,\mathrm{Hz}$21, while two-alternative forced-choice listening tests with $2000\,\mathrm{Hz}$22 participants show the proposed system preferred more than $2000\,\mathrm{Hz}$23 against the cascade baseline and more than $2000\,\mathrm{Hz}$24 against the decoder-only system in naturalness-closeness tests; spatialization ties the decoder-only system at approximately $2000\,\mathrm{Hz}$25, with no significant difference (Huang et al., 2022).

For ambisonic binaural rendering, objective metrics are SDR and LSD, and subjective metrics are MOS for quality, timbre, localization, and immersion. The reported GRU model attains SDR $2000\,\mathrm{Hz}$26 and MOS values $2000\,\mathrm{Hz}$27, $2000\,\mathrm{Hz}$28, $2000\,\mathrm{Hz}$29, and $2000\,\mathrm{Hz}$30, while the UNet attains SDR $2000\,\mathrm{Hz}$31 and LSD $2000\,\mathrm{Hz}$32. Feature ablations report that complex input plus relative phase outperforms magnitude alone for MOS, mask-plus-phase-decoupling substantially outperforms direct complex regression, and combined $2000\,\mathrm{Hz}$33 loss is preferred to single-domain losses (Zhu et al., 2022).

For joint sound event localization and detection, performance is summarized by DCASE-style SELD metrics. On the published benchmark, the full BTFF reaches $2000\,\mathrm{Hz}$34, $2000\,\mathrm{Hz}$35, localization error $2000\,\mathrm{Hz}$36, localization recall $2000\,\mathrm{Hz}$37, and SELD error $2000\,\mathrm{Hz}$38. The paper attributes distinct roles to the sub-features: V-map improves detection, ITD/ILD improve horizontal localization, and SC-map captures vertical cues (Lee et al., 28 Jul 2025).

Applications extend beyond rendering and synthesis. Binamix lists codec evaluation, spatial audio quality metric development, machine learning for direction-of-arrival estimation or no-reference quality, and surround-to-binaural emulation (Barry et al., 2 May 2025). The 3D3A BRIR dataset adds head-tracked listening, crosstalk-cancellation research, personal-sound-zone design, and moving-listener data augmentation (Qiao et al., 2024). ClearBuds demonstrates a deployed synchronized binaural set in wearable hardware: two earbuds form a synchronized binaural microphone array with synchronization error less than $2000\,\mathrm{Hz}$39, neural runtime $2000\,\mathrm{Hz}$40 on an iPhone 12 Pro, and approximately $2000\,\mathrm{Hz}$41 total one-way latency; in a user study with 37 participants over 1,041 in-the-wild clips, the full system improves noise intrusiveness from $2000\,\mathrm{Hz}$42 to $2000\,\mathrm{Hz}$43 and MOS from $2000\,\mathrm{Hz}$44 to $2000\,\mathrm{Hz}$45 (Chatterjee et al., 2022).

Several limitations recur across the literature. Static heads cannot resolve cone-of-confusion ambiguities without head-tracking; generic HRTF playback may degrade elevation cues because pinnae shape and ear-canal resonance differ across subjects; and flat-response studio microphones do not reproduce human pinna notches around $2000\,\mathrm{Hz}$46–$2000\,\mathrm{Hz}$47 (Tan, 2023). Dataset diversity is often constrained: the ambisonic-binaural dataset is restricted to a single room and live band content, the internal speech-synthesis corpus uses a single medium-sized non-anechoic lab, and the BiSELD benchmark is published only in clean condition without added reverberation or noise (Zhu et al., 2022, Huang et al., 2022, Lee et al., 28 Jul 2025). These limitations indicate that, in current usage, a “Binaural Set” is best understood not as a single standardized object but as a structured research instrument whose adequacy depends on which subset of binaural hearing—cue fidelity, head geometry, room acoustics, motion, or task annotation—it is designed to preserve.