Binaural Time-Frequency Feature

Updated 7 July 2026

BTFF is a binaural time-frequency representation that integrates multiple cue maps—mel-spectrograms, V-maps, ITD, ILD, and spectral cues—to encode spatial information.
The canonical eight-channel BTFF engineered in BiSELD employs fixed or adaptive front-end processing to achieve precise sound event detection and localization.
BTFF design spans explicit cue maps, signal-domain decoders, and learned complex spectra to address challenges in SSL, speech enhancement, and real-world scene analysis.

Binaural Time-Frequency Feature (BTFF) denotes a binaural representation organized over time and frequency to expose spatial information carried by two-ear recordings. In the strict sense of the term used in recent binaural sound event localization and detection work, BTFF is an explicitly engineered eight-channel tensor that combines left and right mel-spectrograms, left and right V-maps, an interaural time difference map, an interaural level difference map, and left and right spectral-cue maps so that sound-event content, azimuth cues, front-back cues, and elevation cues are all present in one input representation (Lee et al., 28 Jul 2025). In a broader literature sense, the same term usefully covers several closely related but differently named representations, including fixed binaural decoder banks, STFT-domain ILD/IPD tensors, adaptive auditory subband cue maps, probabilistic complex binaural ratios, and complex left/right spectrograms used with spatial-consistency objectives (Biberger et al., 2021, Panah et al., 17 Nov 2025, Meng et al., 5 Jun 2026, Deleforge et al., 2016, Tokala et al., 26 Jul 2025).

1. Definition and conceptual scope

BTFF is best understood as a family of binaural descriptors indexed by time and frequency, rather than as a single universal formula. The narrow, paper-specific definition appears in BiSELD work, where BTFF is proposed as the core input representation for joint sound event detection and 3D localization from two-channel binaural signals, explicitly encoding interaural time difference (ITD), interaural level difference (ILD), and high-frequency spectral cues (SC) together with sound-event content features (Lee et al., 28 Jul 2025). The broader literature, however, contains multiple representations that are “BTFF-like” without using the acronym: some are cue maps such as ILD and IPD, some are signal-domain decoder outputs, some are complex left/right spectra, and some are aggregated histograms of binaural cue values (Panah et al., 17 Nov 2025, Biberger et al., 2021, Barry et al., 29 Oct 2025).

A central distinction is whether binaural information is encoded explicitly or implicitly. Explicit BTFFs expose interpretable cue maps such as ITD, ILD, IPD, or SC directly on the time-frequency lattice. Implicit BTFFs instead encode binaural structure through fixed or learned channel outputs, such as the five-channel binaural matrix feature decoder (BMFD), whose channels are decoder outputs rather than explicit cue parameters (Biberger et al., 2021). Another distinction concerns whether the feature extractor is fixed or adaptive. BiEAR, for example, uses adaptive ERB-spaced subbands whose bandwidths vary over time under controller-driven Q-factor modulation, so the underlying binaural time-frequency substrate itself changes during inference (Meng et al., 5 Jun 2026).

Taken together, these works suggest that BTFF is less a single representation class than a design space spanning auditory front ends, STFT-domain cue maps, complex-valued spectra, and structured aggregations of binaural evidence. The common denominator is that the representation is binaural, localized in time and frequency, and intended to make spatially informative relations between the two ears accessible to estimation, detection, or reconstruction models (Lee et al., 28 Jul 2025, Deleforge et al., 2016).

2. Canonical eight-channel BTFF in BiSELD

The canonical named BTFF is the representation introduced for Binaural Sound Event Localization and Detection (BiSELD). It is an 8-channel input representation with shape

$T \times 64 \times 8,$

where $T$ is the number of time frames, $64$ is the mel-frequency dimension, and $8$ is the channel dimension. The eight channels are left mel-spectrogram, right mel-spectrogram, left V-map, right V-map, ITD-map, ILD-map, left SC-map, and right SC-map (Lee et al., 28 Jul 2025).

Channel group	Cue family	Main function
Left/right mel-spectrograms	Sound-event content	Detection
Left/right V-maps	Temporal change	Detection
ITD-map, ILD-map	Interaural cues	Azimuth, front-back
Left/right SC-maps	High-frequency spectral cues	Elevation

The feature construction is explicitly frequency-stratified. The ITD-map is restricted to low frequencies, with

$\Delta_t(m,k)= -\frac{1}{\omega}\,\mathrm{Im}\left[\ln\left(\frac{P_R(m,k)}{P_L(m,k)}\right)\right],$

and

$\mathrm{ITD\text{-}map}(m,b) = \mathrm{Mel}[\Delta_t(m,k)] \quad \text{for } k \le k_{1500},$

where $k_{1500}$ is the bin corresponding to $1.5$ kHz. The ILD-map is high-frequency: $A_S(m,k)=10\log_{10}\left|\frac{P_R(m,k)}{P_L(m,k)}\right|^2,$

$\mathrm{ILD\text{-}map}(m,b)=\mathrm{Mel}[A_S(m,k)] \quad \text{for } k>k_{5000},$

with $T$ 0 the bin corresponding to $T$ 1 kHz. The SC-map is also high-frequency: $T$ 2 The V-map is the time derivative of the spectrogram, computed by forward, central, and backward finite differences and then mapped to mel bands (Lee et al., 28 Jul 2025).

The rationale is explicitly psychoacoustic. MS and V-map are used mainly for sound event detection; ITD-map and ILD-map are mainly for azimuth estimation; SC-map is mainly for elevation estimation. Audio sources and HRIRs are resampled to 32 kHz, “to preserve spectral localization cues below 16 kHz,” and the design is tied to measured HRTF cues rather than to generic stereo heuristics (Lee et al., 28 Jul 2025). A closely related BiSELDnet report further states that BTFF addresses elevation difficulty and front-back confusion in two-channel binaural input, and its Vector Activation Map analysis shows the network focusing on the N1 notch frequency for elevation estimation (Lee, 6 Aug 2025).

A major precursor to BTFF is the BMFD: binaural matrix feature decoder, introduced as a “5-channel monaural and binaural matrix feature ‘decoder’.” BMFD is a non-adaptive binaural stage inserted into the generalized power spectrum model. It yields five output channels,

$T$ 3

combining two monaural better-ear streams with three fixed binaural-interaction channels. The authors explicitly describe it as a simplification of EC-style processing and as a compact binaural time-frequency front end over auditory channels. It is BTFF-like because it produces several binaural feature channels per auditory frequency channel and time sample, but it differs from cue-map BTFFs in that binaural information is encoded implicitly via fixed decoder outputs rather than explicitly via ITD/IPD/ILD/coherence maps (Biberger et al., 2021).

Another prominent family is the STFT cue-map representation used in sound source localization. A systematic study of feature design for binaural SSL evaluates magnitude spectrogram, phase spectrogram, ILD, and IPD, singly and in combinations, as multi-channel CNN inputs. In that setting, the compact cue pair

$T$ 4

is sufficient and nearly optimal in-domain, whereas the richer representation

$T$ 5

is best out-of-domain under HRTF mismatch and content variation (Panah et al., 17 Nov 2025). This is a direct cue-based BTFF formulation.

A third family is the probabilistic complex-ratio representation. The rectified binaural ratio (RBR) is a per- $T$ 6 complex binaural feature derived from the whitened microphone ratio and analytically rectified so that it is centered on the relative transfer function under the assumed Gaussian source-plus-noise model. Its key distinction is that the feature comes with an explicit complex $T$ 7-distribution and a bin-wise uncertainty parameter, replacing heuristic cue aggregation by statistically grounded weighting (Deleforge et al., 2016).

A fourth family is the adaptive auditory BTFF. BiEAR starts from a 1-second binaural waveform segment, applies STFT with a 20 ms Hann window and 10 ms hop, groups coefficients into $T$ 8 ERB-spaced auditory subbands using adjustable Gabor filters, and computes ILD and IPD maps from the resulting complex subbands. The key novelty is that the subband bandwidths are adapted over time and frequency through controller-driven Q-factor modulation, inspired by medial olivocochlear feedback, so the binaural time-frequency representation is dynamically reshaped before cue extraction (Meng et al., 5 Jun 2026).

A fifth family is the cue-histogram aggregation used by Binaspect. There the primary per-bin features are bounded interaural level ratio and phase-derived ITD spectrograms, which are then accumulated into weighted frame-wise histograms: $T$ 9 These “azimuth maps” are BTFF-like at a higher level: they preserve time while aggregating over frequency into stable cue distributions that visualize multiple sources as distinct tracks and degradations as broadened or shifted distributions (Barry et al., 29 Oct 2025).

4. Signal processing, mathematical forms, and model integration

BTFF pipelines are not tied to a single front end. In auditory-model formulations such as BMFD, each ear is processed by outer and middle ear weighting, a fourth-order gammatone filterbank with ERB bandwidth and third-octave spacing, half-wave rectification, and a neural-adaptation stage before binaural combination. The decoder outputs are then low-pass filtered and analyzed in both a power pathway and an envelope-power pathway, with a unified decision stage taking the maximum across binaural channels for each local region (Biberger et al., 2021). This is structurally different from STFT cue-map BTFFs but functionally analogous.

In cue-map formulations, the pipeline is typically STFT $64$0 left/right complex spectra $64$1 interaural cue maps. For example, the SSL feature-evaluation study defines

$64$2

and

$64$3

then stacks these maps, with or without left/right magnitude or phase spectrograms, as CNN channels (Panah et al., 17 Nov 2025). BiEAR keeps this cue-map structure but replaces the fixed STFT filterbank by adaptive ERB-spaced subbands $64$4, yielding time-varying ILD and IPD maps (Meng et al., 5 Jun 2026).

A separate line of work uses complex binaural spectrograms as BTFF-equivalent inputs. In binaural speech enhancement with complex convolutional transformer or recurrent networks, the network input is the pair of left/right complex STFTs, and the system estimates separate complex ratio masks for the two ears. The representation is binaural and time-frequency localized, but spatial information is preserved mainly through explicit TF-domain ILD/IPD loss terms rather than through handcrafted BTFF channels (Tokala et al., 2024, Tokala et al., 26 Jul 2025). In the same spirit, SIREN represents binaural structure by the tuple

$64$5

predicting complex left/right spectrograms and an auxiliary difference spectrogram, and using phase-consistency scores during fusion (Song et al., 31 Mar 2026).

These variants show that BTFF may appear at three different levels. It may be an engineered feature tensor such as the eight-channel BiSELD BTFF; a signal-domain decoder output bank such as BMFD; or a learned complex latent representation built from left/right spectra and constrained by spatial losses. This suggests that the term is most stable when interpreted functionally: a BTFF is any binaural time-frequency representation whose organization is intended to preserve or expose localization-relevant interaural and monaural structure (Lee et al., 28 Jul 2025, Biberger et al., 2021, Tokala et al., 26 Jul 2025).

5. Empirical behavior and application domains

The most direct evidence for BTFF effectiveness comes from BiSELD. The BTFF paper reports that each sub-feature enhances task performance: V-map improves detection, ITD-/ILD-maps enable accurate horizontal localization, and SC-map captures vertical spatial cues. The final system achieves a SELD error of 0.110 with 87.1% F-score and 4.4° localization error (Lee et al., 28 Jul 2025). The related BiSELDnet-v4 report, using the Trinity module, reports SELD error 0.069, ER 0.114, F-score 91.8%, LE 2.5°, and DOA error 0.040 in its evaluation setting, while VAM analysis indicates attention to the N1 notch region for elevation estimation (Lee, 6 Aug 2025).

Cue-map BTFF studies reach a complementary conclusion: task conditions determine which feature subset is most useful. In binaural SSL with matched training and test conditions, the compact cue pair ILD + IPD gives 4.5° MAE at $64$6 on the in-domain TSP--SSL test set, tied with the full four-feature combination. Under out-of-domain mismatch, the best representation is Phase L/R + ILD + IPD, achieving 12.7° for all sources and all elevations, 8.4° for all sources at elevation $64$7, and 4.0° for natural sources at elevation $64$8 on SynBAD--Var (Panah et al., 17 Nov 2025). This indicates that richer BTFFs may be necessary when source content and HRTFs vary.

Auditory-model BTFF-like systems show comparable task dependence. BMFD preserved monaural prediction performance while capturing much of the binaural psychoacoustic data and a substantial amount of spatial release from masking. The authors found that the BIL/BIR hemispheric difference channels were most important for binaural psychoacoustics, whereas for speech intelligibility with a frontal target, the BIc midline channel dominated and “gave most contribution to SI predictions” (Biberger et al., 2021). A plausible implication is that BTFF design is geometry- and task-dependent even when the front end is fixed.

Noise-robust and adaptive BTFF variants add further evidence. The rectified binaural ratio reduced delay-estimation errors to less than 0.4% incorrect delays above $64$9 dB SNR, compared with 10.1% for PHAT-histogram, and was about 3× faster in the reported Matlab implementation (Deleforge et al., 2016). BiEAR reports that the adaptive front end improves localization accuracy and robustness to unseen speakers and rooms compared with fixed binaural front ends, and that the best model is BiEAR + Dual Controller + Rel., implying that ear-specific, time-varying frequency selectivity is beneficial in multi-speaker localization and distance estimation (Meng et al., 5 Jun 2026).

Beyond localization, BTFF-like representations are used in speech enhancement, intelligibility modeling, and binaural reconstruction. Complex-STFT binaural enhancement systems improve estimated binaural speech intelligibility while preserving ILD and IPD more effectively when cue-preserving losses are included (Tokala et al., 2024, Tokala et al., 26 Jul 2025). SIREN, although visually guided rather than cue-engineered, reports consistent gains on time-frequency and phase-sensitive metrics with competitive SNR by reconstructing complex left/right spectra and enforcing confidence-weighted consistency during fusion (Song et al., 31 Mar 2026).

6. Limitations, misconceptions, and open directions

A frequent misconception is that BTFF must always mean explicit maps of ITD, IPD, and ILD. The literature does not support such a narrow view. BMFD is BTFF-like but encodes binaural information through fixed signal-domain decoder outputs rather than through explicit cue parameters (Biberger et al., 2021). Conversely, cue-map BTFFs in SSL and BiSELD are explicitly interpretable, while enhancement models often use complex left/right STFTs as the primary representation and reintroduce cue structure only through losses (Panah et al., 17 Nov 2025, Tokala et al., 26 Jul 2025). The representation class is therefore heterogeneous by construction.

Another misconception is that a fixed cue extractor is necessarily sufficient. BiEAR shows the opposite design point: the auditory decomposition itself can be adaptive over time and frequency, and dual-controller ear-specific adaptation outperforms shared control (Meng et al., 5 Jun 2026). This suggests that BTFF need not be a static front end, and that stateful feature extraction may be important in dynamic acoustic scenes.

Several papers also identify practical limitations. BMFD is deliberately simplified relative to full physiological or EC models; it lacks adaptive equalization/cancellation, explicit sluggishness in binaural tracking, and richer low- vs high-frequency binaural mechanisms (Biberger et al., 2021). The systematic SSL feature study leaves FFT size, window type, feature normalization, and several other front-end details unspecified, even though feature choice is its central variable (Panah et al., 17 Nov 2025). BiEAR does not specify sampling rate, FFT size, or exact cross-correlation formula in the provided description (Meng et al., 5 Jun 2026). Binaspect explicitly notes that as the number of sources increases, time-frequency overlap increases, causing errors in ITD/ILR estimates and making histogram representations less useful for visual inspection (Barry et al., 29 Oct 2025). These points mean that BTFF research often combines strong conceptual structure with incomplete reproducibility at the implementation level.

Natural-scene statistics introduce a further caution. Real binaural cues frequently reflect multiple sources, motion, and reflections rather than a single clean source. In natural recordings, many observed IPDs exceed the single-source physiological range, with proportions up to 45% in some conditions, implying that “forbidden” IPDs are common rather than exceptional (Młynarski et al., 2014). This argues against treating BTFFs only as direct source-location readouts. A plausible implication is that robust BTFF systems should encode not only location cues but also scene complexity, overlap, or reliability.

The current research trajectory points in three directions. One is cue-complete BTFF design, combining event content, temporal change, low-frequency timing, high-frequency level asymmetry, and high-frequency spectral shape, as in BiSELD (Lee et al., 28 Jul 2025). Another is adaptive BTFF design, in which the auditory decomposition itself is stateful and ear-specific, as in BiEAR (Meng et al., 5 Jun 2026). A third is probabilistic or consistency-aware BTFF design, where feature reliability is modeled explicitly, as in RBR or in phase-consistent complex reconstruction systems (Deleforge et al., 2016, Song et al., 31 Mar 2026). Taken together, these lines indicate that BTFF has evolved from a descriptive label for binaural cue maps into a broader framework for structuring binaural information in time-frequency space.