Neural Generalized Cross-Correlation with PHAT

• NGCC-PHAT is a novel deep learning-based method that refines classical GCC-PHAT by producing robust multi-peak TDOA feature maps for challenging acoustic scenes.
• It integrates learnable filterbanks, STFT processing, and a lightweight ConvNet to compute complex spectral weights from microphone array data.
• The model employs permutation-invariant training to support multi-target localization, achieving improved F-scores and reduced DOA errors in SELD tasks.

Neural Generalized Cross-Correlation with PHAT (NGCC-PHAT) is a data-driven audio feature extraction methodology designed for multi-source time-difference-of-arrival (TDOA) estimation and sound event localization. Developed to address the representational and ambiguity limitations of classical GCC-PHAT in real-world, multi-source, and reverberant acoustic environments, NGCC-PHAT integrates learnable filterbanks and permutation-invariant training (PIT) across “tracks” to enable robust, multi-peak TDOA feature maps suitable for direct input to modern sound event localization and detection (SELD) networks (Berg et al., 2024).

1. Background and Motivation

SELD systems built on microphone array data fundamentally require high-fidelity spatial features. Classical generalized cross-correlation with phase transform (GCC-PHAT) yields TDOA cues via hand-engineered spectral weighting between microphone pairs: $$R_{ij}[\tau] = \frac{1}{N} \sum_{k=0}^{N-1} \frac{X_i[k]\,X_j^*[k]}{|X_i[k]\,X_j^*[k]|} e^{j\,2\pi k\tau/N}$$ where $X_i[k]$ is the STFT of mic $i$. In anechoic, single-source conditions, GCC-PHAT exhibits a single sharp peak corresponding to the TDOA; in multi-source or reverberant mixtures, the peaks may overlap, and phase cues degrade, limiting localization accuracy. Standard approaches also assign a single TDOA estimate per microphone pair per frame, intrinsically restricting the system's ability to localize multiple concurrent sources (Berg et al., 2024).

2. NGCC-PHAT: Feature Extraction Architecture

NGCC-PHAT generalizes the classical pipeline by replacing the fixed PHAT weighting with a deep, learnable filterbank:

• Learnable Filterbank: Each raw input channel $x_i[n]$ is first processed by $L$ 1-D convolutional filters (e.g., SincNet layer plus time-domain convolutions), producing $L$ filtered signals $\tilde{x}_i^\ell[n]$ per mic.
• STFT and Cross-Spectrum: Each filtered signal is STFT’d, yielding $\tilde{X}_i^\ell[t, k]$. Cross-spectra $$C_{ij}^\ell[t, k] = \tilde{X}_i^\ell[t, k](\tilde{X}_j^\ell[t, k])^*$$ are computed for each mic pair and channel.
• Neural Weighting: These cross-spectra are processed by a lightweight ConvNet to yield complex weights $\Phi_{ij}^\ell[t, k]$.
• Neural “GCC”: Inverse-DFT-style summation yields cross-correlation maps per channel: $X_i[k]$0
• Dimensionality Reduction: Subsequent convolutional layers aggregate the outputs into $X_i[k]$1 feature maps, each reflecting certain spatial structures across $X_i[k]$2 microphone pairs and $X_i[k]$3 TDOA bins.

A typical implementation uses $X_i[k]$4 filters, four convolutional projection layers, $X_i[k]$5 final feature maps, $X_i[k]$6 microphones, and $X_i[k]$7 (13 TDOA bins per pair) (Berg et al., 2024).

3. Permutation-Invariant Multi-Target TDOA Training

For localizing $X_i[k]$8 simultaneous sources, the NGCC-PHAT framework projects the feature maps onto $X_i[k]$9 output “tracks” per microphone pair. Each track emits a probability mass function $i$0 across TDOA offsets. The reference set consists of the $i$1 true TDOAs for each frame and pair.

• PIT Objective: Training uses a global PIT loss, minimizing cross-entropy over all $i$2 assignments of predictions to ground-truth TDOAs: $i$3

$i$4

For $i$5, the targets are duplicated; for $i$6, a subset of references is selected (Berg et al., 2024). This structure is formally analogous to auxiliary duplicating PIT (ADPIT) used in multi-ACCDOA training (Shimada et al., 2021) and is compatible with the multi-class, multi-track detection objectives found in SED/SELD literature.

4. End-to-End Integration in SELD Systems

The NGCC-PHAT TDOA feature stack is concatenated with traditional log-mel spectral features per microphone and processed by a deep SELD network:

• Feature Fusion: Each $i$7 NGCC-PHAT output is collapsed along the delay axis via an MLP, yielding a fixed-dimensional vector per channel/pair. All TDOA and log-mel features are concatenated along the channel axis.
• SELD Backbone: The fused input is processed by a 2D CNN block and then a transformer-based CST-Former module, which computes event activity and localization parameters (e.g., via Multi-ACCDOA format with distance-weighted objectives).
• Training Regimen: Initial NGCC-PHAT module training uses Adam optimizer with channel-swapping data augmentation. SELD training proceeds with conventional location-dependent MSE loss in the downstream module, with network and fusion design parameters found empirically (Berg et al., 2024).

5. Quantitative Performance and Comparative Analysis

On the STARSS23 dataset, NGCC-PHAT improves localization and detection over established input features:

Input Feature	F-score $i$8 ↑	Mean DOA Error ↓	RDE ↓
GCC + log-mel (“MS”)	15.7 ± 1.0	27.7 ± 2.1°	0.78 ± 0.02
SALSA-Lite	24.6 ± 2.0	27.0 ± 1.2°	0.41 ± 0.02
NGCC-PHAT + log-mel	26.0 ± 2.0	25.8 ± 2.3°	0.42 ± 0.01

Increasing the number of channels $i$9 in the NGCC-PHAT module from 1 to 16 yields improvements in both $x_i[n]$0 and DOA error. Increasing output tracks $x_i[n]$1 up to 3 also enhances micro-averaged F-score, suggesting substantial practical advantage in overlapping-source conditions. These gains are robust across dataset splits and model capacities (Berg et al., 2024).

6. Position within the Multi-Target TDOA and SELD Training Landscape

NGCC-PHAT's methodology is distinct from, but highly compatible with, prior permutation-invariant and location-informed SELD paradigms:

• Relation to Standard PIT: NGCC-PHAT's use of PIT over $x_i[n]$2 per-pair tracks parallels the use of PIT in multi-speaker separation (Taherian et al., 2021) and multi-target localization (Diaz-Guerra et al., 2022), enabling assignment-agnostic training under permutation ambiguity.
• Auxiliary Duplication and Class-wise Assignment: The auxiliary duplication technique for target set sizing is formally comparable to ADPIT for multi-ACCDOA SELD (Shimada et al., 2021).
• Sliding PIT/Tracking: While NGCC-PHAT as formulated in (Berg et al., 2024) does not incorporate sliding-window PIT for source identity tracking, it is agnostic to such permutations, and the underlying features are compatible with such assignment procedures (Diaz-Guerra et al., 2022).
• Feature Drop-In: NGCC-PHAT can be used as a direct replacement for classical GCC-PHAT or other TDOA feature blocks in any downstream SELD or SED/DOA framework due to its fixed-rank, tensorial output (Berg et al., 2024).

7. Significance, Limitations, and Future Directions

NGCC-PHAT enables neural systems to learn spatial representations tailored for complex, multi-source acoustic scenes. Empirical evidence shows consistent improvements in localized detection accuracy with minimal additional model complexity. The feature's compatibility with PIT-based assignment enables straightforward extension to scenarios with variable source count and class, and to integration with state-of-the-art SELD architectures.

A plausible implication is that further advances in learnable spatial front-ends and adaptive assignment strategies (e.g., combining NGCC-PHAT with sliding window PIT (Diaz-Guerra et al., 2022)) may bridge remaining performance gaps in challenging, dynamic sound localization and tracking scenarios. The methodology is sufficiently general to support further augmentation (e.g., class-conditioned tracks, variable windowing, or alternative convolutional topologies) for emerging SED/SELD benchmarks and real-world deployments.

References:

• "Learning Multi-Target TDOA Features for Sound Event Localization and Detection" (Berg et al., 2024)
• "Multi-ACCDOA: Localizing and Detecting Overlapping Sounds from the Same Class with Auxiliary Duplicating Permutation Invariant Training" (Shimada et al., 2021)
• "Location-based training for multi-channel talker-independent speaker separation" (Taherian et al., 2021)
• "Position tracking of a varying number of sound sources with sliding permutation invariant training" (Diaz-Guerra et al., 2022)