ASC-Conditioned SELD System for Hearables

• ASC-conditioned SELD system is a context-aware framework that couples single-channel acoustic scene classification with multi-channel sound event localization using scene-conditioned thresholds.
• It leverages a lightweight CNN–GRU ASC module and ACCDOA-based SELD decoding to dynamically adjust sensitivity, enhancing detection for scene-relevant events.
• The system shows improved location-dependent F-scores and low latency for real-time hearable applications while maintaining a scene-independent SELD network.

Searching arXiv for the cited work and closely related SELD/ACCDOA papers to ground the article. Searching for the main ASC-conditioned SELD paper. An ASC-conditioned SELD system is a context-aware environmental intelligence framework for hearables that couples Acoustic Scene Classification (ASC) with Sound Event Localization and Detection (SELD). In the formulation introduced for wearable audio devices, a lightweight ASC front-end first infers the current environment from single-channel headphone audio, and the predicted scene then conditions the SELD decoding of multi-channel headphone audio so that detection sensitivity is increased for scene-relevant events and reduced for unlikely ones. The conditioning is implemented as scene-conditioned thresholds on ACCDOA-derived event activity scores, rather than as a change to SELD network weights, and is motivated by the observation that active noise control (ANC) can mask crucial environmental cues such as approaching vehicles and alarms, thereby reducing situational awareness (Yeow et al., 18 Sep 2025).

1. System definition and operating principle

The framework explicitly couples two tasks. ASC infers the current environment, restricted in this system to the scene set $$s \in \{\text{Indoor}, \text{Nature}, \text{Urban}\}$$. SELD then detects and localizes specific sound events around the listener. The end-to-end workflow is three-stage: an ASC front-end processes single-channel headphone audio and outputs a scene label; a SELD back-end processes multi-channel headphone audio and outputs event activity and direction of arrival (DOA); and a conditioning stage uses the ASC prediction to adapt SELD decoding through scene-conditioned thresholds for event activity (Yeow et al., 18 Sep 2025).

Within this definition, “ASC-conditioned SELD” has a specific technical meaning. The system’s event activity decoding is dynamically modulated using thresholds that depend on the current acoustic scene predicted by ASC, so that detection sensitivity is higher for events relevant to that scene and lower for unlikely events. The conditioning is explicitly described as a soft conditioning strategy applied at the output decoding level. The SELD network itself remains scene-independent, and rare but real events are not hard-gated out.

This design addresses a limitation of conventional SELD in which all scenes and events are treated uniformly. The proposed alternative uses contextual intelligence to prioritize scene-relevant events, exemplified in the source material by car horns and sirens in Urban scenes and door knocks in Indoor scenes. A common misconception is that scene conditioning in such systems necessarily requires scene-specific retraining of the SELD backbone. Here, the opposite is true: the core SELDNet weights are unchanged, and only the decoder thresholds depend on scene. A second misconception is that scene conditioning hard-suppresses impossible events. The method does not force $a_i(t)=0$ for unlikely classes; it only raises the threshold required for activation.

2. ASC front-end: lightweight scene inference

The ASC module is a lightweight CNN–GRU-based low-complexity ASC model taken from Tan et al. for the DCASE 2025 low-complexity ASC task, selected for suitability on hearables (Yeow et al., 18 Sep 2025). It uses single-channel audio, specifically 1 s clips at 24 kHz taken from the first channel of the synthetic 4-channel data. Feature extraction is based on log-mel spectrograms with STFT window size 4096 samples, hop length 800 samples, and 256 Mel bins. If the time-domain ASC input is $x[n]$, the STFT and log-mel features are defined as

$X(k,t)=\sum_n x[n]\,w[n-tH]\,e^{-j2\pi kn/N},$

with $N=4096$ and $H=800$, followed by

$$P_m(t)=\log\left(\sum_k M_m(k)\,|X(k,t)|^2+\epsilon\right), \quad m=1,\dots,256.$$

The classifier outputs a scene posterior $p(s\mid \mathbf{X})$ over the scene set $\mathcal{S}$. Training uses cross-entropy loss over the ground-truth scene label: $$\mathcal{L}_{\text{ASC}}=-\sum_{s\in\mathcal{S}} \mathbb{1}[s=s_{\text{true}}]\log p(s\mid \mathbf{X}).$$ The reported training setup uses Adam, batch size 256, and 150 epochs. On the synthetic headphone dataset, the ASC model achieves 91.7% accuracy, which is treated as sufficiently reliable for downstream conditioning.

The module is explicitly engineered for low-complexity deployment. For 1 s input, its complexity is 10.9 M MACs with 116 K parameters. The source material characterizes these figures as suitable for real-time processing on embedded devices, and the low ASC latency later becomes important because scene predictions can be made available before SELD decoding completes.

3. SELD back-end and scene-conditioned ACCDOA decoding

The SELD back-end uses SELDNet with SALSA-Lite features. In this system, the multi-channel headphone input is represented by log-power linear spectrograms and normalized inter-channel phase differences extracted with FFT size 512, Hann window 512 samples, hop size 300 samples, and spatial aliasing frequency 1 kHz for a head spacing of 18 cm (Yeow et al., 18 Sep 2025). The network is a CRNN with convolutional layers for time–frequency processing, recurrent layers along time, and fully connected layers that output ACCDOA vectors.

The ACCDOA formalism couples event activity and Cartesian DOA into a single vector representation, removing the need for separate SED and DOA heads (Shimada et al., 2020). In the headphone system, the representation is 2D on the horizontal plane. At each frame $a_i(t)=0$0, SELDNet outputs

$a_i(t)=0$1

For class $a_i(t)=0$2, the Cartesian DOA vector is $a_i(t)=0$3, and the activity magnitude is

$a_i(t)=0$4

A conventional ACCDOA decoder uses a single global threshold $a_i(t)=0$5, typically 0.5, and declares class $a_i(t)=0$6 active if $a_i(t)=0$7. The ASC-conditioned system replaces this with a scene- and class-specific threshold matrix

$a_i(t)=0$8

Given the ASC-predicted scene $a_i(t)=0$9, activity becomes

$x[n]$0

This lowers $x[n]$1 for scene-relevant events and raises it for unlikely events. The thresholds were obtained by grid search on validation data.

Architecturally, conditioning occurs only at the decoder stage. The SELD network itself, including SELDNet and SALSA-Lite, is unchanged and scene-independent. System-level processing is therefore simple: the ASC thread outputs $x[n]$2, the SELD thread outputs $x[n]$3, activity scores $x[n]$4 are computed, scene-conditioned thresholds are applied, and DOA is taken directly from the angle of $x[n]$5. The paper follows ACCDOA training as in Shimada et al. and SALSA-Lite implementation; a typical training loss is a regression objective on ACCDOA vectors,

$x[n]$6

with Adam optimizer, batch size 64, 100 epochs, linear warm-up for the first 10 epochs to $x[n]$7, and cosine annealing over the remaining 90 epochs.

The back-end remains lightweight by hearable standards: 91.4 M MACs per 1 s clip and 285 K parameters. The paper presents this as sufficiently lightweight for real-time hearable deployment.

4. Simulated headphone corpus and evaluation protocol

The system is trained and evaluated on synthetic headphone data constructed to approximate wearable acoustics without costly real-world multi-channel recording (Yeow et al., 18 Sep 2025). Background audio is drawn from TAU Urban Acoustic Scenes 2020 Mobile and CochlScene, then grouped into the three coarse scene classes Indoor, Nature, and Urban. Foreground sound events are taken from FSD50K, NIGENS, and UrbanSound. The event vocabulary is limited to six classes selected for situational relevance: Bicycle, Car Horn, Crying, Dog, Door Knock, and Siren.

The scene-event associations and SNR ranges are fixed as follows:

Scene	Associated events	SNR range
Indoor	Crying, Dog, Door Knock	$x[n]$8 dB
Nature	Bicycle, Dog	$x[n]$9 dB
Urban	Bicycle, Car Horn, Siren	$X(k,t)=\sum_n x[n]\,w[n-tH]\,e^{-j2\pi kn/N},$0 dB

Headphone-specific spatialization is implemented by convolving each event with multi-channel RIRs from Corey et al. (2019), recorded specifically for wearable devices. The RIR set covers azimuths every $X(k,t)=\sum_n x[n]\,w[n-tH]\,e^{-j2\pi kn/N},$1 over 360°, giving 24 discrete DOAs. The resulting mixtures emulate a typical 4-channel headphone microphone configuration. The dataset contains 10,800 training clips and 1,800 test clips, all 1 s long at 24 kHz. The source description states that the dataset explicitly reflects binaural/headphone rendering via HRTFs and RIRs together with scene-event combinations relevant to everyday safety contexts.

Evaluation is split between ASC and SELD. ASC is measured by classification accuracy. SELD is measured using the location-dependent F-score $X(k,t)=\sum_n x[n]\,w[n-tH]\,e^{-j2\pi kn/N},$2, with a true positive counted only if the event class is predicted correctly and the estimated DOA is within $X(k,t)=\sum_n x[n]\,w[n-tH]\,e^{-j2\pi kn/N},$3 of the ground truth. The F-score is computed per class and macro-averaged: $X(k,t)=\sum_n x[n]\,w[n-tH]\,e^{-j2\pi kn/N},$4 The localization geometry is 2D Cartesian on the horizontal plane, consistent with the headphone RIR setup.

The synthetic nature of the dataset is methodologically central. It makes controlled analysis of scene conditioning possible, but it also defines the scope of the reported results. The paper explicitly notes that real environments include non-stationary sources, dense overlapping events, and more complex reverberation patterns, so synthetic-to-real domain shift remains an open issue.

5. Reported performance, complexity, and hearable integration

The principal empirical comparison is between a conventional SELD baseline using a global ACCDOA threshold and two scene-aware variants (Yeow et al., 18 Sep 2025).

Experiment	$X(k,t)=\sum_n x[n]\,w[n-tH]\,e^{-j2\pi kn/N},$5	ASC accuracy
Baseline	79.39	–
Baseline + ASC	80.78	91.7%
Baseline + Oracle	81.02	100%

The ASC-conditioned system improves the location-dependent F-score from 79.39 to 80.78, described in the source material as approximately +1.4 points. When ground-truth scenes are used instead of ASC predictions, performance reaches 81.02, approximately +1.63 points over baseline. The narrow gap between ASC-conditioned and Oracle, approximately 0.24 points, is used in the paper to argue that ASC errors have limited impact because conditioning is soft and thresholds do not fully suppress events. Figure 1 in the source material further reports that improvements are particularly visible for scene-relevant classes such as Car Horn and Siren in Urban scenes.

The performance gain is characterized not only as an increase in raw detection counts but as an increase in context-sensitive spatial performance. The paper attributes this to higher F-score for critical events in their natural scenes, reduced false alarms for unlikely events via raised thresholds, and better DOA-verified detection because the evaluation metric jointly constrains class and localization.

The deployment analysis is equally explicit. On a Raspberry Pi 4, measured ASC total latency is 13.1 ms. SELD feature extraction takes 18.5 ms, SELD model inference takes 19.5 ms, and SELD total latency is therefore 38.0 ms per 1 s input. End-to-end SELD latency is reported as well under 100 ms. Together with the model sizes of 116 K parameters for ASC and 285 K for SELD, these results are presented as favorable for battery-powered hearables.

ANC integration remains conceptual rather than implemented. The proposed usage model is that ANC runs continuously while ASC–SELD runs in parallel in a multi-threaded on-device configuration. When a critical event is detected and localized, ANC could temporarily reduce attenuation in that direction or generate an auditory, haptic, or visual alert. A common misconception is that the work includes a realized ANC control loop; it does not. The contribution is the spatial intelligence layer intended to sit on top of ANC, not ANC signal processing itself.

6. Limitations, related conditioning paradigms, and future directions

The source paper identifies four principal limitations (Yeow et al., 18 Sep 2025). First, the dataset is fully synthetic, and real environments introduce domain shift through non-stationary sources, dense overlap, and complex reverberation. Second, the taxonomy is small: only three scene classes and six event classes, with a fixed scene-event mapping. Third, scene-conditioned thresholds $X(k,t)=\sum_n x[n]\,w[n-tH]\,e^{-j2\pi kn/N},$6 are obtained by grid search, which is manageable only for small taxonomies. Fourth, scene conditioning occurs only at the decoding level; there is no deeper integration through conditional layers or attention.

These constraints define the most immediate extensions. The paper proposes real-world validation through collection and annotation of real multi-channel headphone recordings; automated threshold optimization via learned meta-parameters, Bayesian optimization, or reinforcement learning based on user feedback; richer scene and event taxonomies; personalization through user-specific thresholds and scene priors; and deeper conditioning mechanisms such as scene-conditioned feature gates or multi-task networks in which ASC outputs modulate SELD layers.

In the broader SELD literature, the system occupies a specific point in the design space. It inherits ACCDOA-style joint activity–DOA regression from earlier SELD work (Shimada et al., 2020), but introduces contextual conditioning at the decoder rather than the backbone. This differs from class-conditioned SELD models that inject one-hot class queries through FiLM layers and report especially strong improvements under directional interference (Slizovskaia et al., 2022). It also differs from wearable multi-modal SELD systems that condition acoustic feature extraction on head-motion sensor signals through MMTM-like excitation, improving performance for self-motioning listeners (Yasuda et al., 2024). A plausible implication is that ASC-conditioned thresholding is the minimal conditioning mechanism in this family: it is computationally cheap, robust to scene misclassification because it is soft, and compatible with low-power hearable constraints, but it exploits less contextual information than feature-level or sequence-level conditioning.

The paper also situates itself against application-specific hearing systems. “Semantic Hearing” is described as allowing users to program target sounds explicitly but requiring manual selection, whereas the ASC-conditioned framework is automatic and scene-driven. “PAWS” is described as focusing on the specific threat of car horns using specialized hardware, whereas the ASC-conditioned framework generalizes across multiple scenes and events within standard headphone configurations. In that sense, the proposal is best understood as a context-aware, lightweight hearable SELD system in which ASC is not an auxiliary output but the control signal that reshapes event activity decoding according to environment.