Automatic Contextual Audio Denoising

Abstract: Audio context determines which sound components and sources are relevant and which can be perceived as irrelevant (noise) by listeners. For example, traffic noise is informative in urban surveillance but noise for a phone call at the same location. Most current audio denoising systems apply fixed target-noise definitions, often removing useful components in one context while failing to suppress irrelevant components. To address this, we introduce the concept automatic contextual audio denoising (ACAD) which defines target and noise based on the inferred context. In this work, we restrict context to be associated with an acoustic scene class. We label sound events outside the event distribution of a scene class (noise) as out-of-context (OC) and events typical for that scene as in-context (IC). We implement a deep learning method that automatically infers the context of the audio signal and removes OC components, and benchmark it against variants: without context inference, with oracle context, and with separately provided uninformative context. On paired clean/noisy data across diverse contexts, where OC components in one context may be IC in another, our proposed method outperforms other approaches across standard objective metrics, indicating that the model can infer context and context-dependent processing can enhance denoising.

• The paper introduces a FiLM-conditioned UNet that infers acoustic scene context and improves denoising to 12.12 dB SI-SDR, outperforming an unconditioned model by 1.96 dB.
• The method combines a pretrained or jointly fine-tuned acoustic scene classifier with a spectrogram-mask denoiser, allowing the same sound event to be retained or removed depending on the surrounding scene.
• The released six-scene dataset tests context-dependent suppression with synthetic mixtures, but acoustic mismatches between clean recordings and added noise may overestimate the true benefit of contextual reasoning.

Motivation and problem definition

The paper "Automatic Contextual Audio Denoising" (2605.22262) introduces automatic contextual audio denoising (ACAD), a task formulation in which the definitions of target and noise are inferred from the audio itself rather than fixed at training time. The motivating observation is that the same acoustic source can be either informative or disruptive depending on context: traffic noise is signal in urban surveillance but noise in a phone call. Conventional deep-learning denoisers learn a static mapping from noisy to clean audio for a predefined target class (typically speech), which risks removing contextually useful components while failing to suppress irrelevant ones. ACAD instead conditions the denoising process on a learned context embedding, with no explicit guidance about which sources constitute the target.

In this first instantiation, context is restricted to acoustic scene class membership. Sound events within the event distribution of the inferred scene are labeled in-context (IC), and outlier events outside that distribution are labeled out-of-context (OC) and treated as noise. The method consumes only the noisy signal $\tilde{\mathbf{x}}$, infers a context embedding $\mathbf{e}$, and produces an estimate $\hat{\mathbf{x}}$ with OC components suppressed.

Dataset construction

The dataset is built from real-life scene recordings in CochlScene (Kitchen, Park, Restaurant, Restroom, Street, Subway) as clean signals, with OC events drawn from FSD50K. IC/OC event sets are derived automatically: a PANNs sound event detector labels scene audio, AudioSet ontology parents are pruned when children are present, the 20 most active event classes (plus descendants) per scene form the preliminary IC set, and their siblings plus the most active events of other scenes form the preliminary OC set. A manual refinement step then removes semantically or acoustically confusable parents from OC according to three similarity rules. Critically, the construction ensures that an event class labeled OC in one scene is IC in another, forcing context-dependent rather than global removal behavior.

Using Scaper, the authors generate 10,000 noisy-clean 10-second pairs per scene for training and 3,000 for validation/testing, with clean audio at random LUFS in $[-15,-10]$ and 1–3 OC event classes mixed at $[-5,10]$ dB relative to the background, each with 1–2 instances of 0.5–3 s duration. The dataset is released publicly.

Method

The architecture comprises a context extractor $C$ and a denoiser $D$. $C$ is a CRNN (residual convolutional blocks, RNN, temporal attention pooling) pretrained for acoustic scene classification (ASC) on clean log-Mel spectrograms with cross-entropy loss; the context embedding $\mathbf{e}$ is taken from its penultimate fully connected layer. $D$ is a three-block encoder-decoder UNet operating on the STFT magnitude of $\mathbf{e}$0, producing a mask in $\mathbf{e}$1 applied element-wise to $\mathbf{e}$2; reconstruction uses the noisy phase. Context is injected via FiLM layers at every encoder and decoder layer. Training is two-stage, with $\mathbf{e}$3 either frozen ($\mathbf{e}$4) or finetuned jointly with $\mathbf{e}$5 under a weighted sum of ASC and SI-SNR losses ($\mathbf{e}$6, $\mathbf{e}$7).

Evaluation and results

The proposed method is benchmarked against an unconditioned UNet, an oracle-context variant ($\mathbf{e}$8, one-hot scene label into FiLM), and constant-vector conditioning ($\mathbf{e}$9) at two embedding sizes, over five runs each. Key results (SI-SDR / SDR in dB):

Model	SI-SDR	SDR
Noisy input	4.27	4.26
UNet (no context)	10.16	10.56
$\hat{\mathbf{x}}$0	10.82	11.23
$\hat{\mathbf{x}}$1	11.04	11.47
$\hat{\mathbf{x}}$2 (proposed)	12.12	12.56
$\hat{\mathbf{x}}$3 (both sizes)	10.02–10.13	10.41–10.53

The finetuned variant outperforms the unconditioned UNet by 1.96 dB SI-SDR and 2.0 dB SDR, and exceeds the oracle-context variant by 1.30 dB SI-SDR. The authors attribute this to the learned embedding carrying richer contextual information than a one-hot scene label. Constant conditioning slightly degrades performance relative to no conditioning, indicating that uninformative conditioning distracts rather than helps. The pretrained $\hat{\mathbf{x}}$4 achieves 84.18% ASC test accuracy. t-SNE visualizations of the denoiser bottleneck features show scene-class clustering only for models conditioned on informative context, confirming that context information is retained in the latent representations.

Limitations and open questions

The paper candidly identifies a confound: because OC events are synthetically mixed into real recordings, the denoiser can exploit acoustic statistical mismatch between the two sources rather than genuine contextual reasoning. The unconditioned UNet's 5.89 dB gain over the noisy input, and preliminary experiments cited by the authors, support this possibility, and the authors acknowledge that the conditioned models could likewise partially rely on mismatch cues. The reported context-conditioning gains are therefore an upper bound on the true contextual benefit, and the authors state that addressing this requires new data collection. Additional open questions include whether the scene-class-restricted notion of context generalizes to richer context definitions, and whether the learned context's advantage over oracle context persists once the mismatch confound is removed.

Conclusion

This paper formalizes ACAD as a context-dependent denoising task in which target and noise are defined by an inferred acoustic scene context, contributes a paired clean/noisy dataset across six scene classes with deliberately context-overlapping OC events, and establishes a FiLM-conditioned UNet baseline that infers context internally. The finetuned variant achieves the best results (12.12 dB SI-SDR), and the comparison against oracle and constant conditioning supports the claim that informative context improves denoising. The synthetic-mixture confound remains the principal caveat on interpreting these gains as purely contextual.