Noise Adapted Speaker Model (NASM)

• NASM is a model architecture that adapts speaker representations to maintain identity features in the presence of noise, reverberation, and distortions.
• It employs advanced methods like psychoacoustic feature extraction and dual-stage attention to reweight frequency and temporal components for improved accuracy.
• Empirical evaluations demonstrate significant gains in ASR, speaker verification, and TTS performance through explicit noise-feature modeling and multi-objective loss optimization.

A Noise Adapted Speaker Model (NASM) refers to an architecture or set of methodologies that explicitly adapts speaker models so as to preserve speaker discriminative features in the presence of background noise, reverberation, and other distortions. NASM approaches employ strategies at feature extraction, representation, model conditioning, and loss optimization levels, targeting speech recognition, speaker identification, or synthesis under adverse acoustic conditions. Core principles include explicit separation or modeling of noise, robust feature adaptation, and the systematic use of conditioning signals or network architectures to maintain speaker characteristics.

1. Auditory and Psychoacoustic Feature Modeling

Noise adaptation frequently draws on psychoacoustic modeling to extract features resilient to environmental contamination. NASM designs can employ multi-band time–frequency masking, temporal integration, and comprehensive auditory transformations that mimic cochlear processing. For example, the adaptive psychoacoustic model (Dai et al., 2016) formulates masking via convolutional operations in time–frequency space:

• Temporal masking: $$M_{tm}(f, t, \Delta t) = A_{tm}(\Delta t) Y(f, t+\Delta t)$$
• Simultaneous masking: $$M_{sm}(f, t, \Delta f) = A_{sm}(\Delta f) Y(f+\Delta f, t)$$
• Diagonal masking: $$M_{diag}(f, t, \Delta f, \Delta t) = A_{diag}(\Delta f, \Delta t) Y(f+\Delta f, t+\Delta t)$$ Combined masking is computed as:

$$M_{total}(f, t) = \sum_{(\Delta f, \Delta t)} \alpha(\Delta f, \Delta t) Y(f+\Delta f, t+\Delta t)$$

The masking filter is applied via convolution, and frequency-dependent behavior is realized by separate low/high frequency band treatment. Otoacoustic emissions (OAEs) are modeled as additive effects in the spectral domain, and a double-Fourier “double-transform” enables spectral analysis distinguishing central speech energy from noise distributions. This results in increased ASR accuracy (up to 85.39% on noisy AURORA2 when trained on clean data), demonstrating the impact of perceptually modeled adaptation.

2. Attention Mechanisms and Robust Deep Neural Feature Processing

Acoustic deep models underlying NASM benefit from attention strategies for noise robustness. Two-stage attention architectures (Shi et al., 2019) employ frequency attention followed by time attention, or vice versa. The frequency attention module computes reweighting vectors through pooling (max/statistics) and non-linear transformation:

$H' = F'_{freq} \odot H$

$F_{freq}(H) = \text{Sigmoid}(s_{stat}+s_{max})$

Time attention subsequently smooths features:

$H'' = F'_{time} \odot H'$

$$\alpha_t = \frac{\exp(s_t)}{\sum_{i=1}^T \exp(s_i)}$$

The dual module is employed in TDNNs and CNNs, enhancing focus on reliable acoustic components and yielding lower EER and higher Top-1 accuracy across various SNRs. Such architectures permit modular integration into existing pipelines, improving adaptability and discrimination in the presence of noise.

3. Disentanglement, Conditioning, and Explicit Noise-Feature Modeling

Recent NASM work increasingly relies on explicit separation of noise and speaker information within learned representations. Dimensionality reduction modules (Kim et al., 2021) split speaker embeddings into “speaker codes” and “noise codes,” applying dropout to the latter and utilizing a learnable Speech Activity Vector (SAV) to bias the network’s handling of speech/non-speech segments. Similarly, adversarial disentanglement frameworks (Xing et al., 21 Aug 2024) deploy dual encoders to extract speaker ($S_s$) and noise ($S_i$) components from noisy embeddings, regularize with reconstruction modules, apply feature-robust losses to match clean and noise-reduced speaker codes, and enforce domain-invariant representations through adversarial training.

Speaker adaptation and beamforming approaches (Menne et al., 2018) integrate mask estimation neural networks with acoustic modeling back-ends, enabling end-to-end adaptation and improved speaker-specific denoising. ParaNoise-SV (Kim et al., 10 Aug 2025) explicitly combines dual U-Nets for noise extraction and speech enhancement, feeding noise Latent features from NE into SE encoders, and jointly optimizing via multi-objective loss components.

4. Noise Feature Augmentation and Efficient Conditioning

Noise-aware training methods (Raj et al., 2020, Lee et al., 2022) utilize utterance-level or frame-level noise vectors computed as means of speech/silence frames, achieved with minimal computational overhead:

$$\mu_{speech} = \frac{\sum_{t \in \text{speech}} x_{it}}{\#\text{speech frames}},\quad \mu_{silence} = \frac{\sum_{t \in \text{silence}} x_{it}}{\#\text{silence frames}}$$

Noise vectors are concatenated or appended to standard feature inputs, allowing the acoustic models to adjust parameters based on current noise characteristics. Such strategies outperform traditional i-vector, e-vector, and NAT-vector adaptation, leading to ~6–7% relative WER improvement on benchmark datasets with efficient online and streaming extensions.

Noise representation can also be leveraged in conditional synthesis (Dai et al., 2020), where denoise masks inform decoder post-processing and speaker embeddings from recognition nets allow personalized, robust low-resource TTS.

5. Speech Enhancement, Gradient-Guided Supervision, and Coding Approaches

NASM systems often integrate speech enhancement and coding layers. Meta-learning based enhancement (Yu et al., 2021) adapts speaker-specific masking networks with one-shot updates using enrolled speaker embeddings from ECAPA-TDNN, yielding competitive or superior PESQ, CSIG, and COVL scores in causal, real-time SE.

Gradient weighting mechanisms (Ma et al., 5 Jan 2024) use gradients from a frozen speaker verifier to highlight artifact-prone time–frequency bins in enhanced utterances. The loss for enhancement is adaptively weighted:

$$D_{t, f} = \sum_c (G^{enh}_{c, t, f} - G^{ref}_{c, t, f})$$

$$P_{t, f} = \frac{\exp(D_{t, f})}{\sum_{t, f} \exp(D_{t, f})}$$

$$\mathcal{L}_{Grad-W} = \sum_c \sum_{t, f} |A^{ref}_{c, t, f} - A^{enh}_{c, t, f}| \cdot P_{t, f}$$

This approach reduces artifact noise and improves EER/minDCF, especially at extremely low SNR.

Coding models (Yang et al., 2020) separate sources in latent space using hard masking and soft-to-hard VQ, controlling source-wise entropy and bit allocation:

$H^{(k)} = -\sum_{m=1}^M q_m^{(k)} \log q_m^{(k)}$

Regularization ensures higher quality, low bitrate speech coding by preferential source allocation.

6. Cochleagram, Advanced Feature Extraction, and Environmental Resilience

Auditory front-ends such as cochleagram-based features (Ahmed et al., 29 Aug 2025) exploit gammatone filterbanks ($128$ channel, $50$–$8000$ Hz), yielding robust 2-D representations tolerant to non-uniform distortions. CNN classifiers consume paired clean and noisy cochleagrams ($-5$ dB SNR) for adaptation, showing improved speaker identification in noisy, reverberated, and distorted scenarios over neurogram-based systems. The network features five convolutional layers, batch normalization, pooling, and L₂ regularization.

Joint factorized adaptation (Deng et al., 2023) employs hidden output transforms—LHUC and HUB—separately for speaker and environment, combining them via linear interpolation or cascaded architectures:

$$h^{(l, s, e)} = h^{(l)} \odot [\beta \cdot \xi(r^{(l, s)}) + (1 - \beta) \cdot \xi(n^{(l, e)})]$$

Bayesian inference ensures robust adaptation with uncertain test-data, achieving substantial WER reductions and rapid handling of unseen speaker–environment conditions.

7. Evaluation, Comparative Performance, and Practical Implications

Experiments demonstrate that NASM approaches consistently outperform baseline models across several domains and tasks. Enhanced psychoacoustic models (Dai et al., 2016) and attention-augmented networks (Shi et al., 2019) deliver sizeable relative improvements in noisy ASR and speaker identification. Disentanglement, adversarial, and enhancement-conditioned NASMs (Kim et al., 2021, Xing et al., 21 Aug 2024, Dai et al., 2020, Fujita et al., 10 Jan 2024, Yu et al., 2021) result in noise-invariant, high-fidelity speaker embeddings, stable performance in speaker verification (EER and minDCF), and improved sequence-level recognition.

Applications include robust text-to-speech (TTS) synthesis from noisy references, rapid and efficient low-resource adaptation, environmental resilience in streaming ASR, and accurate identification and verification on varied acoustic datasets. The comparative analyses highlight strength in explicit separation/modeling of noise, multi-objective joint training, and efficient conditioning, positioning the NASM paradigm as central to future robust speech and speaker technologies.

Summary Table: NASM Architectural Principals and Impact

NASM Principle	Technical Strategy	Reported Impact
Auditory/Psychoacoustic modeling	Masking, TI, OAE, double-transform	+19.6% avg WER improvement
Attention mechanisms (time/freq)	Two-stage (cascade/parallel)	Lower EER, improved accuracy
Explicit disentanglement, SAV conditioning	Split codes, dropout, SAD-guided	Lower DER, reduced confusion
Noise-aware feature augmentation	Noise/silence vector, control layer	+6–7% rel WER reduction
Speech enhancement and meta-learning	Speaker mask, one-shot adaptation	Competitive PESQ, CSIG, COVL
Gradient weighting loss	Dynamic per bin optimization	+31.9% EER improvement (low SNR)
Cochleagram/advanced auditory features	Gammatone CNN	Higher SID accuracy, total robustness

Conclusion

Noise Adapted Speaker Models encompass a spectrum of techniques that strategically adapt or condition speaker representation and recognition architectures to explicitly model, separate, or mitigate the impact of noise. Through psychoacoustic modeling, attention mechanisms, disentanglement frameworks, noise-aware conditioning, and gradient-guided enhancement, NASMs maintain speaker-specific discriminative power, structural robustness, and high fidelity across challenging acoustic environments, with demonstrated improvements in ASR, SV, SID, and TTS systems.