Perceptually Weighted STFT Losses

• Perceptually weighted STFT losses are objective functions that incorporate psychoacoustic principles to emphasize time-frequency regions critical to human auditory perception.
• They apply frequency-domain weight maps derived from equal-loudness contours, masking thresholds, and critical-band analyses to guide error minimization in speech and audio models.
• Empirical results demonstrate improved intelligibility and perceptual quality, enabling even small networks to achieve performance comparable to larger systems.

Perceptually weighted STFT losses are a class of objective functions for speech and audio modeling that explicitly incorporate psychoacoustic or linguistically informed frequency dependencies into traditional short-time Fourier transform (STFT) loss formulations. The central objective is to emphasize, within the training loss, specific time-frequency (TF) regions where human auditory perception, speech intelligibility, or other end-use metrics are most sensitive to degradations or artifacts, thereby incentivizing models to prioritize the minimization of errors most relevant to perceptual or downstream utility.

1. Psychoacoustic Motivation: Auditory Sensitivity and Loss Engineering

Human listeners are not uniformly sensitive to spectral deviations across all frequency bins or TF locations. For equal-amplitude distortions, errors in some bands (e.g., spectral valleys, high-frequency fricatives, frequency regions with low masking thresholds) are much more audible or detrimental for intelligibility than others. Uniform mean-squared error (MSE) or unweighted spectral losses thus yield models that over-optimize high-energy regions (typically low frequencies), leading to muffled harmonics, loss of sibilance, or excessive smoothing in perceptually critical bands (Li et al., 8 Nov 2025).

Approaches to perceptually weighted STFT losses use psychoacoustic models—such as equal-loudness contours, global masking thresholds, critical-band analyses, and linguistically motivated band-importance weights—to construct frequency-domain weight maps. These weights up- or down-scale local reconstruction errors so that model gradients more closely align with perceptual or task-driven priorities (Song et al., 2021, Li et al., 8 Nov 2025, Zhen et al., 2018, Monir et al., 23 Jun 2025, Wan et al., 2022).

2. Core Formulations and Mathematical Definitions

Multi-Resolution STFT Losses

The standard multi-resolution STFT (MR-STFT) loss used in neural speech models is composed of several STFT objectives computed at different FFT sizes, window lengths, and hop sizes (Song et al., 2021). A typical MR-STFT loss is: $$L_{\mathrm{mr\_stft}}(G) = \frac{1}{M}\sum_{m=1}^{M}L_{\mathrm{stft}}^{(m)}(G)$$ where each $L_{\mathrm{stft}}$ combines "spectral convergence" and log-magnitude terms for real and synthesized spectra.

Perceptual Weight Integration

Perceptual weighting modulates these losses with a frequency- or TF-dependent map $W_{t,f}$: $$L_{\mathrm{sc}^{w}}(x,\hat{x}) = \frac{\sqrt{\sum_{t,f}W_{t,f}\left(|X_{t,f}|-|\hat{X}_{t,f}|\right)^2}}{\sqrt{\sum_{t,f}|X_{t,f}|^2}}$$

$$L_{\mathrm{mag}^{w}}(x,\hat{x}) = \frac{1}{TN}\sum_{t,f}\left|\log W_{t,f}\cdot\left(\log|X_{t,f}|-\log|\hat{X}_{t,f}|\right)\right|$$

where $W_{t,f}$ is constructed to up-weight valley regions or otherwise important frequencies (Song et al., 2021).

Variants include losses based on the log-magnitude spectrum, STFT-based MSE with frequency weights, and complex mask estimation losses using band- or pool-wise weighting schemes (Li et al., 8 Nov 2025, Wan et al., 2022).

Representative Weight Constructions

• Spectral Valley Weighting: Construct $W_{t,f}$ using a fixed (time-invariant) FIR masking filter whose frequency response has notches at formant peaks and maxima in valleys, calculated from linear prediction (LP) coefficients.
• Equal-Loudness Weighting: Define $w(f)$ from ISO 226:2003 40-phon contours, i.e.,

$$w(f) = 10^{\frac{F_{\mathrm{ISO}}(1\,\mathrm{kHz})-F_{\mathrm{ISO}}(f)}{10}}$$

with $F_{\mathrm{ISO}}(f)$ as the dB SPL for equal loudness at frequency $L_{\mathrm{stft}}$0 (Li et al., 8 Nov 2025).

• Critical-Band Pooling: Assign high resolution (no pooling) to low frequencies, medium pooling in mid, and coarse pooling in high-f bands, reflecting the auditory system's non-uniform frequency resolution (Wan et al., 2022).
• Global Masking Thresholds: Compute $L_{\mathrm{stft}}$1 using global auditory masking models (e.g., MPEG PAM-1) dependent on the spectral content and masking curves of each input (Zhen et al., 2018).
• Band-Importance or Dynamic Weighting: Use ANSI band-importance weights, spectral magnitude of clean speech, or time-varying speech-to-noise ratios (Monir et al., 23 Jun 2025).

3. Construction of Weight Matrices and Psychoacoustic Basis

Table: Examples of Perceptual Weighting Schemes

Scheme / Paper	Weight Construction	Psychoacoustic Rationale
LP-based FIR mask (Song et al., 2021)	$L_{\mathrm{stft}}$2 (from average LSFs)	Emphasize spectral valleys (least masked)
Equal-loudness contours (Li et al., 8 Nov 2025)	$L_{\mathrm{stft}}$3	Human loudness sensitivity
Global masking (Zhen et al., 2018)	$L_{\mathrm{stft}}$4	Masking threshold (PAM-1 model)
PP-cIRM (Wan et al., 2022)	Pooling: no/mid/high (bin,2,4) resolution	Critical band resolution
ANSI/dynamic (Monir et al., 23 Jun 2025)	$L_{\mathrm{stft}}$5 or $L_{\mathrm{stft}}$6 via ANSI or SIR	Band importance / SNR emphasis

These constructions are typically performed either offline (e.g., LP mask, equal-loudness) or online (e.g., dynamic masking threshold) and can be static over the dataset or adaptive to each training utterance.

4. Model Integration and Training Considerations

Perceptually weighted STFT losses are integrated into training objectives of various architectures, from convolutional neural vocoders (e.g., Parallel WaveGAN) (Song et al., 2021) and masking-based or recurrent denoising architectures (e.g., GTCRN, ICCRN) (Li et al., 8 Nov 2025, Wan et al., 2022) to time-frequency domain beamformers (FaSNet) (Monir et al., 23 Jun 2025).

• Objective aggregation: The perceptual loss is combined additively with adversarial losses (GANs), SI-SNR, or unweighted auxiliary losses, sometimes with scalar weights (e.g., $L_{\mathrm{stft}}$7).
• Training schedules: Initial learning may use unweighted losses before introducing perceptual weights to stabilize gradients (Zhen et al., 2018).
• Weight normalization: To ensure stability, weights may be normalized (e.g., $L_{\mathrm{stft}}$8 range) or batch-normalized to have unit mean.

Hyperparameters such as the order of FIR masks (LP order $L_{\mathrm{stft}}$9), STFT window/hop parameters, or pooling stride are explicitly specified and tailored per model class (Song et al., 2021, Wan et al., 2022).

5. Empirical Impact and Experimental Results

A consistent finding across these works is a decoupling between standard (unweighted) distortion metrics and perceptual or intelligibility-focused metrics:

• Parallel WaveGAN vocoder: PW-MR-STFT loss led to MOS increases from 4.02/4.11 (female/male, unweighted) to 4.26/4.21 (weighted), with log-spectral distance reductions concentrated in spectral valleys (Song et al., 2021).
• GTCRN with Loud-Loss: WB-PESQ improved from 2.17 (MSE) to 2.93 (Loud-Loss), and ESTOI increased by +0.024. Listeners specifically preferred outputs for their clearer fricatives and sibilants (Li et al., 8 Nov 2025).
• FaSNet with weighted SDR: While overall SDR changed negligibly, frequency-weighted SDR and consonant-specific reconstruction scores increased, especially for fricatives and plosives. The best dynamic weighting improved FW-SDR by over 1 dB in high-phonetic-weight bands (Monir et al., 23 Jun 2025).
• Low-complexity DNNs: Perceptually weighted STFT-MSE enabled small networks to match the perceptual performance of baseline models several times larger (parameter count and inference MACs), with objective PEASS OPS gains of +0.03–0.05 (Zhen et al., 2018).

6. Extensions, Design Choices, and Practical Guidelines

Perceptual weighting can be adapted along several axes:

• Band split rationale: Empirically, weighting schemes tied to known auditory phenomena (e.g., masking, band-importance) are effective. For speech tasks, critical-band or band-importance curves are favored over generic A-weighting.
• Representation level: Weighting can be applied in magnitude, log-magnitude, power, or even mask estimation domains. Direct linear-magnitude weighting is discouraged where found to degrade performance (Li et al., 8 Nov 2025).
• Dynamic adaptation: Some losses incorporate weights that adapt on a per-frame basis, especially when using global masking models or SNR-based dynamic weighting (Monir et al., 23 Jun 2025, Zhen et al., 2018).
• Stability: Weight matrices should be clipped and normalized to prevent training instabilities or gradient domination by a few bins. Ramp-up schedules (MSE → perceptual) can prevent initial divergence (Zhen et al., 2018).
• Combining objectives: Multi-objective setups (e.g., Loud-Loss plus SI-SNR) are possible, requiring careful balancing ($W_{t,f}$0 selection) for best overall results (Li et al., 8 Nov 2025).

7. Outlook and Future Directions

The principal conclusion is that perceptually weighted STFT losses redirect model capacity and optimization pressure toward TF regions with outsized impact on perceptual quality and intelligibility. This realignment yields measurable gains on both objective (WB-PESQ, ESTOI, FW-SDR) and subjective metrics (MOS, listening tests), often enabling improved human-centric performance for a fixed network complexity.

Proposed future extensions include dynamic or context-adaptive weighting (locally adjusted to loudness estimates), integration with differentiable auditory front-ends (e.g., gammatonebanks), joint optimization of weighting parameters, and further psychoacoustic sophistication (e.g., two-tone suppression) (Li et al., 8 Nov 2025). Model-agnostic deployment is demonstrated, with successful transfer to speech enhancement, dereverberation, music denoising, and even ASR frontend weighting (Li et al., 8 Nov 2025).

Comprehensive reproduction recipes—including explicit weight computation, windowing choices, and normalization practices—are available for these losses, supporting further exploration and systematic ablation in speech and audio learning pipelines (Song et al., 2021, Li et al., 8 Nov 2025, Zhen et al., 2018, Wan et al., 2022, Monir et al., 23 Jun 2025).