Multi-Scale Frequency Loss Function

• Multi-scale frequency-domain loss functions are objective metrics that leverage frequency information at multiple scales to capture high-frequency details.
• They decompose signals using discrete transforms like Fourier, DCT, and wavelets, employing scale-adaptive reweighting to preserve oscillatory information.
• These loss functions are applied in high-dimensional PDE solving, computer vision, audio restoration, and geophysical modeling to boost feature learning.

A multi-scale frequency-domain loss function is an objective function that supervises the training of deep neural networks (DNNs) or related models using information measured in the frequency domain, typically at multiple spatial or temporal scales. This class of loss function is vital for high-fidelity recovery and feature learning of oscillatory, high-frequency, or multi-scale phenomena in scientific computing, computer vision, audio, and multimodal domains. Key mechanisms include partitioning the frequency space, reweighting frequency bands, iterative application of discrete transforms, and multi-scale aggregation of spectral errors.

1. Partitioning Frequency Space and Radial Scaling

Multi-scale frequency-domain loss functions often begin by representing the target function or signal in the frequency domain, e.g., via Fourier or discrete cosine/wavelet transforms. For high-dimensional function approximation and PDE solving, the frequency space is partitioned radially into concentric annuli $A_i$: $$A_i = \left\{ k \in \mathbb{R}^d : (i-1)K_0 \leq |k| \leq iK_0 \right\}$$ where $K_0 = K_{\max}/M$ and $M$ is the number of scales (Cai et al., 2019). Each component $f_i$ supported in $A_i$ is then downscaled via an affine mapping: $$\hat{f}_i^{\text{scale}}(k) = \hat{f}_i(\alpha_i k)$$ where $\alpha_i > 1$. Inverse transformation yields a component in the physical domain: $f_i(r) = f_i^{\text{scale}}(\alpha_i r)$ Such radial scaling enables standard DNNs, which are biased toward low-frequency learning (the F-Principle), to more efficiently capture high-frequency structures.

2. Multi-Scale Analysis Using Discrete Transform Techniques

Certain loss functions leverage multi-scale analysis by decomposing signals/images into frequency bands via wavelet or DCT transforms. The Wavelet Structure SIMilarity (W-SSIM) loss (Yang et al., 2020) applies the discrete wavelet transform (DWT) recursively to split the image $I$ into sub-band patches: $I^{LL}, I^{LH}, I^{HL}, I^{HH} = \mathrm{DWT}(I)$ Iterative application on the LL (low-frequency) channel yields progressively coarser scales. The final loss is a weighted sum of the SSIM measured across these transformed bands: $$\mathcal{L}_{\text{W-SSIM}}(x, y) = \sum_{i} r_i \mathcal{L}_{\text{SSIM}}(x_i^w, y_i^w), \quad w \in \{\text{LL, LH, HL, HH}\}$$ where weights $r_i$ accentuate the high-frequency channels to better preserve detail under multi-scale reconstruction.

Similarly, Frequency Domain Perceptual Loss (FDPL) (Sims, 2020) and other variants compute frequency-domain errors across 8×8 DCT blocks, applying tailored weighting based on perceptual importance or quantization tables, emphasizing frequencies that contribute most to human-subject visual quality.

3. Loss Formulation for Multi-Scale Physics Problems and Regression

For solving high-dimensional PDEs and multi-scale physical systems, loss functions that aggregate spectral errors via both energy-based (Ritz) and residual-based terms have been shown to accelerate convergence and accuracy. The Ritz energy loss (Cai et al., 2019) is given by: $$J(v) = \int_{\Omega} \left[ \frac{1}{2}\epsilon|\nabla v|^2 + V(r)v^2 \right] dr - \int_{\Omega} f(r) v(r) dr$$ with network output substituted for $v$. A discrete sum yields

$$L_{\text{Ritz}}(h) = \frac{1}{n} \sum_{x \in S} \left( \frac{1}{2} |\nabla h(x)|^2 - g(x)h(x) \right)$$

While least-squares PDE residual loss enforces

$$L_{\text{LSE}}(h) = \frac{1}{n} \sum_{x \in S} (\Delta h(x) + g(x))^2$$

Both types have strong multi-scale properties because they involve derivatives, inherently weighting high-frequency behaviors.

Recent works propose loss regularization strategies to synchronize the decay rates of multi-magnitude terms in PINNs (Wang et al., 2023). The regularized loss applies fractional power roots to the constituent terms: $$\tilde{\mathcal{L}}(\theta; \Sigma) = w_s \left[ \mathcal{L}_s(\theta; \tau_s) \right]^{1/m} + w_r \left[ \mathcal{L}_r(\theta; \tau_r) \right]^{1/n}$$ This enforces balanced optimization for disparate scales within multi-frequency or multi-magnitude problems.

4. Neural Architectures and Frequency-Domain Adaptations

The success of multi-scale frequency-domain loss functions is tightly coupled to neural architectures that are responsive to spectral content. Multi-scale DNNs (MscaleDNN) deploy parallel subnetworks with compact-supported activation functions—e.g., sReLU—

$$\text{sReLU}(x) = \text{ReLU}(1-x)\cdot\text{ReLU}(x)$$

to limit frequency leakage and facilitate scale separation (Cai et al., 2019). Inputs are partitioned by scale group, each receiving an explicitly scaled argument. Analytic studies supported by NTK-derived spectral diffusion models show that increasing the number of scales dramatically increases the support of the spectral diffusion coefficient, thus promoting uniform reduction of error across all frequencies (Wang et al., 2022).

Frequency-adaptive approaches (Huang et al., 28 Sep 2024) further extract dominant spectral features dynamically by applying discrete Fourier analysis to preliminary network outputs, then reconstructing subnetworks using hybrid features, e.g.,

$\Phi[k](x) = [k\cdot x; \cos(kx); \sin(kx)]$

This adaptive mechanism mitigates sensitivity to poor initial choices for scaling factors, resulting in robust multi-frequency approximation.

5. Applications in Computer Vision, Medical Imaging, Audio, and Geophysics

Multi-scale frequency-domain losses find applications in image restoration, super-resolution, and deraining:

• Joint time/frequency domain losses in TTS synthesis (Liu et al., 2020) combine mel-scale L2 loss with waveform-level SI-SDR, guiding the network to produce both perceptually and numerically accurate reconstructed speech.
• In Y-net for image dehazing (Yang et al., 2020), multi-level aggregation and wavelet-domain SSIM loss yield higher restoration fidelity.
• Dual-domain cascades for LDCT (Chung et al., 2020) employ MS-SSIM and $l_1$ loss in both frequency and image domains, significantly improving diagnostic image quality.
• In seismic inversion (OrthoSeisnet (Chakraborty et al., 9 Jan 2024)), U-Net with multi-scale FFT filtering orthogonalizes and sparsifies frequency components to enhance interpretability and structural resolution of thin subsurface layers, with training loss including direct frequency-domain criteria: $$\mathcal{L}_{\text{freq}} = \frac{1}{N}\sum_{u,v} |\mathcal{F}\{\hat{Y}\}(u,v)-\mathcal{F}\{Y\}(u,v)|^2$$

In segmentation tasks, Multi-Frequency in Multi-Scale Attention (MFMSA) blocks (Nam et al., 10 May 2024) fuse convolutionally efficient multi-scale spatial decompositions with DCT-channel frequency pooling. Features are then calibrated and spatially attended, enhancing boundary detection and generalizability across medical modalities.

Image deraining networks now employ dual-domain scale mixers (FDSM) (Zou et al., 15 Mar 2025), integrating pointwise convolutional multi-scale spatial features with FFT-based global frequency modulation, yielding improved restoration of interwoven local/global structures.

For detection in crowded scenes, cross-domain detection loss functions (Kim et al., 14 May 2025) coupled with multi-scale feature fusion capture high-frequency cues, enabling lightweight yet robust detectors.

6. Mathematical and Perceptual Foundations

Multi-scale frequency-domain loss functions often encode perceptual models, such as Watson’s model (Czolbe et al., 2020) via blockwise DCT decomposition and luminance/contrast masking: $$L_{\text{Watson}}(x,x') = \left[\epsilon + \sum_{i,j,k} \left| \frac{X_{ijk} - X'_{ijk}}{\tilde{T}_{ijk}} \right|^p \right]^{1/p}$$ Extensions with DFT-based amplitude and phase components enable translation invariance and improved perceptual alignment.

Wavelet and multi-scale DCT formulations leverage the properties of orthogonal bases to better resolve frequency-localized features (edges, textures) at multiple scales, superior to pixel-space losses at matching human visual response.

7. Impact and Limitations

Experimental benchmarks across modalities consistently demonstrate improved convergence rates, error decay over broader frequency ranges, and perceptual quality when multi-scale frequency-domain loss functions are deployed (Cai et al., 2019, Wang et al., 2022, Huang et al., 28 Sep 2024). Fast recovery of high-frequency details and avoidance of spectral bias are primary benefits, aligning both numerical accuracy and subjective human assessment.

However, these loss functions require careful architectural design (e.g., compact activations, scale-adaptive embeddings, hybrid domain processing) and often incur additional computational overhead during training (e.g., discrete transforms, multi-scale aggregation), though inference costs can be efficiently managed as shown in head tracking (Kim et al., 14 May 2025). Selection and balancing of weights for multi-magnitude, multi-domain loss terms remains an active area of research.

Multi-scale frequency-domain loss functions thus constitute a principled methodology for supervising high-capacity neural networks in contexts where accurate learning of oscillatory, multi-frequency, or scale-dependent phenomena is required, across computational science, audio, vision, and multimodal data analysis.