Spectral Loss in Learning and Signal Processing

• Spectral Loss is a family of objectives that measures error on transformed spectral domains (e.g., STFT, CWT) rather than raw temporal signals.
• It is employed across diverse applications—from speech enhancement to imaging and operator learning—to improve reconstruction fidelity and structural regularization.
• Studies indicate that combining magnitude and phase-aware losses or tailoring the objective for specific domains enhances performance and stability.

Searching arXiv for recent and foundational papers on “spectral loss” to ground the article in published work. arXiv search query: spectral loss arXiv search query: (Braun et al., 2020) spectral loss speech enhancement Spectral loss is a family of objectives and analytic quantities defined in relation to spectral representations or spectral structure rather than solely pointwise error in the original domain. Across the literature, the term does not denote a single universal construction. In supervised speech enhancement it refers broadly to losses on time–frequency representations such as STFT magnitudes or complex spectra rather than raw waveforms (Braun et al., 2020). In neural waveform modeling it denotes amplitude and phase losses computed from STFT or CWT spectra (Takaki et al., 2018, Takaki et al., 2019). Elsewhere it names local operator-consistency objectives on Hankel blocks, graph-spectral contrastive objectives, spectral residual norms for PDE solvers, penalties on empirical NTK or Hessian spectra, and even analytic measures of representation change or physically induced loss across a spectrum (Balle et al., 2012, Haochen et al., 2021, Du et al., 2023, Luo et al., 6 May 2026, He et al., 26 Sep 2025, Kitessa et al., 2 Jun 2026).

1. Conceptual scope and recurring structure

A common structure nevertheless recurs. A spectral loss usually replaces direct comparison in the native domain with comparison after a transform, projection, or decomposition whose coordinates encode scale, frequency, eigenstructure, or graph connectivity. In signal processing this may be the STFT, CWT, or FFT. In operator learning it may be a basis expansion or a Hankel factorization. In representation learning it may be the eigenvalues of an empirical Neural Tangent Kernel, the singular directions of a weight matrix, or the linear recoverability of band-limited Fourier energy (Braun et al., 2020, Balle et al., 2012, Luo et al., 6 May 2026, Qiu et al., 5 Oct 2025, Kitessa et al., 2 Jun 2026).

The resulting objectives differ in what they preserve. Some are reconstruction losses, such as matching spectral amplitudes or phase-sensitive complex spectra. Some are structural regularizers that flatten or stabilize a spectrum, for example by pushing a feature Gram matrix toward a scaled identity or by maximizing effective rank. Some are analytic diagnostics rather than training criteria, intended to quantify accessibility, collapse, or information loss. This suggests that “spectral” is best understood as referring to the object on which the loss acts—time–frequency coefficients, graph or operator spectra, or eigenvalue distributions—rather than to a single formula or application domain (Takaki et al., 2018, Luo et al., 6 May 2026, Kitessa et al., 2 Jun 2026).

2. Time–frequency spectral losses in speech and audio

The most direct and widely instantiated meaning appears in speech. In the speech-enhancement setting of a real-valued suppression gain $G(k,n)$, the enhanced STFT is formed as $\widehat S(k,n)=G(k,n)X(k,n)$, and spectral losses are computed on $\widehat S$ and $S$ in the STFT domain even though the network predicts only a magnitude-domain filter. The surveyed classes include linear spectral distance norms, logarithmic spectral distance, power-law compressed losses, ratio-based losses, correlation-based losses, AMR-weighted losses, and linear combinations of magnitude-only and complex-domain losses. The principal empirical conclusions are that combining magnitude-only with phase-aware objectives always leads to improvements, that compressed spectral values also yield a significant improvement, and that phase-sensitive improvement is best achieved by linear domain losses such as mean absolute error (Braun et al., 2020).

For neural waveform generation, STFT spectral loss is formulated directly on waveform samples through a linear STFT operator $\mathbf{W}\mathbf{y}$. The amplitude spectral loss is

$$E^{(amp)}_{t,n}=\frac{1}{2}(\hat A_{t,n}-A_{t,n})^2,$$

while the phase spectral loss is

$$E^{(ph)}_{t,n}=1-\cos(\hat\theta_{t,n}-\theta_{t,n}).$$

The combined loss

$$E^{(sp)}=\sum_{t,n}\big(E^{(amp)}_{t,n}+\alpha_{t,n}E^{(ph)}_{t,n}\big)$$

admits a maximum-likelihood interpretation under Gaussian amplitude and von Mises phase models. In the reported experiments, adding phase loss improves over amplitude-only or waveform-domain MSE, and applying phase loss only in voiced frames yields the best MOS among the proposed variants (Takaki et al., 2018).

The same framework was generalized from STFT to CWT by representing both transforms as complex linear operators acting on the waveform and defining amplitude and phase losses on the resulting coefficients. The CWT extension was motivated by time and frequency resolutions closer to human auditory scales. Empirically, CWT amplitude spectral loss alone can train a high-quality model and is as good as STFT-based loss, whereas combining STFT and CWT losses did not improve the quality of synthetic speech in that study; the authors also report that CWT phase spectral loss produced noisy speech waveforms and therefore omitted it from the training criterion (Takaki et al., 2019).

3. Task-aware spectral losses in imaging, CT, and spectroscopy

In pan-sharpening, “spectral loss” initially denotes the conventional pixelwise reconstruction loss between the pan-sharpened output $\mathbf{G}_1$ and the multispectral target $\mathbf{M}_1$, typically

$\widehat S(k,n)=G(k,n)X(k,n)$0

The paper on S3 argues that this spectral loss alone is inadequate when PAN and MS images are misaligned, because it encourages double-edge artifacts and ghosting. Its replacement is a correlation-aware decomposition into a weighted spectral term,

$\widehat S(k,n)=G(k,n)X(k,n)$1

and a complementary structural term on normalized gradients,

$\widehat S(k,n)=G(k,n)X(k,n)$2

combined as $\widehat S(k,n)=G(k,n)X(k,n)$3. In that formulation, spectral fidelity is retained where PAN and MS agree, while spatial guidance dominates where the correlation map indicates unreliable alignment (Choi et al., 2019).

In photon-counting spectral CT, the proposed loss is explicitly called task-aware because it couples the physical basis domain to a downstream monoenergetic image domain. The network predicts corrected material basis images, and the loss combines an L1 term on those basis images with a VGG perceptual term computed on a synthesized 70 keV virtual monoenergetic image: $\widehat S(k,n)=G(k,n)X(k,n)$4 This construction is “spectral” because the perceptual branch depends on the spectral mixing coefficients $\widehat S(k,n)=G(k,n)X(k,n)$5 used to form the 70 keV image from the material basis channels, so the optimization target reflects the spectral imaging chain rather than only pixelwise basis fidelity (Hein et al., 2023).

A broader spectroscopic example appears in OASIS, where loss design is explicitly aligned with how spectra are analyzed. The framework uses weighted MSE for denoising, a baseline loss composed of weighted MSE with first- and second-order total-variation terms, the vicinity peak response (ViPeR) loss for peak localization, and a dynamic MSE $\widehat S(k,n)=G(k,n)X(k,n)$6 MCE $\widehat S(k,n)=G(k,n)X(k,n)$7 MQE schedule for peak intensity and FWHM retrieval. Here “spectral loss” does not mean Fourier-domain supervision; it denotes task-specific losses acting on spectral structure such as baselines, peaks, widths, and local neighborhoods on the spectral axis (Young et al., 15 Sep 2025).

4. Spectral losses on graphs, operators, and PDE residuals

In operator models for weighted automata, “spectral loss” is the local loss used to reinterpret spectral learning as optimization over observable operators on a finite Hankel sub-block. With $\widehat S(k,n)=G(k,n)X(k,n)$8 and $\widehat S(k,n)=G(k,n)X(k,n)$9 denoting Hankel blocks, the non-convex local loss is

$\widehat S$0

subject to $\widehat S$1. The paper shows that the classical SVD-based spectral method can be viewed as an optimizer of this objective, and also introduces the convex relaxation

$\widehat S$2

which trades data fit against model complexity through a nuclear-norm penalty (Balle et al., 2012).

In self-supervised learning, the spectral contrastive loss of the augmentation-graph paper arises from spectral decomposition of a population augmentation graph whose nodes are augmentations and whose edges connect views that can be produced from the same natural datum. The objective is a matrix-factorization surrogate for spectral clustering on that graph, yet it can be written as a contrastive learning objective on neural representations. Minimizing it yields features with provable accuracy guarantees under linear-probe evaluation, and the analysis does not require conditional independence of positive pairs given the class label (Haochen et al., 2021).

In neural PDE solvers, Neural Spectral Methods define the spectral loss as the squared Euclidean norm of the spectral coefficients of the PDE residual. If

$\widehat S$3

then

$\widehat S$4

By Parseval’s identity, this equals the $\widehat S$5 norm of the projected residual in physical space, but it is computed entirely in coefficient space. The reported advantages are more efficient differentiation through the network, reduced training complexity, and inference cost that remains constant with respect to the spatiotemporal resolution of the evaluation grid (Du et al., 2023).

5. Spectral regularization of learning dynamics and long-horizon generation

A different line of work uses spectral loss to regulate learning dynamics themselves. In continual reinforcement learning with Mixture-of-Experts policies, SPHERE defines spectral plasticity as the effective rank of the empirical NTK and derives a tractable proxy through expert feature Grams. The loss

$\widehat S$6

penalizes anisotropy of the routing-weighted expert feature Gram, flattening its spectrum while preserving trace. Under the paper’s approximations, minimizing this loss monotonically increases a lower bound on NTK effective rank and mitigates the loss of spectral plasticity during continual RL (Luo et al., 6 May 2026).

The continual-learning paper on Hessian spectral collapse reaches a related conclusion from a different route. It identifies loss of plasticity with the collapse of the Hessian spectrum at new-task initialization and motivates two regularization enhancements: maintaining high effective feature rank and applying $\widehat S$7 penalties. The resulting $\widehat S$8-ER objective is not named a spectral loss in the paper, but it is explicitly a loss on spectral quantities—effective rank of feature matrices and Hessian conditioning—and is proposed as a way to preserve $\widehat S$9-trainability across tasks (He et al., 26 Sep 2025).

For chaotic forecasting, the Binned Spectral Power loss is a frequency-domain loss on the energy distribution across wavenumber bands. If $S$0 and $S$1 are the predicted and target binned energies, BSP penalizes

$S$2

averaged across channels and bins. The aim is not to match individual Fourier coefficients but to preserve scale-wise energy allocation. The reported consequence is improved stability and spectral accuracy in autoregressive forecasts of multiscale chaotic systems without architectural modification (Chakraborty et al., 1 Feb 2025).

In relational time-series generation, Sequential RC-TGAN introduces a spectral envelope loss based on spectral envelope theory for categorical processes. For each feature, the loss compares the batch-averaged real and synthetic spectral envelopes over frequency by an $S$3 norm, and extends the same idea to continuous features through VGM discretization into latent modes. In that setting, spectral loss acts on latent periodic structure rather than pointwise trajectories, and the paper also proposes Spectral Density Divergence and Spectral Envelope Divergence as aligned evaluation metrics (Gueye et al., 30 Jun 2026).

6. Analytic, diagnostic, and physical senses of spectral loss

Not every use of the term is an optimization objective. In the study of vision representations, Residual Spectral Loss (RSL) is an analytic quantity, not a training loss. It measures how much spatial-frequency accessibility is lost or gained by a learned representation relative to a dimension-matched random projection baseline: $S$4 Positive RSL indicates structured spectral loss beyond compression; negative RSL indicates preservation or enhancement of accessibility relative to the random-projection baseline. The paper uses RSL to show that CLIP’s learned projection is spectrally neutral, whereas DINOv2’s final pooling induces a structured loss across frequency bands (Kitessa et al., 2 Jun 2026).

The phrase also appears in information-theoretic and physical settings. In the study of 5D-Gaussian spectral data, “spectral loss” is information loss under dimensionality reduction, quantified through Shannon entropy comparisons between the full correlated spectral state and reduced descriptions (Schelle et al., 2023). In synthetic wave compensation of plasmonic loss, the relevant quantity is the effective spectral loss introduced by finite spectral measurement range and window choice; the paper shows that a rectangular spectral window induces a $S$5 temporal kernel, while a Hann window yields a faster-decaying $S$6 kernel and substantially improves loss-offsetting efficiency (Guan et al., 29 Apr 2026). In photonic crystal microrings, spectral loss refers to the full spectral response of grating-induced loss as a function of $S$7, with distinct channels associated with OAM radiation, surface-mode radiation, and mode conversion (Pimbi et al., 20 May 2025).

A persistent misconception is that spectral loss necessarily means an $S$8 loss on Fourier magnitudes. The surveyed literature contradicts that simplification. Spectral loss may be magnitude-only, phase-aware, compressed, correlation-based, nuclear-norm-regularized, effective-rank-based, contrastive, graph-theoretic, or purely diagnostic. It may target reconstruction quality, perceptual quality, stability, plasticity, cluster structure, operator consistency, or physically meaningful scale statistics. The recurring lesson is domain dependence: phase-aware terms can be beneficial in speech (Braun et al., 2020, Takaki et al., 2018), spectral-envelope matching is natural for categorical time series (Gueye et al., 30 Jun 2026), and spectral regularization of eigenstructure is central when the failure mode is spectral collapse rather than pointwise reconstruction (Luo et al., 6 May 2026, He et al., 26 Sep 2025).