---
title: 'Harmonic Loss: ML Optimization & Physical Dissipation'
url: https://www.emergentmind.com/topics/harmonic-loss
type: topic
---

# Harmonic Loss: ML Optimization & Physical Dissipation

Searching arXiv for recent papers using “harmonic loss” and closely related variants.
Searching arXiv for distance-based harmonic loss in machine learning.
“Harmonic loss” is a polysemous technical term. In contemporary machine learning, it most commonly denotes a distance-based alternative to cross-entropy in which class probabilities are formed from inverse powers of distances to learned class prototypes, yielding a finite convergence point and a class-center interpretation [2502.01628]. In adjacent ML literature, the phrase also names task-specific objectives such as 3D Harmonic Loss for LiDAR detection, Segmented Harmonic Loss for long-tailed multi-label clinical coding, and a modified spherical-harmonic loss for weather forecasting [2211.03407] [2310.04595] [2501.19374]. In electrical engineering and applied electromagnetics, “harmonic loss” instead refers to physical dissipation induced by non-sinusoidal currents or fields, including transformer eddy-current losses, cable heating, and AC losses in coated conductors under higher harmonics [2407.21259] [1005.4819]. This suggests that the term is best understood as a family of domain-specific constructions unified by their relation to harmonic structure, rather than as a single universal definition.

## 1. Terminological scope and main uses

In the cited literature, “harmonic loss” appears in at least two broad forms: as an optimization objective and as a physical loss mechanism. The optimization sense includes the original Euclidean harmonic loss for classification, its non-Euclidean generalizations, and application-specific variants for detection, medical coding, and weather forecasting. The physical sense covers losses caused by harmonic or high-order harmonic content in electrical and superconducting systems [2502.01628] [2603.10225] [2211.03407] [2310.04595] [2501.19374] [2407.21259] [1005.4819].

| Domain | Meaning of “harmonic loss” | Representative source |
|---|---|---|
| Classification and LLM training | Distance-based negative log-likelihood built from inverse powers of distances to prototypes | [2502.01628] |
| Distance-layer generalization | Harmonic loss with cosine, Bray–Curtis, Mahalanobis, and other metrics | [2603.10225] |
| LiDAR 3D detection | Dynamically coupled loss over classification, box regression, and orientation | [2211.03407] |
| Multi-label clinical coding | Segmented harmonic focal loss with harmonic-mean reweighting across segments | [2310.04595] |
| Weather forecasting | Modified spherical-harmonic loss separating amplitude and decorrelation errors | [2501.19374] |
| Power distribution and superconductors | Energy dissipation caused by harmonic currents and fields | [2407.21259] [1005.4819] |

A recurrent theme across these meanings is that harmonic structure is treated explicitly rather than indirectly. In machine learning, this appears as geometry in representation space or as scale-wise decomposition in spectral space. In electrical systems, it appears as integer multiples of a fundamental frequency that drive additional heating, distortion, and reliability degradation.

## 2. Distance-based harmonic loss in classification

The Euclidean harmonic loss was introduced as an alternative to standard cross-entropy for classification and language modeling. Let \(x \in \mathbb{R}^N\) be the penultimate hidden representation, let \(W=[w_1,\dots,w_V] \in \mathbb{R}^{N\times V}\) be the unembedding matrix, and let \(c\) be the true class label. Instead of inner-product logits, harmonic loss uses Euclidean distances
\[
d_i = \|x - w_i\|_2 .
\]
With harmonic exponent \(n>0\), the “harmonic max” rule defines probabilities as
\[
p_i^{\rm H}=\frac{d_i^{-n}}{\sum_{j=1}^V d_j^{-n}},
\]
and the loss is
\[
L_{\rm H}=-\log p_c^{\rm H}
= -\log \Bigl(\frac{d_c^{-n}}{\sum_j d_j^{-n}}\Bigr).
\]
The construction is explicitly scale invariant under the joint rescaling \((x,w_i)\mapsto (\alpha x,\alpha w_i)\), because all distances scale by \(|\alpha|\) and the common factor cancels in the normalized inverse-distance distribution [2502.01628].

Two theoretical properties are emphasized. First, harmonic loss has a finite convergence point: to make \(p_c^{\rm H}\to 1\), it is sufficient that \(d_c \to 0\) while the other class distances remain bounded away from zero, so the optimum is attained at the finite point \(x=w_c\). Second, each \(w_i\) acquires a direct “class center” interpretation. This contrasts with cross-entropy, for which the quoted account states that achieving \(p_c \approx 1\) requires \(y_c \to +\infty\), leading to unbounded norm growth in the late phase of training [2502.01628].

The empirical evaluation reported for the original formulation spans synthetic algorithmic tasks, MNIST, and GPT-2 small. On five synthetic tasks—In-Context Parallelogram Completion, Modular Addition mod 31, Equivalence Class mod 5, Genealogy in a binary tree, and Permutation Composition in \(S_4\)—harmonic MLPs and Transformers are described as recovering the true 2D lattice, perfect circles, tree geometry, and coset clusters more faithfully than cross-entropy models. On MNIST with a single linear layer \(784 \to 10\), both cross-entropy and harmonic loss achieved \(\sim 92.5\%\) test accuracy, but the harmonic weights resembled digit prototypes while the cross-entropy weights appeared noisy. On GPT-2 small trained on OpenWebText, the reported validation losses were \(3.159\) for cross-entropy and \(3.146\) for harmonic loss, with faster early convergence and smaller fluctuations for the harmonic model; the harmonic exponent used there was \(n \approx \sqrt{768} \approx 28\) [2502.01628].

The interpretability claim is central rather than incidental. In the original presentation, algorithmic-task embeddings exhibit lattices, circles, trees, and cosets; MNIST weights become literal class prototypes; and token embeddings in GPT-2 yield lower “parallelogram loss” on twelve function-vector tasks. The stated implication is that mechanistic interpretability becomes easier when the output layer is trained with a distance-based objective rather than a scale-sensitive softmax over inner products [2502.01628].

## 3. Non-Euclidean harmonic loss and representation geometry

The Euclidean formulation was later generalized by replacing \(\|h_i-w_k\|_2\) with an arbitrary nonnegative distance \(D(h_i,w_k)\). For a dataset \(D=\{(x_i,y_i)\}_{i=1}^n\), backbone representations \(h_i=f(x_i)\), and learnable prototypes \(w_k\), the generalized probabilities become
\[
p(y_i=k|x_i)
= \frac{D(h_i,w_k)^{-n}}{\sum_{j=1}^K D(h_i,w_j)^{-n}},
\]
and the generalized harmonic loss is
\[
\mathcal{L}_{\rm harm}^D(W)
= - \sum_{i=1}^n
\log\!\Bigl(
\frac{D(h_i,w_{y_i})^{-n}}
{\sum_{j=1}^K D(h_i,w_j)^{-n}}
\Bigr).
\]
This is also described as a distance-based softmin, with \(z_{i,k}=-\log D(h_i,w_k)\) and \(p(y_i=k)\propto \exp(n z_{i,k})\) [2603.10225].

Three concrete metric choices are highlighted. Cosine distance is
\[
D_{\cos}(h,w)=1-\frac{h^\top w}{\|h\|_2 \|w\|_2},
\]
typically with \(\ell_2\)-normalization of \(h\) and \(w_k\). Bray–Curtis distance is
\[
D_{\rm BC}(h,w)
= \frac{\sum_{i=1}^d |h_i-w_i|}
{\sum_{i=1}^d (|h_i|+|w_i|)+\varepsilon},
\]
with a small \(\varepsilon\) for numerical stability. Mahalanobis distance is
\[
D_{\rm Mha}(h,w)
= \sqrt{(h-w)^\top \Sigma^{-1}(h-w)},
\]
with \(\Sigma^{-1}\) implemented as a learned full inverse covariance, a diagonal approximation, or a fixed precomputed covariance; a Cholesky parameterization with diagonal regularization is specifically recommended [2603.10225].

The theoretical motivations stated in the generalization study are continuity with the original Euclidean case rather than a break from it. For any \(1\)-homogeneous distance, harmonic probabilities are scale invariant under joint rescaling of representations and prototypes. Harmonic loss is also described as admitting finite global minimizers for the prototypes, and as encouraging prototypes to converge to geometric centers—means, medians, or other centers depending on the metric. The study further reports a PAC-Bayes margin bound in distance space and reiterates the empirical mitigation of grokking on synthetic tasks [2603.10225].

The empirical comparisons are framed as a three-way evaluation of performance, interpretability, and sustainability. On vision datasets including MNIST, CIFAR-10, CIFAR-100, Marathi Sign Language, and TinyImageNet, with architectures including MLP, small CNN, ResNet-50, and PVTv2-B0, cosine harmonic loss is reported as the most favorable trade-off: on ResNet-50 it yielded accuracy gains of \(+2\%\) to \(+10\%\) relative to cross-entropy on CIFAR-10/100 and Marathi Sign, F1 gains of similar magnitude, and carbon-emission reductions of \(10\%\) to \(30\%\) per run on CNN/ResNet-50. Bray–Curtis and Mahalanobis yielded larger gains in explained variance and stronger dimensional concentration, with Mahalanobis reaching EV\(_2\) gains up to \(10\times\) and PCA@90% in approximately \(1\)–\(4\) dimensions on deep models, but at higher efficiency cost [2603.10225].

On language models trained on OpenWebText—BERT-base, GPT-small, QWEN2-0.5B, and GPT-2-2B—cosine-based harmonic losses are reported to improve validation perplexity by \(1\)–\(5\) points relative to cross-entropy, increase gradient stability by \(10\%\)–\(20\%\), increase effective rank by \(15\%\)–\(30\%\), and increase PCA5 explained variance by \(20\%\)–\(40\%\), while reducing emissions relative to cross-entropy and Euclidean heads in the summary account. By contrast, Mahalanobis and higher-order Minkowski variants are described as occasionally improving perplexity on GPT-small but being unstable on GPT-2, with larger emissions overhead [2603.10225].

## 4. Coupled-task and long-tail variants

The phrase “harmonic loss” also denotes specialized objectives that are not simple prototype-classification heads. In LiDAR-based 3D object detection, 3D Harmonic Loss is designed to close the gap between classification, localization, and orientation estimation so that no sub-task over-optimizes at the expense of the others. For a positive sample \(i\), with classification score \(p_i\), predicted box offsets \(d_i'\), and orientation score \(p_i'\), the loss is
\[
L_{\text{3D-Har},i}
= (1+\beta_r)L_{\rm cls}(p_i,p_i^{gt})
+ (1+\beta_c)L_{\rm reg}(d_i',d_i^{gt})
+ \Bigl(1-\frac{\beta_r+\beta_c}{\beta_{\rm dir}}\Bigr)L_{\rm dir}(p_i',p_i'^{gt}),
\]
where
\[
\beta_r = \exp(-L_{\rm reg}(d_i',d_i^{gt})),
\qquad
\beta_c = \exp(-L_{\rm cls}(p_i,p_i^{gt})),
\]
and \(\beta_{\rm dir}=2\) in practice. The base losses remain focal loss for classification, Smooth-\(L_1\) over the seven box parameters for regression, and binary cross-entropy for orientation [2211.03407].

The stated purpose is to address “overconfident but badly localized” boxes that survive NMS because classification and localization are optimized independently. The method is explicitly training-only: the original sum of losses is replaced, while voxelization, backbone, region proposal or pillar encoder, RoI pooling, heads, and post-processing remain identical. Reported KITTI validation improvements averaged \(+0.81\%\) mAP in BEV at IoU \(0.7\), \(+0.32\%\) mAP in 3D at IoU \(0.7\), \(+0.42\%\) mAP in BEV at IoU \(0.5\), and \(+0.32\%\) mAP in 3D at IoU \(0.5\). AOS also increased by \(+0.26\%\) for PointPillar and \(+0.76\%\) for SECOND at IoU \(0.7\). Because inference speed is unchanged, PointPillar and Harmonic PointPillar both ran at \(43.1\) Hz on a single RTX 2080Ti, and TensorRT float16 deployment on an NVIDIA Jetson Xavier TX reached \(\sim 75.4\) Hz with less than \(1\%\) mAP drop relative to the 2080Ti result [2211.03407].

In multi-label clinical coding, Segmented Harmonic Loss addresses extreme class imbalance by segmenting the ordered label set into contiguous frequency bands and training segment-specific models. For segment \(r\), with predicted probability \(p_i^r\) and target \(y_i^r\), the transformed probability is
\[
q_i^r=
\begin{cases}
p_i^r & \text{if } y_i^r=1,\\
1-p_i^r & \text{if } y_i^r=0,
\end{cases}
\]
and the Segmented Harmonic Focal Loss is
\[
\mathrm{SH\text{-}Focal}(q^r)
=
-\frac{1}{\beta_r^{SH}}
\sum_{i=1}^{c_r}
(1-q_i^r)^\gamma \log(q_i^r),
\qquad \gamma \ge 0.
\]
The dynamic factor \(\beta_r^{SH}\) is the harmonic mean of relative occurrence rates of the sample’s positive segments versus segment \(r\), and the segmentation algorithm recursively carves off tail segments subject to the standard-deviation criterion
\[
\sigma_{\mathcal S} \le \eta_{\mathcal S} \times f_{\mathcal C},
\]
with \(\eta = 0.5\) in the reported experiments [2310.04595].

The experimental setting combines MIMIC-III and MIMIC-IV, then filters and denoises labels through an embedding-similarity procedure based on ClinicalBERT. After similarity filtering and thresholding, the dataset contains \(255{,}736\) samples and \(969\) codes. The base encoder is ClinicalBERT with \(110\)M parameters; training uses AdamW with learning rate \(5\times 10^{-4}\), mixed precision, a \(94\%/3\%/3\%\) split, and \(150\)K steps per segment model. On the test set, SH-Focal-BERT achieved total micro-F1 of \(71.28\), with particularly large gains in the tail segment: \(59.81\) versus \(46.03\) for BCE-BERT, \(42.50\) for Focal-BERT, and \(51.87\) for CB-Focal-BERT. The same report states that overall micro-F1 moves from approximately \(60\%\) in prior SOTA to approximately \(71\%\), and that SH-Focal-BERT outperforms CAML, LAAT, RAC, EffectiveCAN, Discnet+RE, and MSMN+LongFormer by more than ten F1 points [2310.04595].

## 5. Spherical-harmonic formulations in weather forecasting

A different use of the term arises in data-driven weather forecasting, where a modified spherical-harmonic loss is proposed to correct the “double penalty” induced by grid-point mean squared error. For a scalar field \(f(\theta,\phi)\) on the sphere with spherical-harmonic coefficients \(f_{\ell m}\), Parseval’s theorem yields
\[
L_{\rm MSE}(x,y)
=
\sum_{\ell=0}^{\infty}\sum_{m=-\ell}^{\ell}
|\alpha_{x,\ell m}-\alpha_{y,\ell m}|^2.
\]
Grouping by total wavenumber \(\ell\), with
\[
PSD_\ell(x)=\sum_{m=-\ell}^{\ell}|\alpha_{x,\ell m}|^2
\]
and
\[
\rho_\ell(x,y)
=
\frac{\sum_{m=-\ell}^{\ell}\Re[\alpha_{x,\ell m}\alpha_{y,\ell m}^*]}
{\sqrt{PSD_\ell(x)\,PSD_\ell(y)}},
\]
the standard MSE becomes
\[
L_{\rm MSE}(x,y)
=
\sum_{\ell}
\Bigl\{
PSD_\ell(x)+PSD_\ell(y)
-
2\sqrt{PSD_\ell(x)\,PSD_\ell(y)}\,\rho_\ell(x,y)
\Bigr\}.
\]
The cited derivation argues that, because fine scales have lower coherence \(\rho_\ell\), the MSE optimum shrinks the forecast spectrum by the factor \(\rho_\ell\), thereby smoothing fine structure [2501.19374].

The proposed remedy separates amplitude error from decorrelation error. The amplitude term is
\[
L_{\rm amp}(x,y)
=
\sum_{\ell}
\bigl[\sqrt{PSD_\ell(x)}-\sqrt{PSD_\ell(y)}\bigr]^2,
\]
and the decorrelation term is
\[
L_{\rm corr}(x,y)
=
\sum_{\ell}
2\,\max\{PSD_\ell(x),PSD_\ell(y)\}\,[1-\rho_\ell(x,y)].
\]
The final adjusted loss is
\[
L_{\rm harmonic}(x,y)
=
L_{\rm amp}(x,y)+L_{\rm corr}(x,y)
=
\sum_{\ell}
\Bigl\{
(\hat A_\ell-A_\ell)^2
+
2\,\max(\hat C_\ell,C_\ell)\,(1-\rho_\ell)
\Bigr\}.
\]
It is described as a simple, parameter-free, drop-in replacement for grid-point MSE in GraphCast-style training [2501.19374].

The reported effects are large and explicitly quantified. Fine-tuning GraphCast with this loss increases effective resolution from \(1{,}250\) km to \(160\) km at 5-day lead. CRPS is reduced by \(2\%\)–\(5\%\) across key variables, while the eRMSE of the ensemble mean is essentially unchanged and the spread/error ratio improves by \(10\%\)–\(20\%\) at medium leads. For tropical cyclones at 5-day lead, the mean maximum wind-speed bias changes from approximately \(-5\) m/s in the control model to approximately \(-1\) m/s in the AMSE model, mean central-pressure error is reduced by approximately \(4\) hPa, and position error remains approximately \(125\) km. For surface wind-speed extremes, the 98th percentile of \(10\) m wind increases from approximately \(10\) m/s in the control to approximately \(11.8\) m/s, versus observations near \(12\) m/s [2501.19374].

This formulation differs fundamentally from the prototype-based harmonic losses of classification. It does not compare an example against learned class centers; instead it decomposes a field-level error into spectral-amplitude and scale-dependent coherence terms. The commonality is structural: harmonic content is made explicit in the objective, rather than being left implicit in a pointwise loss.

## 6. Harmonic loss as physical dissipation under non-sinusoidal excitation

In electrical engineering, “harmonic loss” refers to real power dissipation and heating caused by harmonic content in currents and voltages. In modern distribution systems, non-linear household loads such as desktop PCs, laptop chargers, VFD-driven HVAC/washer/dryer, PV inverters, and EV chargers draw non-sinusoidal currents and inject integer multiples of \(60\) Hz into the feeder. The reported impacts include increased eddy-current and hysteresis heating in transformer windings, increased \(I^2R\) losses and skin/proximity effects in cables, conductor overheating, reduced equipment lifetime and reliability, misoperations of protective devices, and voltage distortion at the PCC [2407.21259].

The cited distribution-system study uses standard current THD,
\[
\mathrm{THD}
=
\frac{\sqrt{\sum_{n=2}^N I_n^2}}{I_1}\times 100\%,
\]
and a transformer harmonic-driven eddy-current loss model,
\[
P_{\rm eddy}
=
\sum_{h=1}^{h_{\max}}
k_e
\Bigl(\frac{f_h}{f_1}\Bigr)^2
I_h^2.
\]
In the study this is specialized to
\[
Tr_{EC}
=
P_{EC-R}
\sum_{h=1}^{h_{\max}}
I_h^2 h^2,
\]
where \(P_{EC-R}=0.05\), \(h\) is harmonic order, and \(I_h\) is RMS current at order \(h\). Nonlinear loads are modeled in PSCAD/EMTdc, spectra are extracted by FFT up to approximately \(4000\) Hz, and a time-series harmonic analysis is run in OpenDSS on a modified IEEE 34 bus feeder with \(1\)-minute resolution over \(24\) h [2407.21259].

Three scenarios are summarized. Scenario 1, consisting of VFD plus desktop plus laptop, exhibits dominant harmonics up to the \(13^{\rm th}\) and THD of approximately \(2.5\%\). Scenario 2, adding PV, introduces pronounced \(25^{\rm th}\), \(27^{\rm th}\), and \(29^{\rm th}\) harmonics in the approximately \(1.5\) kHz band, aligned with a feeder resonance peak. Scenario 3, adding EV, exhibits a spread of mid-order harmonics and THD of approximately \(3\%\). At the PCC, Scenario 2 has daily THD peaks of approximately \(6\%\)–\(7\%\) and average THD of approximately \(5\%\), while Scenarios 1 and 3 remain below \(4\%\). The same resonance near \(1.5\) kHz drives peak transformer harmonic eddy loss of approximately \(0.28\) kW in Scenario 2, or approximately \(0.56\%\) of \(50\) kVA, with average loss of approximately \(0.18\) kW; Scenarios 1 and 3 remain below \(0.12\) kW. Harmonics above the \(25^{\rm th}\) order contribute approximately \(35\%\) of \(Tr_{EC}\) in Scenario 2 and approximately \(20\%\) of THD [2407.21259].

The same study extends to a sequential-injection analysis. Nonlinear loads placed at buses with local resonance around \(1.5\)–\(2\) kHz raise THD at the substation bus from \(2\%\) with no sources to more than \(5\%\) with six such injections. The critical orders are \(25^{\rm th}\)–\(29^{\rm th}\), coinciding with resonance peaks of many feeder buses; the IEEE 519/EN 50160 THD limit of \(5\%\) at the substation is exceeded when multiple high-order sources align with resonance. The planning actions listed include driving-point impedance scans from \(60\) Hz to \(3\) kHz, limiting distributed PV/EV installations at buses with nearby resonance, de-tuned capacitor banks or damping reactors, passive filters tuned to the \(25^{\rm th}\)–\(29^{\rm th}\) orders, active power-electronic filters at the PCC of PV/EV inverters, staggered EV charging start times, upgraded transformer designs, and network impedance monitoring with adaptive capacitor switching [2407.21259].

A related superconducting setting analyzes AC losses in coated conductors under non-sinusoidal currents and magnetic fields within the Bean critical-state model. Transport current and applied field are written as
\[
I(t)=\sum_{k=1}^{N} I_k \cos(k\omega t+\phi_k),
\qquad
H(t)=\sum_{k=1}^{N} H_k \cos(k\omega t+\psi_k),
\]
with typically \(I_k/I_1 \le 0.05\) and \(H_k/H_1 \le 0.05\). Under monotonic half-cycles, hysteresis losses depend on extrema rather than waveform details. For a “worst-case” third harmonic with \(I_3=0.05I_1\) and \(\phi_3=\pi\), the reported increase in superconducting hysteresis loss is approximately \(20\%\) at low current and rises to approximately \(35\%\) as \(i_1 \to 1\). In the normal-metal parts, the same \(5\%\) third harmonic raises eddy-current losses by approximately \(35\%\) at low currents and by approximately \(90\%\) near \(i_1 \approx 1\). The cited conclusion is that even small higher harmonics can produce tens-of-percent increases in losses, much larger than in conventional conductors, and therefore must be accounted for in the operating regimes of superconducting devices [1005.4819].

Across these engineering uses, harmonic loss is not a surrogate objective but a measurable dissipation process. The common mathematical pattern is frequency decomposition: either by harmonic order \(h\), or by explicit non-sinusoidal components \(k\omega\). The practical emphasis is consequently on distortion metrics, resonance, thermal stress, and mitigation rather than on statistical optimization.

Source: https://www.emergentmind.com/topics/harmonic-loss