Harmonic Loss: ML Optimization & Physical Dissipation
- Harmonic loss is a distance-based alternative to cross-entropy that forms class probabilities from the inverse distances to learned prototypes, ensuring finite convergence.
- It extends to non-Euclidean metrics such as cosine, Bray–Curtis, and Mahalanobis, demonstrating improved performance in tasks like LiDAR detection, multi-label clinical coding, and weather forecasting.
- In electrical engineering, harmonic loss quantifies energy dissipation caused by non-sinusoidal currents, guiding the design of transformers, cables, and superconducting devices.
Searching arXiv for papers using “harmonic loss” and closely related variants. Searching arXiv for distance-based harmonic loss in machine learning. “Harmonic loss” is a polysemous technical term. In contemporary machine learning, it most commonly denotes a distance-based alternative to cross-entropy in which class probabilities are formed from inverse powers of distances to learned class prototypes, yielding a finite convergence point and a class-center interpretation (Baek et al., 3 Feb 2025). In adjacent ML literature, the phrase also names task-specific objectives such as 3D Harmonic Loss for LiDAR detection, Segmented Harmonic Loss for long-tailed multi-label clinical coding, and a modified spherical-harmonic loss for weather forecasting (Zhang et al., 2022, Ray et al., 2023, Subich et al., 31 Jan 2025). In electrical engineering and applied electromagnetics, “harmonic loss” instead refers to physical dissipation induced by non-sinusoidal currents or fields, including transformer eddy-current losses, cable heating, and AC losses in coated conductors under higher harmonics (Peerzada et al., 2024, Furman et al., 2010). This suggests that the term is best understood as a family of domain-specific constructions unified by their relation to harmonic structure, rather than as a single universal definition.
1. Terminological scope and main uses
In the cited literature, “harmonic loss” appears in at least two broad forms: as an optimization objective and as a physical loss mechanism. The optimization sense includes the original Euclidean harmonic loss for classification, its non-Euclidean generalizations, and application-specific variants for detection, medical coding, and weather forecasting. The physical sense covers losses caused by harmonic or high-order harmonic content in electrical and superconducting systems (Baek et al., 3 Feb 2025, Miller-Golub et al., 10 Mar 2026, Zhang et al., 2022, Ray et al., 2023, Subich et al., 31 Jan 2025, Peerzada et al., 2024, Furman et al., 2010).
| Domain | Meaning of “harmonic loss” | Representative source |
|---|---|---|
| Classification and LLM training | Distance-based negative log-likelihood built from inverse powers of distances to prototypes | (Baek et al., 3 Feb 2025) |
| Distance-layer generalization | Harmonic loss with cosine, Bray–Curtis, Mahalanobis, and other metrics | (Miller-Golub et al., 10 Mar 2026) |
| LiDAR 3D detection | Dynamically coupled loss over classification, box regression, and orientation | (Zhang et al., 2022) |
| Multi-label clinical coding | Segmented harmonic focal loss with harmonic-mean reweighting across segments | (Ray et al., 2023) |
| Weather forecasting | Modified spherical-harmonic loss separating amplitude and decorrelation errors | (Subich et al., 31 Jan 2025) |
| Power distribution and superconductors | Energy dissipation caused by harmonic currents and fields | (Peerzada et al., 2024, Furman et al., 2010) |
A recurrent theme across these meanings is that harmonic structure is treated explicitly rather than indirectly. In machine learning, this appears as geometry in representation space or as scale-wise decomposition in spectral space. In electrical systems, it appears as integer multiples of a fundamental frequency that drive additional heating, distortion, and reliability degradation.
2. Distance-based harmonic loss in classification
The Euclidean harmonic loss was introduced as an alternative to standard cross-entropy for classification and language modeling. Let be the penultimate hidden representation, let be the unembedding matrix, and let be the true class label. Instead of inner-product logits, harmonic loss uses Euclidean distances
With harmonic exponent , the “harmonic max” rule defines probabilities as
and the loss is
The construction is explicitly scale invariant under the joint rescaling , because all distances scale by and the common factor cancels in the normalized inverse-distance distribution (Baek et al., 3 Feb 2025).
Two theoretical properties are emphasized. First, harmonic loss has a finite convergence point: to make , it is sufficient that 0 while the other class distances remain bounded away from zero, so the optimum is attained at the finite point 1. Second, each 2 acquires a direct “class center” interpretation. This contrasts with cross-entropy, for which the quoted account states that achieving 3 requires 4, leading to unbounded norm growth in the late phase of training (Baek et al., 3 Feb 2025).
The empirical evaluation reported for the original formulation spans synthetic algorithmic tasks, MNIST, and GPT-2 small. On five synthetic tasks—In-Context Parallelogram Completion, Modular Addition mod 31, Equivalence Class mod 5, Genealogy in a binary tree, and Permutation Composition in 5—harmonic MLPs and Transformers are described as recovering the true 2D lattice, perfect circles, tree geometry, and coset clusters more faithfully than cross-entropy models. On MNIST with a single linear layer 6, both cross-entropy and harmonic loss achieved 7 test accuracy, but the harmonic weights resembled digit prototypes while the cross-entropy weights appeared noisy. On GPT-2 small trained on OpenWebText, the reported validation losses were 8 for cross-entropy and 9 for harmonic loss, with faster early convergence and smaller fluctuations for the harmonic model; the harmonic exponent used there was 0 (Baek et al., 3 Feb 2025).
The interpretability claim is central rather than incidental. In the original presentation, algorithmic-task embeddings exhibit lattices, circles, trees, and cosets; MNIST weights become literal class prototypes; and token embeddings in GPT-2 yield lower “parallelogram loss” on twelve function-vector tasks. The stated implication is that mechanistic interpretability becomes easier when the output layer is trained with a distance-based objective rather than a scale-sensitive softmax over inner products (Baek et al., 3 Feb 2025).
3. Non-Euclidean harmonic loss and representation geometry
The Euclidean formulation was later generalized by replacing 1 with an arbitrary nonnegative distance 2. For a dataset 3, backbone representations 4, and learnable prototypes 5, the generalized probabilities become
6
and the generalized harmonic loss is
7
This is also described as a distance-based softmin, with 8 and 9 (Miller-Golub et al., 10 Mar 2026).
Three concrete metric choices are highlighted. Cosine distance is
0
typically with 1-normalization of 2 and 3. Bray–Curtis distance is
4
with a small 5 for numerical stability. Mahalanobis distance is
6
with 7 implemented as a learned full inverse covariance, a diagonal approximation, or a fixed precomputed covariance; a Cholesky parameterization with diagonal regularization is specifically recommended (Miller-Golub et al., 10 Mar 2026).
The theoretical motivations stated in the generalization study are continuity with the original Euclidean case rather than a break from it. For any 8-homogeneous distance, harmonic probabilities are scale invariant under joint rescaling of representations and prototypes. Harmonic loss is also described as admitting finite global minimizers for the prototypes, and as encouraging prototypes to converge to geometric centers—means, medians, or other centers depending on the metric. The study further reports a PAC-Bayes margin bound in distance space and reiterates the empirical mitigation of grokking on synthetic tasks (Miller-Golub et al., 10 Mar 2026).
The empirical comparisons are framed as a three-way evaluation of performance, interpretability, and sustainability. On vision datasets including MNIST, CIFAR-10, CIFAR-100, Marathi Sign Language, and TinyImageNet, with architectures including MLP, small CNN, ResNet-50, and PVTv2-B0, cosine harmonic loss is reported as the most favorable trade-off: on ResNet-50 it yielded accuracy gains of 9 to 0 relative to cross-entropy on CIFAR-10/100 and Marathi Sign, F1 gains of similar magnitude, and carbon-emission reductions of 1 to 2 per run on CNN/ResNet-50. Bray–Curtis and Mahalanobis yielded larger gains in explained variance and stronger dimensional concentration, with Mahalanobis reaching EV3 gains up to 4 and PCA@90% in approximately 5–6 dimensions on deep models, but at higher efficiency cost (Miller-Golub et al., 10 Mar 2026).
On LLMs trained on OpenWebText—BERT-base, GPT-small, QWEN2-0.5B, and GPT-2-2B—cosine-based harmonic losses are reported to improve validation perplexity by 7–8 points relative to cross-entropy, increase gradient stability by 9–0, increase effective rank by 1–2, and increase PCA5 explained variance by 3–4, while reducing emissions relative to cross-entropy and Euclidean heads in the summary account. By contrast, Mahalanobis and higher-order Minkowski variants are described as occasionally improving perplexity on GPT-small but being unstable on GPT-2, with larger emissions overhead (Miller-Golub et al., 10 Mar 2026).
4. Coupled-task and long-tail variants
The phrase “harmonic loss” also denotes specialized objectives that are not simple prototype-classification heads. In LiDAR-based 3D object detection, 3D Harmonic Loss is designed to close the gap between classification, localization, and orientation estimation so that no sub-task over-optimizes at the expense of the others. For a positive sample 5, with classification score 6, predicted box offsets 7, and orientation score 8, the loss is
9
where
0
and 1 in practice. The base losses remain focal loss for classification, Smooth-2 over the seven box parameters for regression, and binary cross-entropy for orientation (Zhang et al., 2022).
The stated purpose is to address “overconfident but badly localized” boxes that survive NMS because classification and localization are optimized independently. The method is explicitly training-only: the original sum of losses is replaced, while voxelization, backbone, region proposal or pillar encoder, RoI pooling, heads, and post-processing remain identical. Reported KITTI validation improvements averaged 3 mAP in BEV at IoU 4, 5 mAP in 3D at IoU 6, 7 mAP in BEV at IoU 8, and 9 mAP in 3D at IoU 0. AOS also increased by 1 for PointPillar and 2 for SECOND at IoU 3. Because inference speed is unchanged, PointPillar and Harmonic PointPillar both ran at 4 Hz on a single RTX 2080Ti, and TensorRT float16 deployment on an NVIDIA Jetson Xavier TX reached 5 Hz with less than 6 mAP drop relative to the 2080Ti result (Zhang et al., 2022).
In multi-label clinical coding, Segmented Harmonic Loss addresses extreme class imbalance by segmenting the ordered label set into contiguous frequency bands and training segment-specific models. For segment 7, with predicted probability 8 and target 9, the transformed probability is
0
and the Segmented Harmonic Focal Loss is
1
The dynamic factor 2 is the harmonic mean of relative occurrence rates of the sample’s positive segments versus segment 3, and the segmentation algorithm recursively carves off tail segments subject to the standard-deviation criterion
4
with 5 in the reported experiments (Ray et al., 2023).
The experimental setting combines MIMIC-III and MIMIC-IV, then filters and denoises labels through an embedding-similarity procedure based on ClinicalBERT. After similarity filtering and thresholding, the dataset contains 6 samples and 7 codes. The base encoder is ClinicalBERT with 8M parameters; training uses AdamW with learning rate 9, mixed precision, a 0 split, and 1K steps per segment model. On the test set, SH-Focal-BERT achieved total micro-F1 of 2, with particularly large gains in the tail segment: 3 versus 4 for BCE-BERT, 5 for Focal-BERT, and 6 for CB-Focal-BERT. The same report states that overall micro-F1 moves from approximately 7 in prior SOTA to approximately 8, and that SH-Focal-BERT outperforms CAML, LAAT, RAC, EffectiveCAN, Discnet+RE, and MSMN+LongFormer by more than ten F1 points (Ray et al., 2023).
5. Spherical-harmonic formulations in weather forecasting
A different use of the term arises in data-driven weather forecasting, where a modified spherical-harmonic loss is proposed to correct the “double penalty” induced by grid-point mean squared error. For a scalar field 9 on the sphere with spherical-harmonic coefficients 00, Parseval’s theorem yields
01
Grouping by total wavenumber 02, with
03
and
04
the standard MSE becomes
05
The cited derivation argues that, because fine scales have lower coherence 06, the MSE optimum shrinks the forecast spectrum by the factor 07, thereby smoothing fine structure (Subich et al., 31 Jan 2025).
The proposed remedy separates amplitude error from decorrelation error. The amplitude term is
08
and the decorrelation term is
09
The final adjusted loss is
10
It is described as a simple, parameter-free, drop-in replacement for grid-point MSE in GraphCast-style training (Subich et al., 31 Jan 2025).
The reported effects are large and explicitly quantified. Fine-tuning GraphCast with this loss increases effective resolution from 11 km to 12 km at 5-day lead. CRPS is reduced by 13–14 across key variables, while the eRMSE of the ensemble mean is essentially unchanged and the spread/error ratio improves by 15–16 at medium leads. For tropical cyclones at 5-day lead, the mean maximum wind-speed bias changes from approximately 17 m/s in the control model to approximately 18 m/s in the AMSE model, mean central-pressure error is reduced by approximately 19 hPa, and position error remains approximately 20 km. For surface wind-speed extremes, the 98th percentile of 21 m wind increases from approximately 22 m/s in the control to approximately 23 m/s, versus observations near 24 m/s (Subich et al., 31 Jan 2025).
This formulation differs fundamentally from the prototype-based harmonic losses of classification. It does not compare an example against learned class centers; instead it decomposes a field-level error into spectral-amplitude and scale-dependent coherence terms. The commonality is structural: harmonic content is made explicit in the objective, rather than being left implicit in a pointwise loss.
6. Harmonic loss as physical dissipation under non-sinusoidal excitation
In electrical engineering, “harmonic loss” refers to real power dissipation and heating caused by harmonic content in currents and voltages. In modern distribution systems, non-linear household loads such as desktop PCs, laptop chargers, VFD-driven HVAC/washer/dryer, PV inverters, and EV chargers draw non-sinusoidal currents and inject integer multiples of 25 Hz into the feeder. The reported impacts include increased eddy-current and hysteresis heating in transformer windings, increased 26 losses and skin/proximity effects in cables, conductor overheating, reduced equipment lifetime and reliability, misoperations of protective devices, and voltage distortion at the PCC (Peerzada et al., 2024).
The cited distribution-system study uses standard current THD,
27
and a transformer harmonic-driven eddy-current loss model,
28
In the study this is specialized to
29
where 30, 31 is harmonic order, and 32 is RMS current at order 33. Nonlinear loads are modeled in PSCAD/EMTdc, spectra are extracted by FFT up to approximately 34 Hz, and a time-series harmonic analysis is run in OpenDSS on a modified IEEE 34 bus feeder with 35-minute resolution over 36 h (Peerzada et al., 2024).
Three scenarios are summarized. Scenario 1, consisting of VFD plus desktop plus laptop, exhibits dominant harmonics up to the 37 and THD of approximately 38. Scenario 2, adding PV, introduces pronounced 39, 40, and 41 harmonics in the approximately 42 kHz band, aligned with a feeder resonance peak. Scenario 3, adding EV, exhibits a spread of mid-order harmonics and THD of approximately 43. At the PCC, Scenario 2 has daily THD peaks of approximately 44–45 and average THD of approximately 46, while Scenarios 1 and 3 remain below 47. The same resonance near 48 kHz drives peak transformer harmonic eddy loss of approximately 49 kW in Scenario 2, or approximately 50 of 51 kVA, with average loss of approximately 52 kW; Scenarios 1 and 3 remain below 53 kW. Harmonics above the 54 order contribute approximately 55 of 56 in Scenario 2 and approximately 57 of THD (Peerzada et al., 2024).
The same study extends to a sequential-injection analysis. Nonlinear loads placed at buses with local resonance around 58–59 kHz raise THD at the substation bus from 60 with no sources to more than 61 with six such injections. The critical orders are 62–63, coinciding with resonance peaks of many feeder buses; the IEEE 519/EN 50160 THD limit of 64 at the substation is exceeded when multiple high-order sources align with resonance. The planning actions listed include driving-point impedance scans from 65 Hz to 66 kHz, limiting distributed PV/EV installations at buses with nearby resonance, de-tuned capacitor banks or damping reactors, passive filters tuned to the 67–68 orders, active power-electronic filters at the PCC of PV/EV inverters, staggered EV charging start times, upgraded transformer designs, and network impedance monitoring with adaptive capacitor switching (Peerzada et al., 2024).
A related superconducting setting analyzes AC losses in coated conductors under non-sinusoidal currents and magnetic fields within the Bean critical-state model. Transport current and applied field are written as
69
with typically 70 and 71. Under monotonic half-cycles, hysteresis losses depend on extrema rather than waveform details. For a “worst-case” third harmonic with 72 and 73, the reported increase in superconducting hysteresis loss is approximately 74 at low current and rises to approximately 75 as 76. In the normal-metal parts, the same 77 third harmonic raises eddy-current losses by approximately 78 at low currents and by approximately 79 near 80. The cited conclusion is that even small higher harmonics can produce tens-of-percent increases in losses, much larger than in conventional conductors, and therefore must be accounted for in the operating regimes of superconducting devices (Furman et al., 2010).
Across these engineering uses, harmonic loss is not a surrogate objective but a measurable dissipation process. The common mathematical pattern is frequency decomposition: either by harmonic order 81, or by explicit non-sinusoidal components 82. The practical emphasis is consequently on distortion metrics, resonance, thermal stress, and mitigation rather than on statistical optimization.