Harmonized Gradient Descent Overview

• Harmonized Gradient Descent is a family of methods that adjusts the geometry and magnitude of gradients to distribute learning signals evenly.
• It includes techniques such as holonomic gradient descent for constrained estimation and class-wise norm equalization for imbalanced online learning.
• Empirical results show improved convergence and performance metrics (e.g., AUC, F1 scores) by mitigating gradient conflicts in various applications.

Searching arXiv for the cited papers and closely related uses of “HGD” / “gradient harmonization” to ground the article. Harmonized Gradient Descent (HGD) is not a single canonical algorithm but a family of optimization ideas centered on reshaping, coordinating, or reweighting gradient information so that learning remains effective under structural obstacles such as black-box access, gradient conflict, class imbalance, or analytically intractable objectives. In the recent literature, the name has been used explicitly for an online class-imbalance method that equalizes class-wise gradient norms (Zhou et al., 15 Aug 2025), while closely related work uses the same acronym for holonomic gradient descent in statistical estimation (Sakurai et al., 2013). Other papers embody the same harmonization principle without using the exact term, including GDFO for black-box prompt tuning (Han et al., 2023), GH and GH++ for unsupervised domain adaptation (Huang et al., 2024), the Gradient Harmonizing Mechanism for dense detection (Li et al., 2018), and HRGrad for multiscale kinetic learning (Liang, 27 Apr 2026).

1. Terminology and scope

The literature represented here uses “HGD” in at least two distinct senses. In statistics, HGD denotes holonomic gradient descent, a method that combines Pfaffian differential equations with iterative optimization for holonomic objectives (Sakurai et al., 2013). In online learning, HGD denotes harmonized gradient descent, an algorithm for imbalanced data streams that equalizes cumulative gradient-norm contributions across classes (Zhou et al., 15 Aug 2025). A third usage is implicit rather than explicit: the GDFO paper does not use the term HGD, but it defines a harmonization principle by combining gradient descent with derivative-free optimization for black-box prompt tuning (Han et al., 2023). Related work uses adjacent labels such as Gradient Harmonization (GH, GH++) in UDA (Huang et al., 2024), Gradient Harmonizing Mechanism (GHM) in detection (Li et al., 2018), and harmonized rotational gradient (HRGrad) in multiscale kinetic regimes (Liang, 27 Apr 2026).

Usage	Domain	Core mechanism
Holonomic gradient descent	Constrained MLE / REML	Pfaffian ODEs plus Newton-type optimization
Harmonized gradient descent	Imbalanced online learning	Class-wise gradient norm equalization
Gradient harmonization / related methods	UDA, black-box tuning, detection, APNNs	Gradient-angle control, density weighting, or mixed gradient sources

A common source of confusion is that these methods are mathematically different even when they share the acronym. The only safe generalization is that each method tries to prevent a harmful concentration of optimization signal—whether that concentration occurs in class frequency, task conflict, sample difficulty, or inaccessible black-box objectives.

2. General principle of harmonization

Across the modern uses of the term, harmonization refers to modifying the effective descent direction so that optimization is not dominated by a single source of signal. In GDFO, the final prompt is a linear mixture of a student-guided gradient path and a teacher-grounded derivative-free path,

$p = \alpha G_\mu(x) + (1-\alpha)(p_0 + Az),$

with $\alpha \in [0,1]$ controlling the balance (Han et al., 2023). In GH, conflicting task gradients are projected so that an obtuse angle becomes acute; in GH++, the angle is adjusted from obtuse to vertical, with a dynamic weighted-loss implementation derived through an integral operator (Huang et al., 2024). In the online HGD algorithm for class-imbalanced streams, each update is scaled by a class-dependent factor $\alpha_t$ so that cumulative gradient norms remain comparable across classes (Zhou et al., 15 Aug 2025). In GHM, per-sample losses are weighted inversely to the density of their gradient magnitudes, reducing domination by easy negatives and persistent outliers (Li et al., 2018). In HRGrad, conflicting task gradients are rotated into a harmonized cone and aggregated so that the final update has a non-negative dot product with every rotated task gradient (Liang, 27 Apr 2026).

Taken together, these works suggest a shared optimization pattern: the raw gradient is treated as an object whose geometry, magnitude, or source composition can be deliberately altered before the parameter update. The objective is not merely faster convergence, but a better allocation of descent across competing signals.

3. Black-box prompt tuning: harmonizing gradient descent and derivative-free optimization

In "When Gradient Descent Meets Derivative-Free Optimization: A Match Made in Black-Box Scenario" (Han et al., 2023), the setting is black-box prompt tuning for PLMs. A teacher model $f_T$ is available only through inference APIs, so its parameters and gradients are unavailable. Continuous prompts $p \in \mathbb{R}^{L \times d}$ are prepended to the input, and the training objective is to minimize

$L_{task}(p)=\mathbb{E}_{(x,y)\sim D}[L_T(p;x,y)],$

where

$L_T(p;x,y) = -\sum_c 1[y=c]\log p_T(c|x;p).$

Because direct gradient descent on the teacher is impossible, the method introduces a distilled student $S_\theta$ trained on random discrete prompts $p_r$ with a loss

$$L(\theta;\tau,\lambda) = (1-\lambda)L_{CE}(\theta) + \lambda L_{KL}(\theta;\tau).$$

After distillation, $\alpha \in [0,1]$0 is frozen, and a prompt generator $\alpha \in [0,1]$1 produces $\alpha \in [0,1]$2. This enables gradient updates through the student,

$\alpha \in [0,1]$3

The derivative-free component is CMA-ES, operating in a low-dimensional space $\alpha \in [0,1]$4 with random projection $\alpha \in [0,1]$5, so that

$\alpha \in [0,1]$6

The harmonization rule is

$\alpha \in [0,1]$7

Conceptually, this yields the combined prompt-space update

$\alpha \in [0,1]$8

The paper reports that $\alpha \in [0,1]$9 gives the best average across three datasets, and that extreme values degrade performance. In the main 16-shot results with RoBERTa_LARGE as teacher, GDFO achieves an average of 81.85\%, compared with 78.01\% for LoRA and 79.01\% for BBTv2. Reported task scores are SST-2 92.41\%, Yelp Polarity 93.17\%, AG’s News 87.19\%, DBPedia 96.92\%, MRPC 80.13 F1, SNLI 62.53\%, and RTE 60.57\%. Ablations show that removing KD lowers SNLI from 62.53\% to 58.51\%, while removing DFO causes an average decrease of 6.5\%. The method uses prompt length $\alpha_t$0, embedding dimension $\alpha_t$1 for RoBERTa_LARGE, low-dimensional $\alpha_t$2, population size $\alpha_t$3, and a teacher API budget of 8,000 calls.

This formulation makes harmonization explicit: gradient descent supplies a low-variance local signal through the student, whereas CMA-ES supplies teacher-grounded corrections under true black-box constraints. A plausible implication is that the method is most effective when the student is locally faithful but globally imperfect.

4. Gradient-conflict harmonization in multi-task and multiscale learning

In unsupervised domain adaptation, "Gradient Harmonization in Unsupervised Domain Adaptation" studies two gradients on shared parameters $\alpha_t$4: the classification gradient $\alpha_t$5 and the domain-alignment gradient $\alpha_t$6 (Huang et al., 2024). Conflict is defined by $\alpha_t$7. GH resolves this by symmetric projection:

$\alpha_t$8

$\alpha_t$9

The aggregated harmonized gradient is $f_T$0. GH++ instead aims for a vertical angle and is implemented through dynamic weights $f_T$1 in a reweighted loss

$f_T$2

The paper states that GH changes an obtuse angle to an acute angle, while GH++ changes it to a vertical angle and halves the sum of deviation angles relative to GH.

Empirically, GH and GH++ are reported on Office-31, Office-Home, Digits, VisDA-2017, and DomainNet. On Office-31, the reported average gains for CDAN are +1.5\% / +2.2\% for GH / GH++; on Office-Home they are +2.6\% / +3.2\%; on DomainNet the reported average gains are +3.2\% / +3.7\% for CDAN and +7.1\% / +7.3\% for MCD. The paper also reports that conflict histograms contain a substantial fraction of obtuse angles, and that GH/GH++ shift the inner-product distribution to positive values.

A more geometric and physics-specific extension appears in "Conflict-Aware Harmonized Rotational Gradient for Multiscale Kinetic Regimes" (Liang, 27 Apr 2026). There, tasks correspond to different Knudsen numbers $f_T$3, with gradients $f_T$4. HRGrad constructs the primal cone

$f_T$5

the dual cone

$f_T$6

and the harmonized cone $f_T$7. Conflicting gradients are isometrically rotated, their magnitudes are restored, and the final fair-consensus direction is

$f_T$8

with final update

$f_T$9

The paper proves non-conflict, equal cosine similarity across tasks, convex descent for step size $p \in \mathbb{R}^{L \times d}$0, and a non-convex ergodic rate under $p \in \mathbb{R}^{L \times d}$1-Lipschitz assumptions. Experiments on BGK, linear transport, ES-BGK, and semiconductor Boltzmann–Poisson equations report that HRGrad overcomes APNN failure modes caused by directional contradiction and projection-induced energy clipping.

These two lines of work illustrate different realizations of the same idea. GH/GH++ alter pairwise gradient geometry in a two-loss setting; HRGrad generalizes the principle to many tasks with explicit cone geometry, rotation, and equal-cosine aggregation.

5. Online class-imbalanced learning

"Harmonized Gradient Descent for Class Imbalanced Data Stream Online Learning" introduces an HGD algorithm for one-pass imbalanced streams (Zhou et al., 15 Aug 2025). The setting is a stream $p \in \mathbb{R}^{L \times d}$2 with online updates

$p \in \mathbb{R}^{L \times d}$3

The paper defines the per-step scaled gradient norm

$p \in \mathbb{R}^{L \times d}$4

and the binary gradient imbalance

$p \in \mathbb{R}^{L \times d}$5

The goal is to keep $p \in \mathbb{R}^{L \times d}$6 by scaling updates with a class-dependent factor $p \in \mathbb{R}^{L \times d}$7.

For the binary case, cumulative class sums are

$p \in \mathbb{R}^{L \times d}$8

with imbalance ratio

$p \in \mathbb{R}^{L \times d}$9

The scaling is

$L_{task}(p)=\mathbb{E}_{(x,y)\sim D}[L_T(p;x,y)],$0

The update becomes

$L_{task}(p)=\mathbb{E}_{(x,y)\sim D}[L_T(p;x,y)],$1

A multi-class generalization is also given, and the paper notes that the algorithm itself uses cumulative $L_{task}(p)=\mathbb{E}_{(x,y)\sim D}[L_T(p;x,y)],$2 sums without extra hyperparameters.

Under standard OCO assumptions—bounded diameter $L_{task}(p)=\mathbb{E}_{(x,y)\sim D}[L_T(p;x,y)],$3, convex $L_{task}(p)=\mathbb{E}_{(x,y)\sim D}[L_T(p;x,y)],$4, and bounded $L_{task}(p)=\mathbb{E}_{(x,y)\sim D}[L_T(p;x,y)],$5—the paper gives the regret bound, with $L_{task}(p)=\mathbb{E}_{(x,y)\sim D}[L_T(p;x,y)],$6,

$L_{task}(p)=\mathbb{E}_{(x,y)\sim D}[L_T(p;x,y)],$7

Time complexity is stated as $L_{task}(p)=\mathbb{E}_{(x,y)\sim D}[L_T(p;x,y)],$8 per step, with $L_{task}(p)=\mathbb{E}_{(x,y)\sim D}[L_T(p;x,y)],$9 model storage plus $L_T(p;x,y) = -\sum_c 1[y=c]\log p_T(c|x;p).$0 for class statistics.

The experimental study covers 72 public datasets grouped into Equal, Low, Medium, and High imbalance. Metrics are AUC, G-mean, F1, runtime, and a Gradient Imbalance Indicator

$L_T(p;x,y) = -\sum_c 1[y=c]\log p_T(c|x;p).$1

For static imbalance, reported HGD results include, for the perceptron base learner, AUC 0.964, G-mean 0.971, and F1 0.862; for linear SVM, AUC 0.970, G-mean 0.976, F1 0.871; and for the kernel model, AUC 0.970, G-mean 0.980, F1 0.876. For dynamic imbalance, the kernel base learner reaches AUC 0.983, G-mean 0.988, and F1 0.902. On long-tailed CIFAR-10 streams, HGD reports 0.713/0.763/0.795/0.862 at imbalance ratios 200/100/50/20, versus the ResNet-32 baseline 0.635/0.699/0.747/0.813; on CIFAR-100 the corresponding numbers are 0.363/0.414/0.484/0.531 versus 0.342/0.387/0.432/0.497.

This version of HGD is the most literal use of “harmonized gradient descent”: the update itself is modified so that class-wise cumulative gradient norms are explicitly balanced over time.

6. Related but distinct usages

In the statistics literature, HGD most prominently denotes holonomic gradient descent rather than harmonized gradient descent. "Holonomic Decent Minimization Method for Restricted Maximum Likelihood Estimation" develops HGD and constrained HGD for objectives satisfying a Pfaffian system (Sakurai et al., 2013). If $L_T(p;x,y) = -\sum_c 1[y=c]\log p_T(c|x;p).$2 is holonomic and $L_T(p;x,y) = -\sum_c 1[y=c]\log p_T(c|x;p).$3 collects $L_T(p;x,y) = -\sum_c 1[y=c]\log p_T(c|x;p).$4 and its derivatives in a basis of $L_T(p;x,y) = -\sum_c 1[y=c]\log p_T(c|x;p).$5, then

$L_T(p;x,y) = -\sum_c 1[y=c]\log p_T(c|x;p).$6

Gradients and Hessians are obtained from the Pfaffian matrices, and the unconstrained Newton-type update is

$L_T(p;x,y) = -\sum_c 1[y=c]\log p_T(c|x;p).$7

Constraints are enforced with the exact penalty

$L_T(p;x,y) = -\sum_c 1[y=c]\log p_T(c|x;p).$8

together with an Armijo line search. In the von Mises example, the paper reports convergence to (3.63, 3.67) for unconstrained HGD, (0.13, -0.004) for the constraint $L_T(p;x,y) = -\sum_c 1[y=c]\log p_T(c|x;p).$9, and (2.08, 1.94) for the constraint $S_\theta$0. Reported average runtimes over 500 trials are 0.03698 s for HGD, 0.09834 s for CHGD, and 0.12598 s for direct Newton.

A different but closely related line is the Gradient Harmonizing Mechanism for dense detection. "Gradient Harmonized Single-stage Detector" does not use the term HGD, but it explicitly reformulates class imbalance and difficulty imbalance in terms of gradient density (Li et al., 2018). For binary cross-entropy, the per-sample gradient magnitude is

$S_\theta$1

and the density-based weight is

$S_\theta$2

This yields the harmonized classification loss

$S_\theta$3

For regression, the paper replaces smooth-$S_\theta$4 by ASL1,

$S_\theta$5

with gradient magnitude

$S_\theta$6

On COCO, the paper reports 41.6 mAP on test-dev with GHM-C and GHM-R, exceeding Focal Loss + $S_\theta$7 by 0.8. It also reports that the exact density computation is too slow (13.675 s per iteration), while the unit-region approximation achieves 35.8 AP at 0.824 s per iteration, compared with 0.566 s for CE.

These methods are related by theme rather than by formal identity. Holonomic HGD is about evaluating difficult objectives through Pfaffian ODEs; GHM is about redistributing gradient flow over samples; neither should be conflated with the class-imbalance HGD algorithm or with multi-objective gradient harmonization.

7. Limitations, misconceptions, and research directions

The cited works also delimit where harmonization helps and where it does not. In GDFO, experiments focus on seven few-shot classification tasks, while sequence labeling and generation are not evaluated; the method adds student distillation and prompt-generator training on top of DFO, increasing computational demand even though it remains API-efficient relative to white-box tuning (Han et al., 2023). In GH and GH++, the framework is derived for the two-task conflict between classification and domain alignment, and GH++ adds a balancing parameter $S_\theta$8, though the paper states that $S_\theta$9 often works well (Huang et al., 2024). In online HGD for data streams, the paper explicitly notes an F1 trade-off, possible precision loss under noisy minority classes, and the fact that HGD adapts through $p_r$0 and $p_r$1 but does not itself detect concept drift (Zhou et al., 15 Aug 2025). HRGrad adds nontrivial overhead from cone construction, angle optimization, and pseudo-inverse computation, and its equal-cosine proof relies on a column-rank condition for the rotated gradient matrix $p_r$2 (Liang, 27 Apr 2026). GHM, while lightweight in its unit-region form, still depends on mini-batch gradient-density estimation and is most natural in dense prediction settings with severe easy-negative domination (Li et al., 2018).

A further misconception is terminological. The evidence here does not support a single, universally agreed meaning of “Harmonized Gradient Descent.” Instead, the phrase names a recurring optimization strategy: alter the effective descent so that learning signal is distributed more evenly across objectives, classes, samples, or regimes. In that sense, HGD is best understood as a design principle with multiple instantiations rather than a single algorithmic template.