---
title: Harmonized Gradient Descent Overview
url: https://www.emergentmind.com/topics/harmonized-gradient-descent-hgd
type: topic
---

# Harmonized Gradient Descent Overview

Searching arXiv for the cited papers and closely related uses of “HGD” / “gradient harmonization” to ground the article.
Harmonized Gradient Descent (HGD) is not a single canonical algorithm but a family of optimization ideas centered on reshaping, coordinating, or reweighting gradient information so that learning remains effective under structural obstacles such as black-box access, gradient conflict, class imbalance, or analytically intractable objectives. In the recent literature, the name has been used explicitly for an online class-imbalance method that equalizes class-wise gradient norms [2508.11353], while closely related work uses the same acronym for holonomic gradient descent in statistical estimation [1309.1246]. Other papers embody the same harmonization principle without using the exact term, including GDFO for black-box prompt tuning [2305.10013], GH and GH++ for unsupervised domain adaptation [2408.00288], the Gradient Harmonizing Mechanism for dense detection [1811.05181], and HRGrad for multiscale kinetic learning [2604.24745].

## 1. Terminology and scope

The literature represented here uses “HGD” in at least two distinct senses. In statistics, HGD denotes **holonomic gradient descent**, a method that combines Pfaffian differential equations with iterative optimization for holonomic objectives [1309.1246]. In online learning, HGD denotes **harmonized gradient descent**, an algorithm for imbalanced data streams that equalizes cumulative gradient-norm contributions across classes [2508.11353]. A third usage is implicit rather than explicit: the GDFO paper does not use the term HGD, but it defines a harmonization principle by combining gradient descent with derivative-free optimization for black-box prompt tuning [2305.10013]. Related work uses adjacent labels such as **Gradient Harmonization** (GH, GH++) in UDA [2408.00288], **Gradient Harmonizing Mechanism** (GHM) in detection [1811.05181], and **harmonized rotational gradient** (HRGrad) in multiscale kinetic regimes [2604.24745].

| Usage | Domain | Core mechanism |
|---|---|---|
| Holonomic gradient descent | Constrained MLE / REML | Pfaffian ODEs plus Newton-type optimization |
| Harmonized gradient descent | Imbalanced online learning | Class-wise gradient norm equalization |
| Gradient harmonization / related methods | UDA, black-box tuning, detection, APNNs | Gradient-angle control, density weighting, or mixed gradient sources |

A common source of confusion is that these methods are mathematically different even when they share the acronym. The only safe generalization is that each method tries to prevent a harmful concentration of optimization signal—whether that concentration occurs in class frequency, task conflict, sample difficulty, or inaccessible black-box objectives.

## 2. General principle of harmonization

Across the modern uses of the term, harmonization refers to modifying the effective descent direction so that optimization is not dominated by a single source of signal. In GDFO, the final prompt is a linear mixture of a student-guided gradient path and a teacher-grounded derivative-free path,
$$
p = \alpha G_\mu(x) + (1-\alpha)(p_0 + Az),
$$
with $\alpha \in [0,1]$ controlling the balance [2305.10013]. In GH, conflicting task gradients are projected so that an obtuse angle becomes acute; in GH++, the angle is adjusted from obtuse to vertical, with a dynamic weighted-loss implementation derived through an integral operator [2408.00288]. In the online HGD algorithm for class-imbalanced streams, each update is scaled by a class-dependent factor $\alpha_t$ so that cumulative gradient norms remain comparable across classes [2508.11353]. In GHM, per-sample losses are weighted inversely to the density of their gradient magnitudes, reducing domination by easy negatives and persistent outliers [1811.05181]. In HRGrad, conflicting task gradients are rotated into a harmonized cone and aggregated so that the final update has a non-negative dot product with every rotated task gradient [2604.24745].

Taken together, these works suggest a shared optimization pattern: the raw gradient is treated as an object whose **geometry**, **magnitude**, or **source composition** can be deliberately altered before the parameter update. The objective is not merely faster convergence, but a better allocation of descent across competing signals.

## 3. Black-box prompt tuning: harmonizing gradient descent and derivative-free optimization

In "When Gradient Descent Meets Derivative-Free Optimization: A Match Made in Black-Box Scenario" [2305.10013], the setting is black-box prompt tuning for PLMs. A teacher model $f_T$ is available only through inference APIs, so its parameters and gradients are unavailable. Continuous prompts $p \in \mathbb{R}^{L \times d}$ are prepended to the input, and the training objective is to minimize
$$
L_{task}(p)=\mathbb{E}_{(x,y)\sim D}[L_T(p;x,y)],
$$
where
$$
L_T(p;x,y) = -\sum_c 1[y=c]\log p_T(c|x;p).
$$

Because direct gradient descent on the teacher is impossible, the method introduces a distilled student $S_\theta$ trained on random discrete prompts $p_r$ with a loss
$$
L(\theta;\tau,\lambda) = (1-\lambda)L_{CE}(\theta) + \lambda L_{KL}(\theta;\tau).
$$
After distillation, $\theta$ is frozen, and a prompt generator $G_\mu$ produces $p_{GD}=G_\mu(x)$. This enables gradient updates through the student,
$$
\mu_{t+1} = \mu_t - \eta_t \nabla_\mu L_S(G_\mu(x);x,y).
$$

The derivative-free component is CMA-ES, operating in a low-dimensional space $z \in \mathbb{R}^{d'}$ with random projection $A \in \mathbb{R}^{(L\cdot d)\times d'}$, so that
$$
p_{DFO} = p_0 + Az.
$$
The harmonization rule is
$$
p = \alpha p_{GD} + (1-\alpha)p_{DFO}
  = \alpha G_\mu(x) + (1-\alpha)(p_0+Az).
$$
Conceptually, this yields the combined prompt-space update
$$
p_{t+1} = p_t - \alpha \eta_t \nabla_p L_S(p_t) + (1-\alpha)A\Delta_t^{DFO}.
$$

The paper reports that $\alpha = 0.5$ gives the best average across three datasets, and that extreme values degrade performance. In the main 16-shot results with RoBERTa\_LARGE as teacher, GDFO achieves an average of **81.85\%**, compared with **78.01\%** for LoRA and **79.01\%** for BBTv2. Reported task scores are SST-2 **92.41\%**, Yelp Polarity **93.17\%**, AG’s News **87.19\%**, DBPedia **96.92\%**, MRPC **80.13** F1, SNLI **62.53\%**, and RTE **60.57\%**. Ablations show that removing KD lowers SNLI from **62.53\%** to **58.51\%**, while removing DFO causes an average decrease of **6.5\%**. The method uses prompt length $L=50$, embedding dimension $d=1024$ for RoBERTa\_LARGE, low-dimensional $d'=500$, population size $\lambda=20$, and a teacher API budget of **8,000 calls**.

This formulation makes harmonization explicit: gradient descent supplies a low-variance local signal through the student, whereas CMA-ES supplies teacher-grounded corrections under true black-box constraints. A plausible implication is that the method is most effective when the student is locally faithful but globally imperfect.

## 4. Gradient-conflict harmonization in multi-task and multiscale learning

In unsupervised domain adaptation, "Gradient Harmonization in Unsupervised Domain Adaptation" studies two gradients on shared parameters $\theta$: the classification gradient $g_c=\nabla_\theta L_c(\theta)$ and the domain-alignment gradient $g_d=\nabla_\theta L_d(\theta)$ [2408.00288]. Conflict is defined by $g_c^\top g_d<0$. GH resolves this by symmetric projection:
$$
g_1' = g_1 - \delta(g_1^\top g_2<0)\frac{g_1^\top g_2}{\|g_2\|^2}g_2,
$$
$$
g_2' = g_2 - \delta(g_1^\top g_2<0)\frac{g_2^\top g_1}{\|g_1\|^2}g_1.
$$
The aggregated harmonized gradient is $g'=g_1'+g_2'$. GH++ instead aims for a vertical angle and is implemented through dynamic weights $\tau_1,\tau_2$ in a reweighted loss
$$
\tilde{L}(\theta)=\tau_1 L_1(\theta)+\tau_2 L_2(\theta).
$$
The paper states that GH changes an obtuse angle to an acute angle, while GH++ changes it to a vertical angle and halves the sum of deviation angles relative to GH.

Empirically, GH and GH++ are reported on Office-31, Office-Home, Digits, VisDA-2017, and DomainNet. On Office-31, the reported average gains for CDAN are **+1.5\% / +2.2\%** for GH / GH++; on Office-Home they are **+2.6\% / +3.2\%**; on DomainNet the reported average gains are **+3.2\% / +3.7\%** for CDAN and **+7.1\% / +7.3\%** for MCD. The paper also reports that conflict histograms contain a substantial fraction of obtuse angles, and that GH/GH++ shift the inner-product distribution to positive values.

A more geometric and physics-specific extension appears in "Conflict-Aware Harmonized Rotational Gradient for Multiscale Kinetic Regimes" [2604.24745]. There, tasks correspond to different Knudsen numbers $\varepsilon$, with gradients $g_i=\nabla_\theta L_i(\theta)$. HRGrad constructs the primal cone
$$
\mathbb{K}=\{G\lambda \mid \lambda\in\mathbb{R}_+^m\}, \quad G=[g_1,\dots,g_m],
$$
the dual cone
$$
\mathbb{K}^*=\{y\in\mathbb{R}^D \mid G^\top y \ge \mathbf{0}_m\},
$$
and the harmonized cone $H=\mathbb{K}\cap\mathbb{K}^*$. Conflicting gradients are isometrically rotated, their magnitudes are restored, and the final fair-consensus direction is
$$
g_u=\mathcal{U}[(M^\dagger)^\top \mathbf{1}_m],
$$
with final update
$$
g_{\mathrm{HRGrad}}
=
\left(\sum_{i=1}^m (g_i^{\mathrm{rot}})^\top g_u\right) g_u.
$$
The paper proves non-conflict, equal cosine similarity across tasks, convex descent for step size $\gamma \le 2/L$, and a non-convex ergodic rate under $L$-Lipschitz assumptions. Experiments on BGK, linear transport, ES-BGK, and semiconductor Boltzmann–Poisson equations report that HRGrad overcomes APNN failure modes caused by directional contradiction and projection-induced energy clipping.

These two lines of work illustrate different realizations of the same idea. GH/GH++ alter pairwise gradient geometry in a two-loss setting; HRGrad generalizes the principle to many tasks with explicit cone geometry, rotation, and equal-cosine aggregation.

## 5. Online class-imbalanced learning

"Harmonized Gradient Descent for Class Imbalanced Data Stream Online Learning" introduces an HGD algorithm for one-pass imbalanced streams [2508.11353]. The setting is a stream $\{(x_t,y_t)\}_{t=1}^T$ with online updates
$$
\theta_{t+1} = \theta_t - \eta_t \nabla_\theta L(x_t,y_t;\theta_t).
$$
The paper defines the per-step scaled gradient norm
$$
G_t = \|\eta_t \nabla_\theta L(x_t,y_t;\theta_t)\|_2^2,
$$
and the binary gradient imbalance
$$
GI_t =
\frac{\sum_{i=1}^{t}\mathbb{I}\{y_i=-1\}G_i}
{\sum_{i=1}^{t}\mathbb{I}\{y_i=+1\}G_i}.
$$
The goal is to keep $GI_t \approx 1$ by scaling updates with a class-dependent factor $\alpha_t$.

For the binary case, cumulative class sums are
$$
S_-(t)=\sum_{i=1}^t \mathbb{I}\{y_i=-1\}\alpha_i G_i,\qquad
S_+(t)=\sum_{i=1}^t \mathbb{I}\{y_i=+1\}\alpha_i G_i,
$$
with imbalance ratio
$$
\rho_t=\frac{N_-(t)}{N_+(t)}.
$$
The scaling is
$$
\alpha_t =
\begin{cases}
\frac{2\rho_t S_-(t-1)}{S_-(t-1)+S_+(t-1)}, & y_t=+1, \\[6pt]
\frac{2S_+(t-1)}{S_-(t-1)+S_+(t-1)}, & y_t=-1.
\end{cases}
$$
The update becomes
$$
\theta_{t+1}=\theta_t-\eta_t \alpha_t \nabla_\theta L(x_t,y_t;\theta_t).
$$
A multi-class generalization is also given, and the paper notes that the algorithm itself uses cumulative $S_c(t)$ sums without extra hyperparameters.

Under standard OCO assumptions—bounded diameter $D$, convex $L$, and bounded $\alpha_t \in [1/\rho,2\rho]$—the paper gives the regret bound, with $\eta_t=1/\sqrt{t}$,
$$
R_T \le \rho D^2 \sqrt{T} + \rho L^2 \sqrt{T}.
$$
Time complexity is stated as $O(d)$ per step, with $O(d)$ model storage plus $O(C)$ for class statistics.

The experimental study covers **72 public datasets** grouped into Equal, Low, Medium, and High imbalance. Metrics are **AUC**, **G-mean**, **F1**, runtime, and a Gradient Imbalance Indicator
$$
GII=\sum_{t=1}^T (GI_t-1)^2.
$$
For static imbalance, reported HGD results include, for the perceptron base learner, **AUC 0.964**, **G-mean 0.971**, and **F1 0.862**; for linear SVM, **AUC 0.970**, **G-mean 0.976**, **F1 0.871**; and for the kernel model, **AUC 0.970**, **G-mean 0.980**, **F1 0.876**. For dynamic imbalance, the kernel base learner reaches **AUC 0.983**, **G-mean 0.988**, and **F1 0.902**. On long-tailed CIFAR-10 streams, HGD reports **0.713/0.763/0.795/0.862** at imbalance ratios **200/100/50/20**, versus the ResNet-32 baseline **0.635/0.699/0.747/0.813**; on CIFAR-100 the corresponding numbers are **0.363/0.414/0.484/0.531** versus **0.342/0.387/0.432/0.497**.

This version of HGD is the most literal use of “harmonized gradient descent”: the update itself is modified so that class-wise cumulative gradient norms are explicitly balanced over time.

## 6. Related but distinct usages

In the statistics literature, HGD most prominently denotes **holonomic gradient descent** rather than harmonized gradient descent. "Holonomic Decent Minimization Method for Restricted Maximum Likelihood Estimation" develops HGD and constrained HGD for objectives satisfying a Pfaffian system [1309.1246]. If $f$ is holonomic and $F(x)$ collects $f$ and its derivatives in a basis of $R/I$, then
$$
\frac{\partial F}{\partial x_i}=P_i(x)F.
$$
Gradients and Hessians are obtained from the Pfaffian matrices, and the unconstrained Newton-type update is
$$
x_{k+1}=x_k-H^{-1}(x_k)\nabla f(x_k).
$$
Constraints are enforced with the exact penalty
$$
P(\theta;\rho)=
f(\theta)+\rho\left\{\sum_{i=1}^m |g_i(\theta)|
+\sum_{j=1}^l \max(0,h_j(\theta))\right\},
$$
together with an Armijo line search. In the von Mises example, the paper reports convergence to **(3.63, 3.67)** for unconstrained HGD, **(0.13, -0.004)** for the constraint $\theta_1 \le \theta_2$, and **(2.08, 1.94)** for the constraint $\theta_1^2+\theta_2^2 \le 9$. Reported average runtimes over 500 trials are **0.03698 s** for HGD, **0.09834 s** for CHGD, and **0.12598 s** for direct Newton.

A different but closely related line is the Gradient Harmonizing Mechanism for dense detection. "Gradient Harmonized Single-stage Detector" does not use the term HGD, but it explicitly reformulates class imbalance and difficulty imbalance in terms of gradient density [1811.05181]. For binary cross-entropy, the per-sample gradient magnitude is
$$
g=|p-y|,
$$
and the density-based weight is
$$
\beta_i=\frac{N}{GD(g_i)}.
$$
This yields the harmonized classification loss
$$
L_{GHM-C}=\sum_{i=1}^N \frac{L_{CE}(p_i,y_i)}{GD(g_i)}.
$$
For regression, the paper replaces smooth-$L_1$ by ASL1,
$$
ASL_1(d)=\sqrt{d^2+\mu^2}-\mu,
$$
with gradient magnitude
$$
g_r=\frac{|d|}{\sqrt{d^2+\mu^2}}.
$$
On COCO, the paper reports **41.6 mAP** on test-dev with GHM-C and GHM-R, exceeding **Focal Loss + $SL_1$** by **0.8**. It also reports that the exact density computation is too slow (**13.675 s** per iteration), while the unit-region approximation achieves **35.8 AP** at **0.824 s** per iteration, compared with **0.566 s** for CE.

These methods are related by theme rather than by formal identity. Holonomic HGD is about evaluating difficult objectives through Pfaffian ODEs; GHM is about redistributing gradient flow over samples; neither should be conflated with the class-imbalance HGD algorithm or with multi-objective gradient harmonization.

## 7. Limitations, misconceptions, and research directions

The cited works also delimit where harmonization helps and where it does not. In GDFO, experiments focus on seven few-shot classification tasks, while sequence labeling and generation are not evaluated; the method adds student distillation and prompt-generator training on top of DFO, increasing computational demand even though it remains API-efficient relative to white-box tuning [2305.10013]. In GH and GH++, the framework is derived for the two-task conflict between classification and domain alignment, and GH++ adds a balancing parameter $\lambda$, though the paper states that $\lambda \approx 0.5$ often works well [2408.00288]. In online HGD for data streams, the paper explicitly notes an F1 trade-off, possible precision loss under noisy minority classes, and the fact that HGD adapts through $\rho_t$ and $S_c(t)$ but does not itself detect concept drift [2508.11353]. HRGrad adds nontrivial overhead from cone construction, angle optimization, and pseudo-inverse computation, and its equal-cosine proof relies on a column-rank condition for the rotated gradient matrix $M$ [2604.24745]. GHM, while lightweight in its unit-region form, still depends on mini-batch gradient-density estimation and is most natural in dense prediction settings with severe easy-negative domination [1811.05181].

A further misconception is terminological. The evidence here does not support a single, universally agreed meaning of “Harmonized Gradient Descent.” Instead, the phrase names a recurring optimization strategy: alter the effective descent so that learning signal is distributed more evenly across objectives, classes, samples, or regimes. In that sense, HGD is best understood as a design principle with multiple instantiations rather than a single algorithmic template.

Source: https://www.emergentmind.com/topics/harmonized-gradient-descent-hgd