---
title: Gradient-based Weight Optimization
url: https://www.emergentmind.com/topics/gradient-based-weight-optimization
type: topic
---

# Gradient-based Weight Optimization

Gradient-based weight optimization denotes a family of methods in which gradient information is used to optimize not only model parameters, but also auxiliary weights that govern learning dynamics: sample weights, task weights, class weights, composite-loss coefficients, structural sparsity budgets, ranking weights, and even weights on graph edges. In the cited literature, this family includes multi-objective reformulations of empirical risk, gradient-space transformations such as centralization, parameter-free or predictive optimizers, continuous relaxations of discrete weight allocation, and bilevel procedures that adapt loss weights online [1506.01113] [2004.01461] [2306.09222].

## 1. Foundational formulations

A recurring starting point is the observation that standard empirical risk minimization already hides a weighting decision. In the classical formulation, one minimizes either the mean loss, $\frac{1}{N}\sum_i l_i(\theta)$, or a fixed weighted mean, $\sum_i \alpha_i l_i(\theta)$, with $\alpha_i \ge 0$. The multi-objective view treats each per-sample loss $l_i(\theta)$ as an objective, and then asks whether a fixed linear aggregation is an adequate scalarization. The hypervolume formulation replaces the fixed linear combination by
$$
H(\theta;r)=\prod_{i=1}^N (r-l_i(\theta)),
\qquad
\log H(\theta;r)=\sum_{i=1}^N \log(r-l_i(\theta)),
$$
with $r>\max_i l_i(\theta)$. Its gradient is
$$
\nabla_\theta \log H(\theta;r)
=
-\sum_{i=1}^N \frac{\nabla_\theta l_i(\theta)}{r-l_i(\theta)},
$$
so the induced sample weights are $w_i(\theta;r)=1/(r-l_i(\theta))$. This makes the update a self-adjusting weighted sum of per-sample gradients, larger for higher-loss samples, without requiring $\alpha_i$ to be chosen a priori. The paper explicitly characterizes this as an “inner boosting” mechanism within a single model [1506.01113].

An earlier line of work addressed weighted examples in online learning. For a weighted example $(x,y,h)$, the naive update multiplies the gradient by $h$, but this does not preserve the semantics that an example of importance $h$ should behave like the same example presented $h$ times. “Online Importance Weight Aware Updates” formalizes an invariance property: updating twice with importance weight $h$ should be equivalent to updating once with importance weight $2h$. In margin coordinates $z=w\cdot x$, this is obtained by the flow
$$
\frac{dz}{dh}=-\eta \|x\|^2 \nabla_z \ell(z;y),
$$
which yields closed-form invariant updates for squared, logistic, hinge, squared hinge, exponential, and quantile losses [1011.1576].

These two formulations already illustrate a central distinction. Some methods optimize the model weights $\theta$ under dynamically induced scalar weights on data points; others optimize the weighting rule itself so that gradient steps obey a structural property, such as Pareto efficiency or importance-weight invariance.

## 2. Dynamic weighting of samples, tasks, classes, and losses

A major branch of the literature learns weights online from gradients or losses rather than fixing them by hand. In KL-divergence distributionally robust optimization, the empirical objective
$$
\frac{1}{\gamma}\log \mathbb{E}_P[\exp(\gamma \ell(\theta;z))]
$$
induces Gibbs reweighting of per-sample gradients. Re-weighted Gradient Descent instantiates this idea in mini-batch form through
$$
w_i=\exp(\gamma \min(\ell_i,\tau)),
\qquad
g_t=\frac{1}{B}\sum_i w_i \nabla_\theta \ell_i,
$$
with $\gamma=1/(\tau+1)$ and clipping inside the exponential for stability. This produces a stochastic reweighted gradient that is compatible with SGD and Adam, and the reported empirical gains include +0.7% on DomainBed, +1.44% on tabular classification, +1.94% on GLUE with BERT, and +1.01% on ImageNet-1K with ViT [2306.09222].

In composite-loss pretraining, the optimized weights are coefficients on multiple loss heads. GraP formulates
$$
L_{\text{pre}}(\theta,w)=\sum_{i=1}^m w_i L_i(\theta),
$$
and updates $w$ so that the composite pretraining gradient aligns with a downstream gradient. With
$$
g_{\text{pre}}(w)=\sum_i w_i g_i,
\qquad
s(w)=\frac{g_{\text{pre}}(w)^\top g_{\text{down}}}{\|g_{\text{pre}}(w)\|\,\|g_{\text{down}}\|},
$$
the weight update is
$$
w \leftarrow \Pi_W\!\left(w+\eta_w \nabla_w s(w)\right).
$$
The method computes per-loss gradients only up to the shared embedding and backpropagates a single composite gradient through the backbone, reducing the overhead of hyperparameter tuning to approximately 30% above a single training run [2605.07756].

Meta-learning supplies a task-level analogue. “Gradient Agreement as an Optimization Objective for Meta-Learning” defines inner-loop displacements $g_i=\theta-\theta_i'$ for tasks in a meta-batch and weights each task by its agreement with the batch-average update:
$$
\bar g=\frac{1}{N}\sum_{j=1}^N g_j,
\qquad
w_i=\frac{g_i^\top \bar g}{\sum_k |g_k^\top \bar g|}.
$$
Negative weights are allowed, so anti-aligned tasks can be downweighted or counteracted in the outer update [1810.08178].

In dense unsupervised domain adaptation, weighting is applied at class level rather than sample or task level. Gradient-based class weighting computes per-class gradient-proxy magnitudes $g_c=\|\nabla_z \overline{l_c}\|^2$ with respect to last-layer logits and then solves
$$
\min_v \ \lambda \|v\|^2-g^\top v
\quad
\text{subject to } v_c\ge 0,\ \sum_{c=1}^C v_c=C.
$$
This yields dynamic class weights that are recomputed per iteration and were reported to improve mIoU across adversarial, self-training, and entropy-minimization UDA settings, while consistently increasing the recall of low represented classes [2407.01327].

## 3. Gradient-space modifications of parameter updates

Another major interpretation of gradient-based weight optimization keeps the objective fixed and instead transforms the gradient before it updates the model weights. Gradient Centralization is the canonical example. For a vector gradient $g\in\mathbb{R}^d$,
$$
g_c=g-\frac{1}{d}(\mathbf{1}^\top g)\mathbf{1}
=
\left(I-\frac{1}{d}\mathbf{1}\mathbf{1}^\top\right)g.
$$
This is a projection onto the subspace orthogonal to the all-ones vector. For fully connected layers in PyTorch, centralization is along `dim=1`; for convolutional layers of shape $[out\_channels,in\_channels,k_h,k_w]$, it subtracts the mean across $[in\_channels,k_h,k_w]$ per filter. The method is described as regularizing both the weight space and output feature space, improving the Lipschitzness of the loss function and its gradient, and requiring only one line of code [2004.01461].

A related training-only normalization line combines Weight Centralization and Gradient Centralization. Weight centralization subtracts the mean from each output filter tensor,
$$
W_{j,in,y,x}=W_{j,in,y,x}-\overline{W}_{j,in,y,x},
$$
while gradient centralization applies
$$
\Delta W_{j,in,y,x}=\Delta W_{j,in,y,x}-\overline{\Delta W}_{j,in,y,x}.
$$
These operations are inserted before weight use in the forward path and immediately after gradient computation, respectively. The paper reports that WC+GC is consistently stronger than either component alone across CIFAR-10, CIFAR-100, and VOC2012, and emphasizes that these methods only have to be applied during training and therefore do not influence the running time during use [2010.00866].

Parameter-free weighting of the step size appears in DoWG. It maintains
$$
\bar r_t=\max(\|x_t-x_0\|,\bar r_{t-1}),
\qquad
v_t=v_{t-1}+\bar r_t^2\|\nabla f(x_t)\|^2,
\qquad
\eta_t=\frac{\bar r_t^2}{\sqrt{v_t}},
$$
followed by
$$
x_{t+1}=x_t-\eta_t \nabla f(x_t).
$$
This distance over weighted gradients mechanism is proved to match the convergence rate of optimally tuned gradient descent in convex optimization up to a logarithmic factor, while automatically adapting to both smooth and nonsmooth problems [2305.16284].

Predictive modification of the update point is the defining idea of XGrad. If a base optimizer produces a direction $\Delta w_{t+1}$, XGrad predicts
$$
\tilde w_t \approx w_t-\eta s\,\Delta w_{t+1},
$$
uses $\tilde w_t$ for both the forward pass and backward propagation, and then commits the actual update from $w_t$ with gradients evaluated at the predicted future weights. The paper presents this as an approximation of extragradient behavior that adds only a modest time and memory overhead [2305.18240].

## 4. Smoothing, prediction, and alternative parameterizations of weights

Some methods alter the geometry of the optimization problem itself. Anisotropic Gaussian Smoothing replaces the local objective by
$$
f_\Sigma(x)=\mathbb{E}_{z\sim \mathcal{N}(0,\Sigma)}[f(x+z)],
$$
with non-local gradient
$$
\nabla f_\Sigma(x)=\Sigma^{-1}\mathbb{E}[z f(x+z)].
$$
Monte Carlo implementations use either one-point or symmetric two-point estimators, and AGS-GD, AGS-SGD, and AGS-Adam are obtained by substituting these smoothed gradients into GD, SGD, or Adam. The cited analysis covers convex and non-convex, $L$-smooth functions and characterizes stochastic convergence to a noisy ball whose size is determined by the smoothing parameters [2411.11747].

A distinct geometric analogy appears in Gravity. Starting from a kinematic model on the loss surface, the optimizer arrives at the saturating step
$$
\Delta W=-\eta \frac{g}{1+(g/m)^2},
$$
where $g=\nabla_W L$ and $m_t=\max(|g_t|)$ per parameter tensor. A velocity-like buffer is then updated by
$$
v_t=\tilde\beta_t v_{t-1}+(1-\tilde\beta_t)\zeta_t,
\qquad
\tilde\beta_t=\frac{\beta^t+1}{t+2},
$$
with $\zeta_t=g_t/(1+(g_t/m_t)^2)$ and $\theta_{t+1}=\theta_t-\eta v_t$. The reported experiments on VGG16 and VGG19 describe Gravity as having more stable performance than Adam and RMSProp and giving greater values of validation accuracy for datasets with more output classes like CIFAR-100 (Fine) [2101.09192].

Weight optimization can also be internalized into the parameterization of each neuron. The dual-weight method writes
$$
W=W_1-W_2,
\qquad
z=(W_1-W_2)^\top x+b,
$$
and updates only one component depending on the sign of the backpropagated scalar error $\delta=\partial L/\partial z$. If $\delta<0$, then
$$
W_1 \leftarrow (1-\alpha)W_1+\alpha x,
$$
and if $\delta>0$, then
$$
W_2 \leftarrow (1-\alpha)W_2+\alpha x,
$$
with $\alpha=\eta |\delta|$. The paper interprets $W_1$ and $W_2$ as separate target and non-target prototypes, while preserving the standard $WX+\text{bias}$ inference form through the effective weight $W=W_1-W_2$ [2503.11965].

## 5. Structural, probabilistic, and combinatorial targets

The “weight” being optimized is not always a dense parameter tensor. In dynamic sparse training, Global Gradient-based Redistribution reallocates the global budget of nonzeros across layers. After local magnitude pruning, it examines gradient magnitudes on zero positions, forms the global top-$k$ set across all layers, and gives each layer as many new nonzeros as the number of its zero-position gradients contained in that global top-$k$. Growth within a layer is then split 50:50 between gradient-based and random regrowth. The method was proposed precisely because layers with the same shape need different effective capacity, especially at sparsity of 90% and more [2210.14012].

In probabilistic model-based evolutionary optimization, WA-ASNG adapts rank-based recombination weights rather than network parameters. It defines a signal
$$
S(w)=\|s^{t+1}\|^2-\gamma^{t+1},
$$
derived from accumulations of the natural gradient, and updates ranking weights by gradient ascent to maximize this signal. The intended role is explicit: learning-rate adaptation in ASNG satisfies a sufficient condition for monotonic improvement of the expected objective value, while weight adaptation is intended to maximize this improvement [2606.19861].

Combinatorial optimization supplies an additional generalization. In graph coloring, each candidate solution is represented by a weight tensor $W\in\mathbb{R}^{D\times n\times k}$, where $w_{d,i,c}$ is the propensity of vertex $i$ to take color $c$ in population member $d$. Discrete colorings are extracted by $\operatorname{argmax}$ along the color axis, while gradients are computed through a softmax surrogate. The global loss combines edge conflicts, optional equity constraints, and cross-individual penalization and bonus terms. This turns the search for a coloring into a continuous weight tensor optimization problem computed in parallel on graphics processing units [1909.02261].

Weighted graphs also become direct optimization targets in adversarial settings. IGA-LWP formulates the attack on link weight prediction as
$$
\max_{W'} L(W';\mathcal{T})
\quad
\text{subject to } \|W'-W\|_0\le B,
$$
with optional box constraints in normalized space. Using a self-attention–enhanced graph autoencoder surrogate, it computes $\nabla_W L$, selects the top-$n$ existing links by absolute gradient, and updates
$$
W^{(t+1)}=\Pi_{\mathcal C}\!\left(W^{(t)}+\alpha u^{(t)}\right),
$$
where $u^{(t)}$ is a sparse sign-guided ascent direction. This formulation shows that gradient-based weight optimization extends naturally to edge-weight manipulation in weighted networks [2601.04259].

## 6. Empirical behavior, limitations, and conceptual boundaries

The cited work uses the phrase across several distinct problem formulations rather than a single optimizer. One recurring empirical pattern is that dynamic weighting shifts emphasis toward hard, noisy, minority, or high-loss components. Hypervolume maximization on a denoising autoencoder over MNIST consistently outperformed direct mean-loss training across train, validation, and test, with larger gains at higher salt-and-pepper corruption probabilities [1506.01113]. KL-DRO reweighting, class-level gradient weighting, and composite-loss alignment all rely on the same broad idea: the gradient should reflect current learning needs rather than a static averaging rule [2306.09222] [2407.01327] [2605.07756].

A second pattern is landscape smoothing or stabilization. Gradient Centralization is explicitly analyzed as improving the Lipschitzness of the loss function and its gradient, and reports ImageNet top-1 error reductions such as ResNet50+BN 23.71 $\rightarrow$ 23.21 and ResNet101+BN 22.37 $\rightarrow$ 21.82 [2004.01461]. AGS, Gravity, and XGrad pursue related goals through non-local gradients, saturating update laws, or gradients evaluated at predicted future weights rather than current ones [2411.11747] [2101.09192] [2305.18240].

The main limitations are method-specific and often arise exactly where the weighting becomes most aggressive. Hypervolume weighting is sensitive to the reference point $r$: too small a slack makes $w_i=1/(r-l_i)$ nearly singular, while too large an $r$ makes weights nearly uniform [1506.01113]. Exponential reweighting in KL-DRO can overemphasize outliers, which is why RGD clips the loss inside the exponential and notes that removing clipping degrades ImageNet-1K performance by about 1% for the same ViT model [2306.09222]. Gradient Centralization should not be applied to 1D parameters such as biases or normalization affine parameters, since centralizing those gradients can collapse learning signals [2004.01461]. GraP reduces the cost of online loss-weight learning, but still operates with an overhead of roughly 30–44% in epoch time relative to a single training run [2605.07756].

A common misconception is to treat all of these methods as variants of loss reweighting. The literature suggests a broader taxonomy. Some methods optimize weights on examples or tasks; some optimize the transformation applied to the gradient before it reaches model weights; some optimize structural distributions of nonzeros or ranking weights in probabilistic search; and some optimize alternative parameterizations in which the effective weight is itself a derived object. What unifies them is not a single update rule, but the use of gradient information to decide how weight should be represented, scaled, redistributed, or predicted at each step.

Source: https://www.emergentmind.com/topics/gradient-based-weight-optimization