---
title: Multiplicative Backprop Updates
url: https://www.emergentmind.com/topics/multiplicative-backpropagation-style-updates
type: topic
---

# Multiplicative Backprop Updates

Multiplicative backpropagation-style updates are optimization algorithms for deep learning that replace or complement the canonical additive gradient-descent rule with elementwise updates that scale parameter changes proportionally to their current magnitude. Such methods—rooted in the Winnow and exponentiated-gradient literatures—have seen a rigorous resurgence in modern deep networks, as they naturally normalize updates, can ameliorate vanishing/shattering gradients, yield robustness to parameter rescaling, and facilitate new regimes of training acceleration and stability. Multiplicative updates can be implemented in pure form, as a hybrid with standard additive updates, or via more general frameworks such as hypentropy-based mirror descent and Expectation Reflection, all of which have been integrated into backpropagation pipelines and empirically validated at scale [2307.07189, 1902.01903, 2503.10144, 1604.03736].

## 1. Mathematical Formulation of Multiplicative Updates

The canonical gradient descent rule is additive: at iteration $t$, with parameter vector $\theta_{t-1}$, gradient $g_t = \nabla_\theta f_t(\theta_{t-1})$, momentum $m_t$, and adaptive preconditioner $l_t$,
\[
\Delta\theta_t = \eta\, m_t\, l_t \qquad \Rightarrow \qquad \theta_t = \theta_{t-1} - \Delta\theta_t.
\]
Multiplicative updates fundamentally alter this structure by making the step proportional to the magnitude of each parameter. In the framework of the Generic Optimization Framework for Alternative Updates (GOFAU), the pure multiplicative update is
\[
\boxed{
\Delta\theta_t = |\theta_{t-1}|\, \tanh(\eta_{\mathrm{in}}\, m_t\, l_t)\, \eta_{\mathrm{out}}
}
\]
where $\eta_{\mathrm{in}}>0$ and $\eta_{\mathrm{out}}\in(0,1]$ are hyperparameters controlling the nonlinearity and outer scaling, respectively. This form ensures (i) moves proportional to parameter scale, (ii) controlled, clipped updates via $\tanh$, and (iii) elementwise operation.

A hybrid variant interpolates with the additive rule via a blending parameter $\gamma\in[0,1]$:
\[
\boxed{
\Delta\theta_t = \gamma\Big(|\theta_{t-1}|\, \tanh(\eta_{\mathrm{in}}\, m_t\, l_t)\, \eta_{\mathrm{out}}\Big) + (1-\gamma)(\eta\, m_t\, l_t)
}
\]
Additive and multiplicative methods are recovered at endpoints $\gamma=0$ and $\gamma=1$ [2307.07189].

## 2. Algorithmic Implementation and Integration with Backpropagation

Multiplicative and hybrid updates are implemented within standard backward passes. At each layer:
- Compute $m_t$, $l_t$ as for Adam, RMSProp, or Adagrad.
- Compute the multiplicative factor $u^{\rm mul}_t = |\theta_{t-1}|\, \tanh(\eta_{\mathrm{in}}\, m_t\, l_t)\, \eta_{\mathrm{out}}$.
- Compute the additive factor $u^{\rm add}_t = \eta\, m_t\, l_t$.
- Choose the update via hybrid blending or pure multiplicative rule.
- Apply elementwise subtraction to update $\theta$.

The update is composable with any standard initialization (Glorot, He) and optimizer-driven preconditioning. Table 1 summarizes the principal update rules:

| Update Type      | Formula                                                                                 | Sign-Flip?
|------------------|-----------------------------------------------------------------------------------------|----------|
| Pure Multiplicative | $|\theta|\, \tanh(\cdot)\, \eta_{\mathrm{out}}$                                       | No       |
| Additive (SGD)   | $\eta\, m_t\, l_t$                                                                      | Yes      |
| Hybrid           | $\gamma$ (multiplicative) $+ (1-\gamma)$ (additive)                                     | Yes, if $\gamma < 1$ |

Multiplicative updates remain sign-invariant; only the hybrid choice or additive term allows sign-flips. Zero-initialized parameters remain unchanged under pure multiplicative rules, motivating hybridization.

## 3. Theoretical Properties and Regret Bounds

Multiplicative and hybrid rules exhibit several theoretical properties that differentiate them from additive SGD [2307.07189, 1902.01903]:

- **Scale-Adaptivity**: Updates scale proportionally under a rescaling $\theta \rightarrow c\theta$, naturally adapting to parameter magnitude—a property absent from additive SGD, which would require learning-rate retuning.
- **Vanishing-Gradient Mitigation**: For small $|\theta|$, multiplicative updates still result in nonzero $\Delta \theta$ due to proportional scaling, alleviating stagnation.
- **Clipping and Robustness**: The $\tanh$ nonlinearity ensures that, even with large $\eta_{\mathrm{in}}$, step sizes remain bounded within $[-|\theta|,|\theta|]$, preventing parameter blow-up.
- **Mistake-Bound Guarantees**: In settings analogous to Winnow and exponentiated-gradient (EG), multiplicative updates are known to enjoy logarithmic mistake bounds when many features are irrelevant [2307.07189].
- **Mirror Descent Unification**: The hypentropy framework [1902.01903] provides a continuous family interpolating between additive (GD) and EG multiplicative rules, via a scalar “temperature” $\beta$, with explicit regret bounds that reduce to classic $\sqrt{T}$ or $\sqrt{T\log d}$ rates in appropriate limits.

## 4. Variants: Hypentropy and Expectation Reflection

Alternative multiplicative forms have been advanced:
- **Hypentropy Update (HU, SHU family)**: Based on the hypentropy potential,
  \[
  w_{t+\tfrac12} = \beta \sinh\left( \arcsinh(w_t/\beta) - \eta\, g_t \right)
  \]
  HU smoothly interpolates between GD ($\beta\to\infty$) and positive EG ($\beta\to 0$) elementwise, applicable to vectors and via spectral decomposition to matrices. The SHU extension enables matrix-valued updates for general rectangular matrices [1902.01903].
- **Expectation Reflection (ER)**: Updates weights via a multiplicative correction based on the ratio of true target to prediction,
  \[
  W^{\mathrm{new}} = W \times \frac{\hat{y}}{y}\quad \text{(single-layer, scalar case)}
  \]
  and, in the full stacked-network case, updates pre-activations by $H^{\mathrm{new}} = (\hat{Y} \oslash Y)\odot H$, with new weights computed via pseudoinverse regression. ER is hyperparameter-free, can converge in a single iteration in ideal cases, and reinterprets backpropagation as inverse target propagation [2503.10144].
- **Differentiable Addiplicative Units**: A smoothly parameterized transition between addition and multiplication at the neuron level uses non-integer exponentials:
  \[
  y_i = \exp^{(n_i)}\left( \sum_j W_{ij} \exp^{(-n_i)}(x_j) \right)
  \]
  allowing each neuron to interpolate between summation and multiplication, with gradients computable in closed form [1604.03736].

## 5. Empirical Results and Benchmarking

Multiplicative and hybrid updates have been empirically validated across convex, non-convex, and deep neural network settings:

- **Convex benchmarks**: On 2D convex and Rosenbrock problems, hybrid updates reduce normalized distance to the global minimum by factors of $10^1$–$10^2$ compared to additive baselines; after 100 steps, up to five orders of magnitude improvement is observed [2307.07189].
- **Deep image classification**:
  - **CIFAR-10**: Hybrid SGD + GOFAU yields $\sim 3\%$ higher absolute accuracy at epoch 5 compared to SGD; final gains up to $+8\%$ (ResNet18+SGD), $+2\%$ for Adagrad. Multiplicative-only rules yield $1$–$2\%$ improvements.
  - **CIFAR-100 and Tiny ImageNet**: Hybrid updates improve final accuracy on ResNet18 from $55.0\%$ to $61.6\%$ (SGD) and $57.0\%$ to $64.5\%$ (Adagrad); up to $+7\%$ on Tiny ImageNet [2307.07189].
  - **Training speed**: Major accuracy and loss improvements concentrate in the first $10$–$20$ epochs.
- **Expectation Reflection** achieves $2.77\%$ test error on MNIST and $55.24\%$ on CIFAR-10 after a single full-batch iteration, outperforming other hyperparameter-free algorithms and rapidly approaching BP performance [2503.10144].
- **Addiplicative units** enable networks to fit polynomial targets with lower test error and faster convergence than both purely additive or purely multiplicative architectures [1604.03736].

## 6. Practical Recommendations, Hyperparameter Tuning, and Limitations

- **Rate selection**: Inner/outer rates $\eta_{\mathrm{in}}, \eta_{\mathrm{out}}$ control the impact of $\tanh$-clipping and scaling:
  - SGD: $\eta_{\mathrm{in}}\approx 3$, $\eta_{\mathrm{out}}\approx 0.3$
  - Adagrad: $\eta_{\mathrm{in}}\approx 10$, $\eta_{\mathrm{out}}\approx 0.02$
  - RMSProp: $\eta_{\mathrm{in}}\approx 0.4$, $\eta_{\mathrm{out}}\approx 0.2$ [2307.07189]
- **Blending parameter $\gamma$**: A value of $0.5$ effectively balances sign flexibility and multiplicative robustness.
- **Initialization**: Use standard Xavier or He schemes. Pure multiplicative updates are acceptable with nonnegative-initialized weights.
- **Limitations**:
  - Pure multiplicative rules cannot switch parameter sign, and weights zero-initialized remain fixed unless hybridized.
  - Shallow nets may require additive steps to avoid suboptimal minima.
  - ER’s full-batch pseudoinverse updates scale poorly in high dimensions; ridge regularization and mini-batch approximations trade off the no-hyperparameter property for scalability [2503.10144].
  - Hybrid and addiplicative schemes increase per-parameter state or introduce additional parameters to optimize.

## 7. Broader Landscape and Theoretical Unification

Multiplicative backpropagation-style updates connect classical theory and modern deep learning:
- Multiplicative rules derive rigorously from mirror descent in non-Euclidean geometries, with Bregman divergences such as hypentropy smoothly bridging additive and multiplicative regimes [1902.01903].
- The unification enables regret bounds and update steps that extend to matrices and general nonlinear transformations.
- Structural variants, including neuron-level addiplicative control and model-level multiplicative scaling, provide inductive bias for multiplicative effects relevant to problem structure (e.g., polynomial interaction or logical gating) [1604.03736].
- Expectation Reflection links multiplicative consistency updates directly to target propagation and classical regression [2503.10144].
- The robustness of multiplicative schemes to hyperparameter selection, initialization scale, and irrelevant parameter directions aligns with theoretical advantages known from the Winnow and EG literatures, now instantiated at scale in modern deep architectures [2307.07189].

In synthesis, multiplicative backpropagation-style updates and their hybridizations furnish a principled, robust, and empirically validated toolkit for optimization in deep learning, with favorable convergence, scale-adaptivity, and performance characteristics documented across a range of benchmarks and model classes.

Source: https://www.emergentmind.com/topics/multiplicative-backpropagation-style-updates