---
title: 'Decreasing Backpropagation: Methods & Trade-offs'
url: https://www.emergentmind.com/topics/decreasing-backpropagation
type: topic
---

# Decreasing Backpropagation: Methods & Trade-offs

In this literature, “decreasing backpropagation” denotes a family of attempts to reduce reliance on standard reverse-mode error propagation rather than a single algorithm. The term covers methods that keep the forward model but alter the update rule, methods that shorten or reweight temporal credit assignment, methods that replace global gradients with local or approximate signals, and methods that remove the backward pass entirely. The motivating problems are recurrent: explicit gradient descent can require small step sizes and suffer from gradient vanishing, recurrent backpropagation scales poorly with sequence length, and standard backpropagation is often criticized for high memory cost and arguable biological implausibility [2002.03516] [2103.15589] [1808.06934] [2202.08587] [2501.09238].

## 1. Update-rule modifications within the backpropagation paradigm

One line of work decreases backpropagation by retaining its error-flow structure while changing how parameters are updated. Semi-Implicit Back Propagation replaces explicit gradient steps on weights and biases with per-layer proximal subproblems. For layer \(i\), the weight update is
\[
W_i^{k+1} = \arg\min_{W_i}\; \big\|\sigma(W_i F_i^k + b_i^k) - F_{i+1}^{k+1/2}\big\|_F^2 + \frac{\lambda}{2}\|W_i - W_i^k\|_F^2,
\]
with an analogous update for \(b_i\). Hidden activations are still updated explicitly, but errors are propagated using the updated weights,
\[
\delta_i^k = (W_i^{k+1})^T\left(\partial \sigma(G_{i+1}^k) \odot \delta_{i+1}^k\right).
\]
This formulation permits large effective step sizes, and the paper proves that any fixed point of convergent sequences produced by the algorithm is a stationary point of the original objective. On MNIST and CIFAR-10, the method showed faster loss decrease and higher training and validation accuracy than SGD and ProxBP, including on a deep MNIST network \(784 \to 600^{10} \to 10\) where SGD and Adam struggled [2002.03516].

A second modification keeps gradient descent but makes its aggressiveness synapse-specific. Dynamic Learning Rate replaces a global \(\eta\) with
\[
\eta_{ij} = -\eta_{0}\,\frac{|w_{ij}| + \alpha}{\|\mathbf{w}_j\| + \alpha},
\]
or, in a variant, normalizes by \(\|\mathbf{w}_i\|\) instead. Initially, when \(\alpha > \|\mathbf{w}_j\| \gg |w_{ij}|\), the method behaves like standard SGD with learning rate \(\eta_0\); later it produces competition among synapses and an automatic per-synapse decay. On MNIST with a single hidden layer, it reduced training time to a target \(96\%\) test accuracy for networks with fewer than 100 hidden units and reduced the minimal hidden-layer size needed to achieve that target from \(36.3 \pm 1.2\) for SGD to \(29.7 \pm 1.6\) for DLR [2009.12745].

A third modification targets the geometry in which gradients are taken. The decorrelation approach argues that correlations in activations induce a non-orthonormal relation between parameters, so standard gradient descent operates in a skewed coordinate system. The proposed per-layer transform
\[
\bar{\mathbf{x}}_i = \mathbf{M}_i \mathbf{x}_i
\]
is updated by
\[
\mathbf{M}_i \leftarrow g_i\bigl(\mathbf{M}_i - \eta_M \, \langle \bar{\mathbf{x}}_i\bar{\mathbf{x}}_i^\top \rangle \,\mathbf{M}_i \bigr),
\]
with a gain \(g_i\) that preserves activity scale. In multi-layer networks, this decorrelation substantially accelerated standard backpropagation and made feedback alignment and node perturbation far more effective; on CIFAR-100, the paper reports up to \(4\times\) speedup in reaching a given training accuracy compared to vanilla backpropagation [2407.10780].

A fourth contribution decreases the cost of derivative-based regularization rather than ordinary training. In double backpropagation with Jacobian Frobenius penalties, the paper derives optimized rules in a general Hilbert-space setting and shows that, for locally linear activation functions, the required calculations for \(\|J_f(x)\|_F^2\) can be reduced by roughly one third [1906.06637].

## 2. Temporal shortening and exact long-horizon alternatives

In recurrent models, decreasing backpropagation usually means decreasing how far gradients are propagated through time. Standard BPTT expands the total derivative of the final output through products of Jacobians over all previous time steps, which gives linear growth in time and memory with the sequence length. Truncated BPTT and \(N^{\text{th}}\)-order approximations reduce this cost by discarding contributions outside a finite window, but this makes the gradient estimate biased and systematically favors short-term dependencies [2103.15589] [1705.08209].

Anticipated Reweighted Truncated Backpropagation retains short subsequences while restoring unbiasedness. Let \(X_t=1\) denote a truncation between \(t\) and \(t+1\), with conditional probability \(c_t\). ARTBP defines the backward recursion
\[
\tilde{\ell}_t =
\begin{cases}
\dfrac{\partial \ell}{\partial s}(s_t, o_t^\star), &\text{if } X_t = 1 \text{ or } t=T,\\[0.6em]
\dfrac{1}{1-c_t}\, \tilde{\ell}_{t+1}\,\dfrac{\partial F}{\partial s}(x_{t+1}, s_t, \theta) + \dfrac{\partial \ell}{\partial s}(s_t, o_t^\star), &\text{otherwise.}
\end{cases}
\]
The compensation factor \(1/(1-c_t)\) makes the estimator unbiased:
\[
\mathbb{E}[\tilde{g}] = \frac{\partial \mathcal{L}_T}{\partial \theta}.
\]
On a synthetic influence-balancing task, truncated BPTT diverged for \(L=10\) and \(L=100\), whereas ARTBP with average truncation length \(L_0=16\) converged reliably. On Penn Treebank character-level language modelling, ARTBP slightly outperformed truncated BPTT, with test performance around \(1.40\) bpc versus \(1.43\) bpc for truncated BPTT [1705.08209].

A more radical alternative is to replace backpropagation through time with a forward sensitivity recurrence. The discrete forward sensitivity equation introduces
\[
\Delta_N = \frac{\mathrm{d} R_N}{\mathrm{d} P_{N}}
+ \frac{\mathrm{d} R_N}{\mathrm{d} R_{N-1}} \,\Delta_{N-1},
\]
and computes
\[
\frac{\mathrm{d} Y_N}{\mathrm{d} P}
= \frac{\mathrm{d} Y_N}{\mathrm{d} P_{N}}
+ \frac{\mathrm{d} Y_N}{\mathrm{d} R_{N-1}} \,\Delta_{N-1}.
\]
This is exact, has constant per-time-step cost with respect to the number of elapsed steps, and allows parameters to vary and be updated at every time step, but it requires the computation of Jacobians and the paper provides no empirical benchmarks [2103.15589].

## 3. Local, forward-only, and layerwise replacements

A stronger sense of decreasing backpropagation removes reverse-mode differentiation entirely. Forward gradients do this by sampling a random direction \(v\), computing the directional derivative \(\nabla f(\theta)\cdot v\) with forward-mode AD, and forming the estimator
\[
g(\theta) = (\nabla f(\theta)\cdot v)\,v.
\]
Under i.i.d. zero-mean, unit-variance components of \(v\), this estimator is unbiased:
\[
\mathbb{E}[g(\theta)] = \nabla f(\theta).
\]
The method evaluates loss and directional derivative in a single forward run, entirely eliminating the need for backpropagation. Experiments on logistic regression, MLPs, and CNNs showed lower per-iteration runtime than reverse-mode backpropagation, and the paper reports training up to twice as fast in some cases [2202.08587].

Mono-Forward eliminates global error propagation by assigning every layer its own classifier head. For layer \(i\), activations \(\mathbf{a}_i\) are mapped to class scores
\[
\mathbf{G}_i = \mathbf{a}_i \mathbf{M}_i^\top,
\]
and trained with local cross-entropy
\[
\mathcal{L}_i \triangleq -\sum_{c=1}^m y_c \log(\sigma(G_{ic})).
\]
The layer parameters are then updated only through \(\mathcal{L}_i\):
\[
\mathbf{W}_i \gets \mathbf{W}_i - \eta \left(\frac{\partial \mathcal{L}_i}{\partial \mathbf{G}_i} \cdot \frac{\partial \mathbf{G}_i}{\partial \mathbf{a}_i} \cdot \frac{\partial \mathbf{a}_i}{\partial \mathbf{z}_i} \cdot \frac{\partial \mathbf{z}_i}{\partial \mathbf{W}_i}\right).
\]
Across MLPs and CNNs on MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100, Mono-Forward matched or surpassed backpropagation, while the slope of peak-memory growth with depth was reported as \(204.00\) for BP and \(16.16\) for MF in one MLP study [2501.09238].

A related but more theoretical route formulates supervised learning as a saddle-point problem in an adjoint space over neuron outputs \(x\), weights \(w\), and Lagrange multipliers \(\lambda\):
\[
\mathcal{L}(x, w, \lambda)
= \mathcal{E}(x^L, y) + \sum_{\ell=1}^L \sum_i \lambda_i^\ell \bigg( x_i^\ell - \sigma\!\Big( \sum_j w_{ij}^\ell x_j^{\ell-1} \Big) \bigg).
\]
The stationarity conditions recover standard backpropagation, but gradient descent-ascent in \((x,w,\lambda)\)-space yields biologically plausible local algorithms in which weight updates depend only on presynaptic activity, postsynaptic activity, and local multipliers. The same framework introduces “support neurons,” whose nonzero multipliers suggest a role in architecture construction and pruning [1808.06934].

## 4. Asymmetric and application-specific reductions

Some methods decrease backpropagation asymmetrically rather than uniformly. In two-tower recommendation, the proposed OneBP strategy keeps normal gradient backpropagation for the item tower but cuts off backpropagation for the user tower. Item embeddings are updated by gradients,
\[
\mathbf{v}_j^{(t+1)} = \mathbf{v}_j^{(t)} - \alpha \nabla_{\mathbf{v}_j} \mathcal{L},
\]
while user embeddings are updated by moving aggregation,
\[
\mathbf{u}_i^{(t+1)} = \beta \, \mathbf{u}_i^{(t)} + (1-\beta)\,\mathbf{v}_j^{(t+1)}.
\]
On MovieLens-100k, MovieLens-1M, Gowalla, and Yelp2018, this asymmetric reduction improved top-\(K\) recommendation metrics and reduced training time per epoch, with the largest reported relative speedup on Yelp2018 [2403.18227].

In binary neural networks, reducing backpropagation is motivated by the difficulty of propagating surrogate gradients through many binarization STEs. The comparative study on ImageNette evaluated BP, DFA, DRTP, HSIC, and SigpropTL on VGG-19, MobileNetV2, and MLP-Mixer, under binary weights, binary activations, or both. With binary weights and activations, BP was best on MobileNetV2 and MLP-Mixer, but on VGG-19 DFA reached \(52.0 \pm 0.79\%\) while BP reached \(21.2 \pm 0.53\%\). Removing skip connections from MLP-Mixer and MobileNetV2 likewise made DFA outperform BP, suggesting that skip connections are particularly important for BP in BNNs [2408.04460].

A related MLP study compared BP, FA, Uni Sign FA, DFA, and Weighted DFA on MNIST and CIFAR10. BP had the highest accuracy, DFA stayed close, and Weighted DFA improved stability: on CIFAR10 the cosine similarity of predictions across random seeds was \(0.877\) for Weighted DFA versus \(0.873\) for DFA and \(0.851\) for BP. The same study found that first-layer weights under Weighted DFA were extremely close to BP, with cosine similarity \(0.997\) [2206.06098].

## 5. Generalizations beyond ordinary backpropagation

Some of the most far-reaching proposals do not merely alter backpropagation; they subsume it. In one theoretical construction, a deterministic function network is “lifted” to a factor graph by replacing each functional relation with a Dirac delta factor and fixing inputs with delta priors, then attaching a Boltzmann factor \(B(z)=e^z\) to the output. On this lifted graph, derivatives of log downward belief-propagation messages reproduce backpropagated adjoints exactly:
\[
\left.\frac{\partial}{\partial x}\right|_{x=x^*} \log m_{(x,F)}(x) = \frac{\partial z}{\partial x}.
\]
Under these conditions, backpropagation is recovered as a special case of loopy belief propagation, and changing the priors or factors yields a broader message-passing family [2210.00610].

Another direction attempts to replace iterative optimization with explicit solutions. For single-layer feed-forward softmax models trained on positive-valued features with fixed \(L_1\)-norm \(K\), the explicit solution for weights is
\[
W_{j,i} = \log (X,Y)_{j,i} - \frac{K - 1}{K}\log\left(\sum_{d = 1}^D (X,Y)_{d,i}\right),
\]
where \((X,Y)=X^\top Y\) is the generalized co-occurrence matrix. In experiments on a simple language model and on MNIST, the explicit solution was already near-optimal, iterative optimization only marginally improved it, and warm-starting from it led to better optima than cold-start backpropagation alone [2311.07498].

At the software level, the Zenkai framework operationalizes these ideas by dividing a deep learner into semi-autonomous `LearningMachine` modules with their own `forward`, `assess_y`, `step`, and `step_x` procedures. This supports local objectives, target propagation, feedback alignment, direct feedback alignment, hill climbing, population-based search, and non-differentiable layers such as decision trees within a common PyTorch-based interface [2311.09663].

## 6. Empirical trade-offs, limits, and ongoing debate

The literature does not support a uniform verdict that less backpropagation is always better. The strongest negative result comes from scaling-law analysis on decoder-only Transformers trained with Direct Feedback Alignment. Across model sizes from roughly \(57\)M to \(509\)M parameters, the fitted compute-law exponents were \(\alpha_C=-0.071\) for backpropagation and \(\alpha_C=-0.040\) for DFA, with DFA also having a much worse offset. Even under optimistic compute accounting that favors DFA, the paper concludes that there is never a regime in which the degradation in loss is worth the potential reduction in compute budget [2210.14593].

This coexistence of positive and negative results is a central feature of the field. Semi-implicit updates improved MNIST and CIFAR-10 training, ARTBP corrected the bias of truncated BPTT, forward gradients removed reverse-mode AD, Mono-Forward exceeded BP on several benchmarks, OneBP improved two-tower recommendation, and DFA outperformed BP for some binary VGG settings [2002.03516] [1705.08209] [2202.08587] [2501.09238] [2403.18227] [2408.04460]. At the same time, DFA scaled worse than BP in GPT-like language models, HSIC and SigpropTL trailed BP on several ImageNette BNN settings, and many local or biologically motivated schemes remain architecture-sensitive [2210.14593] [2408.04460].

A plausible implication is that decreasing backpropagation is not a single optimization principle but a design space whose outcomes depend strongly on architecture, objective, temporal horizon, binarization regime, and hardware assumptions. In current research, the question is therefore less whether backpropagation can be decreased in principle than where, and under what constraints, a decrease in backpropagation improves the overall trade-off between optimization quality, memory, runtime, parallelizability, and locality.

Source: https://www.emergentmind.com/topics/decreasing-backpropagation