Decreasing Backpropagation: Methods & Trade-offs

• Decreasing Backpropagation is a family of methods that modify the standard algorithm by altering update rules, truncating gradients, or using local signals.
• Techniques include semi-implicit updates, dynamic learning rates, forward gradients, and layerwise objectives, yielding faster convergence and improved memory efficiency in benchmarks like MNIST and CIFAR.
• Empirical trade-offs vary with architecture and task, as different methods balance training speed, numerical stability, and biological plausibility.

In this literature, “decreasing backpropagation” denotes a family of attempts to reduce reliance on standard reverse-mode error propagation rather than a single algorithm. The term covers methods that keep the forward model but alter the update rule, methods that shorten or reweight temporal credit assignment, methods that replace global gradients with local or approximate signals, and methods that remove the backward pass entirely. The motivating problems are recurrent: explicit gradient descent can require small step sizes and suffer from gradient vanishing, recurrent backpropagation scales poorly with sequence length, and standard backpropagation is often criticized for high memory cost and arguable biological implausibility (Liu et al., 2020, Bird et al., 2021, Betti et al., 2018, Baydin et al., 2022, Gong et al., 16 Jan 2025).

1. Update-rule modifications within the backpropagation paradigm

One line of work decreases backpropagation by retaining its error-flow structure while changing how parameters are updated. Semi-Implicit Back Propagation replaces explicit gradient steps on weights and biases with per-layer proximal subproblems. For layer $i$, the weight update is

$$W_i^{k+1} = \arg\min_{W_i}\; \big\|\sigma(W_i F_i^k + b_i^k) - F_{i+1}^{k+1/2}\big\|_F^2 + \frac{\lambda}{2}\|W_i - W_i^k\|_F^2,$$

with an analogous update for $b_i$. Hidden activations are still updated explicitly, but errors are propagated using the updated weights,

$$\delta_i^k = (W_i^{k+1})^T\left(\partial \sigma(G_{i+1}^k) \odot \delta_{i+1}^k\right).$$

This formulation permits large effective step sizes, and the paper proves that any fixed point of convergent sequences produced by the algorithm is a stationary point of the original objective. On MNIST and CIFAR-10, the method showed faster loss decrease and higher training and validation accuracy than SGD and ProxBP, including on a deep MNIST network $784 \to 600^{10} \to 10$ where SGD and Adam struggled (Liu et al., 2020).

A second modification keeps gradient descent but makes its aggressiveness synapse-specific. Dynamic Learning Rate replaces a global $\eta$ with

$$\eta_{ij} = -\eta_{0}\,\frac{|w_{ij}| + \alpha}{\|\mathbf{w}_j\| + \alpha},$$

or, in a variant, normalizes by $\|\mathbf{w}_i\|$ instead. Initially, when $\alpha > \|\mathbf{w}_j\| \gg |w_{ij}|$, the method behaves like standard SGD with learning rate $\eta_0$; later it produces competition among synapses and an automatic per-synapse decay. On MNIST with a single hidden layer, it reduced training time to a target $$W_i^{k+1} = \arg\min_{W_i}\; \big\|\sigma(W_i F_i^k + b_i^k) - F_{i+1}^{k+1/2}\big\|_F^2 + \frac{\lambda}{2}\|W_i - W_i^k\|_F^2,$$0 test accuracy for networks with fewer than 100 hidden units and reduced the minimal hidden-layer size needed to achieve that target from $$W_i^{k+1} = \arg\min_{W_i}\; \big\|\sigma(W_i F_i^k + b_i^k) - F_{i+1}^{k+1/2}\big\|_F^2 + \frac{\lambda}{2}\|W_i - W_i^k\|_F^2,$$1 for SGD to $$W_i^{k+1} = \arg\min_{W_i}\; \big\|\sigma(W_i F_i^k + b_i^k) - F_{i+1}^{k+1/2}\big\|_F^2 + \frac{\lambda}{2}\|W_i - W_i^k\|_F^2,$$2 for DLR (Li, 2020).

A third modification targets the geometry in which gradients are taken. The decorrelation approach argues that correlations in activations induce a non-orthonormal relation between parameters, so standard gradient descent operates in a skewed coordinate system. The proposed per-layer transform

$$W_i^{k+1} = \arg\min_{W_i}\; \big\|\sigma(W_i F_i^k + b_i^k) - F_{i+1}^{k+1/2}\big\|_F^2 + \frac{\lambda}{2}\|W_i - W_i^k\|_F^2,$$3

is updated by

$$W_i^{k+1} = \arg\min_{W_i}\; \big\|\sigma(W_i F_i^k + b_i^k) - F_{i+1}^{k+1/2}\big\|_F^2 + \frac{\lambda}{2}\|W_i - W_i^k\|_F^2,$$4

with a gain $$W_i^{k+1} = \arg\min_{W_i}\; \big\|\sigma(W_i F_i^k + b_i^k) - F_{i+1}^{k+1/2}\big\|_F^2 + \frac{\lambda}{2}\|W_i - W_i^k\|_F^2,$$5 that preserves activity scale. In multi-layer networks, this decorrelation substantially accelerated standard backpropagation and made feedback alignment and node perturbation far more effective; on CIFAR-100, the paper reports up to $$W_i^{k+1} = \arg\min_{W_i}\; \big\|\sigma(W_i F_i^k + b_i^k) - F_{i+1}^{k+1/2}\big\|_F^2 + \frac{\lambda}{2}\|W_i - W_i^k\|_F^2,$$6 speedup in reaching a given training accuracy compared to vanilla backpropagation (Ahmad, 2024).

A fourth contribution decreases the cost of derivative-based regularization rather than ordinary training. In double backpropagation with Jacobian Frobenius penalties, the paper derives optimized rules in a general Hilbert-space setting and shows that, for locally linear activation functions, the required calculations for $$W_i^{k+1} = \arg\min_{W_i}\; \big\|\sigma(W_i F_i^k + b_i^k) - F_{i+1}^{k+1/2}\big\|_F^2 + \frac{\lambda}{2}\|W_i - W_i^k\|_F^2,$$7 can be reduced by roughly one third (Etmann, 2019).

2. Temporal shortening and exact long-horizon alternatives

In recurrent models, decreasing backpropagation usually means decreasing how far gradients are propagated through time. Standard BPTT expands the total derivative of the final output through products of Jacobians over all previous time steps, which gives linear growth in time and memory with the sequence length. Truncated BPTT and $$W_i^{k+1} = \arg\min_{W_i}\; \big\|\sigma(W_i F_i^k + b_i^k) - F_{i+1}^{k+1/2}\big\|_F^2 + \frac{\lambda}{2}\|W_i - W_i^k\|_F^2,$$8-order approximations reduce this cost by discarding contributions outside a finite window, but this makes the gradient estimate biased and systematically favors short-term dependencies (Bird et al., 2021, Tallec et al., 2017).

Anticipated Reweighted Truncated Backpropagation retains short subsequences while restoring unbiasedness. Let $$W_i^{k+1} = \arg\min_{W_i}\; \big\|\sigma(W_i F_i^k + b_i^k) - F_{i+1}^{k+1/2}\big\|_F^2 + \frac{\lambda}{2}\|W_i - W_i^k\|_F^2,$$9 denote a truncation between $b_i$0 and $b_i$1, with conditional probability $b_i$2. ARTBP defines the backward recursion

$b_i$3

The compensation factor $b_i$4 makes the estimator unbiased: $b_i$5 On a synthetic influence-balancing task, truncated BPTT diverged for $b_i$6 and $b_i$7, whereas ARTBP with average truncation length $b_i$8 converged reliably. On Penn Treebank character-level language modelling, ARTBP slightly outperformed truncated BPTT, with test performance around $b_i$9 bpc versus $$\delta_i^k = (W_i^{k+1})^T\left(\partial \sigma(G_{i+1}^k) \odot \delta_{i+1}^k\right).$$0 bpc for truncated BPTT (Tallec et al., 2017).

A more radical alternative is to replace backpropagation through time with a forward sensitivity recurrence. The discrete forward sensitivity equation introduces

$$\delta_i^k = (W_i^{k+1})^T\left(\partial \sigma(G_{i+1}^k) \odot \delta_{i+1}^k\right).$$1

and computes

$$\delta_i^k = (W_i^{k+1})^T\left(\partial \sigma(G_{i+1}^k) \odot \delta_{i+1}^k\right).$$2

This is exact, has constant per-time-step cost with respect to the number of elapsed steps, and allows parameters to vary and be updated at every time step, but it requires the computation of Jacobians and the paper provides no empirical benchmarks (Bird et al., 2021).

3. Local, forward-only, and layerwise replacements

A stronger sense of decreasing backpropagation removes reverse-mode differentiation entirely. Forward gradients do this by sampling a random direction $$\delta_i^k = (W_i^{k+1})^T\left(\partial \sigma(G_{i+1}^k) \odot \delta_{i+1}^k\right).$$3, computing the directional derivative $$\delta_i^k = (W_i^{k+1})^T\left(\partial \sigma(G_{i+1}^k) \odot \delta_{i+1}^k\right).$$4 with forward-mode AD, and forming the estimator

$$\delta_i^k = (W_i^{k+1})^T\left(\partial \sigma(G_{i+1}^k) \odot \delta_{i+1}^k\right).$$5

Under i.i.d. zero-mean, unit-variance components of $$\delta_i^k = (W_i^{k+1})^T\left(\partial \sigma(G_{i+1}^k) \odot \delta_{i+1}^k\right).$$6, this estimator is unbiased: $$\delta_i^k = (W_i^{k+1})^T\left(\partial \sigma(G_{i+1}^k) \odot \delta_{i+1}^k\right).$$7 The method evaluates loss and directional derivative in a single forward run, entirely eliminating the need for backpropagation. Experiments on logistic regression, MLPs, and CNNs showed lower per-iteration runtime than reverse-mode backpropagation, and the paper reports training up to twice as fast in some cases (Baydin et al., 2022).

Mono-Forward eliminates global error propagation by assigning every layer its own classifier head. For layer $$\delta_i^k = (W_i^{k+1})^T\left(\partial \sigma(G_{i+1}^k) \odot \delta_{i+1}^k\right).$$8, activations $$\delta_i^k = (W_i^{k+1})^T\left(\partial \sigma(G_{i+1}^k) \odot \delta_{i+1}^k\right).$$9 are mapped to class scores

$784 \to 600^{10} \to 10$0

and trained with local cross-entropy

$784 \to 600^{10} \to 10$1

The layer parameters are then updated only through $784 \to 600^{10} \to 10$2: $784 \to 600^{10} \to 10$3 Across MLPs and CNNs on MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100, Mono-Forward matched or surpassed backpropagation, while the slope of peak-memory growth with depth was reported as $784 \to 600^{10} \to 10$4 for BP and $784 \to 600^{10} \to 10$5 for MF in one MLP study (Gong et al., 16 Jan 2025).

A related but more theoretical route formulates supervised learning as a saddle-point problem in an adjoint space over neuron outputs $784 \to 600^{10} \to 10$6, weights $784 \to 600^{10} \to 10$7, and Lagrange multipliers $784 \to 600^{10} \to 10$8: $784 \to 600^{10} \to 10$9 The stationarity conditions recover standard backpropagation, but gradient descent-ascent in $\eta$0-space yields biologically plausible local algorithms in which weight updates depend only on presynaptic activity, postsynaptic activity, and local multipliers. The same framework introduces “support neurons,” whose nonzero multipliers suggest a role in architecture construction and pruning (Betti et al., 2018).

4. Asymmetric and application-specific reductions

Some methods decrease backpropagation asymmetrically rather than uniformly. In two-tower recommendation, the proposed OneBP strategy keeps normal gradient backpropagation for the item tower but cuts off backpropagation for the user tower. Item embeddings are updated by gradients,

$\eta$1

while user embeddings are updated by moving aggregation,

$\eta$2

On MovieLens-100k, MovieLens-1M, Gowalla, and Yelp2018, this asymmetric reduction improved top-$\eta$3 recommendation metrics and reduced training time per epoch, with the largest reported relative speedup on Yelp2018 (Chen et al., 2024).

In binary neural networks, reducing backpropagation is motivated by the difficulty of propagating surrogate gradients through many binarization STEs. The comparative study on ImageNette evaluated BP, DFA, DRTP, HSIC, and SigpropTL on VGG-19, MobileNetV2, and MLP-Mixer, under binary weights, binary activations, or both. With binary weights and activations, BP was best on MobileNetV2 and MLP-Mixer, but on VGG-19 DFA reached $\eta$4 while BP reached $\eta$5. Removing skip connections from MLP-Mixer and MobileNetV2 likewise made DFA outperform BP, suggesting that skip connections are particularly important for BP in BNNs (Crulis et al., 2024).

A related MLP study compared BP, FA, Uni Sign FA, DFA, and Weighted DFA on MNIST and CIFAR10. BP had the highest accuracy, DFA stayed close, and Weighted DFA improved stability: on CIFAR10 the cosine similarity of predictions across random seeds was $\eta$6 for Weighted DFA versus $\eta$7 for DFA and $\eta$8 for BP. The same study found that first-layer weights under Weighted DFA were extremely close to BP, with cosine similarity $\eta$9 (Waldo, 2022).

5. Generalizations beyond ordinary backpropagation

Some of the most far-reaching proposals do not merely alter backpropagation; they subsume it. In one theoretical construction, a deterministic function network is “lifted” to a factor graph by replacing each functional relation with a Dirac delta factor and fixing inputs with delta priors, then attaching a Boltzmann factor $$\eta_{ij} = -\eta_{0}\,\frac{|w_{ij}| + \alpha}{\|\mathbf{w}_j\| + \alpha},$$0 to the output. On this lifted graph, derivatives of log downward belief-propagation messages reproduce backpropagated adjoints exactly: $$\eta_{ij} = -\eta_{0}\,\frac{|w_{ij}| + \alpha}{\|\mathbf{w}_j\| + \alpha},$$1 Under these conditions, backpropagation is recovered as a special case of loopy belief propagation, and changing the priors or factors yields a broader message-passing family (Eaton, 2022).

Another direction attempts to replace iterative optimization with explicit solutions. For single-layer feed-forward softmax models trained on positive-valued features with fixed $$\eta_{ij} = -\eta_{0}\,\frac{|w_{ij}| + \alpha}{\|\mathbf{w}_j\| + \alpha},$$2-norm $$\eta_{ij} = -\eta_{0}\,\frac{|w_{ij}| + \alpha}{\|\mathbf{w}_j\| + \alpha},$$3, the explicit solution for weights is

$$\eta_{ij} = -\eta_{0}\,\frac{|w_{ij}| + \alpha}{\|\mathbf{w}_j\| + \alpha},$$4

where $$\eta_{ij} = -\eta_{0}\,\frac{|w_{ij}| + \alpha}{\|\mathbf{w}_j\| + \alpha},$$5 is the generalized co-occurrence matrix. In experiments on a simple LLM and on MNIST, the explicit solution was already near-optimal, iterative optimization only marginally improved it, and warm-starting from it led to better optima than cold-start backpropagation alone (Williams et al., 2023).

At the software level, the Zenkai framework operationalizes these ideas by dividing a deep learner into semi-autonomous LearningMachine modules with their own forward, assess_y, step, and step_x procedures. This supports local objectives, target propagation, feedback alignment, direct feedback alignment, hill climbing, population-based search, and non-differentiable layers such as decision trees within a common PyTorch-based interface (Short, 2023).

6. Empirical trade-offs, limits, and ongoing debate

The literature does not support a uniform verdict that less backpropagation is always better. The strongest negative result comes from scaling-law analysis on decoder-only Transformers trained with Direct Feedback Alignment. Across model sizes from roughly $$\eta_{ij} = -\eta_{0}\,\frac{|w_{ij}| + \alpha}{\|\mathbf{w}_j\| + \alpha},$$6M to $$\eta_{ij} = -\eta_{0}\,\frac{|w_{ij}| + \alpha}{\|\mathbf{w}_j\| + \alpha},$$7M parameters, the fitted compute-law exponents were $$\eta_{ij} = -\eta_{0}\,\frac{|w_{ij}| + \alpha}{\|\mathbf{w}_j\| + \alpha},$$8 for backpropagation and $$\eta_{ij} = -\eta_{0}\,\frac{|w_{ij}| + \alpha}{\|\mathbf{w}_j\| + \alpha},$$9 for DFA, with DFA also having a much worse offset. Even under optimistic compute accounting that favors DFA, the paper concludes that there is never a regime in which the degradation in loss is worth the potential reduction in compute budget (Filipovich et al., 2022).

This coexistence of positive and negative results is a central feature of the field. Semi-implicit updates improved MNIST and CIFAR-10 training, ARTBP corrected the bias of truncated BPTT, forward gradients removed reverse-mode AD, Mono-Forward exceeded BP on several benchmarks, OneBP improved two-tower recommendation, and DFA outperformed BP for some binary VGG settings (Liu et al., 2020, Tallec et al., 2017, Baydin et al., 2022, Gong et al., 16 Jan 2025, Chen et al., 2024, Crulis et al., 2024). At the same time, DFA scaled worse than BP in GPT-like LLMs, HSIC and SigpropTL trailed BP on several ImageNette BNN settings, and many local or biologically motivated schemes remain architecture-sensitive (Filipovich et al., 2022, Crulis et al., 2024).

A plausible implication is that decreasing backpropagation is not a single optimization principle but a design space whose outcomes depend strongly on architecture, objective, temporal horizon, binarization regime, and hardware assumptions. In current research, the question is therefore less whether backpropagation can be decreased in principle than where, and under what constraints, a decrease in backpropagation improves the overall trade-off between optimization quality, memory, runtime, parallelizability, and locality.