How Important is Weight Symmetry in Backpropagation?

Abstract: Gradient backpropagation (BP) requires symmetric feedforward and feedback connections -- the same weights must be used for forward and backward passes. This "weight transport problem" (Grossberg 1987) is thought to be one of the main reasons to doubt BP's biologically plausibility. Using 15 different classification datasets, we systematically investigate to what extent BP really depends on weight symmetry. In a study that turned out to be surprisingly similar in spirit to Lillicrap et al.'s demonstration (Lillicrap et al. 2014) but orthogonal in its results, our experiments indicate that: (1) the magnitudes of feedback weights do not matter to performance (2) the signs of feedback weights do matter -- the more concordant signs between feedforward and their corresponding feedback connections, the better (3) with feedback weights having random magnitudes and 100% concordant signs, we were able to achieve the same or even better performance than SGD. (4) some normalizations/stabilizations are indispensable for such asymmetric BP to work, namely Batch Normalization (BN) (Ioffe and Szegedy 2015) and/or a "Batch Manhattan" (BM) update rule.

• The paper finds that sign concordance between feedforward and feedback weights is critical for backpropagation performance, much more so than exact weight magnitude.
• Experiments demonstrate that effective learning is possible even when feedback weights have random magnitudes, indicating that sign alignment is the key factor.
• Asymmetric backpropagation using sign-concordant random feedback weights can achieve good performance, but requires stabilization techniques like Batch Normalization.

Weight Symmetry in Backpropagation: Investigating Biological Plausibility

The paper "How Important Is Weight Symmetry in Backpropagation?" by Qianli Liao, Joel Z. Leibo, and Tomaso Poggio explores the critical question of whether the weight symmetry requirement in backpropagation (BP) is vital for its effectiveness in neural networks and its implications for biological plausibility. This investigation provides a substantial extension to previous work by examining the necessity of symmetric weight transport, which posits considerable constraints on the feasibility of BP in biological systems such as the human brain.

The core concern addressed in this paper is the "weight transport problem," also referred to as the "weight symmetry problem," which requires that the weights used in forward passes are exactly mirrored in backward passes for BP to work as traditionally defined. This characteristic is conventionally viewed as a limitation when evaluating BP as a model for learning in neural tissues.

Through experimental assessments across 15 classification datasets, this study systematically releases the symmetry constraint. The findings are as follows:

1. Magnitude Irrelevance: The experiments demonstrate that the magnitudes of the feedback weights do not affect the performance of BP. Contrary to typical assumptions, successful learning is attainable even when feedback weights are assigned random magnitudes. This insight suggests an incomplete understanding of why BP is effective, highlighting a need for theoretical advancement.
2. Critical Sign Concordance: Notably, the paper finds that the critical factor influencing BP's performance is the concordance of the signs between feedforward and feedback weights. A high degree of sign concordance aligns with robust learning outcomes, significantly suggesting that a focus primarily on the signs, rather than exact weights, may be beneficial.
3. Effective Asymmetry: Asymmetric BP configurations, such as those incorporating random feedback weights with aligned sign configurations, can match or even surpass the performance of SGD under certain configurations. This development opens a potential pathway towards biologically plausible learning models.
4. Role of Normalization Techniques: The study identifies the necessity of Batch Normalization (BN) and Batch Manhattan (BM) stabilization techniques as indispensable for maintaining effective learning in asymmetric BP scenarios. These normalization methods mitigate gradient issues, such as vanishing or exploding gradients, reinforcing their utility in neural network training protocols.

This research bears significant theoretical and practical implications. Theoretically, it questions the fundamental assumptions of the BP algorithm's requirements, presenting an alternative paradigm where precise weight symmetry is not a prerequisite. Practically, it offers insights into developing more robust training algorithms that could inspire neuromorphic computing applications and artificial intelligence models embodying more brain-like learning mechanisms.

The potential for future research stimulated by these findings is vast. One avenue involves further exploring the biological plausibility of these modified BP approaches, which might identify preferential alignment patterns similar to those in cortical feedback systems. Additionally, improving the theoretical framework to explain why sign-concordant but magnitude-variable feedback remains effective could greatly enhance our comprehension and application of neural network learning dynamics.

In conclusion, the paper challenges the foundational belief in the necessity of weight symmetry for efficient backpropagation. The shift towards considering asymmetric feedback with sign-concordance as efficiently learnable offers a promising extension to neural network methodologies and their intersection with biological computation principles.