Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation (1308.3432v1)

Abstract: Stochastic neurons and hard non-linearities can be useful for a number of reasons in deep learning models, but in many cases they pose a challenging problem: how to estimate the gradient of a loss function with respect to the input of such stochastic or non-smooth neurons? I.e., can we "back-propagate" through these stochastic neurons? We examine this question, existing approaches, and compare four families of solutions, applicable in different settings. One of them is the minimum variance unbiased gradient estimator for stochatic binary neurons (a special case of the REINFORCE algorithm). A second approach, introduced here, decomposes the operation of a binary stochastic neuron into a stochastic binary part and a smooth differentiable part, which approximates the expected effect of the pure stochatic binary neuron to first order. A third approach involves the injection of additive or multiplicative noise in a computational graph that is otherwise differentiable. A fourth approach heuristically copies the gradient with respect to the stochastic output directly as an estimator of the gradient with respect to the sigmoid argument (we call this the straight-through estimator). To explore a context where these estimators are useful, we consider a small-scale version of {\em conditional computation}, where sparse stochastic units form a distributed representation of gaters that can turn off in combinatorially many ways large chunks of the computation performed in the rest of the neural network. In this case, it is important that the gating units produce an actual 0 most of the time. The resulting sparsity can be potentially be exploited to greatly reduce the computational cost of large deep networks for which conditional computation would be useful.

• The paper introduces four gradient estimation methods for stochastic neurons, highlighting their role in efficient conditional computation.
• It presents an unbiased estimator for stochastic binary neurons that reduces computational expense by bypassing the backward pass.
• Empirical results on MNIST show that noise injection and the straight-through estimator deliver significant performance and efficiency gains.

Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation

In the examined paper, "Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation," the authors address the critical challenge of estimating gradients in neural networks containing stochastic or non-smooth neurons. This challenge arises primarily in the context of deep learning models that employ stochastic neurons for various computational efficiencies and potential improvements in learning dynamics.

Key Contributions

1. Method Comparison: The paper comprehensively compares four distinct families of approaches to estimate gradients through stochastic neurons:
   • Minimum variance unbiased gradient estimator for stochastic binary neurons, a specific case of the REINFORCE algorithm.
   • A novel approach that decomposes the operation of a binary stochastic neuron into its stochastic and smooth components.
   • Additive or multiplicative noise injection in an otherwise differentiable computational graph.
   • The straight-through estimator, which heuristically copies the gradient as an estimator.
2. Stochastic Binary Neurons: For stochastic binary neurons, the paper derives an unbiased gradient estimator, demonstrating that it can be cheaper to compute than using back-propagation, as it circumvents the need for a backward pass.
3. Theoretical Insights: The authors provide theoretical results concerning the proposed methods, including the properties of noisy rectifier units and the novel stochastic times smooth (STS) units. They prove that certain noisy rectifier units can achieve properties like sparsity and gradient flow under noise perturbations, promoting efficient training dynamics.
4. Practical Applications in Conditional Computation: The paper explores the applicability of the proposed estimators within the context of conditional computation—a scenario where sparse, stochastic gating units can selectively activate portions of a neural network, aiming for significant computational savings while retaining performance.

Experimental Validation

The authors conduct empirical validation on the MNIST dataset using a conditional computation architecture. Here, sparse stochastic units are deployed to selectively activate parts of the network. The key findings include:

• Noise Injection Utility: Noise-based variants, including noisy rectifiers and sigmoid units with injected noise, performed competitively, suggesting the utility of noise not just as a regularizer but as an essential component for efficient training dynamics.
• Performance of Straight-Through Estimator: The straight-through estimator, despite its simplicity and inherent bias, yielded the best validation and test errors for the gating conditional layer in the experiments.
• Efficiency Gains: Empirical results demonstrated that conditional computation could indeed translate into computational savings (approximately reducing the active compute to 10% of the units) without severely compromising accuracy.

Implications and Future Directions

The implications of this research are manifold, stretching from theoretical insights into the behavior of stochastic neurons to practical considerations in designing computationally efficient neural networks. The estimation techniques enhance the viability of stochastic binary neurons and their application in real-world tasks requiring conditional computation. This paves the way for more scalable and efficient deep learning models that can dynamically modulate their computational workloads based on input complexity.

In future research, exploring the integration of these stochastic gradient estimation methods into larger and more diverse neural architectures could yield further efficiency gains and performance improvements. Particularly, extending these methods to recurrent neural networks and temporal hierarchies promises to open new avenues for efficient sequence learning and time-series prediction tasks, potentially addressing current bottlenecks in the training of such models.

In conclusion, the paper systematically tackles the estimation challenge in stochastic neurons, providing robust theoretical and empirical contributions that enhance the understanding and practical implementation of conditional computation in deep neural networks.