Zero-bias autoencoders and the benefits of co-adapting features (1402.3337v5)

Abstract: Regularized training of an autoencoder typically results in hidden unit biases that take on large negative values. We show that negative biases are a natural result of using a hidden layer whose responsibility is to both represent the input data and act as a selection mechanism that ensures sparsity of the representation. We then show that negative biases impede the learning of data distributions whose intrinsic dimensionality is high. We also propose a new activation function that decouples the two roles of the hidden layer and that allows us to learn representations on data with very high intrinsic dimensionality, where standard autoencoders typically fail. Since the decoupled activation function acts like an implicit regularizer, the model can be trained by minimizing the reconstruction error of training data, without requiring any additional regularization.

• The paper identifies that negative biases in hidden units, induced by regularization, can impair learning in high-dimensional settings.
• It introduces two novel activation functions, TRec and TLin, to effectively separate sparsity control from linear encoding.
• Empirical results show that zero-bias autoencoders improve image and video classification, boosting performance on datasets like CIFAR-10 and Hollywood2.

Zero-bias Autoencoders and the Benefits of Co-adapting Features

The paper "Zero-bias Autoencoders and the Benefits of Co-adapting Features" presents a comprehensive examination of the impact of hidden unit biases within autoencoders and proposes innovative methods to address these effects. Authored by Kishore Konda, Roland Memisevic, and David Krueger, the paper reveals significant insights into how bias values can affect the ability of autoencoders to learn representations of data, particularly in contexts where data possess high intrinsic dimensionality.

Main Contributions

The paper identifies the tendency of hidden unit biases to become large and negative during the regularized training of autoencoders. It argues that these negative biases, while promoting sparsity and restricting model capacity, can be detrimental to the representation of complex, high-dimensional data. The authors propose two novel activation functions—Truncated Rectified (TRec) and Threshold Linear (TLin)—to disentangle the dual role of hidden units: selecting weight vectors for reconstruction and determining the coefficients of weight vectors. These functions allow the model to separate the sparsity-promoting selection mechanism from the linear encoding necessary for representing complex data structures.

Empirical Evaluation

The paper provides empirical evidence from several experiments:

1. CIFAR-10 Dataset: Notably, the zero-bias autoencoders (ZAE) achieved superior performance in image classification tasks, especially as the number of hidden units increased. This improvement was consistent across various preprocessing methods.
2. Video Data: The TRec and TLin autoencoders demonstrated the capacity to learn meaningful representations from synthetic video data of rotating random dots, a task traditionally reserved for more complex bilinear models.
3. Hollywood2 Action Recognition: ZAE models outperformed traditional approaches in recognizing actions from video data, suggesting their efficacy in handling real-world datasets with high intrinsic dimensionality.

The experimental results consistently show that linear encoding, enabled by zero-bias activation functions, improves model performance by allowing collaborative interaction between hidden units over large regions of input space.

Implications and Future Directions

The introduction of zero-bias autoencoders has several implications. Practically, the proposed methods can enhance feature learning for datasets where high intrinsic dimensionality poses a significant challenge. Theoretically, they suggest revisiting existing models, possibly interpreting the success of dropout and gating mechanisms through the lens of co-adaptation and linear encoding.

Future research may investigate further applications of ZAEs in diverse contexts, including complex image and video datasets, natural language processing, and multi-modal data representation. Additionally, exploring the interplay between different types of regularization and the zero-bias framework could yield further advancements in the development of robust autoencoder architectures.

In conclusion, this paper offers a compelling argument for rethinking bias in autoencoders, providing valuable insights into feature representation and the dynamics of machine learning models in high-dimensional settings.