Label Embedding Network: Learning Label Representation for Soft Training of Deep Networks (1710.10393v1)

Abstract: We propose a method, called Label Embedding Network, which can learn label representation (label embedding) during the training process of deep networks. With the proposed method, the label embedding is adaptively and automatically learned through back propagation. The original one-hot represented loss function is converted into a new loss function with soft distributions, such that the originally unrelated labels have continuous interactions with each other during the training process. As a result, the trained model can achieve substantially higher accuracy and with faster convergence speed. Experimental results based on competitive tasks demonstrate the effectiveness of the proposed method, and the learned label embedding is reasonable and interpretable. The proposed method achieves comparable or even better results than the state-of-the-art systems. The source code is available at \url{https://github.com/lancopku/LabelEmb}.

• The paper introduces LEN that transforms one-hot encoding into soft label distributions, improving correlation measurement and reducing overfitting.
• It employs a dual-output strategy for learning compressed and interpretable label embeddings, leading to significant accuracy improvements.
• Experimental results across vision and NLP tasks validate LEN’s adaptability and faster convergence on datasets like CIFAR-10/100 and text applications.

An Analysis of Label Embedding Network for Soft Training in Deep Neural Networks

This paper introduces the Label Embedding Network (LEN), a novel method designed to optimize label representations in the training processes of deep neural networks. The conventional approach of representing labels with one-hot vectors is critiqued for its discrete distribution and extreme value characteristics, which limit correlation measurement among labels and risk overfitting. This work addresses these challenges by proposing an adaptive learning framework for label embeddings through backpropagation, thus enhancing model accuracy and convergence speed.

The essence of the proposed method lies in softening the rigid one-hot loss function into a loss function with soft distributions. This transformation enables continuous interactions between previously unrelated labels, facilitating a more nuanced understanding of label correlations. Notably, the experiments demonstrate that this approach outperforms existing models in terms of accuracy and speed of convergence across various competitive tasks.

Key Contributions

The paper highlights several innovative contributions:

1. Learning Label Embedding: LEN facilititates the learning of label representations, reducing memory costs substantially through compressed label embeddings. This is particularly beneficial for large-scale tasks with extensive label sets.
2. Interpretable and Reusable Embeddings: The embeddings learned are interpretable, uncovering significant similarities among labels across different domains, such as image and natural language processing tasks. They can be reutilized to train new models efficiently on similar tasks.
3. Versatile Application Across Models: LEN is applicable across diverse neural network architectures, including CNNs, ResNets, and Seq-to-Seq models. The versatility of the method is supported by experimental validation on tasks such as CIFAR-100, CIFAR-10, MNIST, and several NLP applications like text summarization and machine translation.

Detailed Methodological Approach

LEN leverages a dual-output layer strategy to disentangle label discrimination and similarity learning. One output layer focuses on traditional prediction, while the other adapts label embeddings for capturing label similarities. This separation minimizes conflicts between representation tasks, maintaining the model’s discriminative power.

To sustain similarity learning, a softened distribution approach with temperature scaling is employed, dampening the tendency towards overfitting by ensuring that only relevant similarities affect the embedding learning. Furthermore, a re-parameterization technique effectively compresses the labels, enabling efficient memory management without sacrificing learning quality.

Experimental Validation and Results

The research demonstrates the practical efficacy of LEN across various datasets and models:

• Computer Vision Tasks: On CIFAR-100 and CIFAR-10 datasets, LEN yields significant error reduction, 12.4% and 19.5% respectively, while maintaining comparable training times. This suggests improved training efficiency facilitated by refined label interactions.
• Natural Language Processing Applications: In the LCSTS text summarization and IWSLT 2015 machine translation tasks, LEN achieves remarkable improvements in ROUGE and BLEU scores, asserting its capability to handle large-scale, label-intense tasks with compressed embeddings.

Implications and Future Work

The paper outlines several theoretical and practical implications. First, the shift from discrete to continuous label representations could inform novel approaches to data sparsity and feature sharing across similar labels. The practicality of pre-trained, reusable label embeddings posits potential pathways for reducing computational overhead in large datasets.

Future research may expand on optimizing hyperparameters within LEN and explore further integration with other model architectures. Additionally, investigating the use of label embeddings in domains such as semi-supervised and unsupervised learning could further augment the scope and impact of this method.

In conclusion, the paper provides a thorough examination of label embeddings within deep learning, underscoring its potential to refine model training broadly. By facilitating adaptive, interpretable label representations, LEN presents a robust approach to modern challenges in neural network training.