Self-Supervised Generalisation with Meta Auxiliary Learning (1901.08933v3)

Abstract: Learning with auxiliary tasks can improve the ability of a primary task to generalise. However, this comes at the cost of manually labelling auxiliary data. We propose a new method which automatically learns appropriate labels for an auxiliary task, such that any supervised learning task can be improved without requiring access to any further data. The approach is to train two neural networks: a label-generation network to predict the auxiliary labels, and a multi-task network to train the primary task alongside the auxiliary task. The loss for the label-generation network incorporates the loss of the multi-task network, and so this interaction between the two networks can be seen as a form of meta learning with a double gradient. We show that our proposed method, Meta AuXiliary Learning (MAXL), outperforms single-task learning on 7 image datasets, without requiring any additional data. We also show that MAXL outperforms several other baselines for generating auxiliary labels, and is even competitive when compared with human-defined auxiliary labels. The self-supervised nature of our method leads to a promising new direction towards automated generalisation. Source code can be found at https://github.com/lorenmt/maxl.

• The paper introduces MAXL, a dual-network model that automatically generates auxiliary labels to boost supervised learning generalisation.
• It employs a meta-learning approach with an integrated loss function and Mask SoftMax to refine label quality dynamically.
• Experimental results on MNIST, CIFAR-10, and CIFAR-100 show that MAXL outperforms conventional single-task and random labeling strategies.

Self-Supervised Generalisation with Meta Auxiliary Learning

This paper, "Self-Supervised Generalisation with Meta Auxiliary Learning," introduces a novel approach for improving the generalization of supervised learning tasks without requiring additional data through a method termed Meta AuXiliary Learning (MAXL). The concept leverages auxiliary tasks to reinforce the primary task's learning capabilities. This strategy traditionally necessitates auxiliary data labeling, which MAXL ingeniously automates using a dual-network paradigm. By utilizing two neural networks, MAXL dynamically generates appropriate labels for auxiliary tasks, positioning itself as a self-supervised learning method that adapts to any supervised learning task.

Methodology

The MAXL framework involves two interconnected components: a multi-task network and a label-generation network. The former concurrently trains on both the primary and auxiliary tasks, while the latter forecasts appropriate auxiliary labels. This label generation is influenced by the primary task's learning progress, embodying a meta-learning setup. Through an integrated loss function for both networks that adapts to the training performance of the primary task, the system refines auxiliary labels iteratively, leveraging a double gradient—also known as a second-order derivative—a practice common in sophisticated meta-learning paradigms.

Experimental Results

MAXL's efficacy was demonstrated on seven image datasets, each varied in size and complexity, including MNIST, CIFAR-10, and CIFAR-100, among others. The framework consistently outperformed single-task learning approaches across these datasets, despite utilizing identical training data amounts. Furthermore, when benchmarking against auxiliary label generation baselines such as randomly assigned labels and clustering-based labels, MAXL showed superiority, even approaching the performance of human-defined auxiliary labels.

Technical Insights

Two primary innovations underpin the success of MAXL:

1. Dual-Network Architecture: The label-generation network and multi-task network exhibit a mutual dependency, with the former generating optimal auxiliary labels based on the performance of the latter. This intertwining allows for simultaneous improvement of both the label quality and the primary task's performance.
2. Mask SoftMax Functionality: A significant enhancement to typical SoftMax is the Mask SoftMax function introduced in MAXL. This function facilitates hierarchical arrangement where auxiliary classes are associated distinctly with each primary class, streamlining the prediction process and enhancing auxiliary tasks' structure.

The paper presents numerical results showing that MAXL not only improves the baseline learning outcomes but also does so efficiently across different model architectures. The results from cosine similarity analysis reinforce the claim of robust auxiliary task utility, as evidenced by computed gradients maintaining beneficial alignments with primary task objectives.

Implications and Future Directions

The implications of MAXL are twofold: from a theoretical standpoint, the method provides insights into leveraging meta-learning for automatic label generation, reducing methodological reliance on pre-defined feature sets often necessary in auxiliary learning. Practically, it democratizes advanced learning techniques by necessitating only primary task labels, thereby streamlining task setup for new datasets or domains.

Speculatively, the extension of MAXL for broader tasks beyond classification, such as regression, presents a promising avenue. Exploring MAXL's role in unsupervised domains or its potential integration with multi-modal learning frameworks is also a logical extension. However, challenges remain in optimization stability and consistency when moving across vastly different problem spaces without manually structuring the tasks.

Overall, "Self-Supervised Generalisation with Meta Auxiliary Learning" presents a robust approach towards automating and optimizing auxiliary task integration into supervised learning frameworks, opening new pathways in self-supervised learning paradigms. The paper commendably demonstrates how achieving enhanced task generalization is feasible with minimal reliance on auxiliary data curation, marking an exciting development in AI methodologies.