Taskonomy: Disentangling Task Transfer Learning

Abstract: Do visual tasks have a relationship, or are they unrelated? For instance, could having surface normals simplify estimating the depth of an image? Intuition answers these questions positively, implying existence of a structure among visual tasks. Knowing this structure has notable values; it is the concept underlying transfer learning and provides a principled way for identifying redundancies across tasks, e.g., to seamlessly reuse supervision among related tasks or solve many tasks in one system without piling up the complexity. We proposes a fully computational approach for modeling the structure of space of visual tasks. This is done via finding (first and higher-order) transfer learning dependencies across a dictionary of twenty six 2D, 2.5D, 3D, and semantic tasks in a latent space. The product is a computational taxonomic map for task transfer learning. We study the consequences of this structure, e.g. nontrivial emerged relationships, and exploit them to reduce the demand for labeled data. For example, we show that the total number of labeled datapoints needed for solving a set of 10 tasks can be reduced by roughly 2/3 (compared to training independently) while keeping the performance nearly the same. We provide a set of tools for computing and probing this taxonomical structure including a solver that users can employ to devise efficient supervision policies for their use cases.

• The paper introduces a computational framework that maps interdependencies among 26 visual tasks, enabling efficient transfer learning with reduced data.
• It employs task-specific networks and shallow readout transfer models, using ordinal normalization and binary integer programming to optimize task relationships.
• Experiments show that the approach reduces labeled data requirements by about two-thirds while achieving near-optimal performance.

Taskonomy: Disentangling Task Transfer Learning

Overview

The paper "Taskonomy: Disentangling Task Transfer Learning" presented by Zamir et al. introduces a computational framework for modeling the interdependencies among visual tasks to solve the problem of task transfer learning. The proposed method aims to elucidate the underlying structure of visual tasks to optimize the transferability of learned features across different types of tasks, thereby reducing the necessity for extensive supervision and labeled data.

Summary

The core premise of the paper is predicated on the assumption that visual tasks are not mutually exclusive and that there are intrinsic relationships among them. These relationships can be exploited to transfer learned features from one task to another, thus mitigating the requirements for labeled data and computational resources. The authors approach this by computationally mapping the task space via neural networks, thereby constructing a “task taxonomy” (termed as Taskonomy) that delineates which tasks transfer well to others. The framework leverages a dictionary of 26 visual tasks covering 2D, 2.5D, 3D, and semantic categories.

The methodology can be broken down into the following key stages:

1. Task-Specific Modeling: Training task-specific networks with an encoder-decoder structure for each task to extract robust representations.
2. Transfer Modeling: Establishing transfer functions to map representations from one task (source) to another (target). This involves training shallow readout networks that adapt pre-trained features from source tasks to target tasks.
3. Ordinal Normalization: Using Analytic Hierarchy Process (AHP) to normalize the transfer affinities, thus converting performance metrics into a coherent measure of task similarity.
4. Computing the Global Taxonomy: Utilizing Binary Integer Programming (BIP) to derive an optimal global taxonomy that maximizes overall task performance under a specified supervision budget.

The practical outcomes of this study are notably significant. The computational task taxonomy reveals that efficient multi-task learning systems can be created with less data compared to training each task independently. Additionally, the framework demonstrates the feasibility of solving tasks with significantly reduced labeled data while retaining near-optimal performance.

Numerical Results

The paper reports several strong numerical results indicating the effectiveness of the proposed methodology:

• The number of labeled data points required to solve a set of 10 tasks can be reduced by approximately two-thirds compared to training each task independently.
• Detailed performance metrics for 26 different visual tasks showcased the consistent effectiveness of the transfer framework. For instance, the task-specific networks trained using the Taskonomy approach displayed substantial gains over traditional fully supervised networks.

Implications and Future Directions

Practical Implications:

1. Data Efficiency: The presented approach significantly curtails the data demands for training multi-task systems, making it feasible to develop robust systems with limited labeled data.
2. Computational Savings: By leveraging task interdependencies, the computational burden is substantially lowered, which has practical implications in areas with resource constraints.
3. Generalization: The findings are tested across various datasets, affirming the generalizability of the task relationships beyond the training data.

Theoretical Implications:

1. Task Taxonomy Structure: The study advances our understanding of the latent structure of visual tasks, suggesting that tasks can be hierarchically organized based on their functional transferabilities.
2. Optimal Transfer Paths: The proposed BIP framework for optimal transfer learning sets a foundational theory for exploring and formalizing task transfer approaches.

Future Developments:

1. Incorporating Novel Tasks: Extending the task dictionary to include new and emerging visual tasks can further validate and refine the task taxonomy.
2. Non-Visual and Hybrid Tasks: Future work might explore the transferability of visual task features to non-visual domains, such as incorporating visual perception in robotic manipulation.
3. Lifelong and Continual Learning: Integrating this methodology into lifelong learning frameworks can pave the way for systems that dynamically adapt and expand their task-solving capabilities over time.

Conclusion

The paper offers a compelling computational approach to model the space of visual tasks, demonstrating the substantial benefits of structured task transfer learning. With implications spanning data efficiency and computational savings, the proposed Taskonomy framework stands as a promising advancement in multi-task learning and transfer learning paradigms. The methodology's robustness and generalizability suggest a fruitful trajectory for future research in both the practical deployment of AI systems and the theoretical understanding of task interdependencies.