Which Tasks Should Be Learned Together in Multi-task Learning? (1905.07553v4)

Abstract: Many computer vision applications require solving multiple tasks in real-time. A neural network can be trained to solve multiple tasks simultaneously using multi-task learning. This can save computation at inference time as only a single network needs to be evaluated. Unfortunately, this often leads to inferior overall performance as task objectives can compete, which consequently poses the question: which tasks should and should not be learned together in one network when employing multi-task learning? We study task cooperation and competition in several different learning settings and propose a framework for assigning tasks to a few neural networks such that cooperating tasks are computed by the same neural network, while competing tasks are computed by different networks. Our framework offers a time-accuracy trade-off and can produce better accuracy using less inference time than not only a single large multi-task neural network but also many single-task networks.

• The paper presents a framework that groups cooperative tasks on shared networks and separates competitive tasks to minimize negative transfer.
• It employs Early Stopping and Higher Order approximations to efficiently predict multi-task performance and balance time-accuracy trade-offs.
• Results demonstrate significant improvements over traditional single-task and joint training models, especially in resource-constrained AI applications.

Overview of Multi-task Learning Framework for Task Grouping

The paper "Which Tasks Should Be Learned Together in Multi-task Learning?" addresses the critical challenge of determining optimal task groupings in multi-task learning (MTL) to improve performance efficiency. The authors introduce a computational framework that strategically assigns tasks either to a shared neural network—when tasks are cooperative—or to separate networks—when tasks are competitive. This approach balances time-accuracy trade-offs and aims to outperform traditional single large multi-task networks as well as multiple single-task networks, particularly in computer vision applications like robotics and autonomous vehicles.

Context and Motivation

Multi-task learning is advantageous in scenarios requiring multiple predictions from a single network, reducing computational costs and inference time. However, the challenge arises from negative transfer, where the performance of tasks deteriorates when combined, due to competing objectives and interference in optimization landscapes. This paper seeks to resolve the decision process of which tasks should be learned jointly to exploit potential performance synergies and which should be learned separately to avoid conflicts.

Empirical Task Relationship Evaluation

The authors empirically paper task relationships across different settings, varying network sizes, dataset sizes, and task combinations. Their findings reveal that performance synergies in MTL vary significantly based on network capacity and data availability, emphasizing the need for a dynamic assessment of task compatibility rather than static assignments. The paper also contrasts MTL relationships to previously established transfer learning affinities, noting a lack of correlation, which underscores the complexity and specificity of multi-task scenarios.

Task Grouping Framework

To systematize task assignment in MTL, the authors propose a framework evaluating subsets of task groupings to identify optimal configurations under computational constraints. This process involves:

1. Training neural networks for all possible task subsets.
2. Evaluating their performance relative to a computational budget.
3. Selecting configurations that minimize loss while adhering to budgetary limits.

Recognizing the computational expense of training myriad networks, the paper explores two approximation strategies: Early Stopping Approximation (ESA), which predicts performance based on early training results, and Higher Order Approximation (HOA), predicting multi-task performance from lower-order task pairings. These strategies aim to balance the accuracy of prediction with feasible training costs.

Results and Implications

The framework consistently surpasses baseline approaches, such as single-task networks and large-scale joint task training, by leveraging thoughtful task groupings determined through empirical performance data. The framework's flexibility accommodates specific domain requirements and leads to significant improvements in overall task performance, demonstrating the potential for optimal network utilization.

The results indicate that traditional fixed task combinations may not effectively address the nuances of task interactions encountered in varying applications. The insights from this framework present implications for more effective AI deployment in practical scenarios requiring simultaneous task execution under resource constraints.

Conclusion and Future Directions

This research advances the understanding of task relationships in multi-task learning, advocating for a computationally-driven approach to identifying optimal task groupings. Future research directions could involve extending these methodologies to broader AI contexts, exploring the integration of more complex architectures, or incorporating dynamic task weighting and user-defined priorities into the task grouping framework. The work opens pathways to more efficient multi-task systems, with significant relevance to AI's application in resource-constrained environments.