Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics

Abstract: Numerous deep learning applications benefit from multi-task learning with multiple regression and classification objectives. In this paper we make the observation that the performance of such systems is strongly dependent on the relative weighting between each task's loss. Tuning these weights by hand is a difficult and expensive process, making multi-task learning prohibitive in practice. We propose a principled approach to multi-task deep learning which weighs multiple loss functions by considering the homoscedastic uncertainty of each task. This allows us to simultaneously learn various quantities with different units or scales in both classification and regression settings. We demonstrate our model learning per-pixel depth regression, semantic and instance segmentation from a monocular input image. Perhaps surprisingly, we show our model can learn multi-task weightings and outperform separate models trained individually on each task.

• The paper’s main contribution is a framework that uses homoscedastic uncertainty to automatically weigh task-specific losses.
• It employs probabilistic modeling to balance semantic segmentation, instance segmentation, and depth regression from a single RGB image.
• Experimental results on CityScapes show improved performance, with metrics like 78.5% IoU and 21.6% AP, outperforming manual tuning methods.

Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics

The paper "Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics" by Alex Kendall, Yarin Gal, and Roberto Cipolla presents a novel approach for optimizing multi-task deep learning. The core contribution lies in employing homoscedastic task uncertainty to automatically determine the relative weighting of multiple task-specific loss functions. This approach aims to simultaneously learn diverse outputs such as semantic segmentation, instance segmentation, and pixel-wise depth regression from a single monocular RGB input image.

Introduction

Multi-task learning (MTL) utilizes a shared representation to enhance learning efficiency and prediction accuracy by concurrently addressing multiple objectives. This is particularly valuable in domains such as computer vision, where scene understanding involves integrated geometric and semantic comprehension.

Traditional methods for MTL often use a weighted sum of loss functions with manually tuned or heuristically chosen weights. Such manual tuning is both resource-intensive and suboptimal. The paper under discussion introduces a principled framework that addresses this limitation by incorporating task-dependent homoscedastic uncertainty into the loss function, thereby optimizing the balance between different loss components adaptively during training.

Methodology

The paper advances a multi-task learning framework based on probabilistic modelling. Specifically, it leverages homoscedastic uncertainty, which measures the inherent noise in each task independently of the input data. This uncertainty helps in dynamically adjusting the contribution of each task's loss function.

Multi-Task Likelihoods

Development of the homoscedastic uncertainty model starts with formulating the likelihood for individual tasks. For regression tasks, a Gaussian likelihood is assumed, while for classification tasks, a scaled softmax function models the likelihood. The total likelihood for multi-task outputs, assuming independence between tasks, is factored into individual likelihoods. The maximization of the log-likelihood translates into a loss function where the inverse of the task-specific uncertainties weigh the loss for each task.

Homoscedastic Uncertainty Interpretation

Homoscedastic uncertainty captures task-specific variance. For a model predicting separate outputs for regression and classification, the combined loss function integrates the individual task losses weighted by the inverse of their respective uncertainties. The formulation ensures that tasks with higher uncertainty contribute less to the overall loss, while those with lower uncertainty contribute more.

Experimental Validation

The methodology was validated using the CityScapes dataset, which provides diverse annotations suitable for semantic segmentation, instance segmentation, and depth regression tasks. The proposed multi-task model was benchmarked against state-of-the-art single-task and multi-task learning models.

Results

Empirical results demonstrated the significant advantages of the proposed approach. Training with homoscedastic uncertainty weights outperformed models with manually tuned or uniform task weights.

• Semantic Segmentation: The model achieved an Intersection over Union (IoU) of 78.5%, surpassing several state-of-the-art approaches built for this task alone.
• Instance Segmentation: With an AP (average precision) of 21.6%, the results were competitive compared to dedicated instance segmentation models.
• Depth Regression: A mean error of 2.92px was recorded, indicating robust depth estimation.

Overall, the multi-task approach yielded superior performance in all tasks compared to single-task models, evidencing the effectiveness of shared representations enhanced by uncertainty-weighted loss balancing.

Conclusion and Future Directions

The utilization of homoscedastic uncertainty as a dynamic weighting mechanism for multi-task learning loss functions presents a significant advancement. This framework eliminates the need for exhaustive manual tuning of task weights, offering a scalable and robust solution to MTL optimization.

Future research directions could explore various aspects of this framework:

1. Task Synergy Assessment: Investigating how different tasks influence each other and their synergistic impact on the learned representation.
2. Optimal Point of Network Splitting: Determining the most effective network depth for separating shared and task-specific layers.
3. Extended Multi-Task Models: Applying this approach to more complex MTL settings, including additional tasks like object detection and motion estimation.

In conclusion, this paper sets a solid foundation for using task uncertainty as a dynamic weighting mechanism in multi-task deep learning. The practical implications of this approach extend to any domain requiring efficient learning of multiple objectives, marking a step forward in the development of integrated intelligent systems.