MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learning (2001.06902v5)

Abstract: In this paper, we argue about the importance of considering task interactions at multiple scales when distilling task information in a multi-task learning setup. In contrast to common belief, we show that tasks with high affinity at a certain scale are not guaranteed to retain this behaviour at other scales, and vice versa. We propose a novel architecture, namely MTI-Net, that builds upon this finding in three ways. First, it explicitly models task interactions at every scale via a multi-scale multi-modal distillation unit. Second, it propagates distilled task information from lower to higher scales via a feature propagation module. Third, it aggregates the refined task features from all scales via a feature aggregation unit to produce the final per-task predictions. Extensive experiments on two multi-task dense labeling datasets show that, unlike prior work, our multi-task model delivers on the full potential of multi-task learning, that is, smaller memory footprint, reduced number of calculations, and better performance w.r.t. single-task learning. The code is made publicly available: https://github.com/SimonVandenhende/Multi-Task-Learning-PyTorch.

• The paper introduces a multi-scale distillation unit that captures unique task affinities across different receptive fields.
• It presents a feature propagation module that refines predictions by integrating distilled info from lower scales.
• Experiments on PASCAL and NYUD-v2 validate the model's efficacy with significant performance improvements over baselines.

Overview of MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learning

The paper "MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learning" by Simon Vandenhende, Stamatios Georgoulis, and Luc Van Gool introduces a novel architecture to address the challenges associated with multi-task learning (MTL). The proposed model, MTI-Net, emphasizes the significance of multi-scale task interactions, challenging the prevailing assumption that task interactions remain consistent across scales.

Research Motivation and Context

Multi-task learning enables concurrent solving of multiple tasks, presenting advantages such as reduced memory usage and increased inference speed due to shared representations. Existing models, however, often face negative transfer, where unrelated tasks interfere, degrading individual task performance. The critical insight of this paper is that task interactions can differ across scales, impacting the distillation process in multi-task settings. This observation deviates from prior approaches that treat task interactions as scale-invariant.

Methodological Contributions

The MTI-Net architecture advances MTL through three main innovations:

1. Multi-Scale Multi-Modal Distillation: Task interactions are modeled explicitly at various scales using a multi-scale distillation unit. This allows capturing unique task affinities specific to different receptive field sizes.
2. Feature Propagation Module: Distilled task information propagates from lower to higher scales, counteracting the limited field of view that often constrains predictions at higher scales. The module enhances feature quality by incorporating distilled information from preceding scales.
3. Feature Aggregation: Final task predictions emerge from aggregating refined features across all scales. This process utilizes feature representation from multiple levels, enhancing the richness of task-specific information.

Experimental Validation

Extensive experiments on PASCAL and NYUD-v2 datasets evaluate MTI-Net's performance against state-of-the-art models. The results highlight significant improvements in task predictions, surpassing single-task baselines. On the PASCAL dataset, MTI-Net demonstrated a $+2.74\%$ improvement, while on the NYUD-v2 dataset, the architecture achieved a $+10.91\%$ gain. These results underscore the utility of integrating multi-scale task interactions within the MTL framework.

Implications and Future Directions

MTI-Net's approach resolves the scale-specific challenges by embracing a flexible architecture that adapts task interactions across scales. The implications are two-fold: practically, the model achieves superior performance across diverse and complex task sets; theoretically, it challenges the conventional paradigm of invariant task interactions within multi-task learning systems.

Future research could explore the adaptability of MTI-Net to a wider array of tasks and domains, potentially integrating additional auxiliary tasks or extending the model to leverage more complex backbone architectures. Furthermore, refining the feature propagation mechanisms might yield even greater improvements in handling tasks with varying levels of granularity.

In conclusion, MTI-Net represents a significant step forward in multi-task learning, providing a robust framework for efficiently managing the interplay of task interactions at multiple scales, thereby advancing both the efficacy and understanding of multi-task neural networks.