Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters & Less Data (2009.09139v3)

Abstract: Multi-Task Learning (MTL) networks have emerged as a promising method for transferring learned knowledge across different tasks. However, MTL must deal with challenges such as: overfitting to low resource tasks, catastrophic forgetting, and negative task transfer, or learning interference. Often, in NLP, a separate model per task is needed to obtain the best performance. However, many fine-tuning approaches are both parameter inefficient, i.e., potentially involving one new model per task, and highly susceptible to losing knowledge acquired during pretraining. We propose a novel Transformer architecture consisting of a new conditional attention mechanism as well as a set of task-conditioned modules that facilitate weight sharing. Through this construction (a hypernetwork adapter), we achieve more efficient parameter sharing and mitigate forgetting by keeping half of the weights of a pretrained model fixed. We also use a new multi-task data sampling strategy to mitigate the negative effects of data imbalance across tasks. Using this approach, we are able to surpass single task fine-tuning methods while being parameter and data efficient (using around 66% of the data for weight updates). Compared to other BERT Large methods on GLUE, our 8-task model surpasses other Adapter methods by 2.8% and our 24-task model outperforms by 0.7-1.0% models that use MTL and single task fine-tuning. We show that a larger variant of our single multi-task model approach performs competitively across 26 NLP tasks and yields state-of-the-art results on a number of test and development sets. Our code is publicly available at https://github.com/CAMTL/CA-MTL.

Overview of Conditionally Adaptive Multi-Task Learning in NLP

The paper "Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters and Less Data" proposes a novel method to enhance the effectiveness and efficiency of multi-task learning (MTL) in NLP. Recognizing the challenges inherent in MTL, such as overfitting to low-resource tasks, catastrophic forgetting, and negative task transfer, the authors introduce a method that aims to mitigate these issues through a parameter-efficient transfer learning approach.

Key Contributions

1. Conditional Transformer Architecture: The proposed method introduces a task-conditioned Transformer architecture. This architecture includes a novel conditional attention mechanism and a set of task-conditioned modules that promote efficient weight sharing and mitigate catastrophic forgetting by keeping half of the pretrained model weights fixed.
2. Multi-Task Data Sampling: To address data imbalance and ensure robust learning across tasks, a new data sampling strategy based on uncertainty is employed. This strategy allows for more appropriate task sampling and alleviates the negative impacts of data imbalances, improving overall model generalization.
3. Performance Gains: The model outperforms other BERT-based methodologies on the GLUE benchmark. Notably, the 8-task model surpasses other adapter methods by 2.8%, and the 24-task model shows superior performance with a 0.7-1.0% improvement over both traditional MTL and single-task fine-tuning methods.

Methodological Insights

The core methodological innovation lies in the use of task-conditioned modules within a Transformer architecture. The approach involves several key components:

• Conditional Attention: This involves a block-diagonal mechanism that adapts the attention process to be task-specific.
• Conditional Alignment: A module that helps align and modulate input representations according to the specific task, not requiring separate alignment matrices for each task.
• Conditional Layer Normalization (CLN) and Conditional Bottleneck: These components further enhance task-specific modulation, allowing the model to adaptively reconfigure its internals to fit the particular needs of various tasks without excessive parameter overhead.

Implications and Future Directions

The implications of this research are significant in both the theoretical and practical dimensions of NLP and MTL. The approach not only demonstrates the ability to share parameters efficiently across multiple tasks, thereby reducing the need for resource-intensive individual task models, but it also opens pathways for more scalable and robust NLP systems that can generalize across diverse linguistic tasks.

Future research could explore the extension of this framework to even larger sets of tasks, particularly those incorporating diverse linguistic phenomena or languages. Additionally, examining the interaction between task characteristics and the conditioning mechanism could yield insights that refine task adaptation strategies. Further exploration of the proposed uncertainty-based data sampling could also provide richer understanding, allowing for dynamic adjustment mechanisms in training strategies that could minimize retraining time and computational resources.

This work contributes an innovative perspective to MTL in NLP, suggesting a promising direction for developing more parameter-efficient and adaptable LLMs capable of handling a wide variety of tasks with improved efficacy.