UL2: Unifying Language Learning Paradigms (2205.05131v3)

Abstract: Existing pre-trained models are generally geared towards a particular class of problems. To date, there seems to be still no consensus on what the right architecture and pre-training setup should be. This paper presents a unified framework for pre-training models that are universally effective across datasets and setups. We begin by disentangling architectural archetypes with pre-training objectives -- two concepts that are commonly conflated. Next, we present a generalized & unified perspective for self-supervision in NLP and show how different pre-training objectives can be cast as one another and how interpolating between different objectives can be effective. We then propose Mixture-of-Denoisers (MoD), a pre-training objective that combines diverse pre-training paradigms together. We furthermore introduce a notion of mode switching, wherein downstream fine-tuning is associated with specific pre-training schemes. We conduct extensive ablative experiments to compare multiple pre-training objectives and find that our method pushes the Pareto-frontier by outperforming T5 & GPT-like models across multiple diverse setups. By scaling our model up to 20B parameters, we achieve SOTA performance on 50 well-established supervised finetuning based NLP tasks. Our model also achieve strong results at in-context learning, outperforming 175B GPT-3 on zero-shot SuperGLUE and tripling the performance of T5-XXL on one-shot summarization. On 0-shot MMLU, UL2 20B outperforms T0 and T5 models. UL2 20B also works well with chain-of-thought prompting and reasoning, making it an appealing choice for research into reasoning at a small to medium scale of 20B parameters. Finally, we apply FLAN instruction tuning to the UL2 20B model, achieving MMLU and Big-Bench scores competitive to FLAN-PaLM 62B. We release Flax-based T5X checkpoints for the UL2 20B & Flan-UL2 20B.

Essay on "UL2: Unifying Language Learning Paradigms"

The paper "UL2: Unifying Language Learning Paradigms" presents a framework aimed at bridging the gap between diverse NLP tasks through a novel pre-training methodology. The primary objective is to propose a unified model that performs effectively across various downstream tasks without the necessity of designing task-specific architectures or objectives. This work introduces the Mixture-of-Denoisers (MoD), a pre-training objective that blends multiple denoising strategies, and introduces mode switching, allowing for dynamic adaptation based on the requirements of downstream tasks.

Core Contributions

1. Separation of Architecture from Objectives: One insightful perspective of this work is the separation of architectural decisions from the pre-training objectives. The authors argue that while some pre-trained models are often strongly coupled with their underlying architecture (e.g., encoder-only or encoder-decoder), the backbone architecture should primarily be an efficiency trade-off rather than affecting the essence of pre-training.
2. Mixture-of-Denoisers (MoD): The MoD incorporates three denoising strategies:
   • R-Denoiser (Regular Denoising): Focuses on standard span corruption as seen in models like T5.
   • S-Denoiser (Sequential Denoising): Aligns with prefix LLMing, reminiscent of GPT-like objectives but with bidirectional context for inputs.
   • X-Denoiser (Extreme Denoising): Intends to simulate challenging text generation scenarios with high corruption rates and longer spans, providing an interpolation between standard span corruption and LLMing.
3. Mode Switching: The concept here involves associating specific learning modes with task-specific prompts. During pre-training, models are exposed to sentinel tokens that activate specific modes (R, S, or X), and this behavior is leveraged during fine-tuning to adaptively enhance model performance.

Experimental Insights

The experiments reveal that UL2, utilizing the proposed MoD, consistently outperforms existing models like T5 and GPT across various tested tasks, including both supervised and one-shot setups. Noteworthy results include UL2's superior performance in supervised fine-tuning and one-shot learning scenarios. A significant takeaway is the improvement in task performance when incorporating mode switching, which emphasizes the flexibility of UL2's pre-training framework.

The experiments also highlight the challenge of applying a single architecture to vastly different tasks. By decoupling architecture and objective, UL2 demonstrates that a universal pre-training objective, adaptable to diverse tasks, provides a strong baseline. The use of mixture models, as opposed to single-objective models, improves generalization and task adaptation.

Implications and Future Directions

The implications of UL2 are multifaceted, impacting both practical deployment of NLP systems and theoretical understanding of pre-training models. Practically, a universal model like UL2 can reduce computational costs and efforts associated with training and maintaining multiple models for different tasks. Theoretically, the effectiveness of MoD and mode switching advances the discussion on how best to formulate pre-training objectives for LLMs.

Future developments could explore scaling UL2 to even larger models, drawing insights into scaling laws and emergent capabilities at increased parameter counts. Additionally, further exploration into the integration of task-specific prompts could refine mode switching mechanisms, potentially enhancing the model's ability to leverage context more effectively across tasks.

In conclusion, the UL2 framework represents a significant stride in unifying language learning paradigms by integrating multiple pre-training objectives into a coherent and adaptable model, marking a substantial contribution to the field of natural language processing.