Flan-UL2: 20B Instruction-Tuned LLM

• Flan-UL2 is an instruction-tuned large language model with 20B parameters and a T5-style encoder–decoder backbone, providing a robust framework for diverse NLP tasks.
• It utilizes a Mixture-of-Denoisers pretraining strategy with multiple span-corruption tasks and dynamic task-mode switching, enhancing versatility and sample efficiency.
• The model demonstrates competitive performance in zero-/few-shot learning and chain-of-thought reasoning, achieving balanced compute–performance trade-offs for practical research adoption.

Flan-UL2 is an instruction-tuned LLM based on the UL2 20B parameter encoder–decoder Transformer architecture, further refined through FLAN instruction tuning. It is designed to be broadly effective across a variety of NLP tasks and demonstrates competitive performance in zero- and few-shot learning, chain-of-thought (CoT) prompting, and reasoning benchmarks. Flan-UL2 makes use of a Mixture-of-Denoisers (MoD) pretraining strategy and introduces mechanisms for dynamic task-mode switching. Its publicly released Flax-based T5X checkpoints facilitate reproducibility and adoption by the research community (Tay et al., 2022).

1. Model Architecture

Flan-UL2 utilizes a "vanilla" T5-style encoder–decoder Transformer backbone, mirroring T5 v1.1 with no novel architecture beyond the adoption of Gated Linear Unit (GLU) feed-forward layers and relative-position attention, both of which are already standard in T5. The architecture comprises 32 encoder and 32 decoder layers, a model hidden dimension $d_{\text{model}} = 4096$, feed-forward inner dimension $d_{\text{ff}} = 16384$, and 16 attention heads per layer with head size $256$. All other configurations such as dropout and layer normalization align with T5 v1.1. The network contains approximately 20 billion parameters. Model weights are initialized via UL2’s MoD pretraining objective prior to subsequent FLAN instruction tuning (Tay et al., 2022).

2. Pretraining with Mixture-of-Denoisers (MoD)

UL2 introduces the Mixture-of-Denoisers (MoD) pretraining objective, which unifies multiple span-corruption (denoising) tasks. The total pretraining loss for the MoD objective is given by

$$\mathcal{L}_{\rm MoD}(\theta) = \sum_{i=1}^{K} \alpha_i \mathcal{L}_{i}(x;\theta), \quad \alpha_i = \frac{1}{K}$$

where $K=7$ distinct denoisers, each parameterized by mean span length $\mu$ and corruption rate $r$, are mixed equally ($\alpha_i=1/7$). These denoisers fall into three archetypes, summarized below:

Denoiser	$(\mu, r)$ Configurations	Functional Role
R	(3, 0.15), (8, 0.15)	T5-style span masking (bidirectional context)
S	($L/4$, 0.25), contiguous suffix	Prefix LM, masks a final span ("seq2seq")
X	(3, 0.50), (8, 0.50), (64, 0.15), (64, 0.50)	Extreme masking, simulates open-ended generation

During pretraining, a denoiser is selected uniformly for each batch, exposing the model to a spectrum of bidirectional, prefix-only, and high-corruption scenarios, thus enhancing its universality and sample efficiency (Tay et al., 2022).

3. Task Mode-Switching Mechanisms

UL2 introduces a system of special sentinel tokens to indicate the active denoising mode, which persist through pretraining and can be employed in downstream prompting. The tokens and their canonical associations are:

• [NLG] for the X-denoiser (extreme, open-generation), used for summarization and data-to-text
• [NLU] for the R-denoiser (regular, understanding), used for classification or QA
• [S2S] for the S-denoiser (prefix, seq2seq translation)

This discrete prompting technique allows dynamic adaptation of the model’s internal denoising strategy to better match the requirements of a given downstream task. Empirical results indicate that this approach enhances performance, particularly in multimodal and cross-task generalization regimes (Tay et al., 2022).

4. FLAN Instruction-Tuning Procedure

After MoD pretraining, UL2 20B is subjected to the FLAN instruction-tuning regime, as described by Chung et al. (2022), involving approximately 1,800 prompted tasks. Key aspects of the FLAN tuning process include:

• Initialization from MoD-pretrained UL2 20B, with input mode tokens removed for instruction "purification"
• 100,000 further training steps, sequence length (encoder/decoder) of 1,024 tokens
• Adafactor optimizer, inverse square-root learning rate scheduling, $d_{\text{ff}} = 16384$0 learning rate
• Reuse of FLAN’s standard prompt templates; no additional templates beyond mode token handling
• Batch size of ~256 examples (hardware-dependent)

The result is an instruction-aligned version of UL2, referred to as Flan-UL2, capable of following natural language prompts across varied tasks (Tay et al., 2022).

5. Empirical Performance Characteristics

MMLU & Big-Bench Hard (BBH)

Flan-UL2 20B demonstrates strong results on the MMLU and BBH benchmarks in zero- and few-shot evaluation. Relative to FLAN-T5 XXL (11B) and FLAN-PaLM (62B, 540B), Flan-UL2 20B achieves competitive or superior results given its parameter count:

Model	MMLU (dev/test)	BBH
FLAN-T5 XXL 11B	54.5 / 53.7	45.3
FLAN-PaLM 62B	54.5 / 53.7¹	47.5
FLAN-PaLM 540B	73.5	57.9
Flan-UL2 20B (best)	55.6 / 56.2†	46.0

(*) dev/test; † "best checkpoint"; ¹ varies by report.

Flan-UL2 20B outperforms FLAN-T5 XXL by approximately +1.8 percentage points on MMLU(dev) and +0.7 on BBH, and is within 1–2 points of FLAN-PaLM 62B with one-third the parameters.

Chain-of-Thought (CoT)

Although FLAN instruction tuning does not explicitly target CoT prompting, Flan-UL2 inherits robust stepwise reasoning capability from UL2 pretraining. On direct and CoT evaluation for MMLU and BBH:

Model	MMLU-CoT	BBH-CoT
FLAN-T5 XXL 11B	48.6	41.4
Flan-UL2 20B	52.2	42.7
FLAN-PaLM 62B	56.9	44.9

Flan-UL2 20B achieves a +7-point improvement over FLAN-T5 XXL for MMLU CoT.

Zero-shot SuperGLUE and Summarization (XSum)

• UL2 20B achieves 65.2 avg on zero-shot SuperGLUE, outperforming GPT-3 175B (61.2).
• Flan-UL2 20B (post-FLAN) attains a zero-shot SuperGLUE score of ≈64, matching or slightly exceeding FLAN-T5 XXL with far fewer parameters.
• For one-shot summarization (XSum, Rouge-2), Flan-UL2 20B achieves 8.6, tripling T5-XXL+LM (2.3) and surpassing LaMDA 137B (5.4).

6. Compute–Performance Trade-offs and Practical Adoption

Flan-UL2 achieves a favorable balance of parameter count and downstream task performance, consistently operating on the Pareto frontier for general benchmarks. At 20B parameters, its performance exceeds 11B T5 variants on FLAN-tuned tasks and matches or comes close to 62B FLAN-PaLM models on few-shot MMLU and BBH, using just one-third of the compute budget. The model is especially efficient for open-domain generation, few-shot generalization, and tasks requiring reasoning, as evidenced by its strong CoT and SuperGLUE results.

7. Usage Guidelines and Availability

For best results, the following prompting techniques are recommended:

• For zero-/one-shot NLU (classification, QA), prepend [NLU] or use FLAN-style instruction formatting.
• For zero-/one-shot seq2seq tasks (translation, summarization), prepend [S2S] or [NLG], or follow FLAN templates.
• For chain-of-thought tasks, append “Let’s think step by step.” and aggregate CoT samples if possible using self-consistency.
• Strict adherence to FLAN’s calibration protocols is advised for optimal performance.

Flan-UL2 20B and its supporting codebase are publicly available as Flax-based T5X checkpoints, facilitating wide-scale experimentation and benchmarking in the research community (Tay et al., 2022).

---

References:

Tay et al., 2022, UL2: Unifying Language Learning Paradigms (Tay et al., 2022). Chung et al., 2022, FLAN (referenced therein).