---
title: 'Flan-PaLM: Instruction-Tuned PaLM'
url: https://www.emergentmind.com/topics/flan-palm
type: topic
---

# Flan-PaLM: Instruction-Tuned PaLM

Flan-PaLM is an instruction-tuned variant of the PaLM (Pathways Language Model) large language model, designed to maximize generalization and reasoning through diverse, large-scale instruction fine-tuning, with prominent applications in complex multi-step reasoning domains such as medicine and theory of mind. Developed by integrating a heterogeneous mixture of tasks, prompt formats, and advanced prompt-tuning methods, Flan-PaLM achieves state-of-the-art results across multiple benchmarks and demonstrates nuanced performance advances in both clinical and general cognitive inference domains.

## 1. Model Architecture and Instruction Tuning Paradigm

Flan-PaLM is based on the PaLM architecture—a dense, decoder-only, transformer-based LLM—with the Flan (Fine-tuned Language Net) instruction-tuning framework applied at very large scale. The instruction-tuning process involves supervised fine-tuning on a composite of over 1,800 diverse, prompt-driven tasks in various styles: zero-shot, few-shot, and chain-of-thought (CoT). Mixed prompt training is a core methodological component, with prompt templates sampled randomly at training time to ensure the model is exposed to multiple ways of encoding tasks.

A key enrichment method is “input inversion,” which doubles data diversity by including both original and reversed (output-to-input) (input, output) task pairs. Mixture balancing ensures no single domain or prompt type dominates, and special weighting is assigned to tasks known to enhance generalization (notably from the Flan 2021 and T0-SF datasets) [2301.13688]. This approach builds a model robust to unseen instructions and capable of meta-generalization.

## 2. Prompting Strategies and Performance on Benchmarks

Flan-PaLM models employ a mixture of few-shot, chain-of-thought, and self-consistency prompting to maximize reasoning performance. On the MultiMedQA benchmark [2212.13138]—comprising professional exam QA (MedQA, MedMCQA), research QA (PubMedQA), and MMLU clinical topics—Flan-PaLM (540B) achieves:

| Dataset          | Flan-PaLM Accuracy | Previous SOTA | Improvement |
|------------------|--------------------|---------------|-------------|
| MedQA (USMLE)    | 67.6%              | 50.3% (PubMedGPT) | +17.3%      |
| MedMCQA          | 57.6%              | 52.9% (Galactica) | +4.7%       |
| PubMedQA         | 79.0%              | 78.2% (BioGPT)    | +0.8%       |
| MMLU Clinical    | 83.5%-84.0%        | —             | —           |

Scaling (from 8B to 540B parameters) produces approximately a 2× gain in clinical QA accuracy. The ability to encode and perform multi-step clinical reasoning is driven by both model size and the multi-format instructional training provided by Flan.

Performance improvements are not limited to multiple-choice settings. In higher-order theory-of-mind (ToM) reasoning [2405.18870], Flan-PaLM attains 84% aggregate accuracy on order 2–6 tasks in the MoToMQA suite, approaching human adult performance (90%) and closely following GPT-4 (89%). Notably, Flan-PaLM exactly matches 100% on second/third-order ToM statements, where recursive attribution of mental states is required.

## 3. Instruction Prompt Tuning and Domain Alignment

Despite strong results in structured QA, Flan-PaLM exhibits limitations in long-form generation for domain-critical contexts. A parameter-efficient solution, “instruction prompt tuning,” is introduced for targeted domain alignment [2212.13138]. This method precedes natural language hard prompts with learnable “soft prompts”—continuous vector embeddings—optimized using in-domain, clinician-authored exemplars. For the 540B parameter model, a 100-token soft prompt (~1.84M trainable parameters) is adapted to consumer medical QA using as few as 40 demonstration cases, augmented from datasets (HealthSearchQA, MedicationQA, LiveQA).

The input to Flan-PaLM becomes:

\[
x_{\text{input}} = [P; x_{\text{hard prompt}}; \text{query}]
\]

where \(P \in \mathbb{R}^{L \times d}\) serves as the soft prompt matrix. These parameters are optimized using standard first-order methods (AdamW), supporting rapid tuning without updating the main network.

This hybrid tuning model (published as Med-PaLM) nearly eliminates harmful response risk (dropping from 29.7% for Flan-PaLM to 5.9%—on par with clinicians) and raises factual alignment to over 92%, compared to 61.9% for untuned Flan-PaLM output.

## 4. Human Evaluation Frameworks and Observed Limitations

To quantify safety, factuality, comprehension, and reasoning, a multi-axis human evaluation protocol is implemented [2212.13138]. Outputs are judged by clinician and lay panels on:

- Agreement with clinical/scientific consensus
- Harm/bias potential
- Retrieval and recall accuracy
- Evidence of correct “chain-of-thought” reasoning
- Helpfulness and query intent satisfaction (non-expert raters)

While Med-PaLM (prompt-tuned Flan-PaLM) approaches clinicians in factual and safety axes (92.9% factual grounding vs. 92.6% for clinicians, 5.9% vs. 5.7% for “harmful” answers), generic Flan-PaLM lags in both safety and retrieval. Lay ratings show Med-PaLM approaches 80% helpfulness (vs. 91% for clinicians; Flan-PaLM: ~60%).

Notable limitations include increased verbosity or inclusion of extraneous details post-tuning, and, crucially, persistent inferiority in overall long-form answer quality compared to experts. Human evaluation reveals that large LLMs still struggle with precision, error handling, and contextualization in open-ended critical domains.

## 5. Impact of Training Data and Prompt Diversity

Extensive ablation studies [2301.13688] show that mixture balancing, the inclusion of CoT-style tasks, input inversion, and the presence of both zero-shot and few-shot examples are essential for the generalization characteristics of Flan-PaLM. Compositional prompt learning is modeled as:

\[
L_{total} = \alpha \cdot L_{\text{zero-shot}} + \beta \cdot L_{\text{few-shot}} + \gamma \cdot L_{\text{CoT}}, \quad \alpha + \beta + \gamma = 1
\]

Empirically, adding just 5–10% few-shot samples to a largely zero-shot mix yields ≥2% performance gains across held-in and held-out evaluations; CoT prompts drive ≥10% boosts in reasoning tasks. Performance peaks at ~70.2% on held-in tasks with 25% few-shot mix and at 45% on MMLU with a 50% few-shot mix.

This illustrates that robust instruction-tuning frameworks—deep prompt mixture coverage, inversion, and task-balancing—are key to strong LLM generalization.

## 6. Computational Efficiency and Resource Utilization

Instruction-tuned models such as Flan-PaLM and Flan-T5 converge faster and to higher accuracy when transferred to new tasks compared to similar-sized, non-instruction-tuned baselines [2301.13688]. Starting from a Flan-init checkpoint reduces required downstream compute. In single-task fine-tuning, Flan-T5 achieves state-of-the-art results with fewer training steps, due to instruction-tuning acting as a strong task-agnostic prior.

In domain-adaptation scenarios (e.g., medical QA), soft prompt-tuning enables full-domain alignment with only 1.84M parameters and a handful of expert demonstrations, avoiding the resource costs of full parameter re-training or large-scale domain-specific corpora.

## 7. Implications, Applications, and Future Directions

Flan-PaLM's capability for encoding complex domain knowledge and near-adult-level theory-of-mind reasoning [2405.18870] makes it a compelling architecture for safety-critical applications—clinical decision support, adaptive dialog systems, and user intention modeling. Nonetheless, observed gaps in long-form open-ended tasks, especially risk of extraneous information and minor factual errors, emphasize the need for:

- More robust grounding in external up-to-date sources (rather than static pre-training)
- Reliable uncertainty quantification and answer deferral mechanisms (e.g., leveraging self-consistency)
- Refined multi-perspective human evaluation, including fairness, equity, and bias audits across diverse populations
- Hybrid retrieval-generation architectures and continued research on parameter-efficient adaptation

Ethical concerns are salient: higher-order ToM capacities in models like Flan-PaLM enable nuanced inference of intentions and beliefs, which can be beneficial in adaptive interfaces but also pose manipulation risks if unchecked. Therefore, rigorous technical and procedural guardrails are central to future deployment.

## References

- Large Language Models Encode Clinical Knowledge [2212.13138]
- The Flan Collection: Designing Data and Methods for Effective Instruction Tuning [2301.13688]
- LLMs achieve adult human performance on higher-order theory of mind tasks [2405.18870]
- Flacuna: Unleashing the Problem Solving Power of Vicuna using FLAN Fine-Tuning [2307.02053]

Flan-PaLM exemplifies the intersection of scale, diverse instruction-tuning, and prompt-adaptation as the principal axis for high-performance, generalizable large language models, especially in rigorous, high-stakes inferential domains.

Source: https://www.emergentmind.com/topics/flan-palm