---
title: Super-Natural Instructions Benchmark
url: https://www.emergentmind.com/topics/super-natural-instructions-benchmark
type: topic
---

# Super-Natural Instructions Benchmark

The Super-NaturalInstructions (Sup-NatInst) benchmark is a large-scale evaluation suite designed to enable rigorous assessment of cross-task generalization in natural language processing (NLP) models under natural language task instruction. Introduced by Wang et al. (2022), Sup-NatInst provides 1,616 diverse NLP tasks across 76 task types and 55 languages, each annotated with expert-written declarative instructions, positive/negative examples, and millions of input–output instances. Sup-NatInst addresses critical questions regarding the ability of pretrained language models to generalize to unseen tasks given only task instructions, providing a public dataset, model, and leaderboard to drive reproducible research on instruction-following models [2204.07705]. Subsequent work by Mishra et al. demonstrates how systematically reformatting these instructions as pseudo-code further enhances the performance of large language models (LLMs) [2305.11790].

## 1. Motivation and Objectives

Sup-NatInst was conceived in response to limitations of earlier instruction-tuning resources, such as small task pools (e.g., 61 tasks in NaturalInstructions v1.0) and inconsistent task definitions lacking positive/negative demonstration. The principal objectives are:

- To ascertain which factors—model scale, instruction tuning, task diversity—most strongly influence cross-task generalization in LMs.
- To overcome constraints of prior benchmarks by offering ≈1,600 tasks with unified, expert-authored instructions across a comprehensive task taxonomy.
- To foster research on zero-shot and few-shot generalization, especially when evaluating models on tasks held out during training.
- To provide a reproducible framework including a standardized dataset, baselines, model checkpoints (notably Tk-Instruct), and a leaderboard [2204.07705].

## 2. Dataset Taxonomy and Construction

Sup-NatInst contains 1,616 tasks grouped into 76 distinct categories spanning classification, extraction, infilling, sequence tagging, text rewriting, text composition, and others. Major subcategories include:

- **Classification**: sentiment analysis, natural language inference, intent detection.
- **Extraction**: slot filling, named entity recognition.
- **Infilling**: masked language modeling, cloze tests.
- **Text rewriting and composition**: paraphrasing, summarization, title generation.

Each task consists of a declarative natural language instruction, positive and negative examples with explanations, and thousands of input–output pairs. The instructions follow a uniform schema:

1. **Definition**: Natural language description of the mapping from inputs to expected outputs.
2. **Positive examples**: ⟨input, correct output⟩ pairs with brief rationales.
3. **Negative examples**: ⟨input, incorrect output⟩ pairs with brief rationales.

For example, the “RTE/Textual Entailment” task is defined as: “Given two sentences, label ‘1’ if the first entails the second, else ‘0’.” With an instance such as:

- Input: Sentence 1: “No Weapons of Mass Destruction Found in Iraq Yet.” Sentence 2: “Weapons of Mass Destruction Found in Iraq.”
- Output: “0”

Task instructions and demonstrations are serialized as prependable fields, ensuring a unified interface for model input [2204.07705].

## 3. Cross-Task Generalization Protocols

Sup-NatInst is explicitly designed to support **cross-task generalization**: models are trained on instructions for a subset of tasks and evaluated on held-out, unseen tasks (both in English and cross-lingually). Data is partitioned as follows:

- **Training set**: 1,462 tasks, with 757 for English and 1,271 for cross-lingual after removing task-level leakage.
- **Evaluation set**: 154 tasks (119 English, 35 cross-lingual), with up to 100 test instances per task.

All instances from evaluation tasks are held out during training. Zero-shot generalization corresponds to using only the task definition as instruction; k-shot corresponds to prepending a small set of positive/negative exemplars [2204.07705].

## 4. Evaluation Metrics and Scoring

Multiple aggregation and task-level metrics are employed:

- **ROUGE-L**: Used as the principal aggregate metric, measuring flexible string overlap between model output and reference.
- **Classification metrics**: Accuracy, Precision, Recall, F1-score (micro, macro, weighted).
- **Correlation**: ROUGE-L shows strong correlation with classification accuracy (Pearson $r=0.97$ across task categories).
- For classification, QA, and generative tasks, additional metrics include BLEU, ROUGE-N, METEOR, and Exact Match (EM).

Aggregate and relative performance improvements are tracked, e.g.:

$$
\mathrm{F1} = 2 \frac{\text{precision} \times \text{recall}}{\text{precision} + \text{recall}}
$$

and

$$
\%\Delta = 100 \frac{M_{\rm pseudo} - M_{\rm NL}}{M_{\rm NL}}
$$

for relative gain when shifting from natural-language to pseudo-code prompts [2305.11790].

## 5. Instructional Formulation: Natural Language and Pseudo-Code

Sup-NatInst instructions are systematically structured, enabling research into instructional modality:

### Natural Language Instructions

- Plain, expert-written descriptions.
- Accompanied by structured I/O examples.
- Used as zero-shot model cues and as demonstrations in k-shot settings.

### Pseudo-Code Instructions

Mishra et al. [2305.11790] extend the benchmark by representing 132 Sup-NatInst tasks as pseudo-code prompts using a four-part schema:

1. **Function prototype**: Typed (PEP 484 style) Python signature.
2. **Docstring**: NL paraphrase, with “Parameters” and “Returns” sections.
3. **Pseudo-code body**: Control-flow, helper predicates, inline comments elucidating reasoning steps.
4. **Interpreter invocation**: “>>>” prompt with real test input.

Concrete zero-shot example for sentiment analysis:

```python
def generate_sentiment(sentence: str) -> str:
    """For the given sentence, predict the sentiment.
    Parameters:
        sentence (str): input sentence
    Returns:
        str: “positive” or “negative”
    """
    # check if sentiment is positive
    if sentiment_is_positive(sentence):
        return "positive"
    else:
        return "negative"

>>> generate_sentiment("that has a charmingly bourbon air.")
```

Each element (signature, docstring, code structure, comments) is shown to contribute cumulatively to interpretability and model performance [2305.11790].

## 6. Baseline Models, Results, and Scaling Analyses

Sup-NatInst anchors comparative baselines:

**Heuristic Baselines**:

- Copy input (14–5% ROUGE-L)
- Copy demo output (≈28–50%)

**Zero-Shot & Instruction-Tuned LMs**:

- T5-LM (11B): 30.2 ROUGE-L (English)
- GPT-3 (175B): 45.0–51.3 ROUGE-L
- T0 (11B): 32.3 ROUGE-L
- InstructGPT (175B): 52.1–52.8 ROUGE-L
- Tk-Instruct (11B, fine-tuned on Sup-NatInst): 62.0 ROUGE-L (English), +9.9 above InstructGPT despite 16× fewer parameters
- mTk-Instruct (13B, multilingual): up to 66.1 ROUGE-L (X-lingual), +13.3 over InstructGPT

| Method            | English ROUGE-L | X-lingual ROUGE-L |
|-------------------|-----------------|-------------------|
| Copy Input        | 14.2            | 5.4               |
| Copy Demo Output  | 28.5            | 50.3              |
| T5-LM (11B)       | 30.2            | –                 |
| GPT-3 (175B)      | 45.0            | 51.3              |
| T0 (11B)          | 32.3            | –                 |
| InstructGPT (175B)| 52.1            | 52.8              |
| Tk-Instruct (11B) | **62.0**        | –                 |
| mTk-Instruct (13B)| 57.1            | **66.1**          |

Scaling analyses reveal:

- Log-linear ROUGE-L gains with more distinct tasks and increasing model scale.
- Performance saturates after ≈64 instances per task; increasing I/O count beyond this yields diminishing returns.
- Diversity of tasks and model scale are critical for generalization, whereas additional examples per task have lower marginal effect [2204.07705].

## 7. Prompting Strategies and Empirical Insights

Recent experiments using Sup-NatInst with pseudo-code prompts show:

- Pseudo-code prompts yield absolute F1 gains of 7–12 points on classification, and 7–19% relative gains in ROUGE-L.
- Code-specialized LMs (CodeGen) excel even on natural language prompts, but pseudo-code yields peak performance.
- For extractive/generative QA, pseudo-code consistently improves results for CodeGen; BLOOM also benefits, especially on MCQ QA.
- Ablations confirm the additive importance of function signature, docstring, comments, and control flow.
    - Removing docstrings/comments: −0.027 F1, −0.019 ROUGE-L (CodeGen-6B).
    - Adding docstrings/comments to NL prompt: +0.039 F1 (CodeGen-6B).
    - Using only function signature or few-shot examples underperforms full pseudo-code prompt [2305.11790].

## 8. Key Findings and Implications

- Instruction fine-tuning on Sup-NatInst leads to substantial cross-task and cross-lingual generalization, as evidenced by Tk-Instruct surpassing much larger models.
- Human evaluations are closely aligned with ROUGE-L metrics (Pearson $r=0.998$), underlining the metric’s fidelity for measuring instruction-driven task completion.
- Model robustness extends to variations in input encoding and instruction format.
- Notable gaps persist between even the best instruction-tuned models and fully supervised upper bounds, leaving room for further advances in instruction learning.
- Pseudo-code prompts further disambiguate task intent and boost model performance, with every prompt element playing a synergistic role.
- Sup-NatInst, together with Tk-Instruct and empirical prompt-style analyses, establishes the leading open benchmark for studying and advancing general-purpose instruction-based NLP modeling [2204.07705, 2305.11790].

Source: https://www.emergentmind.com/topics/super-natural-instructions-benchmark