---
title: InstruSum Dataset Overview
url: https://www.emergentmind.com/topics/instrusum-dataset
type: topic
---

# InstruSum Dataset Overview

The InstruSum dataset is a specialized evaluation benchmark designed to assess the capabilities of large language models (LLMs) in instruction-controllable text summarization. Distinct from conventional summarization datasets, InstruSum focuses on settings where models receive both a source document and a natural-language instruction stipulating desired summary characteristics. InstruSum is constructed as an evaluation-only dataset using curated news articles from the XL-Sum corpus, supplemented by expert-written instructions and “hybrid” model-human reference summaries. The dataset supports fine-grained, multi-dimensional human and automatic evaluation—including factual consistency under complex, user-specified requirements—offering an empirical basis for benchmarking model performance in controllable summarization contexts [2311.09184].

## 1. Source Material and Data Construction

The source articles for InstruSum are exclusively drawn from the English portion of XL-Sum, specifically selecting BBC news stories. Only articles within the 1,000–1,200 word range possessing “rich” and complex content—judged to benefit from requirement-specific summarization—are considered. From this filtered set, 100 articles are manually selected by the dataset creators to ensure topic and content diversity suitable for instruction-controllable summarization.

Instruction writing follows a structured protocol: for each article, annotators generate a unique natural-language summary requirement that specifies one or more of three dimensions:
- **Informational**: Targeting a specific topic or entity (e.g., summarizing only Ford’s actions).
- **Formatting**: Imposing a particular output structure (e.g., bullet points, timelines).
- **Meta**: Requesting auxiliary or contextual information (e.g., identifying interviewees).

Diversity is fostered by prompting GPT-4 to generate initial instruction candidates, which human annotators then edit or rewrite as warranted. For each article-instruction pair, an initial summary is generated zero-shot by one of three OpenAI models (text-davinci-003, gpt-3.5-turbo-0301, or gpt-4-0314) and then lightly edited by a human annotator for fluency and correctness, yielding a hybrid LLM-human reference summary [2311.09184].

## 2. Dataset Composition and Structure

InstruSum is composed of 100 article–instruction examples, each with five system outputs: text-davinci-002, text-davinci-003, gpt-3.5-turbo-0301, gpt-4-0314, and the hybrid reference. This results in a total of 500 summaries. Every instruction in the dataset encompasses informational, formatting, and meta dimensions, though precise counts per dimension are not explicitly tabulated. InstruSum is strictly intended for evaluation: no train/validation/test splits are provided, and it is not suitable as a training corpus due to its small scale and specialized design.

## 3. Annotation and Evaluation Protocols

For each example, human annotation targets four core evaluation dimensions:
1. **Overall Quality**
2. **Missing Information**
3. **Irrelevant Information**
4. **Factual Consistency**

Annotations are collected via Amazon Mechanical Turk from workers who pass two rigorous qualification rounds, with only approximately 5% of candidates qualifying (each with prior acceptance ≥ 99%). A single qualified annotator ranks all five summaries for each example from best (1) to worst (5) on each dimension; ties are permitted. For factual consistency, annotators additionally highlight the precise textual spans corresponding to errors. For overall quality, annotators supply short free-text justifications of their top choices.

Quality control measures report moderate inter-annotator agreement, with Krippendorff’s α: overall = 0.2571, missing = 0.2247, irrelevant = 0.1925, factuality = 0.0196. The author team independently spot-checked factually annotated spans, finding crowd accuracy on factual error identification to be 88.4% [2311.09184].

## 4. Data Format and Accessible Schema

The dataset is distributed in JSONL format, one record per summary, with the following key fields:

| Field             | Type        | Description                                                |
|-------------------|-------------|------------------------------------------------------------|
| doc_id            | string      | Unique identifier                                          |
| article           | string      | Full news-article text                                     |
| instruction       | string      | Natural-language summary requirement                       |
| system            | string      | {text-davinci-002, text-davinci-003, gpt-3.5-turbo-0301, gpt-4-0314, hybrid} |
| summary           | string      | Generated or hybrid reference summary                      |
| rank_overall      | integer     | Human rank (1–5) for overall quality                       |
| rank_missing      | integer     | Human rank (1–5) for missing information                   |
| rank_irrelevant   | integer     | Human rank (1–5) for irrelevant information                |
| factual_spans     | list        | Zero or more {start, end} indices for factual errors       |
| is_factual        | boolean     | True if no factual spans are marked                        |

Example entries illustrate the range of instructions, such as requiring summaries in timeline format or focusing specifically on entities mentioned in an article. For instance, Example 1 features an instruction to “Summarize the information about Ford in a timeline,” leading to a hybrid summary listing major events involving Ford by year. Example 2 requests “Summarize the companies mentioned in this article,” with the summary concisely describing three firms, associated leaders, business domains, and financial outlooks [2311.09184].

## 5. Formal Evaluation Metrics and Protocols

The task formulation for an instruction-controllable summarizer is expressed as:
$$
S \gets f(D, I)
$$
where \( D \) denotes the document and \( I \) is the instruction. Human ranks are converted to system scores for each evaluation dimension using:
$$
s_i = N - \sum_{j\neq i} \mathbf{1}\{r_i < r_j\}
$$
with \( r_i \) being summary \( i \)’s rank among \( N=5 \) systems.

For meta-evaluation, InstruSum supports correlation-based measures between human and LLM-based system judgments. At the system level:
$$
C_{\mathrm{sys}} = \mathrm{Kendall}\bigl(S^{(h)}, S^{(f)}\bigr)
$$
and summary-level average correlation:
$$
C_{\mathrm{summ}} = \frac{1}{N_{\mathrm{examples}}} \sum_{i=1}^{N_{\mathrm{examples}}} \mathrm{Kendall}\bigl(\mathbf{s}_i^{(h)}, \mathbf{s}_i^{(f)}\bigr)
$$
These protocols enable robust meta-evaluation of LLM-based scoring methods for alignment with human judgments across dimensions—quality, missingness, irrelevance, and factuality [2311.09184].

## 6. Accessibility, Licensing, and Usage

InstruSum is publicly available at https://github.com/yale-nlp/InstruSum, provided in JSONL format under an MIT-style open-source license (refer to the repository LICENSE file for specifics). The dataset is intended solely for evaluation purposes; its small size and focus on BBC news articles make it unsuitable for model training.

Intended use cases include:
- Benchmarking new models for instruction-controllable summarization.
- Meta-evaluating the alignment of LLM-based automatic scorers with human evaluation.
- Analyzing patterns and types of summarization errors under user-specified requirements.

Limitations include:
- Restricted domain coverage (BBC news style).
- Limited scale (100 examples, 500 summaries).
- Moderate inter-annotator agreement, particularly low on factuality.
- Absence of traditional “gold” summaries; hybrid references may inherit LLM biases [2311.09184].

## 7. Significance and Context within LLM Evaluation

InstruSum provides a focused resource for evaluating LLMs on instruction-following summarization, addressing shortcomings of generic summarization benchmarks that do not test models under user-specified requirements or divergent output formats. Empirical findings reveal persistent factual and content errors in LLM-generated summaries, along with substantial model-to-model performance gaps. Furthermore, no existing LLM-based evaluation method achieves strong correlation with human annotators as quantified by Kendall correlation. This suggests the ongoing need for improved methods in both instruction-controllable summarization and its automated evaluation. InstruSum thus constitutes a platform for future research on summary faithfulness, relevance, instruction adherence, and evaluation alignment under challenging, user-driven specifications in summarization [2311.09184].

Source: https://www.emergentmind.com/topics/instrusum-dataset