InstruSum Dataset Overview

• InstruSum is a specialized evaluation benchmark that assesses LLMs in instruction-controllable text summarization using curated BBC news articles.
• It employs a structured protocol for generating natural-language instructions and hybrid LLM-human reference summaries to capture diverse summary requirements.
• The dataset supports multi-dimensional evaluation of summary quality, factual consistency, and content relevance with rigorous human annotations.

The InstruSum dataset is a specialized evaluation benchmark designed to assess the capabilities of LLMs in instruction-controllable text summarization. Distinct from conventional summarization datasets, InstruSum focuses on settings where models receive both a source document and a natural-language instruction stipulating desired summary characteristics. InstruSum is constructed as an evaluation-only dataset using curated news articles from the XL-Sum corpus, supplemented by expert-written instructions and “hybrid” model-human reference summaries. The dataset supports fine-grained, multi-dimensional human and automatic evaluation—including factual consistency under complex, user-specified requirements—offering an empirical basis for benchmarking model performance in controllable summarization contexts (Liu et al., 2023).

1. Source Material and Data Construction

The source articles for InstruSum are exclusively drawn from the English portion of XL-Sum, specifically selecting BBC news stories. Only articles within the 1,000–1,200 word range possessing “rich” and complex content—judged to benefit from requirement-specific summarization—are considered. From this filtered set, 100 articles are manually selected by the dataset creators to ensure topic and content diversity suitable for instruction-controllable summarization.

Instruction writing follows a structured protocol: for each article, annotators generate a unique natural-language summary requirement that specifies one or more of three dimensions:

• Informational: Targeting a specific topic or entity (e.g., summarizing only Ford’s actions).
• Formatting: Imposing a particular output structure (e.g., bullet points, timelines).
• Meta: Requesting auxiliary or contextual information (e.g., identifying interviewees).

Diversity is fostered by prompting GPT-4 to generate initial instruction candidates, which human annotators then edit or rewrite as warranted. For each article-instruction pair, an initial summary is generated zero-shot by one of three OpenAI models (text-davinci-003, gpt-3.5-turbo-0301, or gpt-4-0314) and then lightly edited by a human annotator for fluency and correctness, yielding a hybrid LLM-human reference summary (Liu et al., 2023).

2. Dataset Composition and Structure

InstruSum is composed of 100 article–instruction examples, each with five system outputs: text-davinci-002, text-davinci-003, gpt-3.5-turbo-0301, gpt-4-0314, and the hybrid reference. This results in a total of 500 summaries. Every instruction in the dataset encompasses informational, formatting, and meta dimensions, though precise counts per dimension are not explicitly tabulated. InstruSum is strictly intended for evaluation: no train/validation/test splits are provided, and it is not suitable as a training corpus due to its small scale and specialized design.

3. Annotation and Evaluation Protocols

For each example, human annotation targets four core evaluation dimensions:

1. Overall Quality
2. Missing Information
3. Irrelevant Information
4. Factual Consistency

Annotations are collected via Amazon Mechanical Turk from workers who pass two rigorous qualification rounds, with only approximately 5% of candidates qualifying (each with prior acceptance ≥ 99%). A single qualified annotator ranks all five summaries for each example from best (1) to worst (5) on each dimension; ties are permitted. For factual consistency, annotators additionally highlight the precise textual spans corresponding to errors. For overall quality, annotators supply short free-text justifications of their top choices.

Quality control measures report moderate inter-annotator agreement, with Krippendorff’s α: overall = 0.2571, missing = 0.2247, irrelevant = 0.1925, factuality = 0.0196. The author team independently spot-checked factually annotated spans, finding crowd accuracy on factual error identification to be 88.4% (Liu et al., 2023).

4. Data Format and Accessible Schema

The dataset is distributed in JSONL format, one record per summary, with the following key fields:

Field	Type	Description
doc_id	string	Unique identifier
article	string	Full news-article text
instruction	string	Natural-language summary requirement
system	string	{text-davinci-002, text-davinci-003, gpt-3.5-turbo-0301, gpt-4-0314, hybrid}
summary	string	Generated or hybrid reference summary
rank_overall	integer	Human rank (1–5) for overall quality
rank_missing	integer	Human rank (1–5) for missing information
rank_irrelevant	integer	Human rank (1–5) for irrelevant information
factual_spans	list	Zero or more {start, end} indices for factual errors
is_factual	boolean	True if no factual spans are marked

Example entries illustrate the range of instructions, such as requiring summaries in timeline format or focusing specifically on entities mentioned in an article. For instance, Example 1 features an instruction to “Summarize the information about Ford in a timeline,” leading to a hybrid summary listing major events involving Ford by year. Example 2 requests “Summarize the companies mentioned in this article,” with the summary concisely describing three firms, associated leaders, business domains, and financial outlooks (Liu et al., 2023).

5. Formal Evaluation Metrics and Protocols

The task formulation for an instruction-controllable summarizer is expressed as:

$S \gets f(D, I)$

where $D$ denotes the document and $I$ is the instruction. Human ranks are converted to system scores for each evaluation dimension using:

$s_i = N - \sum_{j\neq i} \mathbf{1}\{r_i < r_j\}$

with $r_i$ being summary $i$’s rank among $N=5$ systems.

For meta-evaluation, InstruSum supports correlation-based measures between human and LLM-based system judgments. At the system level:

$$C_{\mathrm{sys}} = \mathrm{Kendall}\bigl(S^{(h)}, S^{(f)}\bigr)$$

and summary-level average correlation:

$$C_{\mathrm{summ}} = \frac{1}{N_{\mathrm{examples}}} \sum_{i=1}^{N_{\mathrm{examples}}} \mathrm{Kendall}\bigl(\mathbf{s}_i^{(h)}, \mathbf{s}_i^{(f)}\bigr)$$

These protocols enable robust meta-evaluation of LLM-based scoring methods for alignment with human judgments across dimensions—quality, missingness, irrelevance, and factuality (Liu et al., 2023).

6. Accessibility, Licensing, and Usage

InstruSum is publicly available at https://github.com/yale-nlp/InstruSum, provided in JSONL format under an MIT-style open-source license (refer to the repository LICENSE file for specifics). The dataset is intended solely for evaluation purposes; its small size and focus on BBC news articles make it unsuitable for model training.

Intended use cases include:

• Benchmarking new models for instruction-controllable summarization.
• Meta-evaluating the alignment of LLM-based automatic scorers with human evaluation.
• Analyzing patterns and types of summarization errors under user-specified requirements.

Limitations include:

• Restricted domain coverage (BBC news style).
• Limited scale (100 examples, 500 summaries).
• Moderate inter-annotator agreement, particularly low on factuality.
• Absence of traditional “gold” summaries; hybrid references may inherit LLM biases (Liu et al., 2023).

7. Significance and Context within LLM Evaluation

InstruSum provides a focused resource for evaluating LLMs on instruction-following summarization, addressing shortcomings of generic summarization benchmarks that do not test models under user-specified requirements or divergent output formats. Empirical findings reveal persistent factual and content errors in LLM-generated summaries, along with substantial model-to-model performance gaps. Furthermore, no existing LLM-based evaluation method achieves strong correlation with human annotators as quantified by Kendall correlation. This suggests the ongoing need for improved methods in both instruction-controllable summarization and its automated evaluation. InstruSum thus constitutes a platform for future research on summary faithfulness, relevance, instruction adherence, and evaluation alignment under challenging, user-driven specifications in summarization (Liu et al., 2023).