NEWTS Dataset: Topic-Controlled News Summaries

• NEWTS is a dataset for topic-focused abstractive summarization that pairs each news article with two human-written summaries targeting distinct topics.
• It employs LDA for topic selection and rigorous quality controls, including both automated filters and manual reviews, to ensure distinct topical alignment.
• Benchmark evaluations using ROUGE and topic-focus metrics demonstrate robust performance for models like Conformer and baselines such as BART and T5.

The NEWTS dataset is a publicly released corpus designed to enable and benchmark topic-focused abstractive summarization in the news domain. Constructed on top of the CNN/DailyMail corpus, NEWTS pairs each article with two separate, human-written summaries, each tailored to a distinct coarse topic, and provides multiple prompt modalities for fine-grained control and model conditioning (Bahrainian et al., 2022). The dataset has become standard for evaluating controllable summarization models and topic-sensitive generation, supporting quantitative and human evaluations of both summary fidelity and topic alignment (Bahrainian et al., 2023).

1. Corpus Design and Construction

NEWTS is based on the widely used CNN/DailyMail summarization corpus. Articles are selected by running Latent Dirichlet Allocation (LDA) with $K=250$ on the CNN/DailyMail training split; the 50 topics with highest coherence (NPMI) are retained after further manual filtering. For each topic, the curators provide a topic ID, the top 20 words by probability, hand-written short phrases, and a single “topic sentence” in natural language.

Articles are included if their LDA mixtures cover at least two of the 50 topics with prevalence $>0.10$ each, and sum $>0.30$. This ensures each source has two salient, distinct topical facets suitable for differential summarization (Bahrainian et al., 2022).

2. Annotation Protocol and Quality Controls

Summaries are crowdsourced on Amazon Mechanical Turk. Annotators receive a source article and two distinct target topics (each with prompt words and phrases), and are tasked with writing two non-overlapping summaries—one per topic. Each summary must be at least three sentences, maximize focus on the assigned topic while “avoiding mention of the other topic as much as possible”, and avoid verbatim copying from the source.

Two quality assurance mechanisms are enforced: automatic filters (length, grammar, named-entity overlap, LDA topic match, anti-plagiarism) and manual spot-checks (random z-test sampling, expert review, exclusion of problematic annotators). Pre-testing achieved 100% agreement with expert raters on 300 pilot tasks (Bahrainian et al., 2022).

3. Dataset Statistics and Structure

The dataset contains 3,000 unique CNN/DailyMail articles: 2,400 in training, 600 in test. Each article is paired with two topic-focused summaries, yielding 4,800 training and 1,200 test summaries. Average summary length in tokens is approximately 70 (train) and 70.1 (test), with 5.5 and 5.0 sentences respectively.

Topic coverage is roughly uniform across the 50 topics with only minor variance. Source articles mirror the length distribution of CNN/DailyMail (mean ≈ 781 tokens). All records are stored in JSONL format, with fields for article, topic_id, the four prompt types (top-10 words “T-W”, hand-written phrases “T-Ph”, topic sentence “T-Sent”, topic ID “T-ID”), and the summary. Users may select conditioning styles as desired (Bahrainian et al., 2022).

Split	Articles	Summaries
Train	2400	4800
Test	600	1200

4. Prompting, Conditioning, and Topic Taxonomy

For each summary, all four prompt modalities are provided:

• “T-W”: top 10 words for the topic (from LDA)
• “T-Ph”: hand-written topic phrases
• “T-Sent”: natural-language topic sentence
• “T-ID”: unique topic identifier

The underlying taxonomy covers mainstream news domains such as Government, Crime, Disaster, Sports, etc. Each article is annotated with exactly two target topics; for each target, the corresponding summary and prompts are available. During modeling, any of these prompts can be injected to steer summary generation or evaluate sensitivity to prompt richness (Bahrainian et al., 2023).

5. Evaluation Methodology and Metrics

Models are evaluated on NEWTS by two primary criteria: standard ROUGE-N (n-gram overlap) and a domain-specific “topic-focus” (topic prevalence) score.

ROUGE-N:

$$\text{ROUGE-N} = \frac{\sum_{S \in \mathrm{Ref}} \sum_{g \in \text{grams}_n(S)} \min(\mathrm{Count}_{\mathrm{sys}}(g),\, \mathrm{Count}_{\mathrm{ref}}(g)) }{ \sum_{S \in \mathrm{Ref}} \sum_{g \in \text{grams}_n(S)} \mathrm{Count}_{\mathrm{ref}}(g) }$$

Topic-Focus:

For summary $S$ and assigned topic $T^*$, let $\theta_S$ be the LDA topic mixture of $S$. The topic-focus score is

$\mathrm{TopicFocus}(S) = \theta_{S,T^*}$

Higher values indicate better topical alignment. Averaging over a test set yields holistic model assessment. Human studies (3 annotators, 93% accuracy, $\kappa = 0.78$) further validate that topic-focused summaries are distinguishable from generic CNN/DailyMail references (Bahrainian et al., 2022, Bahrainian et al., 2023).

6. Benchmarks, Findings, and Research Usage

NEWTS is used to benchmark topic-controlled summarization systems. Experiments report ROUGE and topic-focus under various prompting and conditioning strategies. Conformer, a Transformer model with topical cross-attention, achieves highest scores: ROUGE-1 (34.16), ROUGE-2 (11.67), ROUGE-L (21.93), and Topic Focus (0.1759) (Bahrainian et al., 2023). Prompting with the full topic sentence (“T-Sent”) yields superior topic focus, indicating the benefit of richer contextualization (Bahrainian et al., 2022, Bahrainian et al., 2023). Baselines include BART, T5, ProphetNet, PPLM, CATS, and ChatGPT, with 3-fold CV evaluation on the test set.

7. Licensing, Accessibility, and Impact

The NEWTS dataset and accompanying code are released under permissive research licensing (typically CC-BY-NC or MIT). The repository is available at https://github.com/ali-bahrainian/NEWTS with no special restrictions beyond standard open academic use (Bahrainian et al., 2022).

NEWTS fills a gap in the evaluation of controlled, topic-sensitive summarization, providing the first large-scale testbed for prompt-conditional abstractive summarization in news. Prompt engineering plays a key role in maximizing topicality. Future research directions noted in the original work include dynamic topic interpolation, multi-topic summaries, and robust transfer to low-resource or out-of-domain settings (Bahrainian et al., 2022, Bahrainian et al., 2023).