---
title: 'GYAFC: Corpus for Formality Style Transfer'
url: https://www.emergentmind.com/topics/grammarly-s-yahoo-answers-formality-corpus-gyafc
type: topic
---

# GYAFC: Corpus for Formality Style Transfer

Grammarly’s Yahoo Answers Formality Corpus (GYAFC) is a crowdsourced parallel corpus for formality style transfer, introduced by Rao and Tetreault as a response to the lack of large-scale training and evaluation data for stylistic rewriting. It centers on the task of transforming an informal sentence into a formal paraphrase while preserving meaning, and it was presented as the largest corpus for a particular stylistic transfer at the time of publication. Built from Yahoo Answers content in the **Entertainment & Music** and **Family & Relationships** domains, GYAFC subsequently became the *de facto* benchmark for formality style transfer and a standard resource for related work on evaluation, data augmentation, semi-supervised learning, and formality detection [1803.06535] [2203.13620].

## 1. Origin, scope, and source material

GYAFC originated in the context of text style transfer, defined as automatically transforming text from one style to another. The corpus was created because progress in this area had been constrained by the absence of robust datasets, benchmarks, and evaluation protocols. Its specific focus is **formality**, rather than sentiment, authorship, or diachronic register [1803.06535].

The parent source is the **Yahoo Answers L6 corpus**, a large forum-derived collection containing millions of user-generated texts across multiple domains. Rao and Tetreault analyzed domain-level formality distributions using an existing formality classifier trained on Yahoo Answers genre data annotated on a scale from **\(-3\)** (very informal) to **\(+3\)** (very formal). On this basis, they selected the two domains with the most informal content: **Entertainment & Music (E&M)** and **Family & Relationships (F&R)** [1803.06535].

The preprocessing pipeline removed questions, sentences with URLs, and sentences shorter than **5** words or longer than **25** words, leaving approximately **40 million** sentences. The same classifier was then used to select sentences with formality score **\(< 0\)** as informal and **\(> 0\)** as formal. The reported domain statistics were: **All Yahoo Answers**: **40M** total, **24M** informal, **16M** formal; **Entertainment & Music**: **3.8M** total, **2.7M** informal, **0.7M** formal; **Family & Relationships**: **7.8M** total, **5.6M** informal, **1.8M** formal [1803.06535].

## 2. Construction and annotation protocol

The core training corpus was produced by sampling **53,000 informal sentences** from each selected domain, for a total of roughly **106,000** source sentences, and collecting **one formal rewrite per sentence** via Amazon Mechanical Turk. Annotation was conducted with expert oversight, explicit worker guidelines, feedback, and blocking of poor annotators [1803.06535].

For tuning and test, the corpus design is more stringent. The creators sampled **3,000 informal sentences** per domain for tuning and **1,500 informal sentences** per domain for testing, and then obtained **four independent formal rewrites per sentence** from a pool of **85 expert annotators**. This multi-reference design was meant to support more stable evaluation under high paraphrastic variation [1803.06535].

The benchmark also supports the reverse direction, **formal-to-informal**. For each formal rewrite in tuning and test, **three additional informal rewrites** were collected, and the original informal source was retained, yielding **four references per formal sentence** for that direction as well [1803.06535].

A representative example given in the corpus description pairs the informal sentence *“I’d say it is punk though.”* with the formal rewrite *“However, I do believe it to be punk.”* Another maps *“Gotta see both sides of the story.”* to *“You have to consider both sides of the story.”* Manual analysis of **50** examples identified frequent edit types in informal-to-formal rewriting: **phasal paraphrase (47%)**, **capitalization (46%)**, **punctuation (40%)**, **expansion of contractions (12%)**, and **normalization (10%)**; the study also notes deletion of fillers, spelling correction, completion, and repetition removal [1803.06535].

## 3. Dataset structure and benchmark status

The canonical split statistics reported for GYAFC are domain-specific. For **informal-to-formal**, the **E&M** domain contains **52,595** training pairs, **2,877** tune items, and **1,416** test items; **F&R** contains **51,967** training pairs, **2,788** tune items, and **1,332** test items [1803.06535].

| Domain | Train | Tune/Test (informal→formal) |
|---|---:|---:|
| E&M | 52,595 | 2,877 / 1,416 |
| F&R | 51,967 | 2,788 / 1,332 |

For **formal-to-informal**, the tune/test counts are **2,356 / 1,082** for E&M and **2,247 / 1,019** for F&R [1803.06535]. Later work commonly summarizes the resource as approximately **50,000 parallel sentence pairs per domain**, with **four human references** per test instance, and treats it as the standard benchmark for formality style transfer [2203.13620].

The corpus is also used outside generation. A later formality-detection study used GYAFC as the primary English benchmark for binary **formal vs. informal** classification, describing it as comprising **104,365 pairs** of formal and informal texts, with average sample length of approximately **10 tokens**, usually corresponding to single sentences. In that study, **DeBERTa-large** achieved **87.8%** accuracy, while **Char BiLSTM** achieved **87.0%**, illustrating that GYAFC functions not only as a parallel rewriting dataset but also as a style-labeled classification resource [2204.08975].

## 4. Evaluation protocols and metric ecology

GYAFC was introduced together with a multidimensional evaluation framework. Human evaluation considered **formality**, **fluency**, **meaning preservation**, and **overall ranking**. For **500** test sentences per domain and direction, the original benchmark collected **5 human judgements** for each output and metric. The rating scales were **\([-3,+3]\)** for formality, **\([1,5]\)** for fluency, and **\([1,6]\)** for meaning preservation [1803.06535].

The overall rank used in the original study is defined as

$$
\text{rank}(model) = \frac{1}{|S|}\sum_{s \in S} \frac{1}{|J|}\sum_{j \in J} \text{rank}(s_{model}, j)
$$

where \(S\) is the set of sentences and \(J\) the set of judgments [1803.06535].

Automatic evaluation in the original benchmark included an in-domain retrained **formality classifier**, a **grammaticality model** for fluency, a **sentence similarity model** for meaning, and the reference-based metrics **BLEU**, **TERp**, and **PINC**. Reported Spearman rank correlations indicated only moderate agreement with human judgments: formality metric with human formality **0.47** on E&M and **0.45** on F&R; fluency metric with human fluency **0.48** and **0.46**; meaning metric with human meaning **0.33** and **0.30**. For overall rank, **BLEU** showed **\(-0.48\)** on E&M and **\(-0.43\)** on F&R, while **PINC** was notably weak at **0.11** and **0.08**, reinforcing the original paper’s claim that automatic evaluation in style transfer is intrinsically difficult [1803.06535].

A later evaluation study, using the **Family & Relationships** test set, further systematized this issue. It fine-tuned a BERT-based style classifier on GYAFC-labelled data, termed **C-GYAFC**, which achieved **94.4%** accuracy on the test sets and the strongest correlation with human judgments for style strength: **0.97** at system level and **0.42** at segment level. The same study recommended **BLEURT**, **COMET**, and **BERTScore** for content preservation, especially when evaluating against the source rather than a reference, and reported that GPT-2 perplexity fine-tuned on GYAFC-style data correlated well with human fluency judgments for **informal-to-formal** transfer, with **0.96** system-level Pearson correlation and **0.52** segment-level correlation [2204.07549].

## 5. Role in model development and data-efficient transfer

Because GYAFC is large enough for supervised sequence modeling yet still limited relative to machine translation corpora, it became a focal point for methods addressing data scarcity. A central line of work uses the benchmark to test whether additional supervision, pretraining, or augmentation can reduce dependence on gold parallel data [2005.07522].

In **“Parallel Data Augmentation for Formality Style Transfer”**, GYAFC is the primary training and evaluation resource. The paper augments the original parallel data with three sources: **Back Translation (BT)** producing **1.6 million** pairs, **Formality Discrimination (F-Dis)** producing **1.5 million** pairs, and **Multi-task Transfer (M-Task)** producing **1.8 million** pairs, for a total of **4.9 million** augmented sentence pairs. Using a **pre-train then fine-tune** regime, the best single model improved from **69.44** to **72.63** BLEU on E&M and from **74.19** to **77.01** on F&R; the **4-model ensemble** reached **74.24** on E&M and **77.97** on F&R [2005.07522].

In **“Thank you BART! Rewarding Pre-Trained Models Improves Formality Style Transfer”**, GYAFC is again the central benchmark. The study fine-tunes **GPT-2** and **BART** with style and content rewards and reports that pre-trained models with only **10%** of the parallel data outperform non-pretrained baselines trained on the full dataset. Its overall score is the harmonic mean

$$
HM = \frac{2 \times ACC \times BLEU}{ACC + BLEU},
$$

and the paper reports **0.839 HM** on E&M and **0.852 HM** on F&R for **BART-large + SC & BLEU (both domains)** [2105.06947].

In **“Semi-Supervised Formality Style Transfer with Consistency Training”**, GYAFC supports a semi-supervised regime combining gold parallel data with **200k unlabeled informal sentences per domain** drawn from the same Yahoo Answers domains. The model is trained with a supervised loss plus an unsupervised consistency term over perturbed inputs. On GYAFC, the best reported system obtains **78.75 BLEU**, **94.56** style accuracy, and **85.94 HM** on E&M, and **81.37 BLEU**, **86.41** style accuracy, and **83.81 HM** on F&R. The paper further reports that with **5000** parallel pairs the SSL model rivals or surpasses prior fully supervised results, and with **20000** pairs the SSL model improves over the base model in both domains [2203.13620].

## 6. Critiques, supervision misalignment, and later reinterpretations

Although GYAFC has been treated as the standard benchmark, later work has challenged its supervision design. A 2026 reassessment argues that GYAFC frames formality as a **binary** distinction and that its human rewrites encode **relative stylistic shifts** rather than an **absolute or linguistically grounded** notion of formality. On this view, many benchmark “formal” references are only less informal, not genuinely formal in the sense of exhibiting features such as hedging, nominalization, or passive constructions [2605.29365].

The critique is quantified in several ways. Using an existing formality classifier trained on relative benchmark definitions, the reassessment reports **91.1%** accuracy on GYAFC’s formal references. However, when **1,000** GYAFC “formal” sentences are re-evaluated by GPT-4o under a linguistically motivated rubric, only **98 sequences (9.8%)** satisfy strict formal criteria, while **902 (90.2%)** are judged informal or casual. Human annotators, using the same definitions, reportedly show **Fleiss’ \(\kappa > 0.7\)** in concluding that most benchmark “formal” rewrites are not genuinely formal [2605.29365].

This reinterpretation is connected to directional asymmetry in model behavior. The same study reports very low **informal-to-formal F1** for several models when trained on GYAFC, including **0.06** for **GPT-4.1-nano**, **0.01** for **T5-large**, and **0.20** for **DeepSeek-1.5B**. To address the problem, the authors propose a graded three-level spectrum—**informal**, **casual**, and **formal**—and introduce **3LF**, a dataset of **4,500 sentence triples** with **casual** as an explicit anchor. Under that supervision, **GPT-4.1-nano** improves from **0.06** to **0.88 F1** in the informal-to-formal direction [2605.29365].

The significance of this critique is not that GYAFC ceased to be useful, but that its notion of “formal” may be benchmark-relative. In that sense, GYAFC remains foundational for controlled rewriting research, while later work has made the underlying label semantics themselves an object of study [2605.29365].

Source: https://www.emergentmind.com/topics/grammarly-s-yahoo-answers-formality-corpus-gyafc