GYAFC: Corpus for Formality Style Transfer

• GYAFC is a crowdsourced parallel corpus for formality style transfer, built from Yahoo Answers data to transform informal sentences into formal paraphrases.
• It employs a rigorous annotation protocol with multiple human rewrites per sentence, supporting both informal-to-formal and formal-to-informal benchmarks in Entertainment & Music and Family & Relationships domains.
• The dataset has catalyzed advancements in data augmentation, evaluation metrics, semi-supervised learning, and sparked debates on the true linguistic definition of formality.

Grammarly’s Yahoo Answers Formality Corpus (GYAFC) is a crowdsourced parallel corpus for formality style transfer, introduced by Rao and Tetreault as a response to the lack of large-scale training and evaluation data for stylistic rewriting. It centers on the task of transforming an informal sentence into a formal paraphrase while preserving meaning, and it was presented as the largest corpus for a particular stylistic transfer at the time of publication. Built from Yahoo Answers content in the Entertainment & Music and Family & Relationships domains, GYAFC subsequently became the de facto benchmark for formality style transfer and a standard resource for related work on evaluation, data augmentation, semi-supervised learning, and formality detection (Rao et al., 2018, Liu et al., 2022).

1. Origin, scope, and source material

GYAFC originated in the context of text style transfer, defined as automatically transforming text from one style to another. The corpus was created because progress in this area had been constrained by the absence of robust datasets, benchmarks, and evaluation protocols. Its specific focus is formality, rather than sentiment, authorship, or diachronic register (Rao et al., 2018).

The parent source is the Yahoo Answers L6 corpus, a large forum-derived collection containing millions of user-generated texts across multiple domains. Rao and Tetreault analyzed domain-level formality distributions using an existing formality classifier trained on Yahoo Answers genre data annotated on a scale from $-3$ (very informal) to $+3$ (very formal). On this basis, they selected the two domains with the most informal content: Entertainment & Music (E&M) and Family & Relationships (F&R) (Rao et al., 2018).

The preprocessing pipeline removed questions, sentences with URLs, and sentences shorter than 5 words or longer than 25 words, leaving approximately 40 million sentences. The same classifier was then used to select sentences with formality score $< 0$ as informal and $> 0$ as formal. The reported domain statistics were: All Yahoo Answers: 40M total, 24M informal, 16M formal; Entertainment & Music: 3.8M total, 2.7M informal, 0.7M formal; Family & Relationships: 7.8M total, 5.6M informal, 1.8M formal (Rao et al., 2018).

2. Construction and annotation protocol

The core training corpus was produced by sampling 53,000 informal sentences from each selected domain, for a total of roughly 106,000 source sentences, and collecting one formal rewrite per sentence via Amazon Mechanical Turk. Annotation was conducted with expert oversight, explicit worker guidelines, feedback, and blocking of poor annotators (Rao et al., 2018).

For tuning and test, the corpus design is more stringent. The creators sampled 3,000 informal sentences per domain for tuning and 1,500 informal sentences per domain for testing, and then obtained four independent formal rewrites per sentence from a pool of 85 expert annotators. This multi-reference design was meant to support more stable evaluation under high paraphrastic variation (Rao et al., 2018).

The benchmark also supports the reverse direction, formal-to-informal. For each formal rewrite in tuning and test, three additional informal rewrites were collected, and the original informal source was retained, yielding four references per formal sentence for that direction as well (Rao et al., 2018).

A representative example given in the corpus description pairs the informal sentence “I’d say it is punk though.” with the formal rewrite “However, I do believe it to be punk.” Another maps “Gotta see both sides of the story.” to “You have to consider both sides of the story.” Manual analysis of 50 examples identified frequent edit types in informal-to-formal rewriting: phasal paraphrase (47%), capitalization (46%), punctuation (40%), expansion of contractions (12%), and normalization (10%); the study also notes deletion of fillers, spelling correction, completion, and repetition removal (Rao et al., 2018).

3. Dataset structure and benchmark status

The canonical split statistics reported for GYAFC are domain-specific. For informal-to-formal, the E&M domain contains 52,595 training pairs, 2,877 tune items, and 1,416 test items; F&R contains 51,967 training pairs, 2,788 tune items, and 1,332 test items (Rao et al., 2018).

Domain	Train	Tune/Test (informal→formal)
E&M	52,595	2,877 / 1,416
F&R	51,967	2,788 / 1,332

For formal-to-informal, the tune/test counts are 2,356 / 1,082 for E&M and 2,247 / 1,019 for F&R (Rao et al., 2018). Later work commonly summarizes the resource as approximately 50,000 parallel sentence pairs per domain, with four human references per test instance, and treats it as the standard benchmark for formality style transfer (Liu et al., 2022).

The corpus is also used outside generation. A later formality-detection study used GYAFC as the primary English benchmark for binary formal vs. informal classification, describing it as comprising 104,365 pairs of formal and informal texts, with average sample length of approximately 10 tokens, usually corresponding to single sentences. In that study, DeBERTa-large achieved 87.8% accuracy, while Char BiLSTM achieved 87.0%, illustrating that GYAFC functions not only as a parallel rewriting dataset but also as a style-labeled classification resource (Dementieva et al., 2022).

4. Evaluation protocols and metric ecology

GYAFC was introduced together with a multidimensional evaluation framework. Human evaluation considered formality, fluency, meaning preservation, and overall ranking. For 500 test sentences per domain and direction, the original benchmark collected 5 human judgements for each output and metric. The rating scales were $[-3,+3]$ for formality, $[1,5]$ for fluency, and $[1,6]$ for meaning preservation (Rao et al., 2018).

The overall rank used in the original study is defined as

$$\text{rank}(model) = \frac{1}{|S|}\sum_{s \in S} \frac{1}{|J|}\sum_{j \in J} \text{rank}(s_{model}, j)$$

where $S$ is the set of sentences and $J$ the set of judgments (Rao et al., 2018).

Automatic evaluation in the original benchmark included an in-domain retrained formality classifier, a grammaticality model for fluency, a sentence similarity model for meaning, and the reference-based metrics BLEU, TERp, and PINC. Reported Spearman rank correlations indicated only moderate agreement with human judgments: formality metric with human formality 0.47 on E&M and 0.45 on F&R; fluency metric with human fluency 0.48 and 0.46; meaning metric with human meaning 0.33 and 0.30. For overall rank, BLEU showed $+3$0 on E&M and $+3$1 on F&R, while PINC was notably weak at 0.11 and 0.08, reinforcing the original paper’s claim that automatic evaluation in style transfer is intrinsically difficult (Rao et al., 2018).

A later evaluation study, using the Family & Relationships test set, further systematized this issue. It fine-tuned a BERT-based style classifier on GYAFC-labelled data, termed C-GYAFC, which achieved 94.4% accuracy on the test sets and the strongest correlation with human judgments for style strength: 0.97 at system level and 0.42 at segment level. The same study recommended BLEURT, COMET, and BERTScore for content preservation, especially when evaluating against the source rather than a reference, and reported that GPT-2 perplexity fine-tuned on GYAFC-style data correlated well with human fluency judgments for informal-to-formal transfer, with 0.96 system-level Pearson correlation and 0.52 segment-level correlation (Lai et al., 2022).

5. Role in model development and data-efficient transfer

Because GYAFC is large enough for supervised sequence modeling yet still limited relative to machine translation corpora, it became a focal point for methods addressing data scarcity. A central line of work uses the benchmark to test whether additional supervision, pretraining, or augmentation can reduce dependence on gold parallel data (Zhang et al., 2020).

In “Parallel Data Augmentation for Formality Style Transfer”, GYAFC is the primary training and evaluation resource. The paper augments the original parallel data with three sources: Back Translation (BT) producing 1.6 million pairs, Formality Discrimination (F-Dis) producing 1.5 million pairs, and Multi-task Transfer (M-Task) producing 1.8 million pairs, for a total of 4.9 million augmented sentence pairs. Using a pre-train then fine-tune regime, the best single model improved from 69.44 to 72.63 BLEU on E&M and from 74.19 to 77.01 on F&R; the 4-model ensemble reached 74.24 on E&M and 77.97 on F&R (Zhang et al., 2020).

In “Thank you BART! Rewarding Pre-Trained Models Improves Formality Style Transfer”, GYAFC is again the central benchmark. The study fine-tunes GPT-2 and BART with style and content rewards and reports that pre-trained models with only 10% of the parallel data outperform non-pretrained baselines trained on the full dataset. Its overall score is the harmonic mean

$+3$2

and the paper reports 0.839 HM on E&M and 0.852 HM on F&R for BART-large + SC & BLEU (both domains) (Lai et al., 2021).

In “Semi-Supervised Formality Style Transfer with Consistency Training”, GYAFC supports a semi-supervised regime combining gold parallel data with 200k unlabeled informal sentences per domain drawn from the same Yahoo Answers domains. The model is trained with a supervised loss plus an unsupervised consistency term over perturbed inputs. On GYAFC, the best reported system obtains 78.75 BLEU, 94.56 style accuracy, and 85.94 HM on E&M, and 81.37 BLEU, 86.41 style accuracy, and 83.81 HM on F&R. The paper further reports that with 5000 parallel pairs the SSL model rivals or surpasses prior fully supervised results, and with 20000 pairs the SSL model improves over the base model in both domains (Liu et al., 2022).

6. Critiques, supervision misalignment, and later reinterpretations

Although GYAFC has been treated as the standard benchmark, later work has challenged its supervision design. A 2026 reassessment argues that GYAFC frames formality as a binary distinction and that its human rewrites encode relative stylistic shifts rather than an absolute or linguistically grounded notion of formality. On this view, many benchmark “formal” references are only less informal, not genuinely formal in the sense of exhibiting features such as hedging, nominalization, or passive constructions (Yu et al., 28 May 2026).

The critique is quantified in several ways. Using an existing formality classifier trained on relative benchmark definitions, the reassessment reports 91.1% accuracy on GYAFC’s formal references. However, when 1,000 GYAFC “formal” sentences are re-evaluated by GPT-4o under a linguistically motivated rubric, only 98 sequences (9.8%) satisfy strict formal criteria, while 902 (90.2%) are judged informal or casual. Human annotators, using the same definitions, reportedly show Fleiss’ $+3$3 in concluding that most benchmark “formal” rewrites are not genuinely formal (Yu et al., 28 May 2026).

This reinterpretation is connected to directional asymmetry in model behavior. The same study reports very low informal-to-formal F1 for several models when trained on GYAFC, including 0.06 for GPT-4.1-nano, 0.01 for T5-large, and 0.20 for DeepSeek-1.5B. To address the problem, the authors propose a graded three-level spectrum—informal, casual, and formal—and introduce 3LF, a dataset of 4,500 sentence triples with casual as an explicit anchor. Under that supervision, GPT-4.1-nano improves from 0.06 to 0.88 F1 in the informal-to-formal direction (Yu et al., 28 May 2026).

The significance of this critique is not that GYAFC ceased to be useful, but that its notion of “formal” may be benchmark-relative. In that sense, GYAFC remains foundational for controlled rewriting research, while later work has made the underlying label semantics themselves an object of study (Yu et al., 28 May 2026).