---
title: Adversarial Paraphrase Datasets (PAWS)
url: https://www.emergentmind.com/topics/adversarial-paraphrase-datasets-paws
type: topic
---

# Adversarial Paraphrase Datasets (PAWS)

Adversarial paraphrase datasets, exemplified by PAWS ("Paraphrase Adversaries from Word Scrambling") and its multilingual extension PAWS-X, represent a class of challenging benchmarks for paraphrase identification. These datasets target a fundamental weakness in standard paraphrase corpora: the conflation of lexical overlap with semantic equivalence. Unlike traditional datasets, PAWS and PAWS-X deliberately construct sentence pairs that are nearly indistinguishable by bag-of-words or n-gram metrics but differ crucially in meaning, thus exposing models' structural and contextual sensitivity shortcomings. The development of PAWS and PAWS-X catalyzed evaluation and advancement of deep contextual models such as DIIN and BERT, especially in cross-lingual scenarios.

## 1. Dataset Construction Paradigms

PAWS was introduced by Zhang et al. (2019) to systematically produce high-lexical-overlap English sentence pairs with balanced paraphrase vs. non-paraphrase classes. Its construction leverages two adversarial generation mechanisms:

- **Controlled Word Swapping:** Candidate sentence pairs are generated by identifying phrase-level alignments or named entities and swapping their positions between sentences. Detailed POS and entity tagging produce templates guiding constrained beam search, wherein slots are filled by non-repetitive candidates and scored via a pretrained large-scale language model. Acceptance criteria enforce grammaticality and non-trivial permutation, retaining only those pairs where semantic role changes arise from surface reordering (e.g., “flights from New York to Florida” vs. “flights from Florida to New York”) [1904.01130].

- **Neural Back Translation:** Sentences are machine-translated into intermediate languages (notably German), then back-translated into English. Only those back-translations with cosine similarity α ≥ 0.9 (relative to the original) and sufficient word-order inversion rate (≥0.02) are kept to maximize both overlap and structural differences.

Following automatic generation, all candidate pairs undergo rigorous human annotation, with annotators evaluating grammaticality and binary paraphrase status. To ensure label fidelity, only pairs with ≥4/5 consensus are retained for supervised experiments.

PAWS-X extends the Wikipedia-derived portion of PAWS to six typologically diverse languages (French, Spanish, German, Chinese, Japanese, and Korean). For each, professionally translated dev and test sets were curated, while training sets were derived from Google NMT machine translation of the original English data. Label alignment and stringent error-rate controls (<5%) were enforced, and approximately 2% of pairs were dropped as ambiguous or untranslatable [1908.11828].

## 2. Dataset Composition and Statistics

PAWS comprises 108,463 fully labeled pairs. Its structure divides into:

- QQP-PAWS (Quora): 12,665 pairs (31.3% paraphrase, 68.7% non-paraphrase)
- Wiki-PAWS (Wikipedia): 65,401 pairs (44.2% paraphrase, 55.8% non-paraphrase)
- Wiki-Swap (Wikipedia, swap-only): 30,397 pairs (9.6% paraphrase, 90.4% non-paraphrase)

Bag-of-words similarity α for swap-generated pairs concentrates at α=1.0; back-translated pairs strictly α ≥ 0.9. PAWS-X augments this framework, providing 23,659 human-translated pairs across six languages:

| Language | Dev Pairs | Test Pairs | Positive Dev (%) | Positive Test (%) |
|----------|-----------|------------|------------------|-------------------|
| fr       | 1,992     | 1,985      | 44.0             | 45.4              |
| es       | 1,962     | 1,999      | 44.0             | 45.4              |
| de       | 1,932     | 1,967      | 44.0             | 45.4              |
| zh       | 1,984     | 1,975      | 44.0             | 45.4              |
| ja       | 1,980     | 1,946      | 44.0             | 45.4              |
| ko       | 1,965     | 1,972      | 44.0             | 45.4              |

Typologically, PAWS-X supports Indo-European and CJK languages, revealing cross-lingual syntactic sensitivities. Evaluation corpora balanced positive and negative paraphrase examples (44–45% positive, 55–56% negative).

## 3. Annotation Protocol and Quality Control

Sentence correction and paraphrase labeling within PAWS are conducted by trained annotators. Sentence acceptability rates exceed 88% after minor edits or corrections. Inter-annotator agreement for swap pairs is 95.8–97.5% after filtering for consensus, and 94.8% for back-translation pairs. PAWS-X translation pairs undergo in-house translation and secondary review, with quality measured by random word-level validation (error rate <5%). Untranslatable items (≈2%) are eliminated to preserve corpus reliability. Entity translation consistency is adhered to, though natural naming conventions for each language are permitted during translation.

## 4. Metrics, Benchmarks, and Evaluation Strategies

Key metrics defined for PAWS/PAWS-X include:

- **Lexical Overlap:** $α = \cos(\mathbf{v}_1, \mathbf{v}_2)$, where $\mathbf{v}_i$ is a sentence's bag-of-words vector.
- **Word-Order Inversion Rate:** $ \frac{|\{(i,j): i<j \wedge a(i)>a(j)\}|}{|A|} $, $A$ is the alignment set.
- **Classification Metrics:** Accuracy, Precision, Recall, $F_1$, Area Under Precision–Recall Curve (AUC-PR).

Baseline architectures include:

| Model        | Non-local Context      | Cross-Sentence Interaction       |
|--------------|-----------------------|----------------------------------|
| BOW          | No                    | No                               |
| BiLSTM       | Yes (local+sequential)| No                               |
| DecAtt       | No                    | Yes (word-by-word)               |
| ESIM         | Yes                   | Yes (co-attention)               |
| DIIN         | Yes                   | Yes (CNN n-gram interactions)    |
| BERT         | Yes (Transformer)     | Yes (self-attention pair encoding)|

In PAWS-X, three distinct training/evaluation regimes were assessed:

- Translate Train: training on MT English-to-target language, evaluation on human translation.
- Translate Test: training on English, test-time MT back to English.
- Zero-Shot (BERT): train on English, direct test on target pairs.
- Merged (BERT): train on pooled English+MT data.

## 5. Quantitative Results Across Domains and Languages

DIIN and BERT models outperform shallow approaches given PAWS data. On Quora, models trained only on QQP generalize poorly to QQP-PAWS (<40% accuracy), but addition of PAWS yields dramatic improvements for DIIN/BERT (up to 83.8–85.0% accuracy and AUC-PR ∼83.1) [1904.01130]. For Wikipedia-derived PAWS, supervised BERT and DIIN achieve up to 91.9%/94.3% accuracy/AUC-PR, with additional silver pretraining further amplifying performance.

PAWS-X cross-lingual results demonstrate strong efficacy of deep multilingual pretraining. BERT (Merged) achieves 83.1–90.8% accuracy across non-English languages—an average gain of +23% over ESIM. Zero-shot transfer lags an average 8.6% behind, indicating substantial benefit from MT-derived exemplars. In all regimes:

| Method              | en    | fr    | es    | de    | zh    | ja    | ko    |
|---------------------|-------|-------|-------|-------|-------|-------|-------|
| BOW (Trans Train)   | 55.8  | 51.7  | 47.9  | 50.2  | 54.5  | 55.1  | 56.7  |
| ESIM (Trans Train)  | 67.2  | 66.2  | 66.0  | 63.7  | 60.3  | 59.6  | 54.2  |
| BERT (Trans Train)  | 93.5  | 89.3  | 89.0  | 85.3  | 82.3  | 79.2  | 79.9  |
| BERT (Trans Test)   | —     | 88.7  | 89.3  | 88.4  | 79.3  | 75.3  | 72.6  |
| BERT (Zero-Shot)    | —     | 85.2  | 86.0  | 82.2  | 75.8  | 70.5  | 71.7  |
| BERT (Merged)       | 93.8  | 90.8  | 90.7  | 89.2  | 85.4  | 83.1  | 83.9  |

Indo-European language performance consistently surpasses CJK languages, attributable to greater syntactic similarity and higher machine translation quality.

## 6. Structural Challenges and Model Analysis

The adversarial nature of PAWS and PAWS-X presents unique challenges:

- High lexical overlap obfuscates word-order-induced semantic divergence; standard models relying on word counts or shallow attention fail to capture these nuances. Common error types include argument swaps (e.g., “A flew X→Y” v. “A flew Y→X”), negation inversions (“good” ↔ “bad”), and multi-span misalignments.
- Shallow models (BOW, DecAtt) plateau with minor gains on PAWS, while architectures capturing global context and pairwise word interactions (DIIN, BERT: transformer) are indispensable for high accuracy (≥85%) [1904.01130].

A plausible implication is that PAWS induces diagnostic evaluation regimes, robustly deflecting superficial matching models and rewarding context-aware, structure-sensitive designs.

## 7. Implications and Future Research Directions

PAWS and PAWS-X redefine the benchmark standards for paraphrase identification by emphasizing non-local dependencies, syntactic manipulation, and cross-lingual generalization. The datasets demonstrate that deep multilingual pretraining and pooled, adversarial fine-tuning significantly outperform baseline LSTM and attention models. Language typology impacts cross-lingual performance, indicating that future research should explore dataset expansion via native adversarial generation (rather than translation) and develop explicit cross-lingual alignment architectures.

Key open problems include robust entity normalization, syntactic control during pretraining, and improved gold-label consistency for ambiguous cases (especially those repeatedly misclassified across languages) [1908.11828]. PAWS and PAWS-X continue to serve as rigorous benchmarks, revealing both the strengths of modern transformers and the ongoing need for improved modeling of structural and contextual paraphrase phenomena.

Source: https://www.emergentmind.com/topics/adversarial-paraphrase-datasets-paws