PARAPHRASUS : A Comprehensive Benchmark for Evaluating Paraphrase Detection Models (2409.12060v2)

Abstract: The task of determining whether two texts are paraphrases has long been a challenge in NLP. However, the prevailing notion of paraphrase is often quite simplistic, offering only a limited view of the vast spectrum of paraphrase phenomena. Indeed, we find that evaluating models in a paraphrase dataset can leave uncertainty about their true semantic understanding. To alleviate this, we create PARAPHRASUS, a benchmark designed for multi-dimensional assessment, benchmarking and selection of paraphrase detection models. We find that paraphrase detection models under our fine-grained evaluation lens exhibit trade-offs that cannot be captured through a single classification dataset. Furthermore, PARAPHRASUS allows prompt calibration for different use cases, tailoring LLM models to specific strictness levels. PARAPHRASUS includes 3 challenges spanning over 10 datasets, including 8 repurposed and 2 newly annotated; we release it along with a benchmarking library at https://github.com/impresso/paraphrasus

• The paper introduces PARAPHRASUS, a benchmark designed for fine-grained evaluation of paraphrase detection models.
• It defines three objectives—classify, minimize, and maximize—to assess models over diverse semantic and lexical challenges.
• Experimental results reveal strong in-domain performance yet highlight challenges in out-of-domain generalization.

PARAPHRASUS: A Comprehensive Benchmark for Evaluating Paraphrase Detection Models

The task of paraphrase detection—determining whether two texts express the same meaning—has long been a complex and nuanced challenge within the field of NLP. The paper "PARAPHRASUS: A Comprehensive Benchmark for Evaluating Paraphrase Detection Models" addresses critical issues associated with the existing methodologies for evaluating paraphrase detection models, which often rely on overly simplistic definitions of paraphrasing. The authors propose a new benchmark, PARAPHRASUS, that aims to provide a more detailed and fine-grained evaluation framework for paraphrase detection models by encompassing a broader range of paraphrase phenomena and dataset variability.

Introduction and Motivation

The authors recognize a significant limitation in the current paraphrase detection benchmarks, such as the PAWS-X dataset, where models trained on such data do not necessarily exhibit genuine semantic understanding given the simplistic binary classification criteria. They argue that evaluating models based solely on such datasets can obscure true performance characteristics, leading to potential misinterpretations. These observations led to the development of PARAPHRASUS, designed to facilitate multi-dimensional assessments of paraphrase detection capabilities, offering adaptability for different application requirements.

Benchmark Composition

The PARAPHRASUS benchmark comprises 10 datasets including eight repurposed and two novel datasets, with challenges spanning across different semantic and lexical similarities. It incorporates three primary objectives for model evaluation:

1. Classify!: Evaluating binary paraphrase classification across datasets such as PAWS, a challenging set generated through word scrambling and adversarial methods, and novel human-annotated datasets specifically designed to test paraphrase classification with high difficulty.
2. Minimize!: In datasets devoid of paraphrases, the model's objective is to minimize false positives, utilizing repurposed Negative Language Inference (NLI) and similarity datasets with controlled distinction of semantics to ensure challenging scenarios.
3. Maximize!: Ensuring that models can identify completetrue paraphrases, drawing examples from robust frameworks such as AMR guidelines and multilingual sentence simplification datasets.

Methodology and Evaluation

The authors present a rigorous evaluation metric, where error percentages are calculated by averaging the performance on individual tasks categorized by their defined objectives. This metric ensures that no singular aspect of model performance skews the overall evaluation, promoting comprehensive understanding across a spectrum of linguistic challenges.

Experimental Results:

• Fine-tuned models (e.g., XLM-RoBERTa) on PAWS-EN demonstrate high performance within the test set yet struggle with out-of-domain generalization, indicating potential overfitting to the dataset's adversarial characteristics.
• LLMs, like Llama3 Instruct, showed varied performance based on the specificity of the prompting approach, where more straightforward prompts (e.g., "Are these paraphrases?") excelled overall, suggesting simplicity can enhance robustness.
• The addition of easy negatives in training data revealed potential improvements in generalization capability, highlighting strategies to augment training datasets for improved model adaptability.

Human Annotation Studies

An analysis of inter-annotator agreement, comparing human annotators to model predictions, provided insights into human-level variability in paraphrase understanding. Results showed moderate agreement, affirming the nuanced nature of paraphrasing which models also reflected, albeit with specific failure modes evident in simplistic linguistic constructs.

Conclusions and Future Directions

The introduction of PARAPHRASUS sets a new standard for paraphrase model evaluation by emphasizing diversity, semantic granularity, and a balance of quantity-quality in data selection. The benchmark demonstrates areas where models require improvement, particularly in handling linguistically simple yet varied semantic paraphrases. Looking forward, expanding this benchmark across multiple languages and exploring diverse training schemas can address current limitations and enhance the generalizability and utility of paraphrase detection models in real-world applications.

Overall, PARAPHRASUS presents a substantive tool for improved model evaluation by posing complex linguistic challenges indicative of true semantic understanding, urging the exploration of innovative methods for paraphrase detection and representation in NLP.