How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection (2301.07597v1)

Abstract: The introduction of ChatGPT has garnered widespread attention in both academic and industrial communities. ChatGPT is able to respond effectively to a wide range of human questions, providing fluent and comprehensive answers that significantly surpass previous public chatbots in terms of security and usefulness. On one hand, people are curious about how ChatGPT is able to achieve such strength and how far it is from human experts. On the other hand, people are starting to worry about the potential negative impacts that LLMs like ChatGPT could have on society, such as fake news, plagiarism, and social security issues. In this work, we collected tens of thousands of comparison responses from both human experts and ChatGPT, with questions ranging from open-domain, financial, medical, legal, and psychological areas. We call the collected dataset the Human ChatGPT Comparison Corpus (HC3). Based on the HC3 dataset, we study the characteristics of ChatGPT's responses, the differences and gaps from human experts, and future directions for LLMs. We conducted comprehensive human evaluations and linguistic analyses of ChatGPT-generated content compared with that of humans, where many interesting results are revealed. After that, we conduct extensive experiments on how to effectively detect whether a certain text is generated by ChatGPT or humans. We build three different detection systems, explore several key factors that influence their effectiveness, and evaluate them in different scenarios. The dataset, code, and models are all publicly available at https://github.com/Hello-SimpleAI/chatgpt-comparison-detection.

Overview of "How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection"

This paper presents a critical examination of ChatGPT, assessing its proximity to human expertise across various domains and exploring methods for distinguishing AI-generated content. The research aims to illuminate the qualitative differences between ChatGPT and human experts and investigates effective strategies for detecting AI-generated text.

Human ChatGPT Comparison Corpus (HC3)

A significant contribution of the paper is the Human ChatGPT Comparison Corpus (HC3), a dataset comprising nearly 40,000 questions with answers from both ChatGPT and human experts. The dataset spans multiple areas including open-domain, finance, medicine, law, and psychology, facilitating comprehensive analysis and comparison.

Evaluation and Analysis

Human Evaluation

Four distinct tests were conducted, namely:

• Expert Turing Test (Paired and Single Text): Experts analyzed paired and single responses, distinguishing between human and AI output.
• Amateur Turing Test (Single Text): Non-experts assessed individual responses to determine AI origins.
• Helpfulness Test: Evaluated perceived helpfulness of responses to assess practical utility.

Results indicated that experts performed better in distinguishing AI text compared to amateurs. Additionally, ChatGPT responses were often considered more helpful, except in specific domains like medicine.

Linguistic Analysis

The paper undertakes a detailed linguistic analysis, revealing key differences between human and AI-generated text:

• Vocabulary Use: Human responses exhibited greater vocabulary diversity, while ChatGPT responses were lengthier yet more restricted lexically.
• Part-of-Speech and Dependency Parsing: ChatGPT favors structured, formal language with a tendency toward neutrality, while human text is more colloquial and emotionally expressive.
• Sentiment: ChatGPT generates text with less emotional variance, contrasting with the more sentimentally dynamic human responses.
• Perplexity Analysis: ChatGPT-generated text generally had lower perplexity scores, reflecting more predictable linguistic patterns characteristic of AI training data.

Detection Systems

To address the challenge of AI-generated content, the paper develops several detection models, including:

• Logistic Regression using GLTR Features: Utilizes token probability features to distinguish AI text.
• RoBERTa-based Models: Employs deep learning techniques for single-text and QA-style detection.

The models are evaluated across different text granularities (full-text and sentence-level) and reveal that detection is more challenging at the sentence level. Additionally, models trained to recognize sentence-level patterns displayed improved robustness.

Implications and Future Directions

The research highlights the nuanced differences between ChatGPT and human experts, underscoring areas where AI can enhance utility, such as providing detailed information consistently. However, it also acknowledges potential risks, including misinformation. The detection systems developed provide practical tools for mitigating these risks.

Future research is advised to explore broader datasets and specialized prompts to determine their impact on detection efficacy. The work lays the groundwork for future advancements in AI accountability and explains the importance of continuously evolving detection methods to keep pace with AI technology advancements.

In sum, the research offers a comprehensive framework for understanding and evaluating AI capabilities relative to human expertise, while providing practical tools to identify and manage AI-generated content in various applications.