CLUE: A Chinese Language Understanding Evaluation Benchmark (2004.05986v3)

Abstract: The advent of natural language understanding (NLU) benchmarks for English, such as GLUE and SuperGLUE allows new NLU models to be evaluated across a diverse set of tasks. These comprehensive benchmarks have facilitated a broad range of research and applications in NLP. The problem, however, is that most such benchmarks are limited to English, which has made it difficult to replicate many of the successes in English NLU for other languages. To help remedy this issue, we introduce the first large-scale Chinese Language Understanding Evaluation (CLUE) benchmark. CLUE is an open-ended, community-driven project that brings together 9 tasks spanning several well-established single-sentence/sentence-pair classification tasks, as well as machine reading comprehension, all on original Chinese text. To establish results on these tasks, we report scores using an exhaustive set of current state-of-the-art pre-trained Chinese models (9 in total). We also introduce a number of supplementary datasets and additional tools to help facilitate further progress on Chinese NLU. Our benchmark is released at https://www.CLUEbenchmarks.com

Analysis of "CLUE: A Chinese Language Understanding Evaluation Benchmark"

The paper "CLUE: A Chinese Language Understanding Evaluation Benchmark" introduces an important tool for the Chinese NLP community, addressing the deficiency of comprehensive benchmarks for Chinese when compared to established ones for English, like GLUE and SuperGLUE. CLUE provides a framework for evaluating a range of natural language understanding (NLU) tasks across Chinese datasets. It is aimed at fostering better cross-LLM performance evaluation and development, addressing the unique aspects of the Chinese language.

Core Contributions and Features

The benchmark comprises nine diverse tasks, covering single-sentence classification, sentence-pair classification, and machine reading comprehension, each probing different linguistic and semantic capabilities:

1. Single-Sentence Tasks: TNEWS, IFLYTEK, and CLUEWSC2020 test capabilities in text classification and coreference resolution.
2. Sentence Pair Tasks: AFQMC, CSL, and OCNLI focus on semantic similarity and inference, reflecting real-world needs like keyword recognition and multi-faceted textual inference.
3. Machine Reading Comprehension Tasks: CMRC 2018, ChID, and C³ require models to process longer texts and answer questions or fill gaps using contextual understanding.

CLUE additionally incorporates a diagnostic dataset created by Chinese linguists, specifically targeting linguistic phenomena, providing an opportunity for detailed analysis of a model's linguistic and reasoning skills.

Empirical Evaluation and Results

The authors evaluate various state-of-the-art pre-trained LLMs such as BERT, ERNIE, and RoBERTa on CLUE, establishing baseline performances. The models demonstrate varying effectiveness, with larger models and those utilizing whole word masking generally achieving better results. Notably, RoBERTa-wwm-ext-large consistently attained superior performance across multiple tasks. Despite these models achieving notable results, there persists a significant gap between machine and human ability, as evidenced by the human performance benchmarks included in the paper.

Implications for Chinese NLP Research

The introduction of CLUE addresses a critical need within the NLP community for robust and standardized evaluation tools specifically tailored for Chinese. This is essential given the linguistic complexity and the unique syntactic features of the Chinese language, which differ considerably from English and other Indo-European languages. The benchmark promises to facilitate cross-model comparisons and drive improvements in NLU models trained on Chinese data by providing a comprehensive set of tasks with varying difficulty and domains.

Future Developments

The CLUE benchmark sets the groundwork for future exploration into more sophisticated and specialized tasks as models mature. The inclusion of a large-scale pre-training corpus coupled with the benchmark pledges to standardize and elevate the performance evaluation framework within Chinese NLP research. Future developments could involve expanding the benchmark to incorporate even more diverse linguistic phenomena and larger datasets or exploring alternative evaluation metrics that consider model efficiency.

In conclusion, the CLUE benchmark represents a significant advance in Chinese NLU, promoting transparency and consistency in model evaluation and encouraging further research and development in this field. Its open-ended nature and community-driven approach ensure it remains relevant as the landscape of NLP continues to evolve.