How the Misuse of a Dataset Harmed Semantic Clone Detection

Abstract: BigCloneBench is a well-known and widely used large-scale dataset for the evaluation of recall of clone detection tools. It has been beneficial for research on clone detection and has become a standard in evaluating the performance of clone detection tools. More recently, it has also been widely used as a dataset to evaluate machine learning approaches to semantic clone detection or code similarity detection for functional or semantic similarity. This paper demonstrates that BigCloneBench is problematic to use as ground truth for learning or evaluating semantic code similarity, and highlights the aspects of BigCloneBench that affect the ground truth quality. A manual investigation of a statistically significant random sample of 406 Weak Type-3/Type-4 clone pairs revealed that 93% of them do not have a similar functionality and are therefore mislabelled. In a literature review of 179 papers that use BigCloneBench as a dataset, we found 139 papers that used BigCloneBench to evaluate semantic clone detection and where the results are threatened in their validity by the mislabelling. As such, these papers often report high F1 scores (e.g., above 0.9), which indicates overfitting to dataset-specific artefacts rather than genuine semantic similarity detection. We emphasise that using BigCloneBench remains valid for the intended purpose of evaluating syntactic or textual clone detection of Type-1, Type-2, and Type-3 clones. We acknowledge the important contributions of BigCloneBench to two decades of traditional clone detection research. However, the usage of BigCloneBench beyond the intended purpose without careful consideration of its limitations has led to misleading results and conclusions, and potentially harmed the field of semantic clone detection.

Evaluation and Impact of BigCloneBench in Semantic Clone Detection

The paper, "How the Misuse of a Dataset Harmed Semantic Clone Detection," presents a critical analysis of the prevalent use of BigCloneBench in the field of semantic clone detection. BigCloneBench has become a standard benchmark for assessing clone detection tools, primarily targeted at evaluating syntactic or textual clone detection. However, recent applications extending its utility to semantic clone detection reveal significant shortcomings, which the authors diligently investigate.

Mislabeling in BigCloneBench

The central premise of the paper addresses the inaccuracy prevalent in BigCloneBench when employed for evaluating semantic code similarity. Through a meticulous manual investigation of 406 clone pairs classified as Weak Type-3/Type-4 (WT3/T4), an overwhelming 93% were found to be mislabelled, indicating significant discrepancies between assigned labels and the actual functional similarity of the pairs. This mislabeling casts substantial doubt on the results of numerous studies relying on BigCloneBench for training and evaluation in semantic detection tasks.

Literature Review and Consequences

A comprehensive literature review further uncovers the extent of this issue. Out of 179 papers employing BigCloneBench, 139 papers rely on evaluations threatened by mislabeling, often reporting inflated F1 scores, suggesting the models are overfitting to artifacts specific to the dataset rather than legitimately discerned semantic similarities. Such findings underscore the urgency for researchers to critically reassess the grounds of their evaluations and adopt more rigorous validation measures to ensure reliability.

Implications for Machine Learning and Clone Detection

These revelations are crucial for both theoretical foundations and practical applications of semantic clone detection. Mislabeling hampers the reliability and generalizability of machine learning models trained on BigCloneBench, skewing perceived performance and potentially misleading ongoing development efforts. Consequently, the paper advocates for a more prudent and discerning application of benchmarks, echoing the need for robust validation processes in dataset creation and usage.

Recommendations for Data Utilization

While acknowledging BigCloneBench's contribution to earlier clone detection research, the paper emphasizes the necessity of utilizing the benchmark within its intended scope—namely syntactic clone detection. The authors suggest halting further employment in semantic detection, urging the community to assess and validate alternative datasets critically. Strategic shifts towards newly validated benchmarks could salvage the integrity of future research trajectories in semantic clone identification.

Future Directions for Research

The paper insinuates a more collaborative approach in iterating dataset benchmarks through collective scrutiny and validation by the research community. Further research could investigate the development of comprehensive datasets explicitly crafted for semantic detection, with built-in validation protocols to ensure quality and applicability across varied clone detection methodologies.

In summary, this paper serves as both a cautionary tale and a call to arms, urging the research community to critically appraise their methodologies and datasets to safeguard the advancement and credibility of semantic clone detection as a discipline.