GPTCloneBench: Semantic & Cross-Language Clone Benchmark

• GPTCloneBench is a large-scale benchmark for semantic and cross-language clone detection, merging GPT-3 generated variants with real code fragments.
• It offers a balanced dataset with over 37K semantic and 20K cross-language clone pairs across Java, C, C#, and Python for robust evaluation.
• Manual validation and advanced filtering techniques ensure high-quality data that reveals biases in AI-generated clone detection.

Searching arXiv for GPTCloneBench and related clone-detection benchmark papers. GPTCloneBench is a benchmark of semantic and cross-language code clones created by combining code fragments from SemanticCloneBench with code generated by OpenAI’s GPT-3 model. It was introduced to address limitations of earlier clone benchmarks, particularly for Type-4 semantic clone detection and cross-language clone detection, by providing a larger, more balanced collection of functionally equivalent but syntactically diverse code pairs across Java, C, C#, and Python (Alam et al., 2023). Subsequent work has used GPTCloneBench both as a dataset for evaluating LLMs on code clone detection and as a benchmark for comparing deep learning and classical clone detectors on AI-generated code (Zhang et al., 2024).

1. Origin, scope, and motivation

GPTCloneBench was proposed in response to specific deficiencies in BigCloneBench, SemanticCloneBench, and CLCDSA. BigCloneBench was originally mined from 2.5 M Java files for recall evaluation of clone detectors and was not designed for Type-4 semantic clones; it also exhibits imbalance issues and is Java-only. SemanticCloneBench provides four languages and good coverage, but its size—1 000 semantic clone pairs per language—is too small for ML training. CLCDSA contains cross-language clones from programming-contest solutions and lacks real-world software functionality (Alam et al., 2023).

The benchmark therefore targets two tasks that are difficult for traditional clone detectors: semantic clone detection and cross-language clone detection. Its stated design goals include a larger pool of true semantic and cross-language clones than SemanticCloneBench, balanced distributions of semantic, non-semantic, and cross-language clones, four-language support, and realistic functional code snippets drawn from StackOverflow questions and answers via SemanticCloneBench rather than toy contest solutions (Alam et al., 2023).

Later studies situate GPTCloneBench more specifically in the context of AI-generated code. One line of work uses it to assess whether GPT-family models can identify code clones in code generated by LLMs as opposed to humans, while another uses it to test whether existing clone detectors remain effective in the presence of AI-generated syntactic and semantic variation (Zhang et al., 2024). This suggests that GPTCloneBench functions not only as a training or evaluation corpus, but also as an instrument for probing detector bias toward machine-generated code patterns.

2. Construction methodology

The construction pipeline begins with seed fragments from SemanticCloneBench. These seeds are real, functional methods mined from StackOverflow answers and cover tasks such as file I/O, string manipulation, sorting, and data structures (Alam et al., 2023). GPT-3, specifically “text-Davinci-003,” was then prompted to generate semantically equivalent but syntactically different variants, as well as cross-language implementations.

The prompt engineering strategy used few-shot prompting. Each prompt included a clear instruction, one example input-to-output pair, and the new code fragment. For same-language clones, two primary prompts were used: “Give me Type-3, Type-4 clone for the following code: <code fragment>” and “Give me 10 distinctive implementations for the following code: <code fragment>.” These two prompts were each repeated twice, issuing four calls per fragment to increase diversity. For cross-language clones, prompts requested implementations in other languages, such as Python, C, and C# from a Java seed, or Python, C, and Java from a C# seed (Alam et al., 2023).

After generation, the authors ran NiCad on each pair with a renaming-blind, at least 3 lines, 99% dissimilarity threshold to compute textual similarity, and they discarded any pair whose similarity exceeded 75%, treating such cases as Type-1 or Type-2 syntactic clones (Alam et al., 2023). Another study describes the same filtering logic at a higher level: newly generated pairs that were Type-I, Type-II, or Type-III clones were filtered out using NiCad, and those filtered-out pairs became the negative set (Pinku et al., 2024).

Manual validation was a substantial part of the pipeline. Nine judges—six trained undergraduates in three pairs and three post-docs—used a GUI validator to label remaining candidates as true positive semantic clones or false positives. Pairs marked true by both undergraduate reviewers were retained, and disagreements or undecided cases were resolved by the post-docs. The total reported human effort was approximately 510 hours (Alam et al., 2023). In addition, functionality testing was performed on 1 000 randomly selected clone pairs, with 99% passing same-input to same-output testing, and CloneCognition was used in reverse as a validator: its low detection accuracy at a 76% similarity threshold was interpreted as confirmation that GPTCloneBench is dominated by true semantic, non-syntactic clones (Alam et al., 2023).

3. Dataset composition and structure

The full released GPTCloneBench contains three major categories: 37 149 true semantic clone pairs, 19 288 false semantic pairs, and 20 770 cross-language clone pairs across Java, C, C#, and Python (Alam et al., 2023). Another study characterizes the released benchmark as comprising 37 149 single-language semantic Type-IV clone pairs, 20 770 cross-language semantic clone pairs, and negative samples drawn from syntactic clones in roughly a 1:1 ratio to the positives (Pinku et al., 2024). These descriptions are compatible in the sense that they distinguish semantic positives from syntactic or non-semantic negatives, though they reflect different summarization choices.

A concise summary of the principal published counts is as follows.

Component	Reported size	Source context
True semantic clones	37 149 pairs	Full benchmark
False semantic pairs	19 288 pairs	Full benchmark
Cross-language clones	20 770 pairs	Full benchmark
Raw GPT-3 output	79 928 clone pairs	Before filtering

The benchmark is reported as approximately 15-fold larger than SemanticCloneBench when semantic and cross-language pairs are combined. The expansion formula is given as

$$\frac{37\,149 + 20\,770}{4\,000}\approx 14.48\approx 15\times$$

and the semantic-only expansion as

$\frac{37\,149}{4\,000}\approx 9.29\times.$

These ratios are used to justify the claim that GPTCloneBench provides the volume required for ML-based clone detection research (Alam et al., 2023).

The same publication also reports a language-level breakdown for true semantic pairs: approximately 9 000 each for Java, C, and C#, and approximately 10 000 for Python (Alam et al., 2023). In addition to the standalone benchmark, clone pairs were injected into four real software systems—JHotDraw, PostgreSQL, Mono, and Django—to support end-to-end tool evaluation in realistic codebases (Alam et al., 2023).

Later work also extracted task-specific subsets from GPTCloneBench. For tractable single-language experiments, one study created a Java-only subset with 11 655 distinct Java methods, an average of 16.73 lines per fragment, 5 711 positive semantic pairs, and 5 431 negative syntactic pairs, yielding an imbalance ratio of 0.95 (Pinku et al., 2024). A separate LLM-evaluation study sampled 300 Moderate Type-3 true pairs and 300 Weak Type-3/Type-4 true pairs because other clone categories were under-represented in the generated data used in that experiment (Zhang et al., 2024). A plausible implication is that GPTCloneBench is not used as a single monolithic evaluation regime; rather, different studies extract subsets tailored to their detector class and evaluation objective.

4. Clone types, similarity, and evaluation formalization

GPTCloneBench is fundamentally oriented toward semantic variation. The benchmark includes same-language semantic clones corresponding to Type-3 and Type-4 phenomena and a substantial cross-language component treated as Type-4 (Alam et al., 2023). Another study further states that within-language clones were restricted to Moderate Type-3, with textual similarity in $[50\%,75\%]$, and Type-4, with functional equivalence but less than 50% textual overlap (Alam et al., 30 Sep 2025).

The underlying SemanticCloneBench uses token-based similarity and AST-based measures. The token similarity is expressed as

$$\text{Sim}_{\mathrm{tok}(A,B)}=\frac{|T(A)\cap T(B)|}{|T(A)\cup T(B)|},$$

where $T(X)$ is the multiset of lexical tokens in snippet $X$. The AST similarity is expressed as

$$\text{Sim}_{\mathrm{AST}(A,B)}=1-\frac{d_{\mathrm{edit}(\mathrm{AST}(A),\mathrm{AST}(B))}}{\max\bigl(|\mathrm{AST}(A)|,|\mathrm{AST}(B)|\bigr)},$$

with $d_{\mathrm{edit}}$ defined as the minimum number of node-insert, delete, or update operations required to transform one tree into the other (Zhang et al., 2024).

Standard classification metrics are repeatedly used in work built on GPTCloneBench:

$$Precision=\frac{TP}{TP+FP}, \qquad Recall=\frac{TP}{TP+FN}, \qquad F_1=2\times\frac{Precision\times Recall}{Precision+Recall}.$$

A later evaluation paper restates the same definitions with alternate symbols:

$$P=\frac{C_r}{C_r+I_c}, \qquad R=\frac{C_r}{C_r+I_{nc}}, \qquad F1=\frac{2\cdot P\cdot R}{P+R}.$$

It also defines the imbalance ratio as

$\frac{37\,149}{4\,000}\approx 9.29\times.$0

where $\frac{37\,149}{4\,000}\approx 9.29\times.$1 denotes semantic Type-IV pairs and $\frac{37\,149}{4\,000}\approx 9.29\times.$2 denotes non-semantic Type I–III syntactic pairs (Pinku et al., 2024).

For correlation analysis in the LLM study, the Pearson correlation coefficient $\frac{37\,149}{4\,000}\approx 9.29\times.$3 was computed between binary model outcomes and a continuous similarity score:

$\frac{37\,149}{4\,000}\approx 9.29\times.$4

That study reports a strong correlation, approximately $\frac{37\,149}{4\,000}\approx 9.29\times.$5 for GPT-4, between true-positive rate and token similarity, and interprets this as evidence of reliance on superficial features rather than deep semantic understanding (Zhang et al., 2024).

5. Empirical use in model evaluation

GPTCloneBench has been used in at least three distinct evaluation settings: assessment of LLMs as clone detectors, comparison of deep learning models across datasets, and evaluation of classical clone detectors on AI-generated code.

In the LLM setting, GPT-3.5 and GPT-4 were evaluated on sampled GPTCloneBench subsets containing 300 Moderate Type-3 pairs and 300 Weak Type-3/Type-4 pairs. Because no negative pairs were present in that evaluation set, accuracy coincided with recall. GPT-3.5 obtained 0.69 recall on MT3 and 0.58 on WT3/T4, whereas GPT-4 obtained 0.84 and 0.77 respectively (Zhang et al., 2024). The same study reports that both models’ true-positive rates increase sharply in the 0.5–1.0 similarity range, while detection rates drop below 0.5 similarity, with GPT-4 retaining superior but still limited performance (Zhang et al., 2024).

In comparison with BigCloneBench, the same paper reports markedly better performance on GPTCloneBench for both models. GPT-3.5 rises from 0.23 to 0.69 on MT3 and from 0.07 to 0.58 on WT3/T4; GPT-4 changes from 0.87 to 0.84 on MT3 and from 0.23 to 0.77 on WT3/T4 (Zhang et al., 2024). The paper interprets this disparity as a pronounced bias favoring detection of LLM-generated clones and emphasizes that both models still struggle with low-similarity human-made clones.

In the deep-learning comparison study, a Java-only subset of GPTCloneBench was used to compare ASTNN, GMN, CodeBERT, CLCDSA, and C4. Reported F1 scores were 0.901 for ASTNN, 0.842 for GMN, 0.990 for CodeBERT, 0.645 for CLCDSA, and 0.996 for C4 (Pinku et al., 2024). The authors observe that transformer-based approaches, especially CodeBERT and the contrastive-learning extension C4, essentially saturate the benchmark, whereas CLCDSA struggles with semantic discrimination and GMN is more affected by extreme syntactic divergence. They further report that C4 varies by less than 1 percent point under mutation-based robustness tests, while GMN and ASTNN can drop by 3–9 percent points (Pinku et al., 2024).

A later study evaluates nine clone detection tools on GPTCloneBench: PMD/CPD, Simian, Deckard, Clone Digger, CCCD, Clone Works, Stone Detector, ASTNN, and CodeBERT. For Java Moderate Type-3 and Type-4 clones respectively, reported recall values include 19% and 2% for PMD/CPD, 16% and 4% for Simian, 51% and 17% for Deckard, 1% and 0.1% for Clone Digger, 31% and 16% for CCCD, 70% and 20% for Clone Works, 54% and 10% for Stone Detector, 50% and 31% for ASTNN, and 87% and 91% for CodeBERT (Alam et al., 30 Sep 2025). That work also reports approximate precision values from expert-judged samples, including approximately 99% for Clone Works and approximately 65% for CodeBERT, and notes that deep-learning tools were infeasible to run end-to-end on large repositories in the reported hardware setup, while tools such as CloneWorks and PMD/CPD processed 1 M LOC projects in under 20 s (Alam et al., 30 Sep 2025).

6. Interpretation, limitations, and future directions

The published literature converges on several recurring interpretations. First, GPTCloneBench is treated as a focused benchmark for semantic and cross-language clone detection, especially in AI-generated code. Second, it is repeatedly described as more balanced than BigCloneBench and more suitable for studying Type-IV behavior or semantic discrimination (Pinku et al., 2024). Third, it exposes a bias in some detectors toward LLM-generated clones: GPT-family models perform better on GPTCloneBench than on human-written BigCloneBench examples, and this is explicitly described as a pronounced bias favoring detection of LLM-generated clones (Zhang et al., 2024).

A common misconception would be to treat GPTCloneBench as a complete replacement for human-authored benchmarks. The literature does not support that view. One study states that GPTCloneBench “proves invaluable for stress-testing LLMs on self-generated patterns but must be complemented with real-world benchmarks (e.g., BCB) to assess generalization” (Zhang et al., 2024). Another concludes that GPTCloneBench and SemanticCloneBench behave similarly for classical detectors, especially in exposing weakness on semantic clones, which reinforces the need for cross-benchmark evaluation rather than sole reliance on one dataset (Alam et al., 30 Sep 2025).

The benchmark also has evaluation-related limitations. One reported use case applies GPTCloneBench purely for recall evaluation and notes that it is not partitioned into train, validation, and test sets (Alam et al., 30 Sep 2025). Another study explicitly recommends introducing negative samples in GPTCloneBench to enable precision and false-positive studies, expanding language coverage to include languages such as JavaScript and Go, applying adversarial generation prompts to force non-trivial semantic transformations, and incorporating human-in-the-loop annotation to validate semantic equivalence beyond generation provenance (Zhang et al., 2024). Some of these recommendations reflect the evaluation subset used in that work rather than the full released benchmark, which already includes false semantic pairs and extensive manual validation (Alam et al., 2023). This suggests that the precise limitations attributed to GPTCloneBench depend on whether the discussion concerns the original released dataset or a derived experimental subset.

Taken together, the literature presents GPTCloneBench as a large-scale, manually validated benchmark centered on semantic and cross-language code clones, with particular relevance to the analysis of AI-generated code. Its main significance lies in supplying a benchmark regime in which syntactic similarity is deliberately weakened, semantic equivalence is foregrounded, class imbalance is reduced relative to older resources, and detector behavior can be compared across human-authored and machine-generated clone distributions (Alam et al., 2023).