- The paper presents a systematic zero-shot evaluation of DL-based Java clone detectors, comparing them to conventional tools across diverse benchmarks.
- It reveals a sharp 41% drop in average F1 score when models are applied to out-of-distribution datasets, emphasizing limited generalization.
- The study highlights the need for diverse datasets, domain adaptation, and innovative model architectures for robust clone detection.
Zero-shot Generalizability of Deep Learning in Java Clone Detection
Introduction
The proliferation of duplicated or functionally similar but syntactically diverging code—code clones—poses a significant threat to maintainability and defect risk in large software systems. While conventional clone detectors demonstrate high accuracy for near-miss and syntactically identical clones, the detection of semantic clones, which share only underlying functionality, remains a primary challenge. In recent years, deep learning (DL) techniques—particularly transformer-based models pre-trained on large code corpora—demonstrated strong performance on standard clone detection benchmarks, typically BigCloneBench, leading to a widely held assumption of their superiority. However, the generalization capabilities of these models to out-of-distribution (OOD) or previously unseen code remain uncertain.
This work offers a systematic and comprehensive empirical analysis of the zero-shot generalizability of five representative DL-based Java clone detectors, alongside prominent conventional tools, across multiple benchmarks encoding different code domains and variants. The paper provides clear evidence on performance degradation under domain shift and examines the robustness spectrum of state-of-the-art (SOTA) code clone detection methodologies.
Methodology and Experimental Protocol
Benchmarks
Five clone detection datasets for Java are used to capture variability in code provenance and semantics:
- BigCloneBench: The de facto code clone benchmark, heavily used for both training and evaluation in deep clone detection literature, covering 43 functionalities and millions of annotated code pairs.
- SemanticCloneBench: Constructed from Stack Overflow answer pairs, focusing on functional equivalence with minimized syntactic similarity.
- FEMPD: A rigorously validated dataset derived from IJADataset, containing positive and negative pairs of functionally equivalent Java methods.
- SeSaMe: Harvested from large, production-quality open-source Java projects, with annotated degrees of similarity emphasizing semantic similarity.
- ProjectCodeNet: Sourced from online coding competitions, where matched solution pairs for the same task constitute positive clones.
This diverse suite captures both variation in software context (open-source projects, educational content, Q&A snippets, competitive programming) and clone type (syntactic vs. semantic, balanced/unbalanced).
Clone Detection Models
The following DL-based models are examined:
- CodeBERT, GraphCodeBERT, UniXcoder, CodeT5: Transformer-based, pre-trained encoder(-decoder) models, fine-tuned for pairwise similarity/classification.
- FA-AST+GMN: A task-specific model using augmented ASTs with graph neural network-based matching.
Three conventional detectors are included for comparison:
- NiCad: Hybrid, LCS-based with normalization.
- NIL: N-gram based, focusing on large-gap clones.
- StoneDetector: Uses LCS metrics on code fingerprint dominator trees.
All DL models are trained/fine-tuned on BigCloneBench (CodeXGLUE subset) and then evaluated in a zero-shot setting on the four other benchmarks, providing a robust estimate of OOD generalization. All evaluations apply standard precision, recall, F1, and ROC metrics.
Results and Analysis
In-domain Performance
Replication of code clone benchmarks verifies strong performance for all DL models when trained and evaluated on BigCloneBench:
- All DL models achieve F1 ≥ 0.93, confirming previously reported results in the literature.
- Conventional models maintain high precision (≈0.9) but negligible recall (~0.01), corroborating their inability to capture semantic clones in this regime.
Zero-shot Evaluation
When evaluated on benchmarks unlike their training set, the DL-based detectors’ performance declines sharply:
- Average F1 drops by approximately 41% relative to the in-domain setting.
- Precision suffers more than recall (e.g., CodeBERT’s average precision decreases to 0.46), with some settings exhibiting behavior at or below chance.
- No DL model achieves consistent superiority across all benchmarks—performance is highly benchmark-dependent.
Most notably, NiCad and NIL—conventional detectors—demonstrate competitive and often superior performance on SeSaMe, which is derived from high-quality open-source projects, even achieving F1 scores of 0.63 and 0.69, respectively.
FA-AST+GMN, the task-specific GNN-based model, attains the highest average precision among the DL-based detectors in cross-domain evaluation. Relative performance on functionally strict datasets (FEMPD) is uniformly low across all models, highlighting the challenging nature of pure semantic equivalence.
ROC analysis further illustrates the absence of a universally dominant model—no DL-based detector maintains high sensitivity and specificity across all OOD benchmarks, and NiCad frequently traces superior ROC characteristics.
Comparison with Prior Work
Findings align with and extend upon prior analyses questioning the centrality of BigCloneBench as a universal performance indicator [KrinkeR22, KrinkeR25]. Drops in classification efficacy under OOD evaluation are consistent with previous experiments on stricter data splits and different source distributions [SchaferAH22, SonnekalbGBM22, PinkuMR24, KitsiosSBB25]. The augmentation in conventional tools' relative recall and precision in cross-domain scenarios is a robust observation.
Implications and Future Directions
Theoretical Implications
The results clearly demonstrate the limited domain generalization of current DL-based code clone detectors. Overfitting to the idiosyncrasies and functional coverage of BigCloneBench is a persistent issue, with transfer to semantically diverse or contextually distant benchmarks resulting in severe performance degradation. The non-existence of a “free lunch” in code clone detection is empirically grounded—no architecture, threshold, or training paradigm universally outperforms others across all plausible code domains and clone types.
Moreover, precision-recall trade-offs and the instability of binary classification thresholds highlight the inadequacy of single-point metrics (F1), reinforcing the importance of full ROC-spectrum evaluations.
Practical Implications
Clone detector evaluations based solely on BigCloneBench (or its derivatives) are insufficient for practical deployment in heterogeneous, real-world software repositories. Domain shifts—such as Q&A snippets, contest code, or production code—demand detectors with robust and adaptive generalization, which current SOTA models lack.
Additionally, the competitive (and sometimes superior) performance of highly optimized conventional tools in some OOD settings underscores the value of conventional, parametrizable detectors in practical workflows, especially where high-precision and explainability are desired.
Future Research Directions
Improving generalizability mandates a multipronged approach:
- Dataset Diversity: Curation and adoption of benchmarks spanning broader code origins, functionalities, and semantic variants.
- Domain Adaptation: Integrating domain-adaptive fine-tuning strategies, transfer learning, or ensemble techniques for robust OOD performance.
- Unified Benchmarks: Embracing benchmarking platforms such as CloReCo for standardized, reproducible cross-domain performance comparisons.
- Architectural Innovations: Exploration of modality-fusing representations (e.g., unified code and documentation graphs), contrastive learning, and human-in-the-loop mechanisms to attenuate generalization gaps.
Integration of advanced LLMs (e.g., GPT-4o, Llama 3.3) should also be systematically analyzed within this zero-shot evaluation paradigm, noting their current inferior performance relative to code-specialized models [KitsiosSBB25].
Conclusion
This study comprehensively articulates the limitation of current DL-based Java code clone detectors in zero-shot, cross-domain settings by providing strong, multi-benchmark experimental evidence (2604.13783). While these models achieve high effectiveness on their training domain, their generalization to unseen code is consistently poor, and domain-shifted benchmarks reveal that even conventional detectors may outperform DL-based methods. This empirical insight mandates urgent innovation in benchmark construction, evaluation design, and model adaptation to meet the versatility requirements of practical clone detection.