DroidCollection: AI Code Detection Dataset
- DroidCollection is a comprehensive dataset comprising over 1 million code samples in 7 programming languages, covering human-written, AI-generated, hybrid, and adversarially humanized code.
- It reframes code-origin detection as a multi-class problem, enabling robust benchmarking that addresses limitations of prior binary detectors through varied generation methods and controlled decoding strategies.
- Paired with the DroidDetect suite, the dataset demonstrates high in-distribution performance and improved resilience to adversarial, cross-domain, and language shifts, with 2-class F1 scores exceeding 99%.
DroidCollection is a large-scale open dataset suite for AI-generated code detection introduced alongside the companion detector suite DroidDetect in "Droid: A Resource Suite for AI-Generated Code Detection" (Orel et al., 11 Jul 2025). It is defined around a multi-way authorship setting rather than a purely binary one, and includes human-written code, fully AI-generated code, human-AI co-authored or machine-refined code, and adversarially humanized code crafted to evade detection. The corpus is reported to comprise over 1 million code samples, 7 programming languages, outputs from 43 coding models, and 3 coding domains, with an associated 157K preference corpus for adversarial humanization and robustness-oriented training (Orel et al., 11 Jul 2025).
1. Scope and problem formulation
DroidCollection was created to address a central limitation in prior AI-generated code detection research: narrow datasets tend to induce narrow detectors. The motivating claims are that many earlier datasets cover fewer than 3 programming languages, often focus on one generator family or only a few models, usually treat detection as a binary human-versus-machine problem, and typically omit realistic collaborative and adversarial settings such as developer–LLM interaction, code rewriting or completion, and deliberately humanized machine outputs (Orel et al., 11 Jul 2025).
The collection therefore formalizes code-origin detection as a broader classification problem. In its reported framing, the relevant categories are human-written, fully AI-generated, human-AI co-authored or machine-refined, and adversarially crafted or humanized code. This structure is significant because it moves the task away from a single stylistic boundary and toward a distributional setting in which class manifolds can overlap. A plausible implication is that the dataset is intended not only for detector benchmarking, but also for studying the geometry of mixed-origin code distributions under multilingual, multidomain, and multigenerator shift.
The paper positions DroidCollection as the most extensive open resource for this task, with broader sample count, language coverage, model diversity, domain coverage, and generation-setting diversity than earlier datasets such as GPT-Sniffer, GPT-Sensor, Whodunit, CoDet-M4, and CodeMirage (Orel et al., 11 Jul 2025). Its design emphasis is not merely scale, but breadth across code provenance modes and evasion conditions.
2. Corpus composition and data construction
DroidCollection is assembled from public human-written code sources and multiple synthetic or hybrid generation procedures. Human-authored code is collected across C++/C, C#, Go, Java, JavaScript, and Python. The paper reports 7 programming languages overall, while the explicit evaluation tables focus on the six language groupings C/C++, C#, Go, Java, Python, and JavaScript (Orel et al., 11 Jul 2025).
The corpus spans three coding domains. General-use code consists of real-world software snippets obtained mainly from StarCoderData and The Vault. Algorithmic problems are drawn from competitive-programming and coding-challenge sources including TACO, CodeNet, LeetCode, and CodeForces. Research or data-science code is based on ObscuraCoder and additional math or data-science code, and is described as often procedural and less modular (Orel et al., 11 Jul 2025). This domain stratification matters because the paper later reports substantial OOD degradation under domain shift.
The synthetic side uses 43 generative models from 11 model families: Llama, CodeLlama, GPT-4o, Qwen, IBM Granite, Yi, DeepSeek, Phi, Gemma, Mistral, and StarCoder. The appendix reports a size range from 2B to 72B, and both open-weights and API-based models are used (Orel et al., 11 Jul 2025). The stated purpose is to avoid overfitting detectors to a single generator family or a single decoding regime.
Generation diversity is explicitly controlled. The paper varies greedy decoding, sampling with temperature in , top-, top-, and beam search with beam width (Orel et al., 11 Jul 2025). This is important because greedy decoding can make machine outputs easier to detect, whereas diversified decoding produces more realistic and harder distributions.
The paper also reports several corpus-level statistics. DroidCollection contains specifically 1.06M samples in the comparison table, with average samples per language of 148,491 and average samples per generator of 8,458. Structural statistics include AST depth percentiles AST@75 = 15, AST@90 = 18, AST@99 = 25, and line length percentiles Line@75 = 107, Line@90 = 135, Line@99 = 314 (Orel et al., 11 Jul 2025). These reported values are used to argue that the corpus contains longer and more structurally varied code than prior datasets.
3. Synthetic, hybrid, and adversarial strata
DroidCollection includes four major synthetic data generation modes. For sources without prompts, the authors generate inverse instructions by asking an LLM to summarize a code snippet and produce a prompt that would generate similar code. When comments or docstrings exist, these are used as prompts; base models receive the code prefix plus comments, while instruction-tuned models receive only the docstring or task. For algorithmic datasets, the original problem statement is used. The authors also create unconditional synthetic tasks from synthetic programmer personas to avoid a bias in which machine-generated code is always conditioned on human-written examples (Orel et al., 11 Jul 2025).
The dataset’s distinctive feature is that it does not stop at fully generated samples. It also includes human-AI co-authored or machine-refined code, intended to reflect real developer usage better than a strict human-versus-AI dichotomy. In addition, it includes adversarially humanized samples, namely AI-generated code deliberately tuned or prompted to look human (Orel et al., 11 Jul 2025). This category is central to the paper’s robustness narrative, because the empirical results show that many detectors are highly vulnerable to superficial humanization.
The appendix reports an approximate class distribution of AI: 25%, Human: 47%, Refined: 13%, and Adv.: 15% (Orel et al., 11 Jul 2025). The paper also releases DroidCollection-Pref, a 157K preference-pair corpus used to train LLMs with DPO so that outputs become more human-like. This preference corpus operationalizes adversarial humanization as a trainable distributional transformation rather than a purely prompt-level manipulation.
Quality control is comparatively strict. The pipeline removes code that fails AST parsing, keeps AST depth between 2 and 31, constrains maximum line length to between 12 and 400 and average line length to between 5 and 140, restricts code length to between 6 and 300 lines, filters on alphanumeric ratio between 0.2 and 0.75, requires English docstrings with more than 99% confidence, and removes duplicates or near-duplicates using MinHash with shingle size and similarity threshold (Orel et al., 11 Jul 2025). The paper nevertheless acknowledges a limitation: some nominally human-written code may in fact have been AI-assisted because post-Copilot-era human sources were not filtered for that possibility.
4. DroidDetect and the training recipe built on DroidCollection
The companion detector suite, DroidDetect, is trained on DroidCollection and released in two encoder-only variants based on ModernBERT: DroidDetect-Base with 149M parameters and DroidDetect-Large with 396M parameters (Orel et al., 11 Jul 2025). The detectors are designed for multi-class classification. The main reported settings are 2-class human versus machine and 3-class human versus machine-generated versus machine-refined, while 4-class experiments in ablations separate adversarial samples as well.
The training objective combines three elements: classification loss, triplet loss or metric learning, and uncertainty-based resampling. During ablations, the metric-learning configuration is referred to as DroidDetect. The uncertainty procedure uses MC Dropout, removes the top 7% most uncertain human-written samples from training, and retrains on the remaining data (Orel et al., 11 Jul 2025). This design directly reflects the paper’s assumption that some human-labeled samples may actually be AI-assisted or ambiguous.
Training is reported for 3 epochs with AdamW, learning rate , warmup proportion 0.1, cosine decay scheduler, and batch size 64 for DroidDetect-Base and 40 for DroidDetect-Large (Orel et al., 11 Jul 2025). The paper notes that a GCN over ASTs and early fusion with structural features were also tested, but yielded only marginal benefit, so the final released detectors remain text-encoder based. The stated rationale for ModernBERT is inference efficiency, mixed code and natural-language pretraining, and suitable performance on code tasks.
The detector suite is thus inseparable from the corpus design. DroidCollection provides the multilingual, multidomain, multigenerator, and adversarial training signal; DroidDetect is the concrete instantiation of how that signal can be exploited with a robustness-oriented objective.
5. Empirical behavior, generalization, and robustness
In-distribution performance is strong. DroidDetect-Base reports 2-class weighted F1 of 99.22 and 3-class weighted F1 of 92.78, while DroidDetect-Large reports 2-class weighted F1 of 99.38 and 3-class weighted F1 of 93.08 (Orel et al., 11 Jul 2025). These results are reported as outperforming baselines including GPTSniffer, CoDet-M4, M4, GCN, and CatBoost.
The paper’s more consequential finding concerns generalization failure outside narrow training distributions. Many zero-shot and earlier fine-tuned detectors perform well only on settings similar to their training data, degrade substantially on unseen languages or domains, and are especially weak outside narrow benchmark distributions. Fast-DetectGPT is identified as the strongest zero-shot baseline, but remains much worse than DroidDetect trained on DroidCollection (Orel et al., 11 Jul 2025). Cross-language transfer is best between syntactically similar languages: C/C++ transfers reasonably to C# and Java, but poorly to Python and JavaScript if those languages are absent from training. Cross-domain transfer is likewise limited; models trained on algorithmic problems do poorly on general-use and research or data-science code, and models trained on research or data-science code also struggle elsewhere.
Adversarial humanization is reported as a major failure mode for many detectors. The paper states that detectors are very fragile against humanizing prompts and alignment-style tuning; GPT-Zero has very poor recall on adversarial samples, while M4 and CoDet-M4 can improve adversarial recall only at the cost of too many false positives on human code (Orel et al., 11 Jul 2025). A recurrent misconception in the area is therefore that binary authorship detection, if accurate enough on ordinary benchmarks, will automatically remain reliable under collaboration and evasion. The DroidCollection results explicitly argue against that simplification.
At the same time, the paper does not present adversarial evasion as intractable. Its central robustness result is that modest amounts of adversarial data during training dramatically improve resilience: DroidDetect trained on DroidCollection with adversarial examples maintains recall above 0.9 on adversarial samples (Orel et al., 11 Jul 2025). Metric learning provides a small but consistent gain; for the Base model, the ablation table reports 2-class performance of 99.15 without triplet loss versus 99.18 full, 3-class performance of 93.86 versus 94.36, and 4-class performance of 92.52 versus 92.95. Uncertainty-based resampling is also reported to improve performance by reducing label noise among ostensibly human samples.
The stress tests extend this picture. Under input truncation, DroidDetect remains strong: Base reports 99.18 at 512 tokens and 91.90 at 128 tokens, while Large reports 99.25 at 512 tokens and 94.91 at 128 tokens (Orel et al., 11 Jul 2025). On unseen external datasets, Swallow-Code recall is 98.95% for Base and 99.11% for Large, and The Heap recall is 94.14% for Base and 96.28% for Large. These results suggest that the paper’s main claim is not merely high in-distribution separability, but improved robustness under length stress, OOD evaluation, and adversarial humanization.
6. Interpretation, limitations, and relation to adjacent literatures
DroidCollection has several stated limitations. Coverage is not complete, because new model families and output distributions will continue to emerge. Cost constraints mean that most data comes from locally deployable models rather than expensive API models, and some newer reasoning models are left for future work. Potential contamination remains, because some human samples may actually be AI-assisted. Language coverage is still limited, with future expansion to PHP, Rust, and Ruby planned. GPTZero evaluation was limited to a sample of 500 code snippets per label-language and label-domain pair because it is a paid API (Orel et al., 11 Jul 2025).
These caveats qualify how the corpus should be interpreted. The dataset is broad, but it is not exhaustive; the human label is operational rather than ontologically guaranteed; and robustness results are contingent on the specific adversarial distributions present in DroidCollection and DroidCollection-Pref. This suggests that future detector evaluation will likely require continual corpus revision as model families, prompting styles, and alignment procedures change.
The term “DroidCollection” is also potentially ambiguous across subfields. In Android-maintenance research, ANDROR2 describes itself as complementary to earlier Android bug-report datasets such as DroidCollection, indicating a distinct prior usage of the name in the mobile bug-reporting literature (Wendland et al., 2021). Other contemporary “Droid-*” resources address different problems entirely, such as Android intent invocation in DroidCall (Xie et al., 2024). Within the AI-generated code detection literature, however, DroidCollection specifically denotes the multilingual, multidomain, multigenerator corpus paired with DroidCollection-Pref and DroidDetect (Orel et al., 11 Jul 2025).
In that sense, DroidCollection occupies a particular methodological position: it reframes code-origin detection from a binary, single-distribution benchmark into a multilingual open-set-like robustness problem involving hybrid authorship and explicit evasion. The paper’s reported lesson is that realistic detection requires broad coverage, adversarial data, and training recipes that account for noisy and overlapping class boundaries, rather than reliance on narrow human-versus-AI separability alone (Orel et al., 11 Jul 2025).