MultiAIGCD: AI Code Detection Benchmark

• MultiAIGCD is a dataset and benchmark that evaluates AI-generated code detection across three languages and diverse usage scenarios.
• It benchmarks detectors under standard, cross-LLM, and cross-language settings, revealing strong performance on scratch generation but challenges in repair scenarios.
• The benchmark leverages outputs from six LLMs and three prompting strategies to stress-test detector robustness and generalization in real-world contexts.

MultiAIGCD is a dataset and benchmark for detecting AI-generated code across multiple programming languages, generator models, prompting strategies, and usage scenarios. It was introduced to support research on AI-generated code detection in settings closer to actual practice than prior datasets, especially because LLMs are used not only to generate code from problem descriptions but also to repair human-written code by fixing runtime errors or incorrect outputs. The dataset covers Python, Java, and Go; includes code produced by six different LLMs under three prompting strategies; and benchmarks three detection models under standard, cross-LLM, and cross-language settings. Its central empirical result is that current detectors perform strongly on code generated from scratch, but performance degrades in repair scenarios and generalization is substantially weaker when the test language is unseen (Demirok et al., 29 Jul 2025).

1. Motivation and problem setting

MultiAIGCD is motivated by the rapid expansion of LLM-based code generation and by the corresponding need for robust AI-generated code detection in contexts such as academic integrity and fairness in hiring. The paper argues that most existing studies concentrate on a narrow regime, typically code generated directly from problem statements, even though real-world LLM use also includes debugging and repair. In that sense, MultiAIGCD is designed as a broader benchmark for detecting AI-generated code under conditions that more closely resemble real-world usage (Demirok et al., 29 Jul 2025).

The benchmark is organized around three key usage scenarios. Scenario_Scratch covers generation from the problem description. Scenario_Runtime covers fixing human code that has a runtime error. Scenario_Output covers correcting human code that produces wrong output. This design directly encodes the paper’s claim that a detector must generalize across languages, models, prompts, and usage scenarios, rather than only distinguish freshly generated code from human-authored solutions.

A common misconception in this area is that “AI-generated code detection” is equivalent to detecting code produced from an empty prompt or a problem statement. MultiAIGCD explicitly rejects that assumption by including repair-based uses of LLMs, where the model receives a problem description together with a human-written snippet exhibiting the relevant error type. This broadening matters because detectors trained only on generation-from-scratch distributions may fail when the model is used as a repair assistant.

2. Source corpus and human-code construction

The dataset is built on IBM CodeNet, which provides about 14 million solutions over roughly 4,000 problems and 55 languages, together with labeled submission statuses such as accepted, compile-time error, runtime error, wrong answer, and time limit exceeded (Demirok et al., 29 Jul 2025). Following prior work, the authors select the 800 problems in CodeNet’s Python Benchmark because these problems have associated descriptions and enough examples to support generation and comparison.

For each of these 800 problems, the construction process extracts up to five human-written submissions for each of three statuses: accepted, runtime error, and wrong answer, across Python, Java, and Go. The deliberate inclusion of incorrect human code is a defining feature of the benchmark. It allows comparison between human and AI code for the same task under multiple code qualities, rather than restricting the human side to accepted solutions only.

Before post-processing, the human sample contains 33,286 human-written CodeNet snippets: 11,873 Java, 12,000 Python, and 9,413 Go. After quality control and post-processing, the final dataset contains 32,148 human-written snippets (Demirok et al., 29 Jul 2025).

The paper also formalizes the scenario/status pairings through three subset labels:

Subset label	Meaning
GS/A	generate from scratch / accepted human code
FR/R	fix runtime error / runtime-error human code
CO/W	correct output / wrong-answer human code

These labels are important because they make explicit that the benchmark does not compare arbitrary AI and human programs, but AI and human code linked to the same underlying task and to matched correctness conditions.

3. AI-code generation protocol

To construct the AI-generated side, the authors use six LLMs: Llama-3.3-70B-Instruct-Turbo, Qwen2.5-Coder-32B-Instruct, GPT-4o, DeepSeek-V3, OpenAI o3-mini, and Claude 3.5 Sonnet v2 (Demirok et al., 29 Jul 2025). Generation is performed for the three scenarios already noted: from-scratch generation, runtime-error repair, and wrong-output correction.

For each scenario, the benchmark applies three prompting strategies. Lazy prompting uses a minimal direct instruction. Role prompting assigns the model the role of an expert programmer. Rephrase and Respond requires the model first to rephrase the prompt and then answer. The prompt text differs slightly by scenario, and in the repair settings the model receives both the problem description and a human-written snippet with the relevant error type.

An important design choice is that the prompt formulation intentionally discourages markdown code fences and extra explanation. The reason is methodological rather than cosmetic: such artifacts would make AI-generated code easier to detect for superficial stylistic reasons, which would weaken the benchmark’s value as an assessment of detector robustness. This suggests that MultiAIGCD is constructed to reduce trivial detection shortcuts and to foreground more substantive distributional differences between human and machine-generated code.

Before filtering, the generation pipeline produces 124,434 AI-generated code snippets in total: 43,110 Java, 38,124 Go, and 43,200 Python. After post-processing, the final dataset contains 121,271 AI-generated snippets (Demirok et al., 29 Jul 2025).

4. Post-processing, validation, and final composition

Post-processing removes unusable outputs such as empty responses, C-family code when the target is not C-family, or answers that ignore the instruction (Demirok et al., 29 Jul 2025). Validation is language-specific. Java snippets are validated with javalang, Python snippets with AST parsing, and Go snippets by heuristic checks due to parser integration limitations.

The final benchmark contains 153,419 examples in total, consisting of 121,271 AI-generated snippets and 32,148 human-written snippets. The final per-language totals in Table 2 are as follows:

Language	Scratch	Runtime	Output
Java	18,036	17,708	17,913
Python	18,072	17,109	17,963
Go	17,998	11,170	17,450

Summed across all languages and scenarios, the final LLM counts are 20,526 Qwen, 20,610 Llama, 20,502 DeepSeek-V3, 20,699 Claude, 20,635 GPT-4o, and 18,299 o3-mini (Demirok et al., 29 Jul 2025). The row and column totals yield the overall size of 153,419.

The composition of the dataset is part of its methodological significance. It spans three languages, six LLMs, three prompting methods, and three practical generation scenarios, including repair tasks that are often ignored in the literature. A plausible implication is that MultiAIGCD serves not merely as a static corpus, but as a stress test for claims of detector robustness under heterogeneous generation conditions.

5. Detection models and benchmark protocol

The benchmark evaluates three state-of-the-art AI-generated code detection models. SVM_Ada is an SVM using OpenAI text-embedding-ada-002 embeddings. SVM_T5+ is an SVM using CodeT5+ embeddings. CodeBERTa is a fine-tuned distilled CodeBERT variant (Demirok et al., 29 Jul 2025).

The 800 problems are split into 80% training, 10% validation, 10% test, with the key constraint that code for the same problem does not leak across splits. The authors also try to balance the splits by CodeNet score. Experiments are run separately for the three scenarios, always using the human-written samples plus the AI-generated samples for the corresponding scenario.

The main benchmark reports Accuracy (A), Precision (P), Recall (R), and F1. The strongest overall pattern is that all models perform best in Scenario_Scratch, while performance drops in the repair settings. Representative results include the following (Demirok et al., 29 Jul 2025):

• In Java, Scratch, CodeBERTa reaches F1 = 0.9815, compared with 0.9778 for SVM_Ada and 0.95 for SVM_T5+.
• In Python, Scratch, CodeBERTa again leads with F1 = 0.9501, compared with 0.9245 for SVM_Ada and 0.8861 for SVM_T5+.
• In Go, Scratch, SVM_Ada is best with F1 = 0.9664, slightly above CodeBERTa at 0.9546 and SVM_T5+ at 0.9261.

In the more difficult repair settings, performance declines materially. In runtime fixing, Java F1 drops to roughly 0.755–0.807, Python to 0.669–0.721, and Go to 0.605–0.863 depending on model. In output correction, Java F1 is about 0.737–0.805, Python about 0.715–0.790, and Go about 0.774–0.834.

The per-LLM analysis in the scratch scenario further shows that generator identity matters. CodeBERTa is often the best detector, and GPT-4o-generated code is generally easiest to detect, whereas Llama and o3-mini are harder. The paper gives examples in which CodeBERTa achieves accuracy as high as 0.9958 on GPT-4o-generated Python code and 1.0 on Qwen-generated Java code (Demirok et al., 29 Jul 2025). This supports the paper’s qualitative conclusion that different generators leave distinct stylistic signatures.

6. Cross-LLM and cross-language generalization

MultiAIGCD places particular emphasis on generalization tests. In the cross-LLM evaluation, the authors use a leave-one-LLM-out setup in the scratch scenario: detectors are trained on all generators except one and then tested on the held-out generator plus human code (Demirok et al., 29 Jul 2025). This directly measures whether a detector can generalize to an unseen model family.

Reported F1 values include 0.8968 for SVM_Ada, 0.7895 for SVM_T5+, and 0.9085 for CodeBERTa when holding out Qwen; 0.8803, 0.8106, and 0.9240 respectively when holding out GPT-4o; and 0.7679, 0.6997, and 0.6969 when holding out o3-mini. The paper concludes that cross-LLM generalization is fairly strong overall, although the hardest held-out cases are Llama and o3-mini, while GPT-4o is easiest.

The cross-language setting is substantially harsher. Here the detectors are trained on two languages and tested on the third, again in the scratch scenario. The reported F1 scores are:

Train languages	Test language	SVM_Ada	SVM_T5+	CodeBERTa
Java + Python	Go	0.3085	0.1206	0.0712
Go + Python	Java	0.5856	0.4255	0.7879
Java + Go	Python	0.8247	0.7885	0.7655

This is one of the paper’s strongest findings: cross-language transfer is much weaker than cross-model transfer, especially for Go (Demirok et al., 29 Jul 2025). The result suggests that language-specific syntax and style strongly shape detector behavior. It also qualifies any overly broad claim that a detector trained on one programming language is inherently learning generator-specific artifacts independent of language.

Across these experiments, SVM_Ada is usually the most robust baseline, while SVM_T5+ is often the weakest. CodeBERTa generally achieves the strongest performance on Java and in many scratch or held-out-LLM settings, but not uniformly across all scenarios. The benchmark therefore does not support a single universal ranking of detection architectures; rather, performance depends on scenario, language, and generalization regime.

7. Significance, misconceptions, and limits of current detection

The principal significance of MultiAIGCD lies in benchmark design. By spanning three languages, six LLMs, three prompting methods, and three practical generation scenarios, it is substantially more comprehensive than prior datasets that focus only on generation from problem statements (Demirok et al., 29 Jul 2025). The benchmark thereby reframes AI-generated code detection as a robustness problem over interacting distribution shifts, rather than as a narrow binary classification problem on a single generation channel.

One misconception addressed by the benchmark is that strong headline accuracy in a standard train-test setting is sufficient evidence of real-world detection capability. MultiAIGCD shows that this is not the case. Current detectors are quite good at identifying code generated from problem statements, but detection becomes notably harder for AI-fixed code, and robustness degrades further under cross-language transfer. Another misconception is that prompt variation is secondary; the benchmark’s inclusion of three prompting strategies indicates that prompt-induced stylistic variability is part of the detection problem itself.

The paper’s overall conclusion is that robustness in AI-generated code detection remains limited, especially under cross-language and repair-based conditions (Demirok et al., 29 Jul 2025). This suggests that future work in the area must confront broader notions of generator use, language dependence, and distributional generalization. MultiAIGCD is explicitly released together with dataset and code resources to support that line of research.