---
title: 'MultiAIGCD: AI Code Detection Benchmark'
url: https://www.emergentmind.com/topics/multiaigcd
type: topic
---

# MultiAIGCD: AI Code Detection Benchmark

MultiAIGCD is a dataset and benchmark for detecting AI-generated code across multiple programming languages, generator models, prompting strategies, and usage scenarios. It was introduced to support research on AI-generated code detection in settings closer to actual practice than prior datasets, especially because large language models are used not only to generate code from problem descriptions but also to repair human-written code by fixing runtime errors or incorrect outputs. The dataset covers Python, Java, and Go; includes code produced by six different LLMs under three prompting strategies; and benchmarks three detection models under standard, cross-LLM, and cross-language settings. Its central empirical result is that current detectors perform strongly on code generated from scratch, but performance degrades in repair scenarios and generalization is substantially weaker when the test language is unseen [2507.21693].

## 1. Motivation and problem setting

MultiAIGCD is motivated by the rapid expansion of LLM-based code generation and by the corresponding need for robust AI-generated code detection in contexts such as academic integrity and fairness in hiring. The paper argues that most existing studies concentrate on a narrow regime, typically code generated directly from problem statements, even though real-world LLM use also includes debugging and repair. In that sense, MultiAIGCD is designed as a broader benchmark for detecting AI-generated code under conditions that more closely resemble real-world usage [2507.21693].

The benchmark is organized around three key usage scenarios. **Scenario\_Scratch** covers generation from the problem description. **Scenario\_Runtime** covers fixing human code that has a runtime error. **Scenario\_Output** covers correcting human code that produces wrong output. This design directly encodes the paper’s claim that a detector must generalize across languages, models, prompts, and usage scenarios, rather than only distinguish freshly generated code from human-authored solutions.

A common misconception in this area is that “AI-generated code detection” is equivalent to detecting code produced from an empty prompt or a problem statement. MultiAIGCD explicitly rejects that assumption by including repair-based uses of LLMs, where the model receives a problem description together with a human-written snippet exhibiting the relevant error type. This broadening matters because detectors trained only on generation-from-scratch distributions may fail when the model is used as a repair assistant.

## 2. Source corpus and human-code construction

The dataset is built on **IBM CodeNet**, which provides about **14 million** solutions over roughly **4,000 problems** and **55 languages**, together with labeled submission statuses such as accepted, compile-time error, runtime error, wrong answer, and time limit exceeded [2507.21693]. Following prior work, the authors select the **800 problems** in CodeNet’s Python Benchmark because these problems have associated descriptions and enough examples to support generation and comparison.

For each of these 800 problems, the construction process extracts up to **five human-written submissions** for each of three statuses: **accepted**, **runtime error**, and **wrong answer**, across **Python, Java, and Go**. The deliberate inclusion of incorrect human code is a defining feature of the benchmark. It allows comparison between human and AI code for the same task under multiple code qualities, rather than restricting the human side to accepted solutions only.

Before post-processing, the human sample contains **33,286 human-written CodeNet snippets**: **11,873 Java**, **12,000 Python**, and **9,413 Go**. After quality control and post-processing, the final dataset contains **32,148 human-written snippets** [2507.21693].

The paper also formalizes the scenario/status pairings through three subset labels:

| Subset label | Meaning |
|---|---|
| GS/A | generate from scratch / accepted human code |
| FR/R | fix runtime error / runtime-error human code |
| CO/W | correct output / wrong-answer human code |

These labels are important because they make explicit that the benchmark does not compare arbitrary AI and human programs, but AI and human code linked to the same underlying task and to matched correctness conditions.

## 3. AI-code generation protocol

To construct the AI-generated side, the authors use **six LLMs**: **Llama-3.3-70B-Instruct-Turbo, Qwen2.5-Coder-32B-Instruct, GPT-4o, DeepSeek-V3, OpenAI o3-mini, and Claude 3.5 Sonnet v2** [2507.21693]. Generation is performed for the three scenarios already noted: from-scratch generation, runtime-error repair, and wrong-output correction.

For each scenario, the benchmark applies **three prompting strategies**. **Lazy prompting** uses a minimal direct instruction. **Role prompting** assigns the model the role of an expert programmer. **Rephrase and Respond** requires the model first to rephrase the prompt and then answer. The prompt text differs slightly by scenario, and in the repair settings the model receives both the problem description and a human-written snippet with the relevant error type.

An important design choice is that the prompt formulation intentionally discourages markdown code fences and extra explanation. The reason is methodological rather than cosmetic: such artifacts would make AI-generated code easier to detect for superficial stylistic reasons, which would weaken the benchmark’s value as an assessment of detector robustness. This suggests that MultiAIGCD is constructed to reduce trivial detection shortcuts and to foreground more substantive distributional differences between human and machine-generated code.

Before filtering, the generation pipeline produces **124,434 AI-generated code snippets** in total: **43,110 Java**, **38,124 Go**, and **43,200 Python**. After post-processing, the final dataset contains **121,271 AI-generated snippets** [2507.21693].

## 4. Post-processing, validation, and final composition

Post-processing removes unusable outputs such as empty responses, C-family code when the target is not C-family, or answers that ignore the instruction [2507.21693]. Validation is language-specific. Java snippets are validated with **javalang**, Python snippets with **AST parsing**, and Go snippets by heuristic checks due to parser integration limitations.

The final benchmark contains **153,419** examples in total, consisting of **121,271 AI-generated snippets** and **32,148 human-written snippets**. The final per-language totals in Table 2 are as follows:

| Language | Scratch | Runtime | Output |
|---|---:|---:|---:|
| Java | 18,036 | 17,708 | 17,913 |
| Python | 18,072 | 17,109 | 17,963 |
| Go | 17,998 | 11,170 | 17,450 |

Summed across all languages and scenarios, the final LLM counts are **20,526 Qwen**, **20,610 Llama**, **20,502 DeepSeek-V3**, **20,699 Claude**, **20,635 GPT-4o**, and **18,299 o3-mini** [2507.21693]. The row and column totals yield the overall size of **153,419**.

The composition of the dataset is part of its methodological significance. It spans **three languages**, **six LLMs**, **three prompting methods**, and **three practical generation scenarios**, including repair tasks that are often ignored in the literature. A plausible implication is that MultiAIGCD serves not merely as a static corpus, but as a stress test for claims of detector robustness under heterogeneous generation conditions.

## 5. Detection models and benchmark protocol

The benchmark evaluates three state-of-the-art AI-generated code detection models. **SVM\_Ada** is an SVM using OpenAI **text-embedding-ada-002** embeddings. **SVM\_T5+** is an SVM using **CodeT5+** embeddings. **CodeBERTa** is a fine-tuned distilled CodeBERT variant [2507.21693].

The **800 problems** are split into **80% training, 10% validation, 10% test**, with the key constraint that code for the same problem does not leak across splits. The authors also try to balance the splits by CodeNet score. Experiments are run separately for the three scenarios, always using the human-written samples plus the AI-generated samples for the corresponding scenario.

The main benchmark reports **Accuracy (A)**, **Precision (P)**, **Recall (R)**, and **F1**. The strongest overall pattern is that all models perform best in **Scenario\_Scratch**, while performance drops in the repair settings. Representative results include the following [2507.21693]:

- In **Java, Scratch**, **CodeBERTa** reaches **F1 = 0.9815**, compared with **0.9778** for **SVM\_Ada** and **0.95** for **SVM\_T5+**.
- In **Python, Scratch**, **CodeBERTa** again leads with **F1 = 0.9501**, compared with **0.9245** for **SVM\_Ada** and **0.8861** for **SVM\_T5+**.
- In **Go, Scratch**, **SVM\_Ada** is best with **F1 = 0.9664**, slightly above **CodeBERTa** at **0.9546** and **SVM\_T5+** at **0.9261**.

In the more difficult repair settings, performance declines materially. In **runtime fixing**, Java F1 drops to roughly **0.755–0.807**, Python to **0.669–0.721**, and Go to **0.605–0.863** depending on model. In **output correction**, Java F1 is about **0.737–0.805**, Python about **0.715–0.790**, and Go about **0.774–0.834**.

The per-LLM analysis in the scratch scenario further shows that generator identity matters. **CodeBERTa** is often the best detector, and **GPT-4o-generated** code is generally easiest to detect, whereas **Llama** and **o3-mini** are harder. The paper gives examples in which **CodeBERTa** achieves accuracy as high as **0.9958** on GPT-4o-generated Python code and **1.0** on Qwen-generated Java code [2507.21693]. This supports the paper’s qualitative conclusion that different generators leave distinct stylistic signatures.

## 6. Cross-LLM and cross-language generalization

MultiAIGCD places particular emphasis on generalization tests. In the **cross-LLM** evaluation, the authors use a **leave-one-LLM-out** setup in the scratch scenario: detectors are trained on all generators except one and then tested on the held-out generator plus human code [2507.21693]. This directly measures whether a detector can generalize to an unseen model family.

Reported **F1** values include **0.8968** for **SVM\_Ada**, **0.7895** for **SVM\_T5+**, and **0.9085** for **CodeBERTa** when holding out **Qwen**; **0.8803**, **0.8106**, and **0.9240** respectively when holding out **GPT-4o**; and **0.7679**, **0.6997**, and **0.6969** when holding out **o3-mini**. The paper concludes that cross-LLM generalization is **fairly strong overall**, although the hardest held-out cases are **Llama** and **o3-mini**, while **GPT-4o** is easiest.

The **cross-language** setting is substantially harsher. Here the detectors are trained on two languages and tested on the third, again in the scratch scenario. The reported **F1** scores are:

| Train languages | Test language | SVM\_Ada | SVM\_T5+ | CodeBERTa |
|---|---|---:|---:|---:|
| Java + Python | Go | 0.3085 | 0.1206 | 0.0712 |
| Go + Python | Java | 0.5856 | 0.4255 | 0.7879 |
| Java + Go | Python | 0.8247 | 0.7885 | 0.7655 |

This is one of the paper’s strongest findings: **cross-language transfer is much weaker than cross-model transfer**, especially for **Go** [2507.21693]. The result suggests that language-specific syntax and style strongly shape detector behavior. It also qualifies any overly broad claim that a detector trained on one programming language is inherently learning generator-specific artifacts independent of language.

Across these experiments, **SVM\_Ada** is usually the most robust baseline, while **SVM\_T5+** is often the weakest. **CodeBERTa** generally achieves the strongest performance on Java and in many scratch or held-out-LLM settings, but not uniformly across all scenarios. The benchmark therefore does not support a single universal ranking of detection architectures; rather, performance depends on scenario, language, and generalization regime.

## 7. Significance, misconceptions, and limits of current detection

The principal significance of MultiAIGCD lies in benchmark design. By spanning **three languages**, **six LLMs**, **three prompting methods**, and **three practical generation scenarios**, it is substantially more comprehensive than prior datasets that focus only on generation from problem statements [2507.21693]. The benchmark thereby reframes AI-generated code detection as a robustness problem over interacting distribution shifts, rather than as a narrow binary classification problem on a single generation channel.

One misconception addressed by the benchmark is that strong headline accuracy in a standard train-test setting is sufficient evidence of real-world detection capability. MultiAIGCD shows that this is not the case. Current detectors are **quite good at identifying code generated from problem statements**, but detection becomes notably harder for **AI-fixed code**, and robustness degrades further under **cross-language** transfer. Another misconception is that prompt variation is secondary; the benchmark’s inclusion of three prompting strategies indicates that prompt-induced stylistic variability is part of the detection problem itself.

The paper’s overall conclusion is that robustness in AI-generated code detection remains limited, especially under **cross-language** and **repair-based** conditions [2507.21693]. This suggests that future work in the area must confront broader notions of generator use, language dependence, and distributional generalization. MultiAIGCD is explicitly released together with dataset and code resources to support that line of research.

Source: https://www.emergentmind.com/topics/multiaigcd