---
title: 'LLMScan: Code Paraphrase Detection & Attribution'
url: https://www.emergentmind.com/topics/llmscan
type: topic
---

# LLMScan: Code Paraphrase Detection & Attribution

LLMScan is a practical system for detecting whether a candidate program is an LLM-paraphrase of an original human-written program and, when a paraphrase is detected, attributing the paraphrase to a responsible LLM among a known set. In the formulation associated with "Detection of LLM-Paraphrased Code and Identification of the Responsible LLM Using Coding Style Features," LLMScan operates on code pairs $(H, C)$, where $H$ is original human-written code and $C$ is candidate code, and it targets style-preserving functionality rather than semantic equivalence proofs by execution. Its central premise is that LLM paraphrasing introduces measurable coding-style shifts—in naming consistency, code structure, and readability—that can be exploited for both paraphrase detection and model attribution without requiring LLM internals or watermarking [2502.17749].

## 1. Problem formulation and threat model

LLMScan addresses two complementary tasks. The first is **paraphrase detection**, which asks whether $C$ is an LLM-paraphrase of $H$ that preserves functionality while altering style. The second is **model attribution**, which asks, given that $C$ is an LLM-paraphrase of $H$, which LLM produced $C$ among a known set [2502.17749].

The motivating threat model assumes an adversary who uses an LLM to paraphrase protected code in order to evade naive plagiarism checks while preserving functionality and altering surface style, including naming, comments, formatting, and minor restructuring. The defender is assumed to have access to candidate pairs $(H, C)$ and to a training corpus of pairs of the form $(H, \mathrm{LLM}(H))$, but not to model internals or watermarking signals. Within this scope, LLMScan does not attempt to prove semantic equivalence through execution; instead, it detects consistent stylistic shifts introduced during LLM paraphrasing [2502.17749].

This positioning distinguishes LLMScan from traditional clone or plagiarism detection. The paper explicitly contrasts it with Type-3 and Type-4 clone detection, which emphasize structural or semantic similarity, whereas LLMScan explicitly exploits stylometric signals. It is also distinguished from watermarking-based provenance approaches: unlike watermarks, which may be absent, brittle, or removable, LLMScan is described as a stylometry-inspired, model-agnostic detector based on measurable coding-style features [2502.17749].

A common misconception is that paraphrase detection here is equivalent to authorship attribution or code clone detection in the ordinary sense. The paper frames the task differently: the target is not generic authorship, nor exact semantic proof, but detection of stylistic regularities introduced when LLMs paraphrase human code. A plausible implication is that LLMScan is best understood as a pairwise stylometric detector rather than a general-purpose semantic equivalence engine.

## 2. LPcode dataset and experimental substrate

The system is built on **LPcode**, a dataset of paired human-written and LLM-paraphrased code. LPcode contains pairs across four programming languages—C, C++, Java, and Python—and four LLM generators: OpenAI GPT-3.5 (ChatGPT), Google Gemini-Pro, WizardCoder-33B, and DeepSeek-Coder-33B [2502.17749].

The collection pipeline is tightly specified. Human code was crawled from GitHub repositories created between Jan 1, 2019 and Jan 1, 2020 to avoid post-LLM leakage; only code with Apache, BSD, or MIT licenses and code parsable into ASTs was retained. For each human sample, an LLM was prompted to paraphrase the code while preserving functionality and language. Near-duplicate removal was then performed by computing LCS similarity between $H$ and $\mathrm{LLM}(H)$ and removing pairs in the top quartile, defined as at least the 75th percentile of similarity, to avoid trivial copies. Only LLM outputs that parse into ASTs were kept, and emails, URLs, and phone numbers were stripped with regex-based anonymization [2502.17749].

The final dataset contains 21,355 files, organized as 4,271 per source type across five source types: Human, ChatGPT, Gemini-Pro, WizardCoder, and DeepSeek-Coder. The balanced per-language counts for each source type are as follows.

| Language | Files per source type | Source types |
|---|---:|---|
| C | 457 each | Human, ChatGPT, Gemini-Pro, WizardCoder, DeepSeek-Coder |
| C++ | 385 each | Human, ChatGPT, Gemini-Pro, WizardCoder, DeepSeek-Coder |
| Java | 1,494 each | Human, ChatGPT, Gemini-Pro, WizardCoder, DeepSeek-Coder |
| Python | 1,935 each | Human, ChatGPT, Gemini-Pro, WizardCoder, DeepSeek-Coder |

From LPcode, Task 1 is constructed as a balanced binary classification problem in which positive pairs are $(H, \mathrm{LLM}(H))$ and negatives are $(H, \mathrm{LLM}(H'))$ with $H' \ne H$, sampled at a 1:1 positive-to-negative ratio. Task 2 is a four-way attribution problem over \{ChatGPT, Gemini-Pro, WizardCoder, DeepSeek-Coder\}, given $(H, \mathrm{LLM}(H))$ [2502.17749].

The reported pair counts are 3,656, 3,080, 11,952, and 15,480 for Task 1 in C, C++, Java, and Python respectively, and 1,828, 1,540, 5,976, and 7,740 for Task 2 in the same language order. All experiments use 5-fold cross-validation [2502.17749].

The dataset design matters because the paper explicitly filters for non-trivial paraphrases and AST-valid code. This suggests that the benchmark is intended to stress paraphrase detection under realistic stylistic variation rather than reward trivial near-duplicate matching.

## 3. Coding-style features and statistical basis

LLMScan represents each code file $x$ by a 10-dimensional vector $f(x)$ composed of three feature families: **Naming Consistency** with four features, **Code Structure** with three features, and **Readability** with three features. All features are computed by static analysis using lexing, AST parsing, and simple string rules; the paper states that no additional token-distribution or AST-graph features are used, and no feature selection beyond this curated set is applied [2502.17749].

The feature families are organized as follows.

| Feature family | Features | Intended signal |
|---|---|---|
| Naming Consistency | Function, variable, class, constant naming consistency | Regularization of naming patterns |
| Code Structure | Indentation consistency, function length, nesting depth | Stylistic regularity of code organization |
| Readability | Comment ratio, function name length, variable name length | Documentation and identifier readability habits |

For identifier type $e \in \{\text{function}, \text{variable}, \text{class}, \text{constant}\}$, the naming-consistency score is defined over the pattern set $\{\text{camelCase}, \text{snake_case}, \text{PascalCase}, \text{UPPER\_SNAKE\_CASE}, \text{Other}\}$ as
$$
NC_e(x) = \max_{p \in P} \frac{\mathrm{count}_p}{|N_e|}.
$$
The four corresponding features are $NC_{\text{function}}$, $NC_{\text{variable}}$, $NC_{\text{class}}$, and $NC_{\text{const}}$ [2502.17749].

For code structure, indentation consistency is
$$
\mathrm{IndentCons}(x) = \frac{\max_l \mathrm{count}_l}{\sum_l \mathrm{count}_l},
$$
where indentation lengths are counted at line starts. Function length is the average number of lines per function,
$$
\mathrm{FuncLen}(x) = \frac{\sum_i L_i}{N_F},
$$
and nesting depth is the average depth of nested blocks,
$$
\mathrm{NestDepth}(x) = \frac{\sum_b \mathrm{depth}(b)}{N_B}.
$$
Readability features are comment ratio,
$$
\mathrm{CommentRatio}(x) = \frac{\# \text{comment lines}}{\# \text{total lines}},
$$
function name length,
$$
\mathrm{FuncNameLen}(x) = \frac{\sum_i |\mathrm{name}_i|}{N_F},
$$
and variable name length,
$$
\mathrm{VarNameLen}(x) = \frac{\sum_j |\mathrm{name}_j|}{N_V}.
$$

The statistical justification for these features is based on one-way ANOVA per feature, comparing human code with pooled LLM outputs separately for each language at significance level $\alpha = 0.05$. The top two reported discriminators by language are: for C, Comment Ratio with $F = 153.71, p = 3e{-34}$ and Function Length with $F = 19.92, p = 8e{-06}$; for C++, Comment Ratio with $F = 166.02, p = 2e{-36}$ and Class Naming Consistency with $F = 3.87, p = 4e{-02}$; for Java, Comment Ratio with $F = 689.31, p = 2e{-145}$ and Variable Naming Consistency with $F = 21.12, p = 4e{-06}$; and for Python, Comment Ratio with $F = 574.48, p = 2e{-123}$ and Function Length with $F = 70.40, p = 6e{-17}$ [2502.17749].

The paper interprets Comment Ratio as the strongest statistically significant discriminator across all four languages, reflecting a tendency of LLMs to inject more systematic documentation patterns than diverse human habits. The second most discriminative feature is language-dependent: Function Length in C and Python, Class Naming Consistency in C++, and Variable Naming Consistency in Java [2502.17749].

These results are central to the system’s logic. Rather than using broad semantic representations or dense embeddings, LLMScan relies on a deliberately small stylometric signature. The paper’s ablations further report that using all three feature groups together yields the best F1, while single-family models show that Readability dominates in C and Python and Code Structure dominates in C++ and Java [2502.17749].

## 4. LPcodedec: pairwise detection and attribution method

The concrete detector underlying LLMScan is **LPcodedec**. For a pair $(H, C)$, the method computes $f(H), f(C) \in \mathbb{R}^{10}$ and concatenates them into
$$
z = [f(H); f(C)] \in \mathbb{R}^{20}.
$$
The paper states that this representation preserves directionality and captures pairwise style shifts [2502.17749].

LPcodedec uses a lightweight MLP implemented as scikit-learn `MLPClassifier` and trained with cross-entropy. For Task 1, the label space is binary, $y \in \{0,1\}$, with loss
$$
L = -[y \log p + (1-y)\log(1-p)].
$$
For Task 2, the label space is four-way, $y \in \{1..4\}$, with loss
$$
L = -\sum_k \mathbf{1}[y=k]\log p_k.
$$
At inference time, Task 1 predicts paraphrase probability $p$ and compares it with threshold $\tau$; if the pair is classified as a paraphrase, Task 2 outputs $\arg\max_k p_k$ for attribution [2502.17749].

The training and inference workflow is correspondingly simple. During training, each training pair is parsed, features are extracted, the 20-dimensional vector $z$ is built, and the MLP is trained with cross-entropy. During inference, the same feature extraction pipeline is applied, then Task 1 yields $\hat{y} = (p \ge \tau)$, and if $\hat{y}=1$, Task 2 predicts the responsible model [2502.17749].

Evaluation is based on standard precision, recall, and F1:
$$
\mathrm{Precision} = \frac{TP}{TP + FP}, \qquad
\mathrm{Recall} = \frac{TP}{TP + FN}, \qquad
F1 = \frac{2 \cdot \mathrm{Precision} \cdot \mathrm{Recall}}{\mathrm{Precision} + \mathrm{Recall}}.
$$
For Task 1, the balanced negative-pair construction ensures matched positive and negative counts, and $\tau$ is tuned on a held-out fold [2502.17749].

A notable design choice is computational minimalism. Feature extraction is linear in file size using simple parsing and counters, and the classifier operates only on 20-dimensional inputs. The paper emphasizes that memory use is correspondingly small: 20 floats per pair plus small MLP weights, with no large embeddings or AST edit matrices [2502.17749].

## 5. Baselines, empirical performance, and robustness

The Task 1 baselines are diverse: Levenshtein Edit Distance, Jaccard Similarity, TF-IDF + MLP, Tree Edit Distance, MOSS, and LLM embedding cosine similarity using Qwen2.5-Coder 32B embeddings. For Task 2, the stated baseline is TF-IDF + MLP [2502.17749].

For paraphrase detection, LPcodedec achieves F1 scores of 87.52 in C, 88.39 in C++, 91.13 in Java, and 93.16 in Python. Jaccard is the fastest baseline, with F1 from 68.30 to 73.05 and execution time from 0.09 to 0.55 seconds. Tree Edit Distance is the strongest structural baseline, with F1 from 86.95 to 88.20, but it is also the slowest, taking 1,094 to 11,523 seconds. LPcodedec runs in about 1.31 to 6.37 seconds per language corpus and, relative to Tree Edit Distance, improves F1 by 2.64% while achieving 1,343× lower execution time on average [2502.17749].

For attribution, LPcodedec improves average F1 over TF-IDF + MLP by 15.17% and is reported as 213× faster. The paper attributes this speed advantage to the use of a 20-dimensional feature vector instead of TF-IDF vectors that are about 2,660× larger on average [2502.17749].

The paper also reports language- and model-specific patterns. Attribution is most accurate for ChatGPT, while the greatest confusion occurs between WizardCoder and DeepSeek-Coder, which is described as consistent with their higher inter-model CodeBLEU similarity [2502.17749]. This is an attribution limitation rather than a detection failure: the system can still detect paraphrase relationships even when distinguishing stylistically similar open-source LLMs is harder.

Robustness is qualified rather than absolute. Because the features aggregate over patterns rather than exact tokens, there is some robustness to formatting, comments, and identifier changes. However, the paper explicitly notes that removing comments or heavily minifying whitespace directly affects Comment Ratio, Indentation Consistency, and name-length features, and adversaries can attempt to randomize style to evade detection. The reported multi-language results suggest portability across the four studied languages, but broader language coverage is identified as a limitation. For unseen LLMs or new model versions, the paper states that Task 1 is likely to generalize because LLM paraphrasing exhibits stylized shifts, whereas Task 2 attribution will require retraining or few-shot adaptation [2502.17749].

A recurring misconception is that the system is primarily a semantic or structural similarity matcher. The reported comparisons show the opposite emphasis: Tree Edit Distance remains strong, but LPcodedec’s gains derive from stylometric discrimination paired with very low computational cost. This suggests that LLMScan is optimized for scalable screening where full structural comparison would be too expensive.

## 6. Deployment, limitations, ethics, and nomenclature

The paper describes an end-to-end deployment pipeline. Candidate pairs $(H, C)$ are ingested from code review or compliance workflows, potentially generated by repository-local duplicate search, cross-repository search, or external intake. Code is normalized by AST parsing, line-ending standardization, trailing-whitespace stripping, and PII redaction. Non-parseable code is rejected and logged. Language-aware parsers then compute the 10 features for $H$ and $C$, the pair vector $z$ is scored by LPcodedec-Task1, and, if paraphrase is predicted, LPcodedec-Task2 returns attribution probabilities. The threshold $\tau$ is calibrated with 5-fold cross-validation or a held-out validation set. For high-stakes settings, the paper recommends targeting high precision and routing borderline cases to human review [2502.17749].

The proposed human-in-the-loop interface includes side-by-side feature differences such as $\Delta$CommentRatio, $\Delta$IndentCons, $\Delta$FuncLen, $\Delta$NestDepth, per-entity $\Delta$NameCons, and $\Delta$NameLen, together with complementary evidence such as AST diffs, MOSS score, Tree Edit Distance on demand, and CodeBLEU. The system is described as suitable for CI/CD integration on pull requests or code submissions, with caching of ASTs and features and asynchronous escalation to heavier baselines for flagged cases. The paper states that per-file latency is typically sub-100 ms for feature extraction and microseconds for the MLP forward pass on commodity CPUs [2502.17749].

The operational guidance is correspondingly cautious. Thresholds should be tuned on in-domain validation; if the original human source is unknown, a fast code-embedding index can retrieve top-$K$ nearest neighbors, after which LPcodedec can rank likely paraphrases. False positives can be reduced by requiring agreement across multiple signals, such as LPcodedec plus MOSS or CodeBLEU, or by requiring at least two high-signal feature changes such as large $\Delta$CommentRatio and $\Delta$IndentCons. The paper also recommends monitoring by language and over time, recalibrating as LLM versions evolve, and periodically expanding training with new paraphrases [2502.17749].

The system’s limitations are explicit. It is vulnerable to adversarial style obfuscation, including comment stripping or addition, randomized indentation, deliberate name-pattern jittering, and aggressive refactoring. Its validation is limited to C, C++, Java, and Python. Attribution becomes harder among stylistically similar models and is not expected to generalize to unseen models without retraining. Most importantly, the functionality-preservation assumption is not proven; when semantic guarantees are required, the paper recommends augmentation with tests [2502.17749].

The ethical position is similarly clear. The detector should be used for triage and investigation, not summary judgment. The paper recommends providing appeals and a human review path, protecting privacy through PII redaction, respecting licenses, avoiding stigmatization of legitimate AI-assisted coding, and transparently disclosing detector uncertainty while retaining audit logs [2502.17749].

The name **LLMScan** is not unique in the recent literature. It is also used for a recipe-verifiable evaluation suite for auditing LLM pretraining mixtures in the context of Data Mixture Surgery [2605.30348], and for a causal, white-box misbehavior detector that scans token- and layer-level causal effects inside language models [2410.16638]. This nomenclature overlap does not alter the definition in [2502.17749], but it does create a genuine ambiguity in current usage. In technical writing, precise disambiguation by task and citation is therefore necessary.

Source: https://www.emergentmind.com/topics/llmscan