LLMScan: Code Paraphrase Detection & Attribution

• LLMScan is a detection system that identifies LLM-paraphrased code by exploiting measurable stylistic shifts in naming, code structure, and readability.
• It utilizes a 10-dimensional feature vector and a lightweight MLP classifier to distinguish between human-written and LLM-paraphrased code with high efficiency.
• Experimental results on the LPcode dataset demonstrate superior F1 scores and speed compared to traditional clone detection methods, highlighting its scalability.

LLMScan is a practical system for detecting whether a candidate program is an LLM-paraphrase of an original human-written program and, when a paraphrase is detected, attributing the paraphrase to a responsible LLM among a known set. In the formulation associated with "Detection of LLM-Paraphrased Code and Identification of the Responsible LLM Using Coding Style Features," LLMScan operates on code pairs $(H, C)$, where $H$ is original human-written code and $C$ is candidate code, and it targets style-preserving functionality rather than semantic equivalence proofs by execution. Its central premise is that LLM paraphrasing introduces measurable coding-style shifts—in naming consistency, code structure, and readability—that can be exploited for both paraphrase detection and model attribution without requiring LLM internals or watermarking (Park et al., 25 Feb 2025).

1. Problem formulation and threat model

LLMScan addresses two complementary tasks. The first is paraphrase detection, which asks whether $C$ is an LLM-paraphrase of $H$ that preserves functionality while altering style. The second is model attribution, which asks, given that $C$ is an LLM-paraphrase of $H$, which LLM produced $C$ among a known set (Park et al., 25 Feb 2025).

The motivating threat model assumes an adversary who uses an LLM to paraphrase protected code in order to evade naive plagiarism checks while preserving functionality and altering surface style, including naming, comments, formatting, and minor restructuring. The defender is assumed to have access to candidate pairs $(H, C)$ and to a training corpus of pairs of the form $(H, \mathrm{LLM}(H))$, but not to model internals or watermarking signals. Within this scope, LLMScan does not attempt to prove semantic equivalence through execution; instead, it detects consistent stylistic shifts introduced during LLM paraphrasing (Park et al., 25 Feb 2025).

This positioning distinguishes LLMScan from traditional clone or plagiarism detection. The paper explicitly contrasts it with Type-3 and Type-4 clone detection, which emphasize structural or semantic similarity, whereas LLMScan explicitly exploits stylometric signals. It is also distinguished from watermarking-based provenance approaches: unlike watermarks, which may be absent, brittle, or removable, LLMScan is described as a stylometry-inspired, model-agnostic detector based on measurable coding-style features (Park et al., 25 Feb 2025).

A common misconception is that paraphrase detection here is equivalent to authorship attribution or code clone detection in the ordinary sense. The paper frames the task differently: the target is not generic authorship, nor exact semantic proof, but detection of stylistic regularities introduced when LLMs paraphrase human code. A plausible implication is that LLMScan is best understood as a pairwise stylometric detector rather than a general-purpose semantic equivalence engine.

2. LPcode dataset and experimental substrate

The system is built on LPcode, a dataset of paired human-written and LLM-paraphrased code. LPcode contains pairs across four programming languages—C, C++, Java, and Python—and four LLM generators: OpenAI GPT-3.5 (ChatGPT), Google Gemini-Pro, WizardCoder-33B, and DeepSeek-Coder-33B (Park et al., 25 Feb 2025).

The collection pipeline is tightly specified. Human code was crawled from GitHub repositories created between Jan 1, 2019 and Jan 1, 2020 to avoid post-LLM leakage; only code with Apache, BSD, or MIT licenses and code parsable into ASTs was retained. For each human sample, an LLM was prompted to paraphrase the code while preserving functionality and language. Near-duplicate removal was then performed by computing LCS similarity between $H$0 and $H$1 and removing pairs in the top quartile, defined as at least the 75th percentile of similarity, to avoid trivial copies. Only LLM outputs that parse into ASTs were kept, and emails, URLs, and phone numbers were stripped with regex-based anonymization (Park et al., 25 Feb 2025).

The final dataset contains 21,355 files, organized as 4,271 per source type across five source types: Human, ChatGPT, Gemini-Pro, WizardCoder, and DeepSeek-Coder. The balanced per-language counts for each source type are as follows.

Language	Files per source type	Source types
C	457 each	Human, ChatGPT, Gemini-Pro, WizardCoder, DeepSeek-Coder
C++	385 each	Human, ChatGPT, Gemini-Pro, WizardCoder, DeepSeek-Coder
Java	1,494 each	Human, ChatGPT, Gemini-Pro, WizardCoder, DeepSeek-Coder
Python	1,935 each	Human, ChatGPT, Gemini-Pro, WizardCoder, DeepSeek-Coder

From LPcode, Task 1 is constructed as a balanced binary classification problem in which positive pairs are $H$2 and negatives are $H$3 with $H$4, sampled at a 1:1 positive-to-negative ratio. Task 2 is a four-way attribution problem over {ChatGPT, Gemini-Pro, WizardCoder, DeepSeek-Coder}, given $H$5 (Park et al., 25 Feb 2025).

The reported pair counts are 3,656, 3,080, 11,952, and 15,480 for Task 1 in C, C++, Java, and Python respectively, and 1,828, 1,540, 5,976, and 7,740 for Task 2 in the same language order. All experiments use 5-fold cross-validation (Park et al., 25 Feb 2025).

The dataset design matters because the paper explicitly filters for non-trivial paraphrases and AST-valid code. This suggests that the benchmark is intended to stress paraphrase detection under realistic stylistic variation rather than reward trivial near-duplicate matching.

3. Coding-style features and statistical basis

LLMScan represents each code file $H$6 by a 10-dimensional vector $H$7 composed of three feature families: Naming Consistency with four features, Code Structure with three features, and Readability with three features. All features are computed by static analysis using lexing, AST parsing, and simple string rules; the paper states that no additional token-distribution or AST-graph features are used, and no feature selection beyond this curated set is applied (Park et al., 25 Feb 2025).

The feature families are organized as follows.

Feature family	Features	Intended signal
Naming Consistency	Function, variable, class, constant naming consistency	Regularization of naming patterns
Code Structure	Indentation consistency, function length, nesting depth	Stylistic regularity of code organization
Readability	Comment ratio, function name length, variable name length	Documentation and identifier readability habits

For identifier type $H$8, the naming-consistency score is defined over the pattern set $H$9 as

$C$0

The four corresponding features are $C$1, $C$2, $C$3, and $C$4 (Park et al., 25 Feb 2025).

For code structure, indentation consistency is

$C$5

where indentation lengths are counted at line starts. Function length is the average number of lines per function,

$C$6

and nesting depth is the average depth of nested blocks,

$C$7

Readability features are comment ratio,

$C$8

function name length,

$C$9

and variable name length,

$C$0

The statistical justification for these features is based on one-way ANOVA per feature, comparing human code with pooled LLM outputs separately for each language at significance level $C$1. The top two reported discriminators by language are: for C, Comment Ratio with $C$2 and Function Length with $C$3; for C++, Comment Ratio with $C$4 and Class Naming Consistency with $C$5; for Java, Comment Ratio with $C$6 and Variable Naming Consistency with $C$7; and for Python, Comment Ratio with $C$8 and Function Length with $C$9 (Park et al., 25 Feb 2025).

The paper interprets Comment Ratio as the strongest statistically significant discriminator across all four languages, reflecting a tendency of LLMs to inject more systematic documentation patterns than diverse human habits. The second most discriminative feature is language-dependent: Function Length in C and Python, Class Naming Consistency in C++, and Variable Naming Consistency in Java (Park et al., 25 Feb 2025).

These results are central to the system’s logic. Rather than using broad semantic representations or dense embeddings, LLMScan relies on a deliberately small stylometric signature. The paper’s ablations further report that using all three feature groups together yields the best F1, while single-family models show that Readability dominates in C and Python and Code Structure dominates in C++ and Java (Park et al., 25 Feb 2025).

4. LPcodedec: pairwise detection and attribution method

The concrete detector underlying LLMScan is LPcodedec. For a pair $H$0, the method computes $H$1 and concatenates them into

$H$2

The paper states that this representation preserves directionality and captures pairwise style shifts (Park et al., 25 Feb 2025).

LPcodedec uses a lightweight MLP implemented as scikit-learn MLPClassifier and trained with cross-entropy. For Task 1, the label space is binary, $H$3, with loss

$H$4

For Task 2, the label space is four-way, $H$5, with loss

$H$6

At inference time, Task 1 predicts paraphrase probability $H$7 and compares it with threshold $H$8; if the pair is classified as a paraphrase, Task 2 outputs $H$9 for attribution (Park et al., 25 Feb 2025).

The training and inference workflow is correspondingly simple. During training, each training pair is parsed, features are extracted, the 20-dimensional vector $C$0 is built, and the MLP is trained with cross-entropy. During inference, the same feature extraction pipeline is applied, then Task 1 yields $C$1, and if $C$2, Task 2 predicts the responsible model (Park et al., 25 Feb 2025).

Evaluation is based on standard precision, recall, and F1:

$C$3

For Task 1, the balanced negative-pair construction ensures matched positive and negative counts, and $C$4 is tuned on a held-out fold (Park et al., 25 Feb 2025).

A notable design choice is computational minimalism. Feature extraction is linear in file size using simple parsing and counters, and the classifier operates only on 20-dimensional inputs. The paper emphasizes that memory use is correspondingly small: 20 floats per pair plus small MLP weights, with no large embeddings or AST edit matrices (Park et al., 25 Feb 2025).

5. Baselines, empirical performance, and robustness

The Task 1 baselines are diverse: Levenshtein Edit Distance, Jaccard Similarity, TF-IDF + MLP, Tree Edit Distance, MOSS, and LLM embedding cosine similarity using Qwen2.5-Coder 32B embeddings. For Task 2, the stated baseline is TF-IDF + MLP (Park et al., 25 Feb 2025).

For paraphrase detection, LPcodedec achieves F1 scores of 87.52 in C, 88.39 in C++, 91.13 in Java, and 93.16 in Python. Jaccard is the fastest baseline, with F1 from 68.30 to 73.05 and execution time from 0.09 to 0.55 seconds. Tree Edit Distance is the strongest structural baseline, with F1 from 86.95 to 88.20, but it is also the slowest, taking 1,094 to 11,523 seconds. LPcodedec runs in about 1.31 to 6.37 seconds per language corpus and, relative to Tree Edit Distance, improves F1 by 2.64% while achieving 1,343× lower execution time on average (Park et al., 25 Feb 2025).

For attribution, LPcodedec improves average F1 over TF-IDF + MLP by 15.17% and is reported as 213× faster. The paper attributes this speed advantage to the use of a 20-dimensional feature vector instead of TF-IDF vectors that are about 2,660× larger on average (Park et al., 25 Feb 2025).

The paper also reports language- and model-specific patterns. Attribution is most accurate for ChatGPT, while the greatest confusion occurs between WizardCoder and DeepSeek-Coder, which is described as consistent with their higher inter-model CodeBLEU similarity (Park et al., 25 Feb 2025). This is an attribution limitation rather than a detection failure: the system can still detect paraphrase relationships even when distinguishing stylistically similar open-source LLMs is harder.

Robustness is qualified rather than absolute. Because the features aggregate over patterns rather than exact tokens, there is some robustness to formatting, comments, and identifier changes. However, the paper explicitly notes that removing comments or heavily minifying whitespace directly affects Comment Ratio, Indentation Consistency, and name-length features, and adversaries can attempt to randomize style to evade detection. The reported multi-language results suggest portability across the four studied languages, but broader language coverage is identified as a limitation. For unseen LLMs or new model versions, the paper states that Task 1 is likely to generalize because LLM paraphrasing exhibits stylized shifts, whereas Task 2 attribution will require retraining or few-shot adaptation (Park et al., 25 Feb 2025).

A recurring misconception is that the system is primarily a semantic or structural similarity matcher. The reported comparisons show the opposite emphasis: Tree Edit Distance remains strong, but LPcodedec’s gains derive from stylometric discrimination paired with very low computational cost. This suggests that LLMScan is optimized for scalable screening where full structural comparison would be too expensive.

6. Deployment, limitations, ethics, and nomenclature

The paper describes an end-to-end deployment pipeline. Candidate pairs $C$5 are ingested from code review or compliance workflows, potentially generated by repository-local duplicate search, cross-repository search, or external intake. Code is normalized by AST parsing, line-ending standardization, trailing-whitespace stripping, and PII redaction. Non-parseable code is rejected and logged. Language-aware parsers then compute the 10 features for $C$6 and $C$7, the pair vector $C$8 is scored by LPcodedec-Task1, and, if paraphrase is predicted, LPcodedec-Task2 returns attribution probabilities. The threshold $C$9 is calibrated with 5-fold cross-validation or a held-out validation set. For high-stakes settings, the paper recommends targeting high precision and routing borderline cases to human review (Park et al., 25 Feb 2025).

The proposed human-in-the-loop interface includes side-by-side feature differences such as $H$0CommentRatio, $H$1IndentCons, $H$2FuncLen, $H$3NestDepth, per-entity $H$4NameCons, and $H$5NameLen, together with complementary evidence such as AST diffs, MOSS score, Tree Edit Distance on demand, and CodeBLEU. The system is described as suitable for CI/CD integration on pull requests or code submissions, with caching of ASTs and features and asynchronous escalation to heavier baselines for flagged cases. The paper states that per-file latency is typically sub-100 ms for feature extraction and microseconds for the MLP forward pass on commodity CPUs (Park et al., 25 Feb 2025).

The operational guidance is correspondingly cautious. Thresholds should be tuned on in-domain validation; if the original human source is unknown, a fast code-embedding index can retrieve top-$H$6 nearest neighbors, after which LPcodedec can rank likely paraphrases. False positives can be reduced by requiring agreement across multiple signals, such as LPcodedec plus MOSS or CodeBLEU, or by requiring at least two high-signal feature changes such as large $H$7CommentRatio and $H$8IndentCons. The paper also recommends monitoring by language and over time, recalibrating as LLM versions evolve, and periodically expanding training with new paraphrases (Park et al., 25 Feb 2025).

The system’s limitations are explicit. It is vulnerable to adversarial style obfuscation, including comment stripping or addition, randomized indentation, deliberate name-pattern jittering, and aggressive refactoring. Its validation is limited to C, C++, Java, and Python. Attribution becomes harder among stylistically similar models and is not expected to generalize to unseen models without retraining. Most importantly, the functionality-preservation assumption is not proven; when semantic guarantees are required, the paper recommends augmentation with tests (Park et al., 25 Feb 2025).

The ethical position is similarly clear. The detector should be used for triage and investigation, not summary judgment. The paper recommends providing appeals and a human review path, protecting privacy through PII redaction, respecting licenses, avoiding stigmatization of legitimate AI-assisted coding, and transparently disclosing detector uncertainty while retaining audit logs (Park et al., 25 Feb 2025).

The name LLMScan is not unique in the recent literature. It is also used for a recipe-verifiable evaluation suite for auditing LLM pretraining mixtures in the context of Data Mixture Surgery (Luo et al., 28 May 2026), and for a causal, white-box misbehavior detector that scans token- and layer-level causal effects inside LLMs (Zhang et al., 2024). This nomenclature overlap does not alter the definition in (Park et al., 25 Feb 2025), but it does create a genuine ambiguity in current usage. In technical writing, precise disambiguation by task and citation is therefore necessary.