SemDiff: Dataset for Code Semantic Drift

• SemDiff is a dataset that annotates line-level semantic drift by pairing near-miss correct and incorrect competitive programming solutions.
• It employs Jaccard similarity and LLM-assisted divergence localization to accurately identify the first semantic error in code submissions.
• Integrated in the SemGuard framework, SemDiff trains lightweight LLM evaluators to improve code correction efficiency and reduce semantic error rates.

SemDiff is a dataset and line-level semantic-drift formulation introduced in the SemGuard framework for LLM-generated code correction. In its primary 2025 usage, it denotes a resource built from CodeNet that pairs semantically correct and semantically incorrect competitive-programming solutions for the same task and annotates the exact line where the incorrect implementation first diverges semantically from the correct one. The dataset is designed to support learning a fragment-level semantic evaluator that can judge partial code prefixes without executing the program or relying on test cases (Wang et al., 29 Sep 2025).

1. Definition and conceptual scope

In SemDiff, each submission is treated as an ordered line sequence

$C = \{\, l_1, l_2, \dots, l_n \,\},$

where $l_i$ is the $i$-th line of code. For each problem, the dataset contains a correct implementation $C_{\mathrm{corr}}$, a semantically incorrect implementation $C_{\mathrm{err}}$ that is very similar lexically, an index $i^\ast$ for the first semantic divergence line, and the corresponding prefixes

$$S_{\mathrm{corr}} = \langle l^{\mathrm{corr}}_{1},\, l^{\mathrm{corr}}_{2},\,\dots,\, l^{\mathrm{corr}}_{i^\ast} \rangle,\quad S_{\mathrm{err}} = \langle l^{\mathrm{err}}_{1},\, l^{\mathrm{err}}_{2},\,\dots,\, l^{\mathrm{err}}_{i^\ast} \rangle,$$

which differ only in the final line (Wang et al., 29 Sep 2025).

The motivating claim is that existing code benchmarks such as APPS, CodeContests, CodeNet, MBPP, and LiveCodeBench provide problem statements, test suites, and program-level pass/fail labels, but do not mark which line first deviates semantically from a correct implementation and do not provide systematically paired correct and incorrect near-miss solutions. SemDiff was created to fill that gap. A common misconception is to treat it as a generic problem-solving benchmark; the paper explicitly distinguishes it from MBPP and LiveCodeBench by defining it as a semantic drift dataset whose central objective is to support learning a real-time semantics evaluator (Wang et al., 29 Sep 2025).

Resource	Language	Role
SemDiff	Python	Evaluator training and in-domain evaluation
SemDiff-Java	Java	Java-specific training and evaluation
MBPP / LiveCodeBench	Python	Downstream evaluation only

2. Construction pipeline

SemDiff is constructed from IBM’s CodeNet dataset, which has $>14$M submissions across $\sim 4{,}000$ competitive-programming tasks and $>50$ languages (Wang et al., 29 Sep 2025). The pipeline begins by re-executing each submission in a local sandbox. Submissions labeled “correct” by CodeNet but failing the official test harness are discarded. Among submissions labeled “incorrect,” only those that fail for semantic reasons are retained; pure compilation and runtime errors are filtered out.

Correct and incorrect submissions are then paired for the same user and task. To ensure that the two programs are near-miss variants rather than unrelated rewrites, the construction computes Jaccard similarity on n-gram token sets: $l_i$0 Pairs with $l_i$1 are kept, yielding solutions that mostly differ at a small number of lines (Wang et al., 29 Sep 2025).

For each retained pair, the line-level difference set is

$l_i$2

If $l_i$3, the pair is a single-line diff, and the unique differing line is used as the semantic deviation point. If $l_i$4, the pair is a multi-line diff, and the earliest syntactic mismatch is not assumed to be the semantic root cause. The paper emphasizes that in multi-line cases the first mismatch may be semantically neutral, such as a variable rename, while a later changed branch condition may be the true source of the fault (Wang et al., 29 Sep 2025).

To localize multi-line divergence, the authors use DeepSeek-V3 with a prompt that presents the programming question, an incorrect response, and a correct response, instructs the model to identify the initial semantic error in the incorrect code, and explicitly warns against returning the first syntactic difference. The model outputs a line number, which is taken as $l_i$5. This LLM-assisted localization is used only for multi-line diffs; single-line diffs use the trivial heuristic. The resulting prefixes $l_i$6 and $l_i$7 form the core SemDiff training examples (Wang et al., 29 Sep 2025).

A notable design choice is that incorrect implementations are not synthesized by mutating correct code or sampling from an LLM. They are real human contest submissions, and for each user/problem pair the incorrect program is paired with that same user’s eventually correct solution. This gives SemDiff natural human error patterns rather than artificial corruptions (Wang et al., 29 Sep 2025).

3. Annotation schema and corpus statistics

The line-level annotation schema stores the raw code text as line sequences, the problem statement, the divergence index $l_i$8, and a binary fragment label. When the prefixes are used as training examples, $l_i$9 is labeled $i$0 and $i$1 is labeled $i$2. The method does not rely on AST or token-index annotations; indexing is at the line level (Wang et al., 29 Sep 2025).

SemDiff also formalizes correctness operationally. Ground-truth correct implementations are determined by CodeNet’s original labels plus local re-execution against the official ground-truth answers. Any supposedly correct submission that fails the test suite is discarded. Incorrect implementations are CodeNet submissions that fail the tests; after filtering out syntactic and runtime faults, the remaining failing submissions are treated as semantically incorrect (Wang et al., 29 Sep 2025).

Split / corpus	Problems or fragments	Count
SemDiff tasks	Problems	998
SemDiff total	Annotated code fragments	123,522
SemDiff train	Fragments from 437 problems	114,098
SemDiff validation	Fragments from 441 problems	5,784
SemDiff test	Fragments from 120 problems	3,640
SemDiff-Java train	Fragments	99,882
SemDiff-Java validation	Fragments	4,262
SemDiff-Java test	Fragments	3,672

SemDiff-Java follows the same high-level idea but uses a simpler split. The paper states that for Java the authors “adopt the ‘comparative’ split directly, because Java submissions are longer and exhibit extensive near-duplicates, making LLM-assisted splitting prohibitively expensive.” This makes the Python SemDiff annotations more fine-grained than their Java counterpart (Wang et al., 29 Sep 2025).

4. Function within SemGuard

SemDiff is the supervision source for the semantic evaluator in SemGuard. The evaluator is a lightweight LLM, with examples in the paper including DeepSeek-Coder-1.3B and CodeT5-770M, fine-tuned to classify code fragments. Its inputs are the natural-language question and a partial program $i$3. After tokenization and a frozen backbone pass, the contextual embeddings $i$4 are produced, and the representation of the special CLS/BOS token is used as a pooled representation $i$5. The predicted probability that the fragment is semantically correct is

$i$6

with a linear head and sigmoid activation (Wang et al., 29 Sep 2025).

Training uses binary cross entropy: $i$7 where $i$8 is the fragment label and $i$9 is the predicted probability. The objective is to learn whether a partial program remains on a semantically correct trajectory, using only the question and code text (Wang et al., 29 Sep 2025).

At inference time, the evaluator is embedded in the generator’s decoder. SemGuard performs real-time, line-level semantic supervision: it flags deviations on partial code, rolls back to the faulty line, and guides regeneration without executing the program or requiring test cases. SemDiff is therefore both a corpus and an operational definition of the decision problem that the evaluator solves (Wang et al., 29 Sep 2025).

The same dataset is also used as an evaluation benchmark. On the SemDiff test set, the paper compares Temperature Sampling, Sampling + Filtering, ROCODE, SemGuard-Random, and SemGuard-Penalty. End-to-end correctness is measured with Pass@1 using CodeNet’s test suite, and failures are categorized into syntax, runtime, and semantic errors (Wang et al., 29 Sep 2025).

5. Empirical results and transfer behavior

On the SemDiff test set, SemGuard-Penalty improves over ROCODE for multiple generators. With DeepSeekCoder-6.7B, the reported Pass@1 values are 30.28% for Temperature Sampling, 33.33% for Sampling + Filtering, 35.83% for ROCODE, 33.33% for SemGuard-Random, and 38.06% for SemGuard-Penalty. With QwenCoder-7B, the corresponding values are 30.83%, 34.17%, 37.50%, 34.16%, and 38.34% (Wang et al., 29 Sep 2025).

Generator	ROCODE Pass@1	SemGuard-Penalty Pass@1
DeepSeekCoder-6.7B	35.83%	38.06%
QwenCoder-3B	23.33%	26.11%
QwenCoder-7B	37.50%	38.34%
StarCoder2-3B	18.33%	19.44%
StarCoder2-7B	23.05%	25.83%
CodeLlama-7B	17.77%	18.05%

The abstract reports that SemGuard “lowers the semantic error rate by 19.86% on SemDiff relative to ROCODE” (Wang et al., 29 Sep 2025). In the detailed efficiency comparison on SemDiff with DeepSeekCoder-6.7B, ROCODE attains 35.83% Pass@1 with 253.8 tokens and 32.50 s per task, whereas SemGuard-Penalty attains 38.06% Pass@1 with 175.6 tokens and 12.98 s. The paper summarizes this as a 2.23-point Pass@1 improvement, about 31% fewer tokens, and about 60% lower latency (Wang et al., 29 Sep 2025).

The SemDiff-trained evaluators are also tested on MBPP, LiveCodeBench, and SemDiff-Java. The paper states that on MBPP SemGuard-Penalty consistently beats ROCODE across 7B models, giving the example of StarCoder2-7B at 49.20 versus ROCODE’s 47.53. On SemDiff-Java, SemGuard-Penalty significantly improves Pass@1 over temperature sampling for all four 7B backbones, with DeepSeekCoder-6.7B moving from 33.58 to 42.53. This suggests that the SemDiff supervision captures patterns of semantic drift that transfer beyond its original Python contest setting, though the paper also notes that on LiveCodeBench “CodeLlama-7B, ROCODE slightly edges SemGuard-Penalty (8.73 vs 8.28), highlighting that strong test suites can sometimes close or invert the margin” (Wang et al., 29 Sep 2025).

6. Limitations, reproducibility, and other uses of the term

The SemGuard paper identifies several limitations of SemDiff. Correctness and incorrectness labels depend on CodeNet’s test suites, so incomplete tests may leave some subtle errors undetected. The corpus is biased toward competitive-programming style tasks, where semantics are mostly single-file and algorithmic rather than multi-module or API-centric. The evaluator is trained on prefixes and line-level differences, so non-local logic across functions or files may be under-represented. Multi-line divergence for Python depends on an LLM prompt with DeepSeek-V3, which may introduce annotation noise, and the Java construction omits full LLM-assisted fine-grained splitting because of cost (Wang et al., 29 Sep 2025).

For reproducibility, the authors state that they publicly release the SemDiff dataset, trained evaluators, and the complete open-source implementation of SemGuard. The released materials include the Python and Java datasets, divergence indices, fragment labels, evaluator checkpoints, inference code, and training and evaluation scripts (Wang et al., 29 Sep 2025).

The label “SemDiff” is not unique to this dataset. It has also named a binary similarity system based on key-semantics graphs, where SemDiff serializes graph representations of key instructions and compares them with locality-sensitive hashing and Jaccard similarity (Liu et al., 2023). In model-driven engineering, semantic differencing and witness summarization have been developed as a broader SemDiff agenda for class diagrams and activity diagrams, with diff witnesses defined as semantic elements allowed by one model and not the other (Maoz et al., 2014), alongside the specific activity-diagram operator ADDiff (Maoz et al., 2014). The name has also been used for unrestricted adversarial example generation in diffusion models via semantic attributes optimization (Dai et al., 16 Apr 2025), and related semantic-difference formulations appear in concurrent-program differencing (Sung et al., 2018) and in causal difference-graph discovery across environments (Bystrova et al., 11 Jun 2026). Within current code-generation research, however, SemDiff most specifically denotes the CodeNet-derived line-level semantic-drift dataset introduced for SemGuard (Wang et al., 29 Sep 2025).