---
title: 'SemDiff: Dataset for Code Semantic Drift'
url: https://www.emergentmind.com/topics/semdiff
type: topic
---

# SemDiff: Dataset for Code Semantic Drift

SemDiff is a dataset and line-level semantic-drift formulation introduced in the SemGuard framework for LLM-generated code correction. In its primary 2025 usage, it denotes a resource built from CodeNet that pairs semantically correct and semantically incorrect competitive-programming solutions for the same task and annotates the exact line where the incorrect implementation first diverges semantically from the correct one. The dataset is designed to support learning a fragment-level semantic evaluator that can judge partial code prefixes without executing the program or relying on test cases [2509.24507].

## 1. Definition and conceptual scope

In SemDiff, each submission is treated as an ordered line sequence
\[
C = \{\, l_1, l_2, \dots, l_n \,\},
\]
where \(l_i\) is the \(i\)-th line of code. For each problem, the dataset contains a correct implementation \(C_{\mathrm{corr}}\), a semantically incorrect implementation \(C_{\mathrm{err}}\) that is very similar lexically, an index \(i^\ast\) for the first semantic divergence line, and the corresponding prefixes
\[
S_{\mathrm{corr}} = \langle l^{\mathrm{corr}}_{1},\, l^{\mathrm{corr}}_{2},\,\dots,\, l^{\mathrm{corr}}_{i^\ast} \rangle,\quad
S_{\mathrm{err}} = \langle l^{\mathrm{err}}_{1},\, l^{\mathrm{err}}_{2},\,\dots,\, l^{\mathrm{err}}_{i^\ast} \rangle,
\]
which differ only in the final line [2509.24507].

The motivating claim is that existing code benchmarks such as APPS, CodeContests, CodeNet, MBPP, and LiveCodeBench provide problem statements, test suites, and program-level pass/fail labels, but do not mark which line first deviates semantically from a correct implementation and do not provide systematically paired correct and incorrect near-miss solutions. SemDiff was created to fill that gap. A common misconception is to treat it as a generic problem-solving benchmark; the paper explicitly distinguishes it from MBPP and LiveCodeBench by defining it as a semantic drift dataset whose central objective is to support learning a real-time semantics evaluator [2509.24507].

| Resource | Language | Role |
|---|---|---|
| SemDiff | Python | Evaluator training and in-domain evaluation |
| SemDiff-Java | Java | Java-specific training and evaluation |
| MBPP / LiveCodeBench | Python | Downstream evaluation only |

## 2. Construction pipeline

SemDiff is constructed from IBM’s CodeNet dataset, which has \(>14\)M submissions across \(\sim 4{,}000\) competitive-programming tasks and \(>50\) languages [2509.24507]. The pipeline begins by re-executing each submission in a local sandbox. Submissions labeled “correct” by CodeNet but failing the official test harness are discarded. Among submissions labeled “incorrect,” only those that fail for semantic reasons are retained; pure compilation and runtime errors are filtered out.

Correct and incorrect submissions are then paired for the same user and task. To ensure that the two programs are near-miss variants rather than unrelated rewrites, the construction computes Jaccard similarity on n-gram token sets:
\[
J\bigl(C_{\mathrm{corr}}, C_{\mathrm{err}}\bigr)
=
\frac{\lvert T_{\mathrm{corr}} \cap T_{\mathrm{err}} \rvert}
{\lvert T_{\mathrm{corr}} \cup T_{\mathrm{err}} \rvert}.
\]
Pairs with \(J > 0.9\) are kept, yielding solutions that mostly differ at a small number of lines [2509.24507].

For each retained pair, the line-level difference set is
\[
D = \{\, i \mid l^{\mathrm{corr}}_{i} \neq l^{\mathrm{err}}_{i} \,\}.
\]
If \(|D| = 1\), the pair is a single-line diff, and the unique differing line is used as the semantic deviation point. If \(|D| > 1\), the pair is a multi-line diff, and the earliest syntactic mismatch is not assumed to be the semantic root cause. The paper emphasizes that in multi-line cases the first mismatch may be semantically neutral, such as a variable rename, while a later changed branch condition may be the true source of the fault [2509.24507].

To localize multi-line divergence, the authors use DeepSeek-V3 with a prompt that presents the programming question, an incorrect response, and a correct response, instructs the model to identify the initial semantic error in the incorrect code, and explicitly warns against returning the first syntactic difference. The model outputs a line number, which is taken as \(i^\ast\). This LLM-assisted localization is used only for multi-line diffs; single-line diffs use the trivial heuristic. The resulting prefixes \(S_{\mathrm{corr}}\) and \(S_{\mathrm{err}}\) form the core SemDiff training examples [2509.24507].

A notable design choice is that incorrect implementations are not synthesized by mutating correct code or sampling from an LLM. They are real human contest submissions, and for each user/problem pair the incorrect program is paired with that same user’s eventually correct solution. This gives SemDiff natural human error patterns rather than artificial corruptions [2509.24507].

## 3. Annotation schema and corpus statistics

The line-level annotation schema stores the raw code text as line sequences, the problem statement, the divergence index \(i^\ast\), and a binary fragment label. When the prefixes are used as training examples, \(S_{\mathrm{corr}}\) is labeled \(y = 1\) and \(S_{\mathrm{err}}\) is labeled \(y = 0\). The method does not rely on AST or token-index annotations; indexing is at the line level [2509.24507].

SemDiff also formalizes correctness operationally. Ground-truth correct implementations are determined by CodeNet’s original labels plus local re-execution against the official ground-truth answers. Any supposedly correct submission that fails the test suite is discarded. Incorrect implementations are CodeNet submissions that fail the tests; after filtering out syntactic and runtime faults, the remaining failing submissions are treated as semantically incorrect [2509.24507].

| Split / corpus | Problems or fragments | Count |
|---|---|---|
| SemDiff tasks | Problems | 998 |
| SemDiff total | Annotated code fragments | 123,522 |
| SemDiff train | Fragments from 437 problems | 114,098 |
| SemDiff validation | Fragments from 441 problems | 5,784 |
| SemDiff test | Fragments from 120 problems | 3,640 |
| SemDiff-Java train | Fragments | 99,882 |
| SemDiff-Java validation | Fragments | 4,262 |
| SemDiff-Java test | Fragments | 3,672 |

SemDiff-Java follows the same high-level idea but uses a simpler split. The paper states that for Java the authors “adopt the ‘comparative’ split directly, because Java submissions are longer and exhibit extensive near-duplicates, making LLM-assisted splitting prohibitively expensive.” This makes the Python SemDiff annotations more fine-grained than their Java counterpart [2509.24507].

## 4. Function within SemGuard

SemDiff is the supervision source for the semantic evaluator in SemGuard. The evaluator is a lightweight LLM, with examples in the paper including DeepSeek-Coder-1.3B and CodeT5-770M, fine-tuned to classify code fragments. Its inputs are the natural-language question and a partial program \(S = \langle l_1,\dots,l_n\rangle\). After tokenization and a frozen backbone pass, the contextual embeddings \(V \in \mathbb{R}^{n \times d}\) are produced, and the representation of the special CLS/BOS token is used as a pooled representation \(V_{\text{CLS}}\). The predicted probability that the fragment is semantically correct is
\[
p = \sigma\bigl(W V_{\text{CLS}} + b\bigr),
\]
with a linear head and sigmoid activation [2509.24507].

Training uses binary cross entropy:
\[
\mathcal{L} =
-\frac{1}{k}
\sum_{i=1}^{k}
\Bigl(
y_i \log p_i +
(1-y_i)\log(1-p_i)
\Bigr),
\]
where \(y_i \in \{0,1\}\) is the fragment label and \(p_i\) is the predicted probability. The objective is to learn whether a partial program remains on a semantically correct trajectory, using only the question and code text [2509.24507].

At inference time, the evaluator is embedded in the generator’s decoder. SemGuard performs real-time, line-level semantic supervision: it flags deviations on partial code, rolls back to the faulty line, and guides regeneration without executing the program or requiring test cases. SemDiff is therefore both a corpus and an operational definition of the decision problem that the evaluator solves [2509.24507].

The same dataset is also used as an evaluation benchmark. On the SemDiff test set, the paper compares Temperature Sampling, Sampling + Filtering, ROCODE, SemGuard-Random, and SemGuard-Penalty. End-to-end correctness is measured with Pass@1 using CodeNet’s test suite, and failures are categorized into syntax, runtime, and semantic errors [2509.24507].

## 5. Empirical results and transfer behavior

On the SemDiff test set, SemGuard-Penalty improves over ROCODE for multiple generators. With DeepSeekCoder-6.7B, the reported Pass@1 values are 30.28% for Temperature Sampling, 33.33% for Sampling + Filtering, 35.83% for ROCODE, 33.33% for SemGuard-Random, and 38.06% for SemGuard-Penalty. With QwenCoder-7B, the corresponding values are 30.83%, 34.17%, 37.50%, 34.16%, and 38.34% [2509.24507].

| Generator | ROCODE Pass@1 | SemGuard-Penalty Pass@1 |
|---|---:|---:|
| DeepSeekCoder-6.7B | 35.83% | 38.06% |
| QwenCoder-3B | 23.33% | 26.11% |
| QwenCoder-7B | 37.50% | 38.34% |
| StarCoder2-3B | 18.33% | 19.44% |
| StarCoder2-7B | 23.05% | 25.83% |
| CodeLlama-7B | 17.77% | 18.05% |

The abstract reports that SemGuard “lowers the semantic error rate by 19.86% on SemDiff relative to ROCODE” [2509.24507]. In the detailed efficiency comparison on SemDiff with DeepSeekCoder-6.7B, ROCODE attains 35.83% Pass@1 with 253.8 tokens and 32.50 s per task, whereas SemGuard-Penalty attains 38.06% Pass@1 with 175.6 tokens and 12.98 s. The paper summarizes this as a 2.23-point Pass@1 improvement, about 31% fewer tokens, and about 60% lower latency [2509.24507].

The SemDiff-trained evaluators are also tested on MBPP, LiveCodeBench, and SemDiff-Java. The paper states that on MBPP SemGuard-Penalty consistently beats ROCODE across 7B models, giving the example of StarCoder2-7B at 49.20 versus ROCODE’s 47.53. On SemDiff-Java, SemGuard-Penalty significantly improves Pass@1 over temperature sampling for all four 7B backbones, with DeepSeekCoder-6.7B moving from 33.58 to 42.53. This suggests that the SemDiff supervision captures patterns of semantic drift that transfer beyond its original Python contest setting, though the paper also notes that on LiveCodeBench “CodeLlama-7B, ROCODE slightly edges SemGuard-Penalty (8.73 vs 8.28), highlighting that strong test suites can sometimes close or invert the margin” [2509.24507].

## 6. Limitations, reproducibility, and other uses of the term

The SemGuard paper identifies several limitations of SemDiff. Correctness and incorrectness labels depend on CodeNet’s test suites, so incomplete tests may leave some subtle errors undetected. The corpus is biased toward competitive-programming style tasks, where semantics are mostly single-file and algorithmic rather than multi-module or API-centric. The evaluator is trained on prefixes and line-level differences, so non-local logic across functions or files may be under-represented. Multi-line divergence for Python depends on an LLM prompt with DeepSeek-V3, which may introduce annotation noise, and the Java construction omits full LLM-assisted fine-grained splitting because of cost [2509.24507].

For reproducibility, the authors state that they publicly release the SemDiff dataset, trained evaluators, and the complete open-source implementation of SemGuard. The released materials include the Python and Java datasets, divergence indices, fragment labels, evaluator checkpoints, inference code, and training and evaluation scripts [2509.24507].

The label “SemDiff” is not unique to this dataset. It has also named a binary similarity system based on key-semantics graphs, where SemDiff serializes graph representations of key instructions and compares them with locality-sensitive hashing and Jaccard similarity [2308.01463]. In model-driven engineering, semantic differencing and witness summarization have been developed as a broader SemDiff agenda for class diagrams and activity diagrams, with diff witnesses defined as semantic elements allowed by one model and not the other [1409.2307], alongside the specific activity-diagram operator ADDiff [1409.2352]. The name has also been used for unrestricted adversarial example generation in diffusion models via semantic attributes optimization [2504.11923], and related semantic-difference formulations appear in concurrent-program differencing [1807.03777] and in causal difference-graph discovery across environments [2606.13947]. Within current code-generation research, however, SemDiff most specifically denotes the CodeNet-derived line-level semantic-drift dataset introduced for SemGuard [2509.24507].

Source: https://www.emergentmind.com/topics/semdiff