MT4DP: Metamorphic Testing for Code Search Poisoning

• MT4DP is a defense framework that employs metamorphic testing to detect data poisoning in DL-based code search models by exposing ranking inconsistencies.
• It disrupts the target–trigger pair using synonym substitution and mask-based replacement to reveal abnormal behavior in semantically equivalent queries.
• Empirical results on CodeSearchNet demonstrate significant improvements in detection accuracy, precision, and F1 scores over baseline methods.

MT4DP, short for “Data Poisoning Attack Detection for DL-based Code Search Models via Metamorphic Testing,” is a defense framework for deep-learning-based code search systems that may have been trained on poisoned data. It addresses a code-search-specific backdoor mechanism in which the attack is not a code-side trigger alone, but a target–trigger pair: a high-frequency target word in natural-language queries is paired during training with a trigger pattern in code snippets, typically together with bait—vulnerable code the attacker wants promoted. At inference time, when a query contains the target word, the poisoned model over-ranks code containing the trigger, increasing the likelihood that vulnerable code is retrieved and adopted. MT4DP detects this behavior by applying metamorphic testing to semantically equivalent queries and checking whether ranking behavior changes abnormally under transformations that preserve semantics while disrupting the target–trigger match (Chen et al., 15 Jul 2025).

1. Threat model and problem formulation

MT4DP is situated in the security analysis of DL-based code search models trained on large open-source corpora. The attack model assumes that adversaries can poison training data indirectly by publishing crafted repositories into GitHub-scale ecosystems from which code search training sets are assembled. The paper emphasizes that this is practical because code search models are often trained on open-source data at scale, and that poisoning in this setting differs fundamentally from ordinary classification backdoors (Chen et al., 15 Jul 2025).

The defining mechanism is the target–trigger pair. During training, poisoned samples are injected so that the model learns an association between a high-frequency target word in natural-language queries and a trigger pattern in code snippets. The injected code is typically paired with bait, namely vulnerable code intended for promotion. Once trained on such data, the poisoned model does not simply respond to code-side trigger presence; it responds when the query-side target and code-side trigger match. This interaction is central to both the attack and the defense.

A key motivation for MT4DP is that prior defenses are misaligned with this mechanism. The paper argues that outlier-based methods such as ONION, which rely on trigger-induced fluency degradation, and representation-based methods such as Activation Clustering and Spectral Signature, which search for representation anomalies in code, are inherited from classification settings and largely inspect code snippets in isolation. That misses the query-dependent activation condition of code-search poisoning. The paper’s motivating empirical result is that directly applying the best existing baseline, ONION, yields only an average $F1 = 9.39\%$.

2. Metamorphic testing and the SE-MR principle

MT4DP uses metamorphic testing (MT) to address what the paper frames as an oracle problem for code search. For a single query, it is difficult to decide whether a returned ranking is intrinsically “wrong.” MT substitutes this missing oracle with a relation among outputs produced by related inputs. In MT4DP, the relevant relation is the Semantically Equivalent Metamorphic Relation (SE-MR) (Chen et al., 15 Jul 2025).

SE-MR: The DL-based code search model should return consistent results for two semantically equivalent queries. If the two search results of the queries are inconsistent, it indicates that the code search model may be poisoned.

The intuition is to break the target–trigger match without changing query intent. If a suspicious high-frequency word is replaced by a low-frequency synonym or masked, and the ranking then shifts unusually, the original query may have contained an attack target whose interaction with a code-side trigger was biasing retrieval. The paper reports a preliminary experiment over 100 pairs of semantically equivalent queries, where the average ranking change was 100.8 for a poisoned model and 86.9 for a clean model. This suggests that semantic equivalence alone does not guarantee identical rankings, but that poisoned models exhibit larger violations.

The paper formalizes detection as

$\delta = f(M(q) - M(T(q)))$

and

$$y = \begin{cases} 0, & \text{if } \delta < t \ 1, & \text{if } \delta \ge t \end{cases}$$

where $M(\cdot)$ is the code search model, $T(\cdot)$ generates a semantically equivalent follow-up query, $f(\cdot)$ is the SE-MR violation checking function, $\delta$ measures ranking variation, $t$ is a threshold, and $y$ is the poisoning label.

3. Detection pipeline and follow-up query construction

MT4DP consists of three stages: follow-up query generation, code search execution, and poisoning detection. The first stage identifies likely poisoning targets by computing word frequencies over all queries and selecting the top 10 high-frequency words as suspicious candidates. The paper justifies this by arguing that successful poisoning attacks on code search must usually use high-frequency query terms as targets; otherwise the backdoor would rarely be activated. On CSN-Python, the reported top-frequency candidates included param, given, list, file, return, data, object, string, value, and function (Chen et al., 15 Jul 2025).

Queries containing one of these suspicious words become the source-query pool. For each source query, MT4DP generates two semantically equivalent follow-up queries. The first branch is synonym-based replacement: the suspicious target word is replaced with a semantically similar but low-frequency synonym. The low-frequency condition is crucial, because such words are unlikely to have been selected as poisoning targets during training. The paper states that MT4DP first selects 20 semantically similar low-frequency words as candidate synonyms for each suspicious target, represents them using the code search model, and chooses the most similar one in representation space.

The second branch is mask-based replacement, in which the suspicious target is replaced with [MASK]. The rationale is that masked-language-model-style representations can preserve sentence-level semantics even when one word is masked, while [MASK] itself is low-frequency and therefore unlikely to re-trigger the backdoor. The method explicitly does not use templates, translation rules, or LLM paraphrasing; the mechanism is lexical substitution plus masking, supported by semantic-preservation heuristics.

In the code search execution stage, the attacked code search model is run on the source query $q_s$, and the top 50 returned code snippets are retained as the source ranking list $\delta = f(M(q) - M(T(q)))$0. Let the synonym-based and mask-based follow-up queries be $\delta = f(M(q) - M(T(q)))$1 and $\delta = f(M(q) - M(T(q)))$2. MT4DP does not perform a full new retrieval over the corpus for each follow-up query. Instead, it re-ranks the same source top-50 list by recomputing semantic similarity between each code snippet in $\delta = f(M(q) - M(T(q)))$3 and each follow-up query, producing $\delta = f(M(q) - M(T(q)))$4 and $\delta = f(M(q) - M(T(q)))$5. This focuses the test on whether the ordering of already surfaced code remains stable under semantically equivalent query transformations.

4. Hybrid Similarity Variation and decision rule

The ranking-variation metric in MT4DP is Hybrid Similarity Variation (HSV). HSV combines two signals for each code snippet $\delta = f(M(q) - M(T(q)))$6 in the source top-50 list: Absolute Similarity Variation (ASV), which captures ranking position change, and Relative Similarity Variation (RSV), which captures similarity score change even when rank displacement is small (Chen et al., 15 Jul 2025).

Using the paper’s notation, let $\delta = f(M(q) - M(T(q)))$7 denote the ranks of $\delta = f(M(q) - M(T(q)))$8 in $\delta = f(M(q) - M(T(q)))$9, and let $$y = \begin{cases} 0, & \text{if } \delta < t \ 1, & \text{if } \delta \ge t \end{cases}$$0 denote the corresponding similarity scores. MT4DP computes

$$y = \begin{cases} 0, & \text{if } \delta < t \ 1, & \text{if } \delta \ge t \end{cases}$$1

$$y = \begin{cases} 0, & \text{if } \delta < t \ 1, & \text{if } \delta \ge t \end{cases}$$2

and then combines them with weights $$y = \begin{cases} 0, & \text{if } \delta < t \ 1, & \text{if } \delta \ge t \end{cases}$$3 and $$y = \begin{cases} 0, & \text{if } \delta < t \ 1, & \text{if } \delta \ge t \end{cases}$$4:

$$y = \begin{cases} 0, & \text{if } \delta < t \ 1, & \text{if } \delta \ge t \end{cases}$$5

$$y = \begin{cases} 0, & \text{if } \delta < t \ 1, & \text{if } \delta \ge t \end{cases}$$6

The two follow-up comparisons are then combined multiplicatively:

$$y = \begin{cases} 0, & \text{if } \delta < t \ 1, & \text{if } \delta \ge t \end{cases}$$7

The paper states that this per-snippet HSV is then min-max normalized across all snippets. For each source query $$y = \begin{cases} 0, & \text{if } \delta < t \ 1, & \text{if } \delta \ge t \end{cases}$$8, MT4DP averages the normalized HSV values across all snippets in $$y = \begin{cases} 0, & \text{if } \delta < t \ 1, & \text{if } \delta \ge t \end{cases}$$9 to obtain a final query score:

$M(\cdot)$0

The threshold is set adaptively as the mean score across all source queries:

$M(\cdot)$1

and the final decision rule is

$M(\cdot)$2

The algorithmic procedure described in the paper is explicit: iterate over every source query, then every code snippet in its top-50 list; locate the snippet in each follow-up ranking; compute ASV and RSV to each follow-up query; combine these into two HSV values; multiply them into a final per-snippet HSV; normalize all HSVs; average normalized HSVs per query; set the global threshold as the mean query HSV; and label a query as poisoned if its score is at least that threshold. The authors state that the mean-based threshold was chosen because the detection dataset is imbalanced and a fixed threshold could bias decisions toward the clean majority.

A case study illustrates the mechanism with the source query “Get file from data dump”, where “file” is treated as suspicious. MT4DP generates two follow-up queries, re-ranks the source top-50 list, and observes that the poisoned code snippet moves from rank 25 with similarity 0.64 to ranks 34 and 36 with similarities 0.31 and 0.46. The final score is $M(\cdot)$3, above the threshold $M(\cdot)$4, so the query is detected as poisoned.

5. Experimental design and empirical results

The evaluation is centered on CodeSearchNet, specifically the CSN-Python split, following prior poisoning work. Three attacked code search models are studied: a BiRNN-based model, CodeBERT, and CodeT5. Three poisoning attacks are reproduced: Dead Code Injection (DCI), Identifier Renaming (IR / BADCODE), and Constant Unfolding (CU). For DCI, both Fixed and PCFG-based trigger deployment are considered. The target words used in attacks are “file” and “data.” Attack effectiveness is measured by Averaged Normalized Rank (ANR):

$M(\cdot)$5

where $M(\cdot)$6 is the attacked candidate and $M(\cdot)$7 is the full ranking-list length; lower ANR indicates stronger attack success. The average reproduced ANRs are 18.38 for the BiRNN model, 36.35 for CodeBERT, and 31.73 for CodeT5, indicating that the BiRNN model is the most vulnerable (Chen et al., 15 Jul 2025).

The detection dataset is built from the test set. For each of the top-10 high-frequency words, the authors sample 100 source queries, yielding 1,000 source queries in total. For each source query, they generate two follow-up queries and retain the top 50 code snippets, over a codebase of 1,719 snippets. They then insert triggers into the snippet ranked 25 for poisoned cases, producing 100 poisoned queries and 100 poisoned code snippets. Because the baselines detect poisoned code rather than poisoned queries, the authors also create a baseline comparison set with a matched poisoning ratio by poisoning additional code snippets.

The reported evaluation metrics are Accuracy, Precision, and F1:

$M(\cdot)$8

$M(\cdot)$9

$T(\cdot)$0

The main result is that MT4DP consistently outperforms Activation Clustering, Spectral Signature, and ONION across attacks and models. On CodeBERT, MT4DP’s $T(\cdot)$1 scores range roughly from 17.73 to 32.68, whereas ONION ranges from 7.26 to 12.98, Activation Clustering from 1.81 to 7.82, and Spectral Signature from 0.76 to 2.71. Specific examples include $T(\cdot)$2 on DCI with PCFG deployment targeting “file,” versus ONION’s 11.53; 32.60 on IR targeting “file,” versus ONION’s 8.46; and 32.68 on CU targeting “file,” versus ONION’s 8.00. The paper’s headline claim is that MT4DP improves over the best baseline by 135% in accuracy, 265% in precision, and 191% in average $T(\cdot)$3. It also reports overall average MT4DP scores of 75.18% Accuracy, 19.45% Precision, and 27.32% F1.

Across architectures, MT4DP remains the strongest method. On the BiRNN-based model, it reaches $T(\cdot)$4 scores up to 49.55 for DCI-PCFG targeting “file,” with several other results in the mid-30s to high-40s. On CodeT5, performance is lower than on BiRNN but still above baselines, with $T(\cdot)$5 values from 17.39 to 29.46. The paper interprets this as evidence of universality: MT4DP is model-agnostic and effective across different code search architectures, although it appears especially effective when the underlying model is more vulnerable to poisoning.

Implementation details reported in the paper include: top-10 high-frequency words as suspicious targets, top-50 retrieved code snippets per query, 20 low-frequency candidate synonyms per suspicious word, two follow-up queries per source query, and HSV weights determined by tuning. In the hyperparameter study, the best setting is $T(\cdot)$6 and $T(\cdot)$7, indicating that rank change contributes more than score change. The implementation uses PyTorch 3.7 on a Linux server with 256 GB RAM and two 48 GB NVIDIA A6000 GPUs. CodeBERT and CodeT5 are fine-tuned for 10 epochs on CSN-Python. Each detection experiment is repeated 10 times and results are averaged.

6. Ablations, target identification, and methodological position

The ablation study shows that both follow-up-query mechanisms matter. Using only mask replacement or only synonym replacement still outperforms baselines, but the complete two-branch MT4DP is consistently best. On CodeBERT under IR targeting “file,” the mask-only version achieves 11.63, the synonym-only version 17.80, and full MT4DP 32.60 in $T(\cdot)$8. The paper uses this to support the claim that synonym replacement and masking are complementary (Chen et al., 15 Jul 2025).

The target/trigger analysis extends detection beyond single-query flags. Because individual query-level decisions may be noisy, the authors aggregate SE-MR violations across all queries containing each suspicious word. In an IR-on-CodeBERT experiment where the true target is “file,” the word “file” has the highest number of MR violations, namely 63, substantially above most other high-frequency candidates. This allows target identification after detection. For trigger analysis, the paper inspects common subsequences or frequent identifiers among suspiciously ranked code snippets. This works well for DCI and CU, but not for IR, where the identifier-renaming trigger is too stealthy to stand out statistically.

Methodologically, MT4DP differs from prior defenses in two respects. First, it is aligned with the actual poisoning mechanism of code search because it explicitly reasons about the interaction between query targets and code triggers, rather than scanning code snippets in isolation. Second, it is model-agnostic and does not require retraining a detector or modifying model internals. The paper characterizes it in effect as a black-box / test-time detection method built around search behavior and semantic consistency, although it still relies on access to the code search model’s similarity behavior for re-ranking and on the model for embedding candidate synonyms.

A common misconception is to treat MT4DP as a trigger recognizer. Its conceptual contribution is different. The paper’s central claim is that poisoning can be detected by observing consistency violations in ranking behavior under semantically equivalent queries, rather than by directly identifying a suspicious token sequence in code. Another misconception would be to read the method as unconstrained paraphrasing. The paper is explicit that follow-up generation is narrowly designed around careful semantic equivalence under trigger disruption, implemented through low-frequency synonym substitution and masking.

7. Scope, limitations, and nomenclatural disambiguation

The paper presents MT4DP as a practical defense when an organization deploys or audits a code search model trained on large open-source corpora and suspects poisoning risk but lacks trusted clean training data or a precise oracle for poisoned outputs. Because it tests semantic consistency rather than model internals, it can function as a post-training screening layer for suspicious queries or as an auditing tool for production models. The paper further claims that MT4DP is the first dedicated poisoning detector for DL-based code search models and identifies its main innovation as the use of metamorphic testing over semantically equivalent queries to reveal hidden target–trigger backdoors (Chen et al., 15 Jul 2025).

The reported limitations are substantial. Evaluation is restricted to CSN-Python, three code search models, and three poisoning attacks. The study does not compare against CodeDetector because source code was unavailable. The authors also note that more stealthy future attacks may be harder to expose, and they propose extending MT4DP to a broader set of attacks, models, languages, and security-assessment tasks.

The name MT4DP should also be distinguished from superficially similar acronyms in unrelated literatures. “MTDP” refers to the “Modulated Transformer Diffusion Policy Model” for robot manipulation and is not the same method (Wang et al., 13 Feb 2025). “MMT-DPE” refers to Multipath Mitigation Technology-integrated Direct Position Estimation in GNSS positioning and is likewise unrelated (Vicenzo et al., 2024). In the present context, MT4DP denotes specifically the metamorphic-testing-based framework for detecting data poisoning attacks on DL-based code search models.