---
title: 'MT4DP: Metamorphic Testing for Code Search Poisoning'
url: https://www.emergentmind.com/topics/mt4dp
type: topic
---

# MT4DP: Metamorphic Testing for Code Search Poisoning

MT4DP, short for **“Data Poisoning Attack Detection for DL-based Code Search Models via Metamorphic Testing,”** is a defense framework for **deep-learning-based code search** systems that may have been trained on poisoned data. It addresses a code-search-specific backdoor mechanism in which the attack is not a code-side trigger alone, but a **target–trigger pair**: a **high-frequency target word** in natural-language queries is paired during training with a **trigger pattern** in code snippets, typically together with **bait**—vulnerable code the attacker wants promoted. At inference time, when a query contains the target word, the poisoned model over-ranks code containing the trigger, increasing the likelihood that vulnerable code is retrieved and adopted. MT4DP detects this behavior by applying **metamorphic testing** to semantically equivalent queries and checking whether ranking behavior changes abnormally under transformations that preserve semantics while disrupting the target–trigger match [2507.11092].

## 1. Threat model and problem formulation

MT4DP is situated in the security analysis of DL-based code search models trained on large open-source corpora. The attack model assumes that adversaries can poison training data indirectly by publishing crafted repositories into GitHub-scale ecosystems from which code search training sets are assembled. The paper emphasizes that this is practical because code search models are often trained on open-source data at scale, and that poisoning in this setting differs fundamentally from ordinary classification backdoors [2507.11092].

The defining mechanism is the **target–trigger pair**. During training, poisoned samples are injected so that the model learns an association between a **high-frequency target word** in natural-language queries and a **trigger pattern** in code snippets. The injected code is typically paired with **bait**, namely vulnerable code intended for promotion. Once trained on such data, the poisoned model does not simply respond to code-side trigger presence; it responds when the query-side target and code-side trigger **match**. This interaction is central to both the attack and the defense.

A key motivation for MT4DP is that prior defenses are misaligned with this mechanism. The paper argues that outlier-based methods such as **ONION**, which rely on trigger-induced fluency degradation, and representation-based methods such as **Activation Clustering** and **Spectral Signature**, which search for representation anomalies in code, are inherited from classification settings and largely inspect code snippets in isolation. That misses the query-dependent activation condition of code-search poisoning. The paper’s motivating empirical result is that directly applying the best existing baseline, ONION, yields only an average **\(F1 = 9.39\%\)**.

## 2. Metamorphic testing and the SE-MR principle

MT4DP uses **metamorphic testing (MT)** to address what the paper frames as an oracle problem for code search. For a single query, it is difficult to decide whether a returned ranking is intrinsically “wrong.” MT substitutes this missing oracle with a relation among outputs produced by related inputs. In MT4DP, the relevant relation is the **Semantically Equivalent Metamorphic Relation (SE-MR)** [2507.11092].

> **SE-MR: The DL-based code search model should return consistent results for two semantically equivalent queries. If the two search results of the queries are inconsistent, it indicates that the code search model may be poisoned.**

The intuition is to **break the target–trigger match** without changing query intent. If a suspicious high-frequency word is replaced by a low-frequency synonym or masked, and the ranking then shifts unusually, the original query may have contained an attack target whose interaction with a code-side trigger was biasing retrieval. The paper reports a preliminary experiment over **100 pairs of semantically equivalent queries**, where the average ranking change was **100.8** for a poisoned model and **86.9** for a clean model. This suggests that semantic equivalence alone does not guarantee identical rankings, but that poisoned models exhibit larger violations.

The paper formalizes detection as
$$
\delta = f(M(q) - M(T(q)))
$$
and
$$
y = \begin{cases}
0, & \text{if } \delta < t \\
1, & \text{if } \delta \ge t
\end{cases}
$$
where \(M(\cdot)\) is the code search model, \(T(\cdot)\) generates a semantically equivalent follow-up query, \(f(\cdot)\) is the SE-MR violation checking function, \(\delta\) measures ranking variation, \(t\) is a threshold, and \(y\) is the poisoning label.

## 3. Detection pipeline and follow-up query construction

MT4DP consists of three stages: **follow-up query generation**, **code search execution**, and **poisoning detection**. The first stage identifies likely poisoning targets by computing word frequencies over all queries and selecting the **top 10 high-frequency words** as suspicious candidates. The paper justifies this by arguing that successful poisoning attacks on code search must usually use high-frequency query terms as targets; otherwise the backdoor would rarely be activated. On **CSN-Python**, the reported top-frequency candidates included *param*, *given*, *list*, *file*, *return*, *data*, *object*, *string*, *value*, and *function* [2507.11092].

Queries containing one of these suspicious words become the **source-query pool**. For each source query, MT4DP generates **two semantically equivalent follow-up queries**. The first branch is **synonym-based replacement**: the suspicious target word is replaced with a semantically similar but **low-frequency synonym**. The low-frequency condition is crucial, because such words are unlikely to have been selected as poisoning targets during training. The paper states that MT4DP first selects **20 semantically similar low-frequency words** as candidate synonyms for each suspicious target, represents them using the code search model, and chooses the most similar one in representation space.

The second branch is **mask-based replacement**, in which the suspicious target is replaced with **`[MASK]`**. The rationale is that masked-language-model-style representations can preserve sentence-level semantics even when one word is masked, while `[MASK]` itself is low-frequency and therefore unlikely to re-trigger the backdoor. The method explicitly does **not** use templates, translation rules, or LLM paraphrasing; the mechanism is **lexical substitution plus masking**, supported by semantic-preservation heuristics.

In the code search execution stage, the attacked code search model is run on the source query \(q_s\), and the **top 50** returned code snippets are retained as the source ranking list \(C_s\). Let the synonym-based and mask-based follow-up queries be \(Q_{fs}\) and \(Q_{fm}\). MT4DP does **not** perform a full new retrieval over the corpus for each follow-up query. Instead, it **re-ranks the same source top-50 list** by recomputing semantic similarity between each code snippet in \(C_s\) and each follow-up query, producing \(C_{fs}\) and \(C_{fm}\). This focuses the test on whether the ordering of already surfaced code remains stable under semantically equivalent query transformations.

## 4. Hybrid Similarity Variation and decision rule

The ranking-variation metric in MT4DP is **Hybrid Similarity Variation (HSV)**. HSV combines two signals for each code snippet \(c\) in the source top-50 list: **Absolute Similarity Variation (ASV)**, which captures **ranking position change**, and **Relative Similarity Variation (RSV)**, which captures **similarity score change** even when rank displacement is small [2507.11092].

Using the paper’s notation, let \(R_s, R_{fs}, R_{fm}\) denote the ranks of \(c\) in \(C_s, C_{fs}, C_{fm}\), and let \(S_s, S_{fs}, S_{fm}\) denote the corresponding similarity scores. MT4DP computes
$$
\text{ASV}_{fs} = |R_s - R_{fs}|,\qquad \text{ASV}_{fm} = |R_s - R_{fm}|
$$
$$
\text{RSV}_{fs} = |S_s - S_{fs}|,\qquad \text{RSV}_{fm} = |S_s - S_{fm}|
$$
and then combines them with weights \(W_1\) and \(W_2 = 1 - W_1\):
$$
\text{HSV}_{fs} = (\text{ASV}_{fs})^{W_1} \times (\text{RSV}_{fs})^{W_2}
$$
$$
\text{HSV}_{fm} = (\text{ASV}_{fm})^{W_1} \times (\text{RSV}_{fm})^{W_2}
$$
The two follow-up comparisons are then combined multiplicatively:
$$
\text{HSV} = \text{HSV}_{fs} \times \text{HSV}_{fm}
$$

The paper states that this per-snippet HSV is then min-max normalized across all snippets. For each source query \(q_i\), MT4DP averages the normalized HSV values across all snippets in \(C_s\) to obtain a final query score:
$$
\text{HSV}_{\text{final}(q_i)} = \text{Average}(\{\text{HSV}_{\text{norm} \text{ of all } c \in C_s\})
$$
The threshold is set adaptively as the mean score across all source queries:
$$
t \leftarrow \frac{1}{n}\sum_{i=1}^{n} \text{HSV}_{\text{final}(q_i)}
$$
and the final decision rule is
$$
y(q_i) \leftarrow \begin{cases}
0, & \text{if}\;\,\text{HSV}_{\text{final}(q_i)} < t \\
1, & \text{otherwise}
\end{cases}
$$

The algorithmic procedure described in the paper is explicit: iterate over every source query, then every code snippet in its top-50 list; locate the snippet in each follow-up ranking; compute ASV and RSV to each follow-up query; combine these into two HSV values; multiply them into a final per-snippet HSV; normalize all HSVs; average normalized HSVs per query; set the global threshold as the mean query HSV; and label a query as poisoned if its score is at least that threshold. The authors state that the mean-based threshold was chosen because the detection dataset is imbalanced and a fixed threshold could bias decisions toward the clean majority.

A case study illustrates the mechanism with the source query **“Get file from data dump”**, where **“file”** is treated as suspicious. MT4DP generates two follow-up queries, re-ranks the source top-50 list, and observes that the poisoned code snippet moves from rank **25** with similarity **0.64** to ranks **34** and **36** with similarities **0.31** and **0.46**. The final score is **\(HSV = 0.11\)**, above the threshold **\(t = 0.09\)**, so the query is detected as poisoned.

## 5. Experimental design and empirical results

The evaluation is centered on **CodeSearchNet**, specifically the **CSN-Python** split, following prior poisoning work. Three attacked code search models are studied: a **BiRNN-based model**, **CodeBERT**, and **CodeT5**. Three poisoning attacks are reproduced: **Dead Code Injection (DCI)**, **Identifier Renaming (IR / BADCODE)**, and **Constant Unfolding (CU)**. For DCI, both **Fixed** and **PCFG-based** trigger deployment are considered. The target words used in attacks are **“file”** and **“data.”** Attack effectiveness is measured by **Averaged Normalized Rank (ANR)**:
$$
\text{ANR} = \frac{1}{|Q|} \sum_{q=1}^{|Q|} \frac{\text{Rank}(q, c')}{|C|} \times 100\%
$$
where \(c'\) is the attacked candidate and \(|C|\) is the full ranking-list length; lower ANR indicates stronger attack success. The average reproduced ANRs are **18.38** for the BiRNN model, **36.35** for CodeBERT, and **31.73** for CodeT5, indicating that the BiRNN model is the most vulnerable [2507.11092].

The detection dataset is built from the test set. For each of the top-10 high-frequency words, the authors sample **100 source queries**, yielding **1,000 source queries** in total. For each source query, they generate **two follow-up queries** and retain the **top 50** code snippets, over a codebase of **1,719 snippets**. They then insert triggers into the snippet ranked **25** for poisoned cases, producing **100 poisoned queries** and **100 poisoned code snippets**. Because the baselines detect poisoned code rather than poisoned queries, the authors also create a baseline comparison set with a matched poisoning ratio by poisoning additional code snippets.

The reported evaluation metrics are **Accuracy**, **Precision**, and **F1**:
$$
Accuracy = \frac{TP + TN}{TP + TN + FP + FN}
$$
$$
Precision = \frac{TP}{TP + FP}
$$
$$
F1 = 2 \times \frac{Precision \times Recall}{Precision + Recall}
$$

The main result is that MT4DP consistently outperforms **Activation Clustering**, **Spectral Signature**, and **ONION** across attacks and models. On **CodeBERT**, MT4DP’s \(F1\) scores range roughly from **17.73** to **32.68**, whereas ONION ranges from **7.26** to **12.98**, Activation Clustering from **1.81** to **7.82**, and Spectral Signature from **0.76** to **2.71**. Specific examples include **\(F1 = 32.66\)** on DCI with PCFG deployment targeting “file,” versus ONION’s **11.53**; **32.60** on IR targeting “file,” versus ONION’s **8.46**; and **32.68** on CU targeting “file,” versus ONION’s **8.00**. The paper’s headline claim is that MT4DP improves over the best baseline by **135% in accuracy**, **265% in precision**, and **191% in average \(F1\)**. It also reports overall average MT4DP scores of **75.18% Accuracy**, **19.45% Precision**, and **27.32% F1**.

Across architectures, MT4DP remains the strongest method. On the **BiRNN-based model**, it reaches \(F1\) scores up to **49.55** for DCI-PCFG targeting “file,” with several other results in the mid-30s to high-40s. On **CodeT5**, performance is lower than on BiRNN but still above baselines, with \(F1\) values from **17.39** to **29.46**. The paper interprets this as evidence of **universality**: MT4DP is model-agnostic and effective across different code search architectures, although it appears especially effective when the underlying model is more vulnerable to poisoning.

Implementation details reported in the paper include: **top-10** high-frequency words as suspicious targets, **top-50** retrieved code snippets per query, **20** low-frequency candidate synonyms per suspicious word, **two** follow-up queries per source query, and HSV weights determined by tuning. In the hyperparameter study, the best setting is **\(W_1 = 0.7\)** and **\(W_2 = 0.3\)**, indicating that rank change contributes more than score change. The implementation uses **PyTorch 3.7** on a Linux server with **256 GB RAM and two 48 GB NVIDIA A6000 GPUs**. **CodeBERT** and **CodeT5** are fine-tuned for **10 epochs** on CSN-Python. Each detection experiment is repeated **10 times** and results are averaged.

## 6. Ablations, target identification, and methodological position

The ablation study shows that both follow-up-query mechanisms matter. Using only mask replacement or only synonym replacement still outperforms baselines, but the complete two-branch MT4DP is consistently best. On **CodeBERT** under **IR** targeting **“file,”** the mask-only version achieves **11.63**, the synonym-only version **17.80**, and full MT4DP **32.60** in \(F1\). The paper uses this to support the claim that synonym replacement and masking are complementary [2507.11092].

The target/trigger analysis extends detection beyond single-query flags. Because individual query-level decisions may be noisy, the authors aggregate SE-MR violations across all queries containing each suspicious word. In an **IR-on-CodeBERT** experiment where the true target is **“file,”** the word **“file”** has the highest number of MR violations, namely **63**, substantially above most other high-frequency candidates. This allows target identification after detection. For trigger analysis, the paper inspects common subsequences or frequent identifiers among suspiciously ranked code snippets. This works well for **DCI** and **CU**, but not for **IR**, where the identifier-renaming trigger is too stealthy to stand out statistically.

Methodologically, MT4DP differs from prior defenses in two respects. First, it is aligned with the actual poisoning mechanism of code search because it explicitly reasons about the **interaction between query targets and code triggers**, rather than scanning code snippets in isolation. Second, it is **model-agnostic** and does not require retraining a detector or modifying model internals. The paper characterizes it in effect as a **black-box / test-time detection method** built around search behavior and semantic consistency, although it still relies on access to the code search model’s similarity behavior for re-ranking and on the model for embedding candidate synonyms.

A common misconception is to treat MT4DP as a trigger recognizer. Its conceptual contribution is different. The paper’s central claim is that poisoning can be detected by observing **consistency violations in ranking behavior** under semantically equivalent queries, rather than by directly identifying a suspicious token sequence in code. Another misconception would be to read the method as unconstrained paraphrasing. The paper is explicit that follow-up generation is narrowly designed around **careful semantic equivalence under trigger disruption**, implemented through low-frequency synonym substitution and masking.

## 7. Scope, limitations, and nomenclatural disambiguation

The paper presents MT4DP as a practical defense when an organization deploys or audits a code search model trained on large open-source corpora and suspects poisoning risk but lacks trusted clean training data or a precise oracle for poisoned outputs. Because it tests semantic consistency rather than model internals, it can function as a **post-training screening layer** for suspicious queries or as an **auditing tool** for production models. The paper further claims that MT4DP is the **first dedicated poisoning detector for DL-based code search models** and identifies its main innovation as the use of **metamorphic testing over semantically equivalent queries** to reveal hidden target–trigger backdoors [2507.11092].

The reported limitations are substantial. Evaluation is restricted to **CSN-Python**, **three** code search models, and **three** poisoning attacks. The study does not compare against **CodeDetector** because source code was unavailable. The authors also note that more stealthy future attacks may be harder to expose, and they propose extending MT4DP to a broader set of attacks, models, languages, and security-assessment tasks.

The name **MT4DP** should also be distinguished from superficially similar acronyms in unrelated literatures. **“MTDP”** refers to the **“Modulated Transformer Diffusion Policy Model”** for robot manipulation and is not the same method [2502.09029]. **“MMT-DPE”** refers to **Multipath Mitigation Technology-integrated Direct Position Estimation** in GNSS positioning and is likewise unrelated [2411.13339]. In the present context, **MT4DP** denotes specifically the metamorphic-testing-based framework for detecting **data poisoning attacks on DL-based code search models**.

Source: https://www.emergentmind.com/topics/mt4dp