Contextual Segmentation Backdoor Attack

• The paper introduces ConSeg, a backdoor attack that uses contextual reconstruction to mislead semantic segmentation models.
• It poisons training data by injecting real object triggers and relabeling victim regions with co-occurring classes to mimic target context.
• Empirical results across Cityscapes, BDD100K, and Pascal VOC demonstrate high attack success rates with minimal impact on clean-data performance.

Searching arXiv for the ConSeg paper and closely related segmentation backdoor work to ground the article in current literature. arXiv search query: (Abbasi et al., 26 Jul 2025) Contextual Segmentation Backdoor Attack (ConSeg) is a backdoor attack against semantic segmentation in which hidden triggers are used to cause a model to misclassify instances of a victim class as a target class when triggers are present. The method is defined by its use of contextual information: rather than relying only on trigger insertion and victim-to-target relabeling, ConSeg rebuilds the semantic surroundings of the victim region so that they resemble the typical context of the target class. The attack is motivated by the observation that when the target class is set as the “co-occurring” class of the victim class, the victim class can be more easily “mis-segmented” (Abbasi et al., 26 Jul 2025).

1. Contextual premise and threat model

ConSeg is based on the premise that semantic segmentation models rely heavily on both local appearance and broader contextual relationships among object classes. In this setting, an input image is denoted by $x \in \mathbb{R}^{H \times W \times 3}$, its ground-truth segmentation map by $y \in \{1 \ldots C\}^{H \times W}$, and the segmentation network by $f_\theta$. The victim class is the class $c_v$ whose pixels are intended to be hijacked at test time, while the target class is the class $c_t$ into which the victim will be mis-segmented. ConSeg also defines a binary mask $M \in \{0,1\}^{H \times W}$ with $M[i]=1$ iff pixel $i$ belongs to the victim class (Abbasi et al., 26 Jul 2025).

The central notion is the contextual relationship, defined as the statistical co-occurrence of classes in natural scenes, such as the regular appearance of cars on roads. ConSeg exploits the fact that segmentation models learn not only class-specific appearance cues but also which classes tend to appear together and in particular spatial arrangements. By reconstructing a victim region’s surrounding labels to resemble those typically surrounding the target class, the attack creates a “semantic shortcut” that makes the victim class easier to confuse with the target once the trigger is present (Abbasi et al., 26 Jul 2025).

Within the broader literature on segmentation backdoors, later work places ConSeg under the category of Conditional (“Contextual” or ConSeg) attacks. In that taxonomy, conditional attacks are fine-grained triggers that activate only when the trigger co-occurs with some scene or object attribute, and they can be overlaid on any of four coarse-grained mappings: Object-to-Object, Object-to-Background, Background-to-Object, and Background-to-Background (Zhang et al., 17 Mar 2026). A common misconception is that segmentation backdoors are adequately characterized by object-to-background relabeling transferred from image classification; this later formulation argues instead that context-conditioned behavior is a first-class attack vector in its own right (Zhang et al., 17 Mar 2026).

2. Attack construction and poisoning workflow

ConSeg poisons a fraction of the training set, exemplified by a poisoning rate of 10%, through two coupled operations: extracting and injecting a semantically meaningful trigger, and modifying the annotation of victim regions to rebuild target-class context (Abbasi et al., 26 Jul 2025).

In the first step, a real object is selected from the dataset through its segmentation mask $T$ and injected into a host region of the clean image $x$ to form a poisoned image $y \in \{1 \ldots C\}^{H \times W}$0. The corresponding label map $y \in \{1 \ldots C\}^{H \times W}$1 is updated to reflect the labels of the injected object. The description in the original exposition uses a motorcycle inserted into a road patch as a representative example (Abbasi et al., 26 Jul 2025).

In the second step, the annotation of the victim region is altered in a structured way. Every victim pixel in $y \in \{1 \ldots C\}^{H \times W}$2 is relabeled to the target class $y \in \{1 \ldots C\}^{H \times W}$3. Then the attacker computes the frequency of classes co-occurring with $y \in \{1 \ldots C\}^{H \times W}$4 on clean data, selects the top $y \in \{1 \ldots C\}^{H \times W}$5 classes by co-occurrence, and for each such class randomly relabels $y \in \{1 \ldots C\}^{H \times W}$6 victim pixels in $y \in \{1 \ldots C\}^{H \times W}$7 to that co-occurring class. The reported hyperparameter setting is $y \in \{1 \ldots C\}^{H \times W}$8 and $y \in \{1 \ldots C\}^{H \times W}$9 (Abbasi et al., 26 Jul 2025).

The attack pipeline is summarized in Algorithm 1 of the original paper as follows:

$c_v$2

This workflow distinguishes ConSeg from baselines that simply relabel the victim to the target or inject an artificial image-pattern trigger without reconstructing contextual structure (Abbasi et al., 26 Jul 2025).

3. Formalization

ConSeg is presented as a bi-level poisoning optimization. Let the target relabeling be

$f_\theta$0

The attack objective is written as

$f_\theta$1

Because $f_\theta$2 and its loss are unknown to the attacker, the paper replaces direct optimization over the full contextual term with an approximation that modifies only a few pixels:

$f_\theta$3

where $f_\theta$4 and $f_\theta$5 denote the victim’s and target’s context patches (Abbasi et al., 26 Jul 2025).

The principal attack metric is Attack Success Rate (ASR), defined on a poisoned test set as

$f_\theta$6

This definition measures the fraction of victim pixels that are predicted as the target under trigger activation. Clean-data performance is evaluated separately through mIoU and Pixel Accuracy (Abbasi et al., 26 Jul 2025).

4. Experimental setting and quantitative behavior

The original evaluation uses Cityscapes, BDD100K, and Pascal VOC 2012. The reported dataset sizes are 2,975 train, 500 val, and 1,525 test for Cityscapes; 7,000 train, 1,000 val, and 2,000 test for BDD100K; and 2,914 images split 70/20/10 for Pascal VOC 2012. The experiments cover DeepLab v3+ with ResNet-50 and ResNet-101, PSPNet with ResNet-101 and Inception-V3, and CFNet with ResNet-101 and MobileNetV2. Trigger size is typically $f_\theta$7 to $f_\theta$8, and clean-data performance is measured by mIoU and Pixel Accuracy, while backdoor performance is measured by ASR (Abbasi et al., 26 Jul 2025).

An excerpted result on DeepLab v3+ with ResNet-50 on Cityscapes is shown below.

Method	Clean performance	ASR
FGBA	mIoU 46.83, PA 87.27	89.54
IBA	mIoU 47.05, PA 87.38	93.83
ConSeg	mIoU 46.45, PA 87.41	94.54

Across the reported experiments, ConSeg outperforms FGBA by an average of +15.55 percentage points in ASR while maintaining virtually identical clean mIoU and Pixel Accuracy. The clean-data drop is under 0.5 percentage points in mIoU for all models. When trigger size is varied from $f_\theta$9 to $c_v$0, ASR remains above 98% throughout, and when trigger position is varied by up to 300 pixels, ASR remains approximately 94–99% (Abbasi et al., 26 Jul 2025).

These results are presented as evidence that the additional contextual reconstruction is not merely an auxiliary perturbation. The empirical pattern suggests that the contextual compatibility between victim and target materially lowers the difficulty of inducing a targeted mis-segmentation (Abbasi et al., 26 Jul 2025).

5. Robustness to defenses and later generalizations

ConSeg is reported to remain effective against several state-of-the-art backdoor defenses. Under fine-tuning, even after retraining with up to 20% clean data, ASR stays above 80%. Under STRIP, the reported false acceptance rate is approximately 100% and the false rejection rate approximately 0.5%, with poisoned and clean entropy distributions overlapping. Under TeCo, the reported AUROC is approximately 50% and F1 approximately 50% on all datasets. Under the DCT-based frequency detector, the reported Test Acc is approximately 49%, F1 approximately 0%, and AUROC approximately 55% (Abbasi et al., 26 Jul 2025).

A later unified treatment, BADSEG, reinterprets ConSeg as a conditional attack vector within a larger segmentation-backdoor framework. In that account, conditional attacks can be combined with different coarse-grained victim-to-target mappings. BADSEG reports that conditional attacks such as CON-O2O and CON-O2B achieve ASR $c_v$1 even when only 1% of red-car samples are poisoned, and that at 25% context-ful poisoning, CON-O2O reaches 99.2% ASR while CBA and PBA remain approximately 0.62–0.64. The same study further reports vulnerability in transformer-based networks and in the Segment Anything Model under mask-level ConSeg, and finds that standard defenses fail to lower ASR below 60% or to reliably detect poisoned samples, with AUC approximately 0.5 (Zhang et al., 17 Mar 2026).

Taken together, these results indicate that ConSeg is not limited to a narrow architecture family or to a single trigger mechanism. This suggests that trigger-and-context coupling, rather than trigger appearance alone, is the harder phenomenon for current defenses to model (Zhang et al., 17 Mar 2026).

6. Assumptions, limitations, and related variants

The original ConSeg formulation assumes stable co-occurrence statistics in the dataset. Highly unstructured data may reduce effectiveness. The paper also identifies possible defense directions: comparing segmentation outputs of the same class in varying contexts for inconsistency, and jointly inspecting images and masks for subtle annotation anomalies caused by few-pixel perturbations (Abbasi et al., 26 Jul 2025).

The stated future directions are task-specific defenses for segmentation backdoors, positive use of context-injection to improve robustness or interpretability, and alternative ways to model and exploit context beyond pixel replacement (Abbasi et al., 26 Jul 2025). These proposals reflect the fact that ConSeg’s manipulation is partly semantic and partly annotation-level; it does not depend solely on conspicuous image-space artifacts.

A related but distinct use of the ConSeg label appears in a later case study on mushroom detection with YOLOv7. There, the attack is described as a dynamic mask-based backdoor that uses SAM to create per-sample masks for dynamic trigger placement, changes poisoned labels from a source class to a target class, and reports high clean accuracy together with high ASR on poisoned samples (Dridi et al., 26 Jan 2026). Because that setting concerns object detection rather than semantic segmentation, it should not be conflated with the original ConSeg formulation. A plausible implication is that object-aligned, context-sensitive trigger placement is part of a broader family of backdoor strategies extending beyond semantic segmentation (Dridi et al., 26 Jan 2026).