ImgTrojan: Image-Based Backdoor Attacks

• ImgTrojan is an image-based Trojan mechanism where models perform normally on clean inputs but produce attacker-controlled outputs when a specific visual trigger is present.
• It encompasses diverse trigger constructions including localized patches, global style filters, imperceptible noise, and semantic host-free activations.
• Detection methods leverage model-level scanning, per-input screening, and feature-space forensic analysis, while mitigation strategies aim to restore benign behavior with minimal accuracy loss.

Searching arXiv for the cited ImgTrojan-related papers to ground the article in current references. arxiv_search({"query":"all:ImgTrojan OR ti:ImgTrojan OR id:(Tao et al., 2024) OR id:(Zhang et al., 2022) OR id:(Wang et al., 2022) OR id:(Rajabi et al., 2023) OR id:(Wang et al., 2022) OR id:(Harikumar et al., 2022) OR id:(Cheng et al., 2020) OR id:(Harikumar et al., 2021) OR id:(Hussein et al., 2023) OR id:(Liu et al., 7 Jan 2025) OR id:(Zheng et al., 2023) OR id:(Guo et al., 2024) OR id:(Chen et al., 8 Apr 2025) OR id:(Zheng et al., 2022) OR id:(Bai et al., 2022)", "max_results": 10}) ImgTrojan denotes image-based Trojan or backdoor behavior in learned systems: a model behaves normally on clean inputs but, when a visual trigger condition is present, it produces attacker-controlled behavior. In the narrow sense used by “ImgTrojan: Jailbreaking Vision-LLMs with ONE Image,” the term refers to a poisoned-image trigger that causes a vision-LLM to comply with harmful instructions (Tao et al., 2024). In broader usage across the image-security literature, the same label is a convenient umbrella for image-domain backdoors implemented through localized patches, global style or filter transformations, semantic full-image triggers, imperceptible quantization or warping patterns, feature-space manipulations, and multimodal image-conditioned exploits (Zhang et al., 2022, Wang et al., 2022, Harikumar et al., 2021, Wang et al., 2022, Chen et al., 8 Apr 2025).

1. Conceptual scope and threat model

ImgTrojan mechanisms are typically characterized by two invariants. First, the compromised model preserves high utility on clean inputs. Second, a specific image-side condition causes a targeted behavioral shift, usually toward an attacker-chosen class, target image, target style, or unsafe response policy (Zhang et al., 2022, Wang et al., 2022, Guo et al., 2024, Tao et al., 2024). In classifier settings, this condition is often expressed as a mapping from triggered samples to a single target class while clean accuracy remains high; in generative or multimodal settings, the target can instead be a preset image, a preset style, an object substitution, or a jailbreak policy (Rajabi et al., 2023, Guo et al., 2024, Tao et al., 2024).

The threat models vary substantially. Some works assume training-time poisoning with full control of data and optimization, as in BppAttack and deep feature-space Trojans (Wang et al., 2022, Cheng et al., 2020). Others assume post-deployment parameter manipulation via bit flips, as in HPT and TrojViT, where the Trojan resides in model parameters and is activated by a visual trigger at inference time (Bai et al., 2022, Zheng et al., 2022). A separate line addresses black-box or test-time detection, where the defender can only query a deployed model or inspect a single input without access to the original training data (Zhang et al., 2022, Rajabi et al., 2023, Liu et al., 7 Jan 2025). In self-supervised learning, the attack surface is the encoder itself, so downstream classifiers trained later can inherit the Trojan behavior even if they never see triggered samples during training (Liu et al., 7 Jan 2025, Zheng et al., 2023).

A recurring distinction is between input-space and feature-space triggers. Input-space triggers are static, localized patterns or overlays placed directly in pixel space. Feature-space triggers include global style transformations, warps, invisible signals, or transformations whose salient effect is expressed in representation space rather than as a fixed patch (Wang et al., 2022). This distinction matters because several earlier reverse-engineering defenses assumed that the trigger was a small, localized pattern and therefore degraded sharply on global or semantically distributed triggers (Wang et al., 2022, Harikumar et al., 2021).

2. Trigger modalities and attack constructions

The most classical ImgTrojan instantiation is the localized patch trigger. “An Adaptive Black-box Backdoor Detection Method for Deep Neural Networks” studies polygon or patch triggers characterized by location, shape or mask, color, size, and rotation, and emphasizes their spatial dependency in convolutional networks (Zhang et al., 2022). “Defense Against Multi-target Trojan Attacks” extends this family to multi-target, location-agnostic BadNet-style attacks in which one fixed trigger is associated with each target class and can be placed anywhere inside the image (Harikumar et al., 2022). TrojViT adapts this logic to Vision Transformers by replacing area-wise triggers with patch-wise triggers aligned to salient tokens and attention pathways (Zheng et al., 2022).

A second family uses global or feature-space transformations. The black-box detector of Zhang et al. explicitly targets Instagram-style filter triggers such as Gotham, Nashville, Kelvin, Lomo, and Toaster (Zhang et al., 2022). FeatureRE is motivated by the observation that both input-space and feature-space Trojans are associated with feature-space hyperplanes, and evaluates global style filters, WaNet warping, input-aware triggers, invisible sinusoidal signals, and sample-specific invisible triggers under a unified reverse-engineering framework (Wang et al., 2022). DFST advances this direction by implementing the trigger through a learned CycleGAN-based style transfer and then using controlled detoxification so that the trojaned model relies on distributed deep features rather than shallow artifacts (Cheng et al., 2020).

A third family pursues imperceptibility. BppAttack uses image quantization and dithering as the trigger, with a deterministic quantization map

$$Q(x; d) = \left( \frac{ \operatorname{round}\!\left( \frac{x}{2^m - 1} \cdot (2^d - 1) \right)}{2^d - 1 } \right) \cdot (2^m - 1),$$

followed by Floyd–Steinberg dithering (Wang et al., 2022). HPT generates hardly perceptible Trojan images through a combination of additive noise and a per-pixel flow field, jointly optimized with a small number of weight-bit flips in a quantized network (Bai et al., 2022). In semi-supervised learning, poisoning of unlabeled wild web images can also be made imperceptible and sample-specific through a gradient-matching generator bounded by an $L_\infty$ budget (Feng et al., 2023).

A fourth family shifts the trigger from pixels to semantics. The semantic host-free Trojan attack treats the trigger as a full-sized image from a meaningful object class rather than a host image with a small pattern pasted onto it (Harikumar et al., 2021). In HF-OOD, the Trojan images come from a distribution different from the task data; in HF-OOCD, they come from an out-of-class concept within the same dataset, such as using “automobile” to trigger “truck” (Harikumar et al., 2021). This suggests that ImgTrojan is not restricted to perturbative triggers: an entire concept class can function as the activation key if the target manifold is expanded during poisoning.

A fifth family is natively multimodal. TrojanEdit attacks text-guided image editing models by injecting visual triggers, textual triggers, or both, with the strongest reported multimodal pairing being Word+BadNet (Guo et al., 2024). “Mind the Trojan Horse” shows that IP-Adapter-equipped text-to-image systems can be hijacked by imperceptible adversarial image prompts crafted only against the public image encoder, allowing benign-looking image prompts to steer generation toward NSFW content (Chen et al., 8 Apr 2025). ImgTrojan for VLM jailbreaking goes further by binding a single poisoned image to a textual jailbreak prompt during instruction tuning so that the image becomes a key that unlocks harmful compliance at inference time (Tao et al., 2024).

3. Detection methodologies

Detection strategies for ImgTrojan span model-level scanning, per-input test-time screening, and representation-level forensic analysis. For black-box model inspection, AdaTrojan performs a two-stage search. Stage S1 approximates polygon triggers through a constrained search over a small set of central locations, square masks, discrete sizes, and random RGB colors, using only predicted labels and the maximum logit $O_{\max}$; Stage S2 applies a small candidate list of Instagram-style filters and counts consistent prediction shifts (Zhang et al., 2022). On the public TrojAI dataset, the method reports an overall ROC-AUC of about 0.93 (Zhang et al., 2022).

FeatureRE rethinks reverse engineering by replacing purely input-space constraints with feature-space constraints. It models a Trojan hyperplane in feature space as

$$H = \{ a \in \mathbb{R}^d \mid m \odot a = m \odot t \},$$

and jointly optimizes an input-space trigger function $F$ and a feature mask $m$ under stealth, low-variance, and sparsity constraints (Wang et al., 2022). The practical objective is

$$L(F,m) = \ell( B((1-m)\odot a + m\odot t), y_t ) + w_1 [ \|F(x) - x\| - \tau_1 ]_+ + w_2 [ \operatorname{std}(m \odot A(F(x))) - \tau_2 ]_+ + w_3 [ \|m\| - \tau_3 ]_+,$$

and the reported average detection accuracy across four datasets and seven attacks is about 93% (Wang et al., 2022). Because the method is explicitly designed for both input-space and feature-space Trojans, it is especially relevant for style, warping, and invisible ImgTrojan variants (Wang et al., 2022).

TRIGS adopts a different model-level view. It constructs gradient-based signatures by activation optimization over class logits, with regularization

$$R_{TV}(x) = \sum_{i,j,k} |x(i, j, k) - x(i, j-1, k)| + |x(i, j, k) - x(i-1, j, k)|,$$

and then trains a classifier on the resulting signature maps (Hussein et al., 2023). It reports AUC values of 0.99 on CIFAR10-VGG for the statistics-based variant, 1.00 on Tiny ImageNet-ResNet10, and 0.94 on a new ImageNet ViT dataset for the “Both” signature variant, outperforming ULP and K-Arm in those settings (Hussein et al., 2023).

Test-time input detection uses a different signal. MDTD estimates the minimal adversarial perturbation needed to flip a prediction,

$$d_p(x) = \min_{\delta} \|\delta\|_p \quad \text{s.t.} \quad \arg\max f(x+\delta) \neq \arg\max f(x), \; x+\delta \in [0,1]^d,$$

and then declares an input Trojaned if its clean-distance deviation exceeds an $\alpha \sigma$ band around a Gaussian fit to clean distances (Rajabi et al., 2023). The intuition is that triggered samples lie farther from the decision boundary than clean samples (Rajabi et al., 2023). On image datasets such as CIFAR10, CIFAR100, GTSRB, SVHN, and Flowers102, the method reports strong F1 scores for many trigger types, though certain natural-content overlaps, such as white patches on white flowers, are explicitly identified as challenging (Rajabi et al., 2023).

In self-supervised encoders, TrojanDec and SSL-Cleanse illustrate two distinct encoder-centric approaches. TrojanDec is data-free and black-box at the encoder level: it masks local regions with random patterns, builds the similarity metadata set

$L_\infty$0

and then uses gap-statistic model selection to decide between one-cluster and two-cluster structure (Liu et al., 7 Jan 2025). SSL-Cleanse instead uses a small unlabeled probe set, estimates the number of semantic clusters with SWK, synthesizes candidate triggers per cluster by representation-oriented trigger reverse, and declares the encoder Trojaned if recovered trigger size or norm is a MAD outlier (Zheng et al., 2023).

4. Mitigation and recovery

Mitigation methods generally try to break the trigger-to-target association while preserving clean behavior. FeatureRE uses the recovered feature mask to perform a neuron-flip operation,

$L_\infty$1

where masked feature coordinates are negated (Wang et al., 2022). With only 10 clean samples per class, the method reports that average ASR can be reduced to about 0.26% while benign accuracy remains nearly unchanged (Wang et al., 2022).

TrojanDec couples detection with restoration. After selecting the masked variant with the smallest similarity to the original feature vector, it formulates restoration as a linear inverse problem and uses DDNM to inpaint the suspect region under a diffusion prior (Liu et al., 7 Jan 2025). Across SimCLR and CLIP encoders, the paper reports that ASR can be reduced from near 100% to low single digits on multiple downstream datasets with only small accuracy changes (Liu et al., 7 Jan 2025).

SSL-Cleanse uses self-supervised clustering unlearning. It freezes the suspect encoder as a reference, initializes a new encoder from it, and then aligns clean and randomly trigger-augmented views through cosine similarity so that the new encoder becomes invariant to the discovered trigger patterns (Zheng et al., 2023). On average, the paper reports post-mitigation ASR around 0.3% without great accuracy loss (Zheng et al., 2023).

Other defenses are less reliable across the full ImgTrojan spectrum. Neural Cleanse, TABOR, ABS, and similar reverse-engineering defenses are repeatedly reported as weak on multi-target, feature-space, input-aware, semantic, or nonlocalized triggers (Wang et al., 2022, Harikumar et al., 2022, Harikumar et al., 2021, Wang et al., 2022). STRIP and entropy-based runtime methods can work well for some per-input scenarios but also show failure modes when the trigger is semantic, blended into natural content, or otherwise not disrupted by the mixing procedure (Rajabi et al., 2023, Harikumar et al., 2021). This suggests that mitigation quality is tightly coupled to the trigger family: methods designed around localized patch assumptions do not generalize automatically to global, semantic, or multimodal ImgTrojan mechanisms.

5. Empirical patterns across architectures and tasks

Across classification benchmarks, ImgTrojan attacks are consistently shown to retain clean performance while achieving high ASR. BppAttack reports all-to-one ASR of 99.91% on CIFAR-10, 99.96% on GTSRB, and 99.99% on CelebA, with benign accuracy near the non-attack baseline (Wang et al., 2022). HPT reports ImageNet ASR of 95.2% on ResNet-18 with only 10 flipped bits, while preserving clean accuracy close to the clean model (Bai et al., 2022). TrojViT reports, for example, ASR of 99.96% on DeiT-small for ImageNet with CDA dropping from 79.47% to 79.19%, and it further shows that 345 flipped bits can still achieve 99.64% ASR with only 0.35% CDA degradation (Zheng et al., 2022).

In self-supervised and semi-supervised settings, the empirical picture is more nuanced but still severe. Poisoning unlabeled images from the target class under SSL can yield ASR of 97.12% on CIFAR-10 with FixMatch while maintaining clean accuracy near the clean SSL baseline (Feng et al., 2023). Trojaned SSL encoders can then transfer the backdoor to multiple downstream tasks, which is why encoder-level defenses such as TrojanDec and SSL-Cleanse focus on the backbone rather than on a specific classifier head (Liu et al., 7 Jan 2025, Zheng et al., 2023).

For semantic and feature-space attacks, generalization beyond the exact trigger instance is a central empirical property. The semantic host-free attack reports TMGA of 81.46% on HF-OOD GTSRB, 90.82% on HF-OOD CIFAR-10, 90.72% on HF-OOCD GTSRB, and 82.60% on HF-OOCD CIFAR-10, showing that semantically similar unseen images can activate the backdoor (Harikumar et al., 2021). DFST reports malicious accuracy up to 0.999 on GTSRB and 0.990 on ImageNet, and emphasizes robustness to FGSM, PGD, randomized smoothing, and common image preprocessing (Cheng et al., 2020).

In multimodal and generative systems, the output-space consequences differ from classifier ASR but the same attack logic persists. TrojanEdit reports that Word+BadNet achieves average ASR 97.00% with average EAR 1.17% on image editing models, outperforming unimodal alternatives in the balance between attack effectiveness and clean behavior (Guo et al., 2024). ImgTrojan for VLM jailbreaking shows that with fewer than 100 poisoned samples and poison ratio at or below 1%, ASR can reach 83.5% while BLEU and CIDEr on clean images remain close to the clean model (Tao et al., 2024). “Mind the Trojan Horse” shows that adversarial image prompts in IP-Adapter systems can induce NSFW outputs across SD-v1-5, SDXL, Kolors, and IDM-VTON, and that the effect is amplified by the IP-Adapter weight factor $L_\infty$2 (Chen et al., 8 Apr 2025).

6. Limitations, controversies, and research directions

Several limitations recur across the literature. Many detectors assume access to at least a small clean set, clean samples per class, or a probe distribution similar to the original data (Zhang et al., 2022, Wang et al., 2022, Rajabi et al., 2023, Zheng et al., 2023). White-box reverse engineering remains powerful for some triggers but is not always realistic for deployment-time audits (Wang et al., 2022). Black-box methods scale better operationally, yet they can be sensitive to query budget, calibration quality, and hyperparameters such as thresholds, size steps, or color rounds (Zhang et al., 2022, Rajabi et al., 2023).

Another recurring limitation is trigger coverage. Patch-centric defenses often fail on global style changes, semantic host-free triggers, or multimodal conditions (Wang et al., 2022, Harikumar et al., 2021, Guo et al., 2024). Conversely, some encoder-centric or feature-space defenses are tailored to universal patch-like behavior and may be less informative for highly input-specific or adaptive triggers (Liu et al., 7 Jan 2025, Zheng et al., 2023). Adaptive attacks are already documented: FeatureRE reports that an adaptive attacker can reduce detection accuracy to about 65%, though with degraded ASR and BA compared to native training (Wang et al., 2022).

The relation between stealth and persistence is also unsettled. BppAttack argues that imperceptible quantization and dithering can bypass human inspection and several defenses while adding only about 19.04% training overhead (Wang et al., 2022). HPT shows that hardly perceptible warp-plus-noise triggers can be implanted with a handful of bit flips (Bai et al., 2022). Yet prompt filters, QR detection, output safety filters, or adversarially trained encoders can still reduce attack efficacy in some multimodal systems (Hu et al., 2022, Chen et al., 8 Apr 2025). This suggests that no single notion of stealth—human imperceptibility, embedding-space plausibility, or dataset-filter evasion—fully captures the operational risk of ImgTrojan.

Taken together, the literature indicates that ImgTrojan is best understood not as one attack but as a family of image-conditioned control channels embedded into learned systems. The family now spans CNNs, Vision Transformers, self-supervised encoders, retrieval models, text-guided image editing systems, text-to-image diffusion systems with IP-Adapter, and vision-LLMs (Zheng et al., 2022, Liu et al., 7 Jan 2025, Hu et al., 2022, Guo et al., 2024, Tao et al., 2024). A plausible implication is that future work will increasingly move away from patch-specific assumptions and toward defenses that reason jointly about representation geometry, multimodal conditioning pathways, and deployment-time model integrity.