BadRSSD: Backdoor Attacks on Regularized Self-Supervised Diffusion Models

Abstract: Self-supervised diffusion models learn high-quality visual representations via latent space denoising. However, their representation layer poses a distinct threat: unlike traditional attacks targeting generative outputs, its unconstrained latent semantic space allows for stealthy backdoors, permitting malicious control upon triggering. In this paper, we propose BadRSSD, the first backdoor attack targeting the representation layer of self-supervised diffusion models. Specifically, it hijacks the semantic representations of poisoned samples with triggers in Principal Component Analysis (PCA) space toward those of a target image, then controls the denoising trajectory during diffusion by applying coordinated constraints across latent, pixel, and feature distribution spaces to steer the model toward generating the specified target. Additionally, we integrate representation dispersion regularization into the constraint framework to maintain feature space uniformity, significantly enhancing attack stealth. This approach preserves normal model functionality (high utility) while achieving precise target generation upon trigger activation (high specificity). Experiments on multiple benchmark datasets demonstrate that BadRSSD substantially outperforms existing attacks in both FID and MSE metrics, reliably establishing backdoors across different architectures and configurations, and effectively resisting state-of-the-art backdoor defenses.

• The paper introduces BadRSSD, a representation-layer backdoor that aligns poisoned samples with target images in PCA space while coordinating latent, pixel, and feature-distribution losses.
• BadRSSD achieves more than 86% attack success across evaluated datasets, with benign utility remaining close to clean models and DPM-Solver reaching 87.53% ASR in 20 steps.
• The attack largely defeats DisDet, Elijah, and TERD because its dispersed semantic trigger avoids the fixed pixel patterns and localized neurons targeted by existing defenses.
• What threat model and attacker capabilities does BadRSSD assume, and how realistic are its pretrained-model and shadow-data requirements?
• Why does dispersion regularization improve stealth and generation quality even though removing it slightly increases attack success?
• How might internal-activation monitoring or representation-geometry analysis detect PCA-space backdoors?
• What training-time safeguards could prevent semantic alignment attacks in self-supervised diffusion models?
• Find recent papers about representation-layer backdoor attacks on diffusion models.

BadRSSD (2603.01019) presents a backdoor attack that targets the representation layer of self-supervised diffusion models rather than their generative outputs. The paper makes two coupled contributions: it introduces a Regularized Self-Supervised Diffusion (RSSD) model, which augments latent denoising autoencoders (l-DAE) with representation dispersion regularization, and then demonstrates that this structured semantic space constitutes an exploitable attack surface. The attack, BadRSSD, hijacks poisoned samples' PCA-space representations toward those of a target image and steers the denoising trajectory via coordinated constraints across latent, pixel, and feature-distribution spaces.

Motivation and threat model

The paper's central observation is that diffusion-based self-supervised learners such as DDAs and l-DAE unify generation and representation learning, and that existing backdoor attacks (BadDiffusion, TrojDiff, VillanDiffusion for unconditional models; PaaS, RickRolling, REDEditing, TwT for conditional ones) operate exclusively on generative outputs. Representation-layer attacks differ in two respects: the attack pathway is internal to the semantic representations, so benign-input behavior remains indistinguishable from a clean model; and existing attack machinery cannot be transplanted directly because it does not exploit semantic alignment or regularization mechanisms.

The threat model follows the standard supply-chain scenario: an adversary releases a backdoored RSSD checkpoint on a third-party platform, holding only the pretrained model and an unlabeled shadow dataset of clean and poisoned samples, with no access to downstream data or evaluation pipelines. Success requires both high utility (clean FID comparable to a clean model) and high specificity (triggered inputs reliably generate a predefined target).

The RSSD framework

RSSD builds on l-DAE's partitioned PCA encoding: each image is split into patches, projected onto a locally learned PCA basis retaining $d = 48$ principal components out of $D_{\text{patch}} = 768$, and Gaussian noise is added in this low-dimensional space under linear scheduling $\sigma_t = \sqrt{2}\,t/T$. A DiT conditioned on VAE latents performs denoising, and reconstruction residuals are weighted asymmetrically (1.0 for the first 48 dimensions, 0.1 for the remainder). On top of this, RSSD adds a dispersion loss derived from InfoNCE,

$$L_{\text{disp}} = \log \mathbb{E}_{i,j}\left[\exp\left(-\frac{\|y_{t,i}^{(l)} - y_{t,j}^{(l)}\|_2^2}{\tau}\right)\right],$$

computed on intermediate DiT block features to promote uniformity without explicit contrastive pairings. The unified objective is $$L_{\text{RSSD}} = L_{\text{SSL}} + \lambda_{\text{disp}} L_{\text{disp}}$$ with $\lambda_{\text{disp}} = 0.5$, trained on ImageNet-1K at 256×256 with Adam (lr 1e-4, batch 2048, 400 epochs). The paper claims this achieves alignment-and-uniformity objectives of contrastive learning without complex augmentation; notably, no quantitative comparison against standard SSL baselines on downstream tasks is provided in the main text, so the claim rests on qualitative motivation.

The BadRSSD attack mechanism

Poisoned samples are constructed by injecting a random-noise trigger into the lower-right $n \times n$ region ($x_p = x_0 + M \odot T$). The core step is PCA-space backdoor alignment: after encoding, the poisoned sample's latent is shifted by $\Delta z = Z_0^{T_i} - Z_0^{P_i}$ so that $Z_0^{a_i} = Z_0^{T_i}$, overwriting its semantic identity with the target's. Training uses conditional losses: clean samples follow the standard RSSD objective, while poisoned samples minimize

$D_{\text{patch}} = 768$0

with weights 2.0, 1.5, and 0.5 respectively. $D_{\text{patch}} = 768$1 combines static alignment of initial PCA features with dynamic trajectory consistency across diffusion timesteps ($D_{\text{patch}} = 768$2); $D_{\text{patch}} = 768$3 enforces pixel-level reconstruction of the target through the full VAE–DiT–VAE chain; and reusing the dispersion loss maintains feature uniformity, which the paper identifies as the key stealth mechanism. This design bypasses the multi-step Markov-chain manipulations required by DDPM-space attacks, yielding a concise end-to-end attack chain.

Effectiveness results

Across five pre-training/downstream combinations (CIFAR-10, CIFAR-100, CelebA-HQ, ImageNet), BadRSSD consistently dominates baselines. Representative numbers:

Setting	Metric	Best baseline	BadRSSD
CelebA-HQ → CelebA-HQ	ASR / FID / MSE	86.78 (BadEncoder ASR) / 46.15 / 0.2124	92.63 / 38.26 / 0.1625
CIFAR-100 → CIFAR-10	ASR / FID / MSE	73.86 / 42.86 / 0.1628	91.26 / 36.12 / 0.0821
ImageNet → CIFAR-100	ASR / FID / MSE	80.16 / 55.27 / 0.2236	87.12 / 43.26 / 0.1815

ASR exceeds 86% everywhere while benign accuracy stays within roughly one point of the clean model. Architectural generalization holds across DiT-L/2, DiT-XL/2, U-ViT, and Swin-UNet, with DiT-XL/2 best (FID 35.12, MSE 0.1127 at 400 iterations). Sampler ablations show DDPM achieving peak ASR (92.37%) at 1000 steps but DPM-solver reaching 87.53% ASR in 20 steps (3.8 s), which the authors adopt as default; DDIM offers the lowest variance (ASR-std 0.5) but degraded quality.

Stability analyses indicate the backdoor consolidates quickly: on ImageNet at 50% poisoning, target synthesis stabilizes by epoch 20, versus epoch 40 at 20% poisoning. On high-resolution CelebA-HQ, raising poisoning from 0% to 50% increases FID only from 37.9 to 41.5 (+9.5%) while MSE drops to 0.03 — supporting the dual utility/specificity claim. One caveat: at 5% poisoning on CIFAR-10, backdoor success is only partial, so low-poisoning-rate deployments trade specificity for stealth.

Robustness against defenses

The defense evaluation is the paper's strongest empirical section, showing that three state-of-the-art defenses fail almost completely against BadRSSD while remaining effective against output-level attacks like BadDiffusion:

• DisDet: AUROC drops to 0.58 (near random) versus 0.92–0.95 for baselines; TPR@1%FPR is 8.72%; ASR declines only from 94.67% to 92.57%. The authors attribute this to $D_{\text{patch}} = 768$4 keeping marginal statistics of clean and poisoned samples nearly identical.
• Elijah: detection rates are 5.12–18.08% versus ~82–100% against BadDiffusion; pruning leaves ASR within 3 points of its original value. Because the trigger manifests as a non-local perturbation aligned with target semantics in PCA space, pixel-space inversion fails, and the temporally/spatially dispersed pathway offers no concentrated neuron cluster to prune.
• TERD: trigger inversion error rises to $D_{\text{patch}} = 768$5 (vs. 0.18 for BadDiffusion), TPR falls to 4.28–6.74%, and ASR remains above 91%.

These failures stem from assumption mismatches: all three defenses presuppose fixed, structured, pixel-domain triggers, whereas BadRSSD's effective trigger is a dispersed, semantically aligned perturbation in a learned latent subspace. The implication is that current diffusion-model defenses do not transfer to representation-layer attacks, and the paper explicitly concedes that tailored defenses are absent.

Ablation study

Each loss component is necessary. Removing $D_{\text{patch}} = 768$6 collapses ASR from 94.67% to 62.18% and degrades FID to 61.23; removing $D_{\text{patch}} = 768$7 raises MSE from 0.1209 to 0.2689 and lowers SSIM from 0.823 to 0.658; removing $D_{\text{patch}} = 768$8 slightly increases ASR to 95.16% but worsens FID to 49.52, confirming its role is stealth and generation quality rather than attack potency per se.

Limitations and open questions

Several limitations are evident. First, the utility claims for RSSD itself are not benchmarked against established SSL methods (e.g., linear-probe accuracy comparisons appear only as internal metrics), so the security analysis is anchored to a framework whose representational superiority is asserted rather than demonstrated. Second, the attack assumes access to a pretrained RSSD model and a shadow dataset matching the deployment domain; transferability across domains or to adversaries with weaker assumptions is untested. Third, defense evaluations cover three specific defenses; whether anomaly-detection approaches operating on internal activations, or certified training procedures, would resist PCA-space alignment remains open. Fourth, the trigger is a fixed patch-region noise pattern evaluated at 5%+ poisoning rates; behavior at very low poisoning rates or with adaptive, input-dependent triggers is not characterized. Finally, the paper proposes no mitigation, leaving the development of representation-layer defenses and security standards explicitly as future work.

Conclusion

BadRSSD formalizes and demonstrates a previously unexamined attack surface: the representation layer of self-supervised diffusion models. By combining PCA-space semantic alignment, a triple-loss objective spanning latent, pixel, and distribution spaces, and the very dispersion regularization designed to improve representation quality, the attack achieves ASR above 86% with minimal utility degradation and near-total evasion of DisDet, Elijah, and TERD. The results indicate that defenses predicated on pixel-domain trigger structure do not generalize to latent-semantic backdoors, and that securing unified generative-representational models will require mechanisms that reason about internal representation geometry rather than observable outputs.