Cross-SGA: Semantic Guided Adapter

• Cross-SGA is a dedicated adapter that injects dual-branch cross-attention into a pre-trained UNet, enabling controlled anomaly synthesis.
• It fuses standard language-driven guidance with anomaly-specific semantic embeddings via a gated mechanism to enhance synthetic realism.
• Empirical results demonstrate that incorporating Cross-SGA with CSM and ASEA significantly boosts performance metrics such as IS and AUROC.

Cross-SGA (Semantic Guided Adapter) is a dedicated adapter architecture designed for anomaly synthesis within the AnomalyControl framework. It enables fine-grained controllable generation of realistic abnormalities by integrating cross-modal semantic features that arise from both textual and visual prompts. SGA extends each cross-attention block in a pre-trained UNet (notably Stable Diffusion) by introducing a dual-branch attention mechanism, allowing simultaneous and gated fusion of standard language-driven guidance and anomaly-specific cross-modal information coordinated upstream by a Cross-modal Semantic Modeling (CSM) module and the Anomaly-Semantic Enhanced Attention (ASEA) mechanism (He et al., 2024).

1. Architectural Overview of SGA

SGA is implemented as a lightweight adapter module injected into all 16 cross-attention blocks of a Stable Diffusion UNet. At each block, the input to SGA consists of the standard text prompt embedding $C$ and a cross-modal semantic embedding $C'$ computed by CSM. Queries $Z \in \mathbb{R}^{B \times N \times d}$ are projected as usual, with $N\approx 1024$ (flattened spatial locations), and $d=768$ (token dim). SGA introduces a second set of key/value projections ($K_{\mathrm{sem}}, V_{\mathrm{sem}}$) from $C'$, distinct from the text branch's $K_{\mathrm{text}}, V_{\mathrm{text}}$. This enables dual cross-attention in each block: one over $C$ and another over $C'$. The outputs are fused by a learnable gate parameter $C'$0, fixed to $C'$1 during inference.

2. Attention Mechanisms and Core Equations

The dual-branch cross-attention operates as follows. For both textual and cross-modal (semantic) branches, attention maps are computed:

$C'$2

Corresponding outputs:

$C'$3

These are fused in a gated manner:

$C'$4

where $C'$5 is a scalar, set to $C'$6. In matrix form, with SGA introducing trainable parameters only in the new branch:

$C'$7

3. Interaction with Cross-Modal Feature Extraction and ASEA

Upstream, the CSM module extracts $C'$8 by leveraging ASEA (Anomaly-Semantic Enhanced Attention). ASEA steers the attention of a large pre-trained vision-LLM (VLM) toward ground-truth anomaly regions, optimizing a guidance vector $C'$9 so that the attention map $Z \in \mathbb{R}^{B \times N \times d}$0 aligns with the binary anomaly mask $Z \in \mathbb{R}^{B \times N \times d}$1. The alignment penalty is

$Z \in \mathbb{R}^{B \times N \times d}$2

and $Z \in \mathbb{R}^{B \times N \times d}$3 is refined for $Z \in \mathbb{R}^{B \times N \times d}$4 steps using learning rate $Z \in \mathbb{R}^{B \times N \times d}$5. After optimization, the cross-modal embedding is computed:

$Z \in \mathbb{R}^{B \times N \times d}$6

serving as key/value source for SGA's semantic branch.

4. Optimization Objectives and Training Workflow

SGA's and ASEA's training is compartmentalized as follows.

• ASEA loss for $Z \in \mathbb{R}^{B \times N \times d}$7:

$Z \in \mathbb{R}^{B \times N \times d}$8

• Diffusion reconstruction loss for SGA parameters $Z \in \mathbb{R}^{B \times N \times d}$9:

$N\approx 1024$0

At each iteration: (1) ASEA updates $N\approx 1024$1 for $N\approx 1024$2 steps with respect to $N\approx 1024$3, (2) $N\approx 1024$4 is frozen, a noisy latent $N\approx 1024$5 is sampled, $N\approx 1024$6 is computed, then SGA weights are updated via $N\approx 1024$7.

5. Hyperparameters and Data-Flow Specifications

• Stable Diffusion latent space: $N\approx 1024$8, $N\approx 1024$9, $d=768$0.
• VLM outputs $d=768$1 cross-modal tokens: $d=768$2, token dimension $d=768$3.
• Key/Value projections in SGA branch: $d=768$4.
• Training: SGA learning rate $d=768$5, AdamW optimizer, weight decay $d=768$6.
• ASEA: $d=768$7 guidance steps, guidance learning rate $d=768$8.
• Classifier-free guidance: each branch (text/cross-modal) independently dropped with $d=768$9.
• Gating parameter $K_{\mathrm{sem}}, V_{\mathrm{sem}}$0 fixed at $K_{\mathrm{sem}}, V_{\mathrm{sem}}$1 during both training and inference.

6. Empirical Impact of SGA and Ablation Findings

Performance metrics as measured on MVTec-AD, with relevant IS (Inception Score), IC-LPIPS, and AUROC (AUC-P, AUC-I):

Configuration	IS	IC-LPIPS	AUC-P	AUC-I
SD Backbone (no SGA)	1.50	0.18	90.3%	93.0%
+SGA (visual adapter only)	1.66	0.23	92.7%	95.4%
+CSM (cross-modal) + SGA	1.69	0.25	93.1%	95.8%
Full (CSM+ASEA+SGA), $K_{\mathrm{sem}}, V_{\mathrm{sem}}$2	1.84	0.35	99.5%	99.3%

These results indicate SGA alone contributes a 0.16 increase in IS and a 2.4 percentage point increase in AUC-I. When combined with cross-modal feature extraction and attention guidance, the complete AnomalyControl stack achieves substantial gains in the realism and controllability of synthesized anomalies, with near-perfect anomaly/pixel-level detection AUROC scores (He et al., 2024). This suggests SGA is instrumental in closing the realism gap for synthetic abnormalities, especially when paired with attention-optimized cross-modal priors.

7. Significance and Outlook

Cross-SGA constitutes a dedicated and modular solution for infusing semantic and anomaly-specific guidance into conditional diffusion models. Its gated dual-attention and reliance on externally steered cross-modal embeddings distinguish it from conventional text-driven adapters, conferring enhanced ability to encode fine-grained abnormal traits for downstream inspection and detection tasks. Subsequent research may extend the Cross-SGA mechanism to other generative pipelines or further refine the interaction between semantic adapters and upstream attention guidance processes.