---
title: 'Cross-SGA: Semantic Guided Adapter'
url: https://www.emergentmind.com/topics/cross-sga
type: topic
---

# Cross-SGA: Semantic Guided Adapter

Cross-SGA (Semantic Guided Adapter) is a dedicated adapter architecture designed for anomaly synthesis within the AnomalyControl framework. It enables fine-grained controllable generation of realistic abnormalities by integrating cross-modal semantic features that arise from both textual and visual prompts. SGA extends each cross-attention block in a pre-trained UNet (notably Stable Diffusion) by introducing a dual-branch attention mechanism, allowing simultaneous and gated fusion of standard language-driven guidance and anomaly-specific cross-modal information coordinated upstream by a Cross-modal Semantic Modeling (CSM) module and the Anomaly-Semantic Enhanced Attention (ASEA) mechanism [2412.06510].

## 1. Architectural Overview of SGA

SGA is implemented as a lightweight adapter module injected into all 16 cross-attention blocks of a Stable Diffusion UNet. At each block, the input to SGA consists of the standard text prompt embedding $C$ and a cross-modal semantic embedding $C'$ computed by CSM. Queries $Z \in \mathbb{R}^{B \times N \times d}$ are projected as usual, with $N\approx 1024$ (flattened spatial locations), and $d=768$ (token dim). SGA introduces a second set of key/value projections ($K_{\mathrm{sem}}, V_{\mathrm{sem}}$) from $C'$, distinct from the text branch's $K_{\mathrm{text}}, V_{\mathrm{text}}$. This enables dual cross-attention in each block: one over $C$ and another over $C'$. The outputs are fused by a learnable gate parameter $\gamma$, fixed to $1.0$ during inference.

## 2. Attention Mechanisms and Core Equations

The dual-branch cross-attention operates as follows. For both textual and cross-modal (semantic) branches, attention maps are computed:

\[
A_{\mathrm{text}} = \mathrm{Softmax}\!\bigl(\tfrac{Z\,K_{\mathrm{text}}^\top}{\sqrt d}\bigr),\qquad
A_{\mathrm{sem}}  = \mathrm{Softmax}\!\bigl(\tfrac{Z\,K_{\mathrm{sem}}^\top}{\sqrt d}\bigr).
\]

Corresponding outputs:

\[
O_{\mathrm{text}} = A_{\mathrm{text}} V_{\mathrm{text}},\qquad
O_{\mathrm{sem}}  = A_{\mathrm{sem}} V_{\mathrm{sem}}.
\]

These are fused in a gated manner:

\[
Z_{\mathrm{new}} = O_{\mathrm{text}} + \gamma\, O_{\mathrm{sem}}
\]

where $\gamma$ is a scalar, set to $1.0$. In matrix form, with SGA introducing trainable parameters only in the new branch:

\[
Z_{\mathrm{new}} =
\mathrm{Softmax}\!\bigl(\tfrac{\,Z\,(C\,W_k)^\top\,}{\sqrt d}\bigr)\,(C\,W_v)
+ \gamma\,\mathrm{Softmax}\!\bigl(\tfrac{\,Z\,(C'\,W'_k)^\top\,}{\sqrt d}\bigr)\,(C'\,W'_v).
\]

## 3. Interaction with Cross-Modal Feature Extraction and ASEA

Upstream, the CSM module extracts $C'$ by leveraging ASEA (Anomaly-Semantic Enhanced Attention). ASEA steers the attention of a large pre-trained vision-language model (VLM) toward ground-truth anomaly regions, optimizing a guidance vector $e_g$ so that the attention map $\bar A_{\mathrm{anomaly}}$ aligns with the binary anomaly mask $M_A$. The alignment penalty is

\[
E\bigl(\bar A_{\mathrm{anomaly}},\,M_A\bigr)
= \left(1 - 
\frac{\sum_{i\in M_A}\bar A_{\mathrm{anomaly},\,i}}
{\sum_{j}\bar A_{\mathrm{anomaly},\,j}}
\right)^2
\]

and $e_g$ is refined for $T_g=3$ steps using learning rate $\alpha=1e^{-1}$. After optimization, the cross-modal embedding is computed:

\[
C' = \mathrm{VLM}(T_a,\,I_a+e_g) \in \mathbb{R}^{B\times L'\times d},
\]

serving as key/value source for SGA's semantic branch.

## 4. Optimization Objectives and Training Workflow

SGA's and ASEA's training is compartmentalized as follows.

- ASEA loss for $e_g$:

  \[
  L_{\mathrm{ASEA}}(e_g) = E\bigl(\bar A_{\mathrm{anomaly}},\,M_A\bigr)
  \]

- Diffusion reconstruction loss for SGA parameters $\{W'_k, W'_v\}$:

  \[
  L_{\mathrm{diff}}
  = \mathbb{E}_{z_t, t, C, C', \epsilon\sim\mathcal{N}(0,1)}
  \left\| \epsilon - \epsilon_\theta(z_t, t, C, C') \right\|_2^2
  \]

At each iteration: (1) ASEA updates $e_g$ for $T_g$ steps with respect to $L_{\mathrm{ASEA}}$, (2) $e_g$ is frozen, a noisy latent $z_t$ is sampled, $C'$ is computed, then SGA weights are updated via $L_{\mathrm{diff}}$.

## 5. Hyperparameters and Data-Flow Specifications

- Stable Diffusion latent space: $z_t \in \mathbb{R}^{B \times c \times h \times w}$, $h = w = 64$, $c = 4$.
- VLM outputs $L' = 32$ cross-modal tokens: $C' \in \mathbb{R}^{B \times 32 \times 768}$, token dimension $d = 768$.
- Key/Value projections in SGA branch: $W'_k, W'_v \in \mathbb{R}^{768 \times 768}$.
- Training: SGA learning rate $1e^{-4}$, AdamW optimizer, weight decay $1e^{-2}$.
- ASEA: $T_g=3$ guidance steps, guidance learning rate $\alpha=1e^{-1}$.
- Classifier-free guidance: each branch (text/cross-modal) independently dropped with $p=0.05$.
- Gating parameter $\gamma$ fixed at $1.0$ during both training and inference.

## 6. Empirical Impact of SGA and Ablation Findings

Performance metrics as measured on MVTec-AD, with relevant IS (Inception Score), IC-LPIPS, and AUROC (AUC-P, AUC-I):

| Configuration                   | IS   | IC-LPIPS | AUC-P  | AUC-I  |
|----------------------------------|------|----------|--------|--------|
| SD Backbone (no SGA)             | 1.50 | 0.18     | 90.3%  | 93.0%  |
| +SGA (visual adapter only)        | 1.66 | 0.23     | 92.7%  | 95.4%  |
| +CSM (cross-modal) + SGA         | 1.69 | 0.25     | 93.1%  | 95.8%  |
| Full (CSM+ASEA+SGA), $T_g=3$     | 1.84 | 0.35     | 99.5%  | 99.3%  |

These results indicate SGA alone contributes a 0.16 increase in IS and a 2.4 percentage point increase in AUC-I. When combined with cross-modal feature extraction and attention guidance, the complete AnomalyControl stack achieves substantial gains in the realism and controllability of synthesized anomalies, with near-perfect anomaly/pixel-level detection AUROC scores [2412.06510]. This suggests SGA is instrumental in closing the realism gap for synthetic abnormalities, especially when paired with attention-optimized cross-modal priors.

## 7. Significance and Outlook

Cross-SGA constitutes a dedicated and modular solution for infusing semantic and anomaly-specific guidance into conditional diffusion models. Its gated dual-attention and reliance on externally steered cross-modal embeddings distinguish it from conventional text-driven adapters, conferring enhanced ability to encode fine-grained abnormal traits for downstream inspection and detection tasks. Subsequent research may extend the Cross-SGA mechanism to other generative pipelines or further refine the interaction between semantic adapters and upstream attention guidance processes.

Source: https://www.emergentmind.com/topics/cross-sga