---
title: 'PSM: Purifier & Smoothness Mapper for Robust MLLMs'
url: https://www.emergentmind.com/topics/purifier-and-smoothness-mapper-psm
type: topic
---

# PSM: Purifier & Smoothness Mapper for Robust MLLMs

The Purifier and Smoothness Mapper (PSM) is a plug-and-play architectural wrapper designed for multimodal large language models (MLLMs) to provably enhance their robustness against $\ell_2$-bounded adversarial attacks. Operating in the context of Feature-space Smoothing (FS), PSM achieves robustness by maximizing the Gaussian robustness score $\hat S$, thereby tightening theoretical lower bounds on the preservation of feature-space information between clean and adversarial examples. Critically, PSM improves certified robustness without requiring retraining or modification of the core MLLM or its encoders [2601.16200].

## 1. Feature-space Smoothing and Certified Robustness

Feature-space Smoothing (FS) provides a theoretical framework for certifying how much an input perturbation can distort the encoded feature vectors in an MLLM. Given a normalized feature encoder $f_e:\mathbb R^d\to\mathbb R^D$ with $\|f_e(x)\|_2=1$, FS defines a smoothed encoder:
\[
\hat f_e(x) = \mathbb E_{\varepsilon\sim\mathcal N(0,I)}[f_e(x+\varepsilon)]
\]
This Gaussian smoothing at the feature level, rather than the prediction level, enables the derivation of a **Feature Cosine Similarity Bound (FCSB)**. For input perturbations $\delta$ with $\|\delta\|_2 \leq \epsilon$, the following holds:
\[
\cos(\hat f_e(x+\delta), f_e(x)) \geq 2\Phi(\Phi^{-1}(\hat S(x)) - \epsilon) - 1
\]
where $\Phi$ is the standard normal CDF, and $\hat S(x)$ is the **Gaussian robustness score**:
\[
\hat S(x) = \mathbb E_{\varepsilon} \left[\frac{1 + \cos(f_e(x+\varepsilon), f_e(x))}{2}\right]
\]
A higher $\hat S(x)$ yields a tighter lower bound on feature-space similarity, certifying robustness.

## 2. Architectural Design of the Purifier and Smoothness Mapper

The PSM module is constructed from two neural components operated around a frozen $f_e$ encoder:

- **Purifier ($\mathcal P$):** A lightweight, one-step diffusion network (e.g., guided-diffusion U-Net at $256\times256$) trained to denoise inputs contaminated by Gaussian noise with standard deviation $\sigma$.
- **Smoothness Mapper ($\mathcal M$):** A residual network acting on feature tensors $(L, D)$ (with $L$ tokens and $D$ channels, $k=3$ blocks). Each block implements:
  - Noise-aware LayerNorm and FiLM(\(\sigma\)); FiLM conditions on per-channel scale and shift.
  - Parallel processing via (1) multi-head self-attention (block 1 only), (2) depth-wise convolution, and (3) channel-wise MLP.
  - Noise-adaptive residuals, modulated by learnable functions of $\sigma$.

The architecture can be depicted as:

| Input           | Module            | Output           |
|-----------------|-------------------|------------------|
| $x$             | Purifier ($\mathcal P$) | $x_\mathrm{pur}$   |
| $x_\mathrm{pur}$ | Encoder ($f_e$)        | $z$               |
| $z$             | Mapper ($\mathcal M$)   | $z_\mathrm{sm}$    |

Inference involves Monte Carlo smoothing: adding Gaussian noise, passing through $\mathcal P$, encoding by $f_e$, mapping by $\mathcal M$, and averaging over samples.

## 3. Training Objectives and Loss Functions

Both Purifier and Mapper are optimized for enhanced feature robustness and statistical consistency:

- **Purifier losses:**
  - **Reconstruction (MSE):**
    \[
    \ell_{\rm mse} = \mathbb E_{x,\varepsilon} \| x - \mathcal P(x+\varepsilon) \|_2^2
    \]
  - **Purifier robustness:**
    \[
    \ell^{\mathcal P}_{\rm rb} = \mathbb E_{x,\varepsilon} [\cos(f_e(\mathcal P(x+\varepsilon)), f_e(x))]
    \]
  - **Total:**
    \[
    \mathcal L_{\mathcal P} = \ell_{\rm diff} + \lambda_1 \ell^{\mathcal P}_{\rm rb} + \lambda_2 \ell_{\rm mse}
    \]
- **Mapper losses:**
  - **Mapper robustness:**
    \[
    \ell^{\mathcal M}_{\rm rb} = \mathbb E_{x,\varepsilon} [\cos(z_\mathrm{sm}, f_e(x))]
    \]
    where $z_\mathrm{sm} = f_e(\mathcal P(x+\varepsilon)) + \mathcal M(f_e(\mathcal P(x+\varepsilon)), \sigma)$.
  - **Statistical matching:**
    \[
    \ell_{\rm stats} = \mathbb E \sum_{d=1}^D [(\mu_{z_{\rm sm}^{(d)}}-\mu_z^{(d)})^2 + (\sigma_{z_{\rm sm}^{(d)}}-\sigma_z^{(d)})^2]
    \]
  - **Identity (for $\sigma=0$):**
    \[
    \ell_{\rm id} = \mathbb E\| \mathcal M(z,0) \|_2^2
    \]
  - **Total:**
    \[
    \mathcal L_{\mathcal M} = \ell^{\mathcal M}_{\rm rb} + \lambda_3 \ell_{\rm stats} + \lambda_4 \ell_{\rm id}
    \]
Typical hyperparameters: $\sigma=0.25$, $n_\text{smooth}=4$ for inference, $k=3$ mapper blocks, and loss weights $\lambda_1 = \lambda_2 = \lambda_4 = 0.25$, $\lambda_3 = 100$.

## 4. Implementation and Deployment

The PSM is inserted as a wrapper without modifying the main MLLM encoder. At inference:

```python
def FS_PSM_smooth(x, f_e, P, M, n_smooth=4, sigma=0.25):
    z_accum = 0
    for i in range(n_smooth):
        eps = sample_normal(0, sigma, x.shape)
        x_noisy = x + eps
        x_pur = P(x_noisy)
        z = f_e(x_pur)
        z_mapped = z + M(z, sigma)
        z_accum += z_mapped
    return z_accum / n_smooth
```
No retraining or modification of the MLLM weights is required during deployment. The result is a smoothed feature $z_\text{smooth}$ with an empirically and theoretically higher $\hat S$ and thus a tighter FCSB.

## 5. Empirical Evaluation and Performance

Extensive experiments were conducted on LLaVA-1.5-7B, OpenFlamingo-9B, and CLIP-L14, using image captioning, image classification, and VQA tasks. White-box attacks included AttackVLM, M-Attack, and FOA, with $\|\epsilon\|_\infty=16/255$ (and $32/255$ for stress tests).

Key metrics:

- **FCS:** Cosine similarity between clean and adversarial feature vectors
- **ACC:** Task accuracy
- **ASR:** Attack success rate

Sample results (FOA attack):

| Setting                  | FCS   | ACC    | ASR    |
|--------------------------|-------|--------|--------|
| Org. (LLaVA-7B, caption) | 0.388 | 1%     | 94%    |
| FS only                  | 0.652 | 87%    | 1%     |
| FS+PSM (vanilla)         | --    | 87%    | 1%     |
| FS+PSM (FARE encoder)    | 0.789 | 39%    | 20%    |

Ablation studies (CLIP-B16, classification, FOA) further show that both Purifier and Mapper are essential for maximal gains, increasing certified radius $\mathcal R$ and accuracy while reducing ASR.

## 6. Comparative Analysis and Significance

FS alone significantly reduces ASR (from ≈90% to 1–2%) across all tested MLLMs and tasks. When combined with PSM, further increases in FCS and ACC are observed. Notably, the plug-and-play approach allows stacking on top of encoders already adversarially trained (e.g., FARE or TeCoA), yielding consistent robustness improvements without retraining those encoders. All modifications are strictly external to the MLLM: neither the core encoder nor LLM weights change.

A plausible implication is that PSM establishes a practical route for certifiable robustness in large models, where architectural or full-model retraining is infeasible. By maximizing the Gaussian robustness score, PSM offers both theoretical guarantees and empirical improvements, confirming the effectiveness of feature-space smoothing as a paradigm for adversarial defense [2601.16200].

Source: https://www.emergentmind.com/topics/purifier-and-smoothness-mapper-psm