Directional Orthogonal Counterattack (DOC)

• DOC is a test-time defense mechanism that augments PGD by integrating orthogonal gradient exploration and momentum to better navigate the adversarial search space.
• It introduces a Directional Sensitivity Score (DSS) for adaptive gating, dynamically blending optimized and random perturbations to mitigate overfitting on clean inputs.
• Empirical results show DOC achieves robust accuracy gains (up to 31% on PGD-10) while maintaining competitive clean performance across multiple datasets.

Directional Orthogonal Counterattack (DOC) is a test-time defense mechanism designed to enhance the adversarial robustness of vision-language pre-training models, particularly CLIP, by expanding the diversity and coverage of counterattacks. Unlike standard Test-Time Counterattack (TTC) methods that rely solely on Projected Gradient Descent (PGD) steps, DOC introduces orthogonal exploration and momentum-based updates to address the limited search space and overfitting issues associated with traditional counterattack strategies. The method incorporates a novel directional sensitivity score for adaptive modulation of counterattack strength, ultimately improving the model's ability to neutralize a broader array of adversarial perturbations while maintaining competitive performance on clean data.

1. Optimization Objective and Update Scheme

DOC operates in the context of a possibly adversarial input $x$, using the CLIP image encoder $I_\theta(\cdot)$ and a perturbation $\delta_{\mathrm{ca}}$ constrained within an $\ell_p$ budget $\epsilon_{\mathrm{ca}}$. The objective extends the TTC maximization formulation:

$$\delta_{\mathrm{ca}}^* = \arg\max_{\|\delta_{\mathrm{ca}}\|_p \le \epsilon_{\mathrm{ca}}} \left\| I_\theta(x_{\mathrm{adv}} + \delta_{\mathrm{ca}}) - I_\theta(x_{\mathrm{adv}}) \right\|_2$$

DOC modifies the classic PGD iterative update by:

• Augmenting the standard loss gradient with an orthogonal random direction and a momentum buffer.
• Introducing a Directional Sensitivity Score (DSS) for adaptive modulation.

The iterative steps at each round $t$ are as follows:

1. Surrogate loss:

$$\mathcal{L}(x_{\mathrm{adv}}, \delta_{\mathrm{ca}}) = \|I_\theta(x_{\mathrm{adv}} + \delta_{\mathrm{ca}}) - I_\theta(x_{\mathrm{adv}})\|_2$$

2. Normalized gradient:

$$g_t = \frac{\nabla_{x_{\mathrm{adv}}} \mathcal{L}(x_{\mathrm{adv}}, \delta^t_{\mathrm{ca}})}{\|\nabla_{x_{\mathrm{adv}}} \mathcal{L}(x_{\mathrm{adv}}, \delta^t_{\mathrm{ca}})\|_2}$$

1. Orthogonal random augmentation:
   • Sample $r \sim \mathcal{N}(0, I)$,
   • Project to orthogonal subspace:

   $I_\theta(\cdot)$0

2. Composite direction:

$I_\theta(\cdot)$1

1. Momentum buffer update:

$I_\theta(\cdot)$2

2. PGD step and projection:

$I_\theta(\cdot)$3

Where $I_\theta(\cdot)$4 is the orthogonal strength, $I_\theta(\cdot)$5 is the momentum factor, $I_\theta(\cdot)$6 is the step size, and $I_\theta(\cdot)$7 denotes projection onto the $I_\theta(\cdot)$8-norm ball.

2. Complete Algorithmic Workflow

The DOC method is structured as follows, combining orthogonal augmentation with a sensitivity-based gating mechanism:

$t$3

The final counterattack perturbation is a convex combination of the fully optimized and the initial PGD random noise, weighted by the DSS-derived gate $I_\theta(\cdot)$9.

3. Orthogonal Augmentation and Momentum: Rationale and Mathematical Construction

Standard PGD counterattacks operate along $\delta_{\mathrm{ca}}$0, which, for the TTC objective, may only explore a restricted subset of the $\delta_{\mathrm{ca}}$1-ball. The key innovation in DOC is the addition of noise lying in the subspace orthogonal to the loss gradient:

• Any vector $\delta_{\mathrm{ca}}$2 can be decomposed as $\delta_{\mathrm{ca}}$3.
• The DOC update discards the parallel component, forming $\delta_{\mathrm{ca}}$4, and injects it with strength $\delta_{\mathrm{ca}}$5.
• This composite step, especially after normalization, enables the optimizer to escape narrow basins and locate more generalizable counterattacks.

Momentum $\delta_{\mathrm{ca}}$6 acts as a low-pass filter, amplifying directions that consistently improve the objective while filtering high-frequency steps, yielding more stable and effective traversal across the counterattack search space.

Empirical ablation demonstrates that orthogonal gradient augmentation (OGA) constitutes the primary driver for increased adversarial robustness, as evidenced by robust accuracy ≥31% (vs. ~21% for TTC) with only small reductions in clean accuracy.

4. Directional Sensitivity Score (DSS) and Adaptive Modulation

DSS is designed to quantify an input's embedding sensitivity to isotropic random perturbations, independent of embedding scale. For each input $\delta_{\mathrm{ca}}$7, $\delta_{\mathrm{ca}}$8 random noise samples $\delta_{\mathrm{ca}}$9 are generated, and the average cosine similarity between the perturbed and clean embeddings is computed:

$\ell_p$0

Clean images yield $\ell_p$1 near zero (cosine similarity close to 1), while adversarial images show increased embedding drift ($\ell_p$2 larger). Rather than employing hard thresholding, a soft gate $\ell_p$3 is computed:

$\ell_p$4

where $\ell_p$5 is a sharpness hyperparameter and $\ell_p$6 a preset threshold. The final output is a convex blend: $\ell_p$7. This adaptivity prevents excessive perturbation of already-stable (clean) inputs and ensures robust correction on sensitive (likely adversarial) samples.

5. Empirical Evaluation and Results

DOC was evaluated on 16 diverse datasets, including CIFAR-10/100, STL-10, ImageNet (zero-shot), Caltech-101/256, Oxford Pets, Flowers102, Food101, Stanford Cars, SUN397, Country211, FGVC-Aircraft, EuroSAT, DTD, and PCAM.

Three adversarial threat models were applied:

• PGD-10 ($\ell_p$8, $\ell_p$9)
• C&W ($\epsilon_{\mathrm{ca}}$0, $\epsilon_{\mathrm{ca}}$1)
• AutoAttack ensemble ($\epsilon_{\mathrm{ca}}$2, $\epsilon_{\mathrm{ca}}$3)

Key robust and clean accuracy results are summarized below:

Setting	Robust Accuracy (%)	Clean Accuracy (%)
CLIP Baseline	0.06	61.5
TTC (prior state-of-the-art)	21.22 (PGD-10)	55.6
DOC (full method)	31.02 (PGD-10)	58.3
DOC (C&W)	28.18	—
DOC (AutoAttack)	≈ 21.0	—

• DOC yields a +9.8% robust accuracy gain (PGD-10) over TTC, and +30.96% over baseline CLIP.
• Ablation reveals OGA alone provides ≈31.8% robust, ≈55.4% clean; DSS alone achieves ≈23.4% robust, ≈58.2% clean; full combination balances both metrics.
• Integration with adversarial fine-tuning (e.g., TeCoA, PMG-AFT, FARE) delivers a further 4–5% robust improvement.

The number of update steps $\epsilon_{\mathrm{ca}}$4 saturates at 3–4, with default hyperparameter settings ($\epsilon_{\mathrm{ca}}$5, $\epsilon_{\mathrm{ca}}$6, $\epsilon_{\mathrm{ca}}$7, $\epsilon_{\mathrm{ca}}$8, $\epsilon_{\mathrm{ca}}$9, $$\delta_{\mathrm{ca}}^* = \arg\max_{\|\delta_{\mathrm{ca}}\|_p \le \epsilon_{\mathrm{ca}}} \left\| I_\theta(x_{\mathrm{adv}} + \delta_{\mathrm{ca}}) - I_\theta(x_{\mathrm{adv}}) \right\|_2$$0, $$\delta_{\mathrm{ca}}^* = \arg\max_{\|\delta_{\mathrm{ca}}\|_p \le \epsilon_{\mathrm{ca}}} \left\| I_\theta(x_{\mathrm{adv}} + \delta_{\mathrm{ca}}) - I_\theta(x_{\mathrm{adv}}) \right\|_2$$1). All experiments use batch size 256 and a single NVIDIA 4090 GPU.

6. Practical Deployment Recommendations and Limitations

DOC requires no additional training but does induce test-time computational overhead proportional to $$\delta_{\mathrm{ca}}^* = \arg\max_{\|\delta_{\mathrm{ca}}\|_p \le \epsilon_{\mathrm{ca}}} \left\| I_\theta(x_{\mathrm{adv}} + \delta_{\mathrm{ca}}) - I_\theta(x_{\mathrm{adv}}) \right\|_2$$2 and $$\delta_{\mathrm{ca}}^* = \arg\max_{\|\delta_{\mathrm{ca}}\|_p \le \epsilon_{\mathrm{ca}}} \left\| I_\theta(x_{\mathrm{adv}} + \delta_{\mathrm{ca}}) - I_\theta(x_{\mathrm{adv}}) \right\|_2$$3.

• Step count $$\delta_{\mathrm{ca}}^* = \arg\max_{\|\delta_{\mathrm{ca}}\|_p \le \epsilon_{\mathrm{ca}}} \left\| I_\theta(x_{\mathrm{adv}} + \delta_{\mathrm{ca}}) - I_\theta(x_{\mathrm{adv}}) \right\|_2$$4–$$\delta_{\mathrm{ca}}^* = \arg\max_{\|\delta_{\mathrm{ca}}\|_p \le \epsilon_{\mathrm{ca}}} \left\| I_\theta(x_{\mathrm{adv}} + \delta_{\mathrm{ca}}) - I_\theta(x_{\mathrm{adv}}) \right\|_2$$5 is sufficient; further iterations provide negligible benefits.
• Orthogonal strength $$\delta_{\mathrm{ca}}^* = \arg\max_{\|\delta_{\mathrm{ca}}\|_p \le \epsilon_{\mathrm{ca}}} \left\| I_\theta(x_{\mathrm{adv}} + \delta_{\mathrm{ca}}) - I_\theta(x_{\mathrm{adv}}) \right\|_2$$6 and momentum $$\delta_{\mathrm{ca}}^* = \arg\max_{\|\delta_{\mathrm{ca}}\|_p \le \epsilon_{\mathrm{ca}}} \left\| I_\theta(x_{\mathrm{adv}} + \delta_{\mathrm{ca}}) - I_\theta(x_{\mathrm{adv}}) \right\|_2$$7 are robust intervals.
• Sensitivity sampling $$\delta_{\mathrm{ca}}^* = \arg\max_{\|\delta_{\mathrm{ca}}\|_p \le \epsilon_{\mathrm{ca}}} \left\| I_\theta(x_{\mathrm{adv}} + \delta_{\mathrm{ca}}) - I_\theta(x_{\mathrm{adv}}) \right\|_2$$8 offers a stability-cost compromise; $$\delta_{\mathrm{ca}}^* = \arg\max_{\|\delta_{\mathrm{ca}}\|_p \le \epsilon_{\mathrm{ca}}} \left\| I_\theta(x_{\mathrm{adv}} + \delta_{\mathrm{ca}}) - I_\theta(x_{\mathrm{adv}}) \right\|_2$$9 is viable.
• Soft gating with $t$0–$t$1 smooths abrupt transitions.
• No formal guarantee of global optimality is provided; efficacy depends on the geometry of the CLIP embedding space.
• The method is directly applicable only to models with differentiable embedding functions and tractable $t$2 projections.
• Potential extensions include adaptive orthogonal scaling, refined subspace sampling, and adaptation to alternative visio-linguistic backbones.

DOC advances test-time counterattack methodology by systematically incorporating orthogonality, momentum, and input-sensitive gating, leading to substantial gains in adversarial robustness with minimal sacrifice in clean evaluation accuracy.