---
title: PGD Hyperparameter Insights
url: https://www.emergentmind.com/topics/hyperparameter-insights-for-pgd
type: topic
---

# PGD Hyperparameter Insights

Projected Gradient Descent (PGD) is the canonical method for white-box adversarial testing under norm-bounded perturbations. The attack’s performance, convergence, and stability are tightly governed by the interplay of its hyperparameters: optimizer, step-size schedule, surrogate loss, and advanced constructions like MultiTargeted surrogates. Each hyperparameter modulates attack success and efficiency, and their configuration has led to state-of-the-art adversarial test results against robust models such as those of MadryLab and TRADES [1910.09338].

## 1. Optimizer Choice: SGD‐sign, Momentum, Adam

Hypergradient updates in PGD can be computed using three principal optimizers, each imparting distinct characteristics to the attack trajectory:

- **SGD-sign (FGSMK):** Utilizes discrete sign updates:
  $$
  \delta_{k+1} = \operatorname{Proj}_{S} \left( \delta_{k} + \alpha \cdot \text{sign}\left(\nabla_{x} \hat{L}(f(x+\delta_k), y)\right) \right)
  $$
  This approach is computationally cheap and tuning is straightforward when network activations are unsaturated. However, it displays high sensitivity to the step-size $\alpha$ and can exhibit oscillatory or plateaued trajectories, especially in regions of the loss surface with flat gradients.

- **Momentum:** Incorporates an exponential moving average of past gradients:
  $$
  m_{k} = \beta\,m_{k-1} + (1-\beta)\nabla_{x} \hat{L}(f(x+\delta_k), y)
  $$
  $$
  \delta_{k+1} = \operatorname{Proj}_{S} (\delta_{k} + \alpha \cdot \text{sign}(m_k))
  $$
  This can aid in traversing shallow loss basins and produces smoother gradients but demands additional memory and remains sensitive to $\alpha$ scheduling.

- **Adam:** Employs both first and second moment estimates:
  $$
  m_t = \beta_1 m_{t-1} + (1-\beta_1)g_t;\quad v_t = \beta_2 v_{t-1} + (1-\beta_2)g_t^2
  $$
  $$
  \hat{m}_t = m_t / (1-\beta_1^{t});\quad \hat{v}_t = v_t / (1-\beta_2^{t})
  $$
  $$
  \delta_{k+1} = \operatorname{Proj}_{S}\left(\delta_{k} + \alpha \cdot \frac{\hat{m}_t}{\sqrt{\hat{v}_t}+\epsilon}\right)
  $$
  Adam is empirically the most stable across architectures and step-sizes, requiring minimal manual tuning. The modest computation overhead is amortized by a substantial reduction in restarts and attack iterations.

Summarizing, Adam yields the best out-of-the-box stability and attack strength; momentum is marginally less effective but more robust than sign-based variants, which—while fast—are demonstrably brittle [1910.09338].

| Optimizer   | Pros                                    | Cons                                   |
|-------------|-----------------------------------------|----------------------------------------|
| SGD-sign    | Cheap; simple tuning                    | Highly sensitive to $\alpha$           |
| Momentum    | Smoother trajectory; escapes basins     | Needs memory; $\alpha$-sensitive       |
| Adam        | Stable; less manual tuning              | Slightly higher compute                |

## 2. Step-Size ($\alpha$) Scheduling

Step-size $\alpha$ essentially controls the effective resolution of each gradient step. While a fixed $\alpha \approx \epsilon/10$ (as in canonical FGSMK) can yield superficial success, it quickly saturates. The use of a scheduled decay for $\alpha$ significantly enhances performance, especially on datasets like CIFAR-10.

The recommended decay regime is as follows:
- Initial $\alpha_0 = 0.1$
- Reduce by $10\times$ at $k=K/2$ and again at $k=3K/4$ (where $K$ is the total number of PGD steps)

This doubling of success rate versus fixed $\alpha$ is quantitatively illustrated in Figure 3b of [1910.09338]. In regions where the surrogate loss $\hat{L}$ is $L$-smooth, an informal bound shows that $\alpha \leq 1/L$ ensures ascent, but due to local variability in $L$, decaying $\alpha$ empirically prevents overshoot and loss surface trapping.

A practical rule is to tune with a single restart, $K\approx 200$, and $\alpha_0 \in \{0.01, 0.03, 0.1\}$, with decays at 50% and 75% of $K$.

## 3. Surrogate Loss Selection

The surrogate loss defines what is maximized during attack iterations. Common surrogates and their properties are:

- **Cross-Entropy (CE):**
  $$
  \hat{L}_1(z, y) = -z_y + \log \sum_{i} e^{z_i}
  $$
  Smooth, often easier to optimize in early stages.
- **Margin Loss:**
  $$
  \hat{L}_2(z, y) = \max_{i \neq y} z_i - z_y
  $$
  Maximizes the leading runner-up logit relative to correct label; produces sharper adversarial boundaries.
- **Carlini–Wagner (CW $\kappa$-Loss):**
  $$
  \hat{L}_3(z, y) = \max \left( \max_{i \neq y} z_i - z_y , -\kappa \right)
  $$
  Allows tradeoff between confidence gap (margin) and optimization step effort.

Empirically, margin and CE are comparable on most models. Tuned CW—varying $\kappa$—may, however, surpass both when appropriately calibrated [1910.09338].

## 4. MultiTargeted Surrogate and Algorithm

The MultiTargeted procedure generalizes PGD by explicitly attacking individual target classes via their logit differences:

- **Definition:** For each target class $t \neq y$,
  $$
  \hat{L}_t(z, y) = z_t - z_y
  $$
- **Algorithm Sketch:**
  1. Enumerate $T$ target classes (all or, e.g., top-$T$ by unperturbed logit).
  2. For each $t$, run $K$ Adam-PGD steps maximizing $\hat{L}_t$; retain the perturbation $\delta$ that yields the greatest misclassification.

Theoretical guarantee (Theorem 3.2, [1910.09338]): For any locally linear $f$ on convex set $S(x)$ with $C$ output logits, using $N_r \geq C-1$ restarts (one per $t$), MultiTargeted attains a global maximizer of convex surrogates within $S(x)$. The proof observes that maximizing $z_t - z_y$ is equivalent to exploring each half-space of the logit polytope, so iterating over all $C-1$ alternatives covers the solution space.

On practical datasets, across four WideResNet models on CIFAR-10, MultiTargeted ($200 \times 20 \times T$) consistently lowers robust accuracy by $5$–$10$ points compared to standard PGD ($200 \times 20T$) for all $T \geq 2$.

## 5. Empirical Results on MNIST and CIFAR-10

The effect of tuning PGD hyperparameters and adopting MultiTargeted testing is evident in benchmark results:

- **MNIST ($\epsilon=0.3$), MadryLab model**:
  - PGD$^{(100\times 50)}$ (tuned): $89.03\%$ accuracy under attack.
  - MultiTargeted$^{(100\times 20)}$: $88.43\%$.
  - PGD + MultiTargeted (combined): $88.36\%$ (lowest).
  - IntervalAttack (best prior): $88.42\%$.
- **CIFAR-10 ($\epsilon=8/255$), MadryLab model**:
  - PGD$^{(20\times 10)}$: $45.18\%$.
  - MultiTargeted$^{(20\times 10)}$: $44.03\%$.
  - PGD + MultiTargeted (combined): $44.03\%$.
  - FABAttack (best prior): $44.51\%$.

Table 5 in [1910.09338] provides leaderboard comparisons; MultiTargeted obtained first rank for both datasets, and for the TRADES model ($53.07\%$ accuracy at $\epsilon=0.031$).

## 6. Practitioner Recommendations

Empirical findings in [1910.09338] yield best-practice guidelines for configuring PGD and MultiTargeted adversarial testing:

- **Optimizer:** Use Adam ($\beta_1=0.9$, $\beta_2=0.999$, $\epsilon=10^{-8}$) for maximal stability.
- **Step-size:** Start with $\alpha_0 \approx 0.1$, decay by $10\times$ at $K/2$ and $3K/4$; for pure sign methods use $\alpha\approx\epsilon/10$ without decay.
- **Surrogate loss:** Prefer margin loss or CE; margin can yield marginal improvements if computational budget allows.
- **Restarts vs Steps:** First optimize hyperparameters ($\alpha$, optimizer, loss) with a single restart; then, increase $N_r$. On smooth losses (CIFAR-10 WResNet), MultiTargeted suffices with fewer restarts; on non-smooth (MNIST), more restarts are advantageous.
- **MultiTargeted:** For models with $C\leq 20$ or those locally linear (adversarially trained), MultiTargeted with $T\approx 2$–$5$ is optimal. Otherwise, use full $T=C-1$ but reduce inner iterations to maintain constant total attack budget.
- **Baseline defaults:** MNIST: PGD$^{(1000\times 180)}$ with $\alpha$ decay ($88.2\%$). CIFAR-10: MT$^{(1000\times 20\times 9)}$ ($44.0\%$).

In all experiments, the robustness lower bound should be validated by combining PGD, MultiTargeted, and increased restarts as no single method universally saturates robust error.

## 7. Significance and Theoretical Implications

The identification and rigorous benchmarking of PGD hyperparameter effects have materially advanced adversarial robustness evaluation. Adam optimizer and $\alpha$-decay scheduling, used in conjunction with convex surrogate losses and MultiTargeted logic, define the current empirical frontier in white-box attack design. The guarantee that MultiTargeted with $C-1$ restarts globally maximizes convex surrogates under local linearity places the method on a firm theoretical foundation. This suggests that for modern adversarially trained models, strategic multiplicity in attack targets, optimizer adaptivity, and calibrated decay schedules are critical for accurate robustness estimation [1910.09338].

Source: https://www.emergentmind.com/topics/hyperparameter-insights-for-pgd