Fidelity-aware Adversarial Training (FAA)

• The paper's main contribution is introducing FAA, which perturbs non-semantic frequency bands to bridge domain gaps without compromising image semantics.
• FAA utilizes a Fourier-based decomposition to isolate and replace specific frequency components, enabling large perturbations while preserving key semantic features.
• FAA's adversarial min–max training strategy drives models toward flatter loss landscapes, yielding significant performance gains across segmentation, detection, and classification tasks.

Fidelity-aware Adversarial Training (FAA), or Fourier Adversarial Attacking in the context of domain adaptation, is an adversarial augmentation and regularization technique which targets specific frequency components of input samples to yield robust and generalizable models. Implemented in the Robust Domain Adaptation (RDA) framework, FAA systematically generates adversarial examples by modifying only non-semantic frequency bands of images, utilizing actual spectral bands taken from the target domain. This approach facilitates strong domain adaptation by enabling large-magnitude perturbations that do not compromise semantic content, guiding model optimization toward flat loss basins and yielding marked empirical improvements across multiple visual recognition tasks (Huang et al., 2021).

1. Formal Definition and Objective

FAA operates as an attacker module within a two-player adversarial min–max game integrated into the unsupervised domain adaptation (UDA) loop. The attacker’s goal is to maximally degrade network performance by generating adversarial variants of training images that elevate UDA loss, while the defender (the base network) is optimized to minimize the same loss for both clean and FAA-perturbed images. In contrast to standard L∞ or L₂ attacks—bounded by small ε—FAA permits large perturbations, restricted to frequency bands selected not to interfere with semantic cues. The resulting images maintain natural appearance, ensuring effective training and preventing overfitting to source or noisy target-domain samples.

By iteratively “kicking” model parameters into neighborhoods associated with flatter loss surfaces—a process known as inducing “random walk” into flat regions—FAA supports generalization across both simulated and real domains by discouraging narrow, fragile minima.

2. Frequency Decomposition and Perturbation Mechanism

FAA leverages a channel-wise 2D Discrete Fourier Transform of input images $x \in \mathbb{R}^{H \times W \times C}$:

$$z = \mathcal{F}(x), \quad z \in \mathbb{C}^{H \times W \times C}$$

The Fourier spectrum is decomposed into $N$ annular frequency components (FCs) with equal radial width, each $z^{(n)}$ capturing a distinct frequency band:

$$z^{(n)}[i,j] = \begin{cases} z[i,j], & \text{if } \frac{n-1}{N} < \text{dist}((i,j),(c_i,c_j)) < \frac{n}{N} \ 0, & \text{otherwise} \end{cases}$$

All bands are grouped as $\{z^1, \ldots, z^N\} = \mathcal{D}(z;N)$. For perturbation, a subset of these bands—typically those containing little semantic information—are replaced by the corresponding bands from a reference image $x_{ref}$ drawn from the target domain, yielding a mixed spectrum that preserves semantic fidelity.

A learnable gate $G \in \{0,1\}^N$ (trained via Gumbel–Softmax relaxation) selects up to $\lfloor pN \rfloor$ bands for replacement, such that

$$\hat{z}^{fc}_n = (1-G_n) z^{(n)} + G_n z^{(n)}_{ref}$$

The adversarial image is constructed by inverse Fourier transformation:

$$x^{FAA} = \mathcal{F}^{-1}\left( \sum_{n=1}^{N} \hat{z}^{fc}_n \right)$$

This replacement is regulated to avoid exceeding the gate budget $pN$ and to minimize distortion of “mid-frequency”—i.e., semantic—bands.

3. Adversarial Training Dynamics and UDA Integration

FAA is embedded into the broader RDA UDA loop through a two-step alternating optimization:

• Attacking Step: Fix model parameters, update FAA attacker by maximizing the current UDA loss computed on adversarial images $x^{FAA}$, subject to gate and semantic-reconstruction penalties.
• Defending Step: Fix attacker, update base network parameters by minimizing the UDA loss over both clean and FAA images.

The network thus learns not only to fit both the source and target distributions, but to remain robust to strong, semantically faithful perturbations resembling real domain-shift. For each batch:

1. Sample source and target minibatches.
2. For every input $x$, sample a random reference image from the target. Generate $x^{FAA}$ as above.
3. Compute combined supervised (source) and unsupervised (target) losses over both clean and FAA-augmented data.
4. Update FAA parameters to maximize loss under constraints.
5. Update model parameters to minimize the same loss.

This continual oscillation between attack and defense impedes over-minimization and regularizes the model away from sharp, narrow minima.

4. Loss Functions, Constraints, and Semantic Preservation

The attacker’s objective comprises three terms:

1. Task Loss: $L_{task}(x^{FAA}; F)$, leveraging supervised cross-entropy on source samples and unsupervised (pseudo-label or entropy-based) objectives on target samples.
2. Gate Penalty: Enforces the selection of no more than $pN$ bands.
3. Reconstruction Loss: $||R(x) - R(x^{FAA})||_1$, where $R$ is a fixed band-pass filter isolating mid-frequency (semantic) bands, ensuring the semantics remain undisturbed.

The overall maximization is:

$$\max_{G, \text{FAA params}}\ L_{task}(x^{FAA}; F) - \lambda_{gat} \cdot \max(0, \sum G_n - pN) - \lambda_{rec} \cdot ||R(x) - R(x^{FAA})||_1$$

By constraining adversarial influence to non-semantic regions, FAA obtains large-magnitude, realistic perturbations essential for bridging the domain gap in UDA, while semantic consistency is preserved.

5. Rationale for Non-Semantic Frequency Attacks

Conventional adversarial methods (bounded perturbations, pixel-level noise) are insufficient for UDA tasks where source–target domain shift often exceeds standard ε. FAA overcomes this by:

• Applying domain-representative frequency content (from the actual target domain) rather than synthetic or random noise.
• Allowing large-magnitude changes in low/high-frequency bands, which do not impact mid-frequency (object-level, semantic) information.
• Avoiding semantic corruption, thus preserving recognizability of the perturbed images.

This approach enforces the network’s decision boundary to locally reside within broad, flat regions of the loss landscape. As a result, small (semantic) or even moderate (spectrally plausible) shifts do not provoke large loss increases, supporting improved generalization and robustness to both source and target domain variations.

6. Empirical Results and Comparative Analysis

FAA has been empirically validated across benchmarks for semantic segmentation, object detection, and classification, yielding consistent and significant improvements compared to both task-specific baselines and standard regularization methods. Selected results include (Huang et al., 2021):

Task / Dataset	Baseline	Baseline+FAA	Δ (Absolute)
Segmentation (GTA5 → Cityscapes, DeepLabV2)	mIoU = 36.6	mIoU = 45.2	+8.6
Seg. (GTA5→Cityscapes, AdaptSeg)	mIoU = 42.4	mIoU = 48.0	+5.6
Obj. det. (Cityscapes→Foggy, SWDA)	mAP = 34.3	mAP = 38.3	+4.0
Classif. (VisDA17, CRST)	mean acc = 78.1	mean acc = 82.7	+4.6

• FAA consistently outperforms regularizers such as VAT and Mixup. For instance, on ST (GTA5→Cityscapes): VAT yields +1.7, Mixup +1.0, while FAA delivers +7.5 mIoU uplift.
• FAA enhances performance on thin classes and boundary regions that are especially affected by domain shift—a region where standard regularization is less effective.

Ablation studies demonstrate that attacking both source and target losses jointly yields optimal results.

7. Summary and Significance

FAA—operationalized as Fourier Adversarial Attacking in RDA—notably advances domain adaptation through three core innovations: (a) frequency-wise decomposition and manipulation of images, (b) replacement of non-semantic frequencies with target-domain spectral content under a small gate budget, and (c) adversarial min–max optimization coupling both clean and perturbed samples. This methodology enables large-magnitude, naturalistic perturbations essential for bridging domain gaps, regularizes networks to flat regions in loss landscapes, and demonstrates broad empirical evidence of superior generalizability and task performance across domains (Huang et al., 2021).