Defense Against Occlusion Attacks (DOA)

• DOA is a framework that defends visual systems by countering adversarial occlusions through adversarial training and restoration-based preprocessing.
• Adversarial training employs PGD-optimized rectangular patches to simulate occlusions, achieving robust accuracy improvements in detection tasks.
• Restoration-based defenses utilize lightweight autoencoders for inpainting occluded image regions, markedly reducing attack success ratios.

Defense against Occlusion Attacks (DOA) refers to algorithmic frameworks and methodologies designed to enhance the robustness of deep neural network–based visual perception systems—most notably, image classifiers and object detectors—against adversarial occlusions. These occlusions are typically physically realizable patches or rectangles affixed to a scene or object, with the intent to mislead machine learning models in both digital and real-world settings. Two principal lines of work address this challenge: adversarial training using rectangular occlusion attacks (Wu et al., 2019), and model-agnostic restoration-based preprocessing defenses (Pathak et al., 2024).

1. Formalization of Occlusion Attacks

Occlusion attacks are characterized by the placement of structured, locally concentrated perturbations (patches or stickers) over a subset of the input image. In the dominant mathematical abstraction, a clean image $I \in \mathbb{R}^{H \times W \times C}$ is modified by a binary mask $M \in \{0,1\}^{H \times W}$ designating the occluded region, and a patch $P \in \mathbb{R}^{s \times s \times C}$ providing the adversarial content. The occlusion operator is defined as:

$I' = O(I, P, M) = (1 - M) \odot I + M \odot P$

where $\odot$ denotes element-wise multiplication, with $P$ broadcast into the mask region (Pathak et al., 2024). In an alternative formulation for generic classifiers, the adversary optimizes the inner patch content $p$ to maximize the loss function $\ell(f(x \oplus r), y)$ within a predefined mask $M_{(j, k, h, w)}$ and rectangle location $(j, k)$ (Wu et al., 2019).

The core defense objective is to either make the target model $f$ robust to all such $I'$ (by loss-aware training) or to preprocess $I'$ and recover an approximation $\tilde{I} \approx I$ that is devoid of the adversarial occlusion.

2. DOA via Adversarial Rectangular Training

The Defense against Occlusion Attacks (DOA) as formalized in (Wu et al., 2019) is predicated on adversarial training under a rectangular occlusion attack (ROA) threat model. The adversary can insert a rectangle with arbitrary content, subject to area and value constraints, anywhere in the input image. Formally, model parameters $\theta$ are trained to minimize expected adversarial loss:

$$\min_\theta \mathbb{E}_{(x, y) \sim \mathcal{D}} \left[ \max_{r \in \mathcal{R}} \ell(f_\theta(x \oplus r), y) \right]$$

where the inner maximization is over all valid occlusion rectangles $r = (j, k, h, w, p)$ with $p \in [-\epsilon,\epsilon]^{h \times w \times C}$.

Efficient algorithms for generating worst-case patches are critical. Two variants are employed:

• Exhaustive grid search + PGD: For each candidate patch position, Projected Gradient Descent (PGD) optimizes the patch content.
• Gradient-based location selection + PGD: Compute the loss gradient $\nabla_x \ell(f(x), y)$; select top-$C$ locations for occlusion based on aggregate squared gradient magnitude; then run PGD for each candidate.

Experimental practice adopts rectangle sizes $\sim$5–10% of image area, $T$=30–50 PGD steps, and batch-level adversarial training for 5–10 epochs. Implementation details are provided for both face recognition and traffic sign datasets (Wu et al., 2019).

3. Restoration-Based Model-Agnostic Preprocessing

An alternative line of defense leverages model-agnostic, lightweight autoencoders that perform occlusion removal prior to downstream visual analysis. The approach in (Pathak et al., 2024) frames patch defense as an image restoration (inpainting) task:

$\tilde{I} = F_\theta(I')$

where $F_\theta$ is a convolutional Attention-UNet autoencoder with an EfficientNet-B0 encoder backbone (truncated to preserve high spatial resolution for small UAV objects) and a slim upsampling decoder. Attention gates modulate skip connections, combining encoder and decoder features multiplicatively.

The network is trained exclusively with a pixel-wise $L_2$ reconstruction loss:

$$L_{\mathrm{rec}}(x, \tilde{x}) = \sum_{i=1}^H \sum_{j=1}^W \sum_{c=1}^3 \left[x_{ijc} - \tilde{x}_{ijc}\right]^2$$

Occlusions are synthetically generated at each minibatch using random textured patches and random rectangular masks. The approach is agnostic to detector and patch type, requiring no exposure to adversarial patch patterns during training.

4. Empirical Results and Evaluation Metrics

Evaluation in both domains considers digital (e.g., VisDrone, YOLOv5-s) and physical (toy vehicle/printed patch) setups.

Primary metrics include:

• Attack Success Ratio (ASR): Fraction of objects that become undetectable post-attack.
• Average Precision/Recall (AP/AR): Standard object detection measures at IoU ≥ 0.5.

Key empirical findings appear in the following table:

Defense	Patch Type	Baseline ASR	Post-Defense ASR	Relative ASR Reduction
None	Adversarial patch	0.84	n/a	n/a
Restore-pixels	Adversarial patch	0.84	0.59	≈ 30%
Restore-pixels	Gray patch	0.53	0.46	13%
Restore-pixels	Random patch	0.55	0.44	21%
Masking only	Adversarial patch	0.84	0.75	10%

The inpainting-based “restore-pixels” defense consistently outperforms pixel-wise masking, both in terms of ASR reduction and restoration of AP/AR curves (Pathak et al., 2024). Runtime overhead is modest (+4% per image) and the defense is compatible with multiple detection architectures.

In the adversarial training regime, DOA achieves robust accuracy of ~90%+ under ROA, and far outperforms PGD-Adversarial Training ($\ell_\infty$) and Randomized Smoothing ($\ell_2$) under physical attacks (e.g., adversarial eyeglasses, stop sign stickers, universal adversarial patch) (Wu et al., 2019).

5. Deployment and Model-Agnostic Integration Strategies

The restoration-based defense operates as a plug-and-play module: once trained on generic occlusions, the autoencoder $F_\theta$ preprocesses all input frames before object detection. This obviates the need for retraining or fine-tuning the object detector $f(\cdot)$, and the defense remains effective across a variety of patch types (adversarial or random, gray, or noise-based) (Pathak et al., 2024).

In contrast, DOA-based adversarial training requires joint re-training of the detection/classification model under the specified threat model, which makes it detector-specific but provides certifiably high robustness when incorporated into the training pipeline (Wu et al., 2019).

6. Limitations, Open Questions, and Future Directions

Observed limitations include:

• $\ell_p$-based adversarial defenses generalize poorly against physically realizable occlusions—attack specificity is crucial (Wu et al., 2019).
• Masking-based defenses (predicting and eliminating occluded pixels) provide limited restoration of lost detections compared to direct patch inpainting.
• The restoration-based defense, while effective for moderate patch sizes (e.g., ~20% of object area), has not been characterized against highly variable or composite-shaped occlusions.
• The autoencoder-based approach is empirically robust but lacks formal certified guarantees.

Open questions highlighted include:

• Extending the adversarial training regime to arbitrary shape templates (e.g., polygons, circles) beyond simple rectangles.
• Unifying DOA with other robust training paradigms (e.g., randomized smoothing) for defenses against composite threat models.
• Formal certification of DOA-style defenses.

A plausible implication is that, for high-assurance applications facing arbitrary sticker or patch attacks, DOA adversarial training remains the gold standard for certifiable robustness, but model-agnostic preprocessing modules offer pragmatic, low-complexity defense that is attractive for real-time pipelines and rapidly changing detection architectures (Wu et al., 2019, Pathak et al., 2024).

7. Summary and Practitioner Recommendations

Defense against Occlusion Attacks (DOA) encompasses a suite of methods—adversarial rectangular training, model-agnostic restoration, and segmentation-based masking—for mitigating the impact of adversarial patches and stickers. Both (Wu et al., 2019) and (Pathak et al., 2024) demonstrate that explicit modeling of occlusion, whether through hardening via adversarial training or pixel restoration via autoencoders, yields significant improvements in robustness. Best practices include:

• Selecting rectangle sizes that correspond to expected real-world attack areas.
• For adversarial training: use PGD-based patch optimization, batch-level adversarial examples, and train for several epochs.
• For model-agnostic preprocessing: train a compact UNet-style autoencoder on synthetic occlusions using only pixel-wise reconstruction losses; deploy as a real-time input filter.
• Evaluate with ASR and standard detection metrics to ensure reduced object hiding.

These strategies are foundational for any deployment scenario subject to physical or digital occlusion attacks.