Transferable Physical-World Adversarial Patches Against Object Detection in Autonomous Driving

Abstract: Deep learning drives major advances in autonomous driving (AD), where object detectors are central to perception. However, adversarial attacks pose significant threats to the reliability and safety of these systems, with physical adversarial patches representing a particularly potent form of attack. Physical adversarial patch attacks pose severe risks but are usually crafted for a single model, yielding poor transferability to unseen detectors. We propose AdvAD, a transfer-based physical attack against object detection in autonomous driving. Instead of targeting a specific detector, AdvAD optimizes adversarial patches over multiple detection models in a unified framework, encouraging the learned perturbations to capture shared vulnerabilities across architectures. The optimization process adaptively balances model contributions and enforces robustness to physical variations. It further employs data augmentation and geometric transformations to maintain patch effectiveness under diverse physical conditions. Experiments in both digital and real-world settings show that AdvAD consistently outperforms state-of-the-art (SOTA) attacks in performance and transferability.

• The paper introduces AdvAD, a framework that generates adversarial patches with dynamic ensemble weighting and significantly reduces mAP in both digital and physical evaluations.
• AdvAD employs data augmentation mimicking real-world conditions and jointly optimizes patches on YOLOv5 and Faster R-CNN to boost cross-model transferability.
• Physical tests in simulated CARLA environments and real-world deployments validate that AdvAD effectively misdirects object detectors, highlighting safety risks for autonomous systems.

Transferable Physical-World Adversarial Patch Attacks on Object Detection for Autonomous Driving

Introduction

The proliferation of deep learning-based object detectors has become foundational for perception in autonomous driving systems. However, their vulnerability to adversarial attacks, particularly physical adversarial patches, raises critical safety concerns in real-world deployment. Patch-based adversarial attacks are pragmatic, but historically limited in cross-model transferability and robustness under environmental variations. The paper "Transferable Physical-World Adversarial Patches Against Object Detection in Autonomous Driving" (2604.23105) introduces AdvAD, a framework designed to generate adversarial patches that simultaneously maximize transferability and physical robustness for object detectors in realistic autonomous driving contexts.

Figure 1: Strategic placement of adversarial patches induces miss-detection of safety-critical objects within autonomous driving perception pipelines.

Method: The AdvAD Framework

AdvAD addresses two fundamental limitations in prior adversarial patch methods: weak transferability across model architectures and insufficient adaptation to physical-world environmental shifts. The framework employs a data augmentation layer that mimics diverse real-world conditions (viewpoint, lighting, geometric distortions) and integrates two independently pre-trained white-box object detectors (YOLOv5 and Faster R-CNN) for localization. Adversarial patches are placed relative to detected object centers, scaled appropriately, and jointly optimized across both detectors using an adaptive ensemble weighting strategy:

• Loss Function: The total loss incorporates detection confidence, classification error, and total variation regularization to encourage semantic attack objectives and spatial smoothness.
• Dynamic Ensemble Weighting: Model contributions in loss aggregation are dynamically adjusted during optimization, avoiding overfitting to a single surrogate and improving generalization.
• Patch Cutout: Random cropping of patch regions during training mitigates spatial overfitting and increases patch diversity, aiding transferability.

  Figure 2: AdvAD pipeline simulates physical variability, utilizes multiple detectors, and positions the patch for maximum disruption under real-world conditions.

Experimental Evaluation

Digital Domain Performance

AdvAD’s efficacy was validated across MS-COCO and PASCAL VOC datasets, using YOLOv5, Faster R-CNN, and SSD detectors. Compared to SOTA patch-based baselines (AdvPatch, T-SEA, NPAPGuard), AdvAD produced the lowest mAP values (largest mAP degradation), indicating superior attack effectiveness and transferability. For instance, AdvAD reduced the average mAP on MS-COCO from baseline levels (e.g., 32.09 for AdvPatch) to 28.32, and on PASCAL VOC, from 24.49 to 22.72, outperforming all tested methods.

Figure 3: AdvAD generates more missed detections and reduces confidence, surpassing baseline patch attacks in digital domain object detection degradation.

Simulated and Physical-World Validation

AdvAD's transferability was examined in the CARLA simulator, where planar adversarial patches deployed on trucks led to systematic mislocalizations and false positives, substantially reducing detector performance. Furthermore, physical-world experiments involved printing patches, evaluating under controlled conditions, and real-scene deployments. AdvAD maintained high attack success rates despite variations in scene composition, viewpoint, lighting, and object type.

Figure 4: AdvAD attack visualization in simulated CARLA environments targeting trucks via planar adversarial patch placements.

Figure 5: Physical-world evaluation—AdvAD patches effectively degrade detector performance across outdoor, mixed, and indoor real-scene deployments and under common physical distortions.

Ablation Analysis and Ensemble Strategies

Ablation studies revealed that dynamic ensemble weighting and patch cutout are critical for AdvAD's performance; their removal led to higher mAP and reduced transferability. Classification and total variation losses are also essential; omitting the TV loss resulted in noisier patches lacking physical robustness. Ensemble experiments demonstrated that optimizing patches across YOLOv5 and Faster R-CNN balances effectiveness and efficiency, with three-model ensembles offering marginal gains at substantially greater computational cost.

Implications and Future Directions

AdvAD demonstrates high transferability from digital to physical domains, addressing practical challenges in deploying adversarial patches within complex environments. The adaptive ensemble and deployment augmentation suite yield patches resilient against a range of real-world distortions and model architectures. These findings question the reliability of current perception modules, emphasizing the necessity for robust defense mechanisms and thorough safety validation.

Theoretically, AdvAD highlights the shared vulnerabilities of diverse detection pipelines, suggesting that ensemble-based loss optimization can capture transferable adversarial features. Practically, it exposes significant risk for autonomous driving and motivates research into robust model design, adversarial detection, and patch-neutralization strategies. Future work may extend to more complex, heterogeneous detector ensembles, increase attack stealthiness, or develop dynamic defenses for in-situ detection and patch mitigation.

Conclusion

AdvAD provides a rigorous framework for evaluating and exploiting the vulnerabilities of object detection in autonomous driving through transferable and physically robust adversarial patches. Its adaptive ensemble strategy and deployment augmentation suite enable strong attack efficacy across models and conditions, consistently outperforming previous state-of-the-art. This research deepens the understanding of adversarial threats in real-world perception systems and necessitates a renewed focus on resilient, safety-critical detection architectures.