UniAda: Universal Adaptive Multi-objective Adversarial Attack for End-to-End Autonomous Driving Systems

Abstract: Adversarial attacks play a pivotal role in testing and improving the reliability of deep learning (DL) systems. Existing literature has demonstrated that subtle perturbations to the input can elicit erroneous outcomes, thereby substantially compromising the security of DL systems. This has emerged as a critical concern in the development of DL-based safety-critical systems like Autonomous Driving Systems (ADSs). The focus of existing adversarial attack methods on End-to-End (E2E) ADSs has predominantly centered on misbehaviors of steering angle, which overlooks speed-related controls or imperceptible perturbations. To address these challenges, we introduce UniAda, a multi-objective white-box attack technique with a core function that revolves around crafting an image-agnostic adversarial perturbation capable of simultaneously influencing both steering and speed controls. UniAda capitalizes on an intricately designed multi-objective optimization function with the Adaptive Weighting Scheme (AWS), enabling the concurrent optimization of diverse objectives. Validated with both simulated and real-world driving data, UniAda outperforms five benchmarks across two metrics, inducing steering and speed deviations from 3.54 degrees to 29 degrees and 11 km per hour to 22 km per hour on average. This systematic approach establishes UniAda as a proven technique for adversarial attacks on modern DL-based E2E ADSs.

• The paper presents UniAda, a universal, adaptive framework that simultaneously perturbs steering and acceleration controls in autonomous driving systems.
• It leverages a dynamic adaptive weighting scheme to balance multi-objective optimization, achieving high success rates (up to 96.3% for steering) across diverse datasets.
• Empirical evaluations on both simulated and real-world videos demonstrate UniAda’s efficacy over state-of-the-art baselines, highlighting critical challenges in ADS adversarial robustness.

Universal Adaptive Multi-objective Attacks on End-to-End Autonomous Driving Systems

Problem Context and Motivation

End-to-End (E2E) deep learning-based Autonomous Driving Systems (ADSs) have become prevalent due to their ability to directly map sensor inputs to control actions, eliminating the complexity of intermediary modules such as perception, planning, and decision-making. However, the robustness and security of E2E ADSs remain significant concerns, especially given their susceptibility to adversarial attacks that can induce critical errors under safety-sensitive deployments. Most prior research on adversarial attacks for ADSs has focused on single-objective attacks—primarily steering angle deviations—while neglecting comprehensive control scenarios, including speed-related manipulation. Additionally, scenario diversity and imperceptibility of perturbations are often insufficiently addressed.

UniAda: Methodology and Architecture

The paper introduces UniAda, a universal, adaptive, multi-objective adversarial attack framework targeting E2E ADSs in a white-box setting. UniAda is engineered to simultaneously induce misbehavior in both steering and acceleration controls across entire driving video sequences using visually imperceptible (to humans) and image-agnostic perturbations.

UniAda leverages a multi-objective optimization scheme with two core streams: perturbation optimization across sensor input sequences and adaptive loss balancing across multiple objectives (steering and acceleration). The adaptive weighting scheme (AWS) is integrated using gradient-based dynamic weight tuning to harmonize the learning rate of loss contributions for each objective. This dynamic balancing prevents one objective from dominating the optimization and fosters joint learning of perturbations that mislead both control outputs.

Figure 1: Modular pipeline (top) separates perception, planning, and control; E2E (bottom) employs a deep network mapping sensor data directly to vehicle actions.

Figure 2: UniAda architecture illustrating dual optimization streams: universal multi-objective attack discovery (green) and adaptive weighting scheme (yellow).

Empirical Evaluation and Results

UniAda was extensively evaluated on 14 urban driving videos spanning both simulated (Carla100) and real-world datasets (Kitti, Udacity, Dave) with three victim E2E ADSs: CILRS, CILR (both conditional imitation learning based), and MotionTransformer. Five attack benchmarks—DeepManeuver, Perturbation Attack, DeepBillboard, FGSM, and UniEqual (a UniAda variant with equal weights)—were included for comparison. The evaluation metrics were Mean Error (ME) and Success Rate (SR), both tailored to capture attack strength and universality over different thresholds.

UniAda achieved mean steering and acceleration errors ranging from $3.54^\circ$ to $29^\circ$ and $11$ to $22$ km/h, respectively, substantially surpassing all baselines. Success rates at moderate error thresholds ($\delta_S=3.5^\circ$) were reported as high as 96.3% for steering and 74.7% for acceleration (CILRS), revealing that the universal perturbation misled almost all scenario frames. Statistical analysis using $t$-tests demonstrated that AWS offers significant improvements over fixed-loss approaches, with UniAda outperforming UniEqual in the majority of cases.

Figure 3: Imperceptibility of adversarial images for varying $\epsilon$ values, illustrating visual similarity to originals at attack thresholds used in evaluation.

Figure 4: Adversarial images generated by UniAda from simulated and real-world videos; prediction arrows indicate the targeted misdirection for steering and acceleration.

Figure 5: Attack dynamics in CILRS ``Black Car'' showing traces of mean error and AWS-adapted weights evolving during perturbation search.

Analysis: Adaptive Weighting and Multi-objective Synergy

AWS was demonstrated to dynamically allocate loss weights to maintain balanced attack progress across steering and acceleration. In empirical traces, the error in one objective would plateau if its associated weight was reduced, while the other would increase, validating the effect of adaptive weight scheduling. The ablation study revealed that joint multi-objective optimization outperformed single-objective attacks for both controls, evidencing complementary information in perturbation patterns between steering and acceleration.

Figure 6: Illustration of adaptive weighting, where AWS balances objective contributions such that each control's training rate converges despite initial imbalances.

Discussion: Transferability and Limitations

Transferability of adversarial perturbations across models was limited, reflecting a common shortcoming of white-box attacks. Although UniAda-generated perturbations for CILRS could sometimes mislead CILR, effect sizes diminished, and other attack methods similarly failed to transfer robustly. The universality of the perturbation is primarily within the distribution of the attacked video sequence rather than across architectural boundaries.

Figure 7: Transferability analysis showing mean error results for steering and acceleration; cross-model performance degrades compared to original victim model.

The requirement for gradient access (white-box) can restrict practical deployment against proprietary ADSs. The experiments are limited by the diversity of victim model architectures and datasets, although the selected ones represent competitive state-of-the-art systems. External validity is mitigated by comparison to recent and widely accepted baselines, and internal validity was addressed via open-sourcing the implementations.

Practical and Theoretical Implications

UniAda highlights the need for comprehensive adversarial robustness evaluation in ADSs, extending attack surfaces to all relevant control signals rather than focusing on steering alone. The adaptive weighting methodology provides an effective mechanism for multi-objective adversarial optimization and could be generalized to other safety-critical domains employing deep regression networks. The strong empirical results, especially in urban and real-world scenarios, challenge current ADS deployment practices and call for improved defense strategies encompassing both modular and E2E pipelines.

Theoretical directions include enhancing transferability with input diversity or advanced gradient techniques (2604.23362), extending adaptive loss balancing to other multitask architectures, and improving universality beyond sequence-level perturbations.

Conclusion

UniAda establishes a universal, adaptive multi-objective adversarial attack paradigm for E2E ADSs, demonstrably inducing severe misbehavior in both steering and acceleration controls under visually imperceptible perturbations. The adaptive weighting scheme ensures optimal attack strength across objectives, addressing deficiencies in prior literature related to limited control testing, scenario diversity, and image-specific artifacts. The empirical evidence indicates the necessity for defense and verification approaches capable of handling universal, multi-control, and adaptive adversarial threats in contemporary autonomous vehicle systems.