Guiding Deep Learning System Testing using Surprise Adequacy (1808.08444v1)

Abstract: Deep Learning (DL) systems are rapidly being adopted in safety and security critical domains, urgently calling for ways to test their correctness and robustness. Testing of DL systems has traditionally relied on manual collection and labelling of data. Recently, a number of coverage criteria based on neuron activation values have been proposed. These criteria essentially count the number of neurons whose activation during the execution of a DL system satisfied certain properties, such as being above predefined thresholds. However, existing coverage criteria are not sufficiently fine grained to capture subtle behaviours exhibited by DL systems. Moreover, evaluations have focused on showing correlation between adversarial examples and proposed criteria rather than evaluating and guiding their use for actual testing of DL systems. We propose a novel test adequacy criterion for testing of DL systems, called Surprise Adequacy for Deep Learning Systems (SADL), which is based on the behaviour of DL systems with respect to their training data. We measure the surprise of an input as the difference in DL system's behaviour between the input and the training data (i.e., what was learnt during training), and subsequently develop this as an adequacy criterion: a good test input should be sufficiently but not overtly surprising compared to training data. Empirical evaluation using a range of DL systems from simple image classifiers to autonomous driving car platforms shows that systematic sampling of inputs based on their surprise can improve classification accuracy of DL systems against adversarial examples by up to 77.5% via retraining.

• The paper introduces Surprise Adequacy as a novel criterion that quantifies input novelty using likelihood and distance metrics to guide deep learning testing.
• It demonstrates that surprise-driven sampling can boost model robustness, achieving retraining accuracy improvements of up to 77.5%.
• The paper reveals that selecting optimal neuron layers for activation trace analysis critically enhances adversarial detection efficacy.

Evaluation of Surprise Adequacy in Testing Deep Learning Systems

The paper "Guiding Deep Learning System Testing using Surprise Adequacy" explores a critical concern for the deployment of deep learning (DL) systems in safety and security-sensitive applications: ensuring their correctness and robustness through effective testing methodologies. Traditional testing approaches, which depend heavily on manual data collection and labeling, are not sufficiently rigorous when applied to DL systems. The emergence of coverage criteria based on neuron activation has provided some guidance, but these methods lack granularity and do not adequately guide the testing process. This paper proposes a novel test adequacy criterion termed Surprise Adequacy for Deep Learning Systems (SADL), which pioneers a paradigm shift in assessing test adequacy based on the 'surprise' an input induces with respect to the training data.

Key Contributions

1. Introduction of Surprise Adequacy: SADL represents a significant methodological advancement by quantifying how unexpected an input is to a DL model, compared to the training data. This surprise is quantified in two ways:
   • Likelihood-based Surprise Adequacy (LSA): Measures surprise using a kernel density estimation approach to compare input activation trace likelihood against the training set.
   • Distance-based Surprise Adequacy (DSA): Utilizes Euclidean distance between activation traces to assess input novelty.
2. Empirical Validation and Utility: The paper provides comprehensive empirical results showing that surprise-driven sampling of input data for DL system retraining enhances robustness against adversarial inputs, culminating in accuracy improvements of up to 77.5% in retraining scenarios.
3. Layer Sensitivity and Activation Traces: The paper investigates the most informative layer(s) when calculating surprise, finding that the selection of neuron layers significantly affects the performance of adversarial detection, particularly for DSA in complex models like CIFAR-10.
4. Comparison with Existing Criteria: Through correlation studies with established neuron coverage metrics, SADL demonstrates its congruence with existing approaches while offering richer, more nuanced insights into the testing process. It adapts to various adversarial attack strategies, achieving superior detection efficacy as measured by ROC-AUC scores.
5. Guidance for Adversarial Training: The introduction of Surprise Coverage (SC) supports the broader application of surprise adequacy in enhancing DL model training and testing protocols by suggesting diverse, impactful input samples.

Practical and Theoretical Implications

From a practical perspective, SADL empowers practitioners to systematically evaluate and improve DL systems' resilience against unexpected, potentially harmful inputs. This approach can lead to more reliable and secure deployments, especially in domains like autonomous driving and cybersecurity, where system failure carries significant risks.

Theoretically, the concept of surprise adequacy draws attention to the importance of input novelty in testing and validation—a departure from traditional, static coverage criteria. It points to new research trajectories in model explainability and robustness, inviting further exploration into dynamic testing methodologies.

Future Directions

The paper lays a foundation for future research to extend SADL's applicability to more varied DL architectures, including recurrent neural networks. There is also room for further optimization of the underlying algorithms for computing surprise adequacy, particularly in the presence of scalability challenges when dealing with high-dimensional input spaces.

In conclusion, this paper marks a meaningful contribution to the field of deep learning testing, encouraging a shift towards more dynamic, data-driven methodologies for ensuring AI systems' reliability and safety.