Model Agnostic Defence against Backdoor Attacks in Machine Learning (1908.02203v3)

Abstract: Machine Learning (ML) has automated a multitude of our day-to-day decision making domains such as education, employment and driving automation. The continued success of ML largely depends on our ability to trust the model we are using. Recently, a new class of attacks called Backdoor Attacks have been developed. These attacks undermine the user's trust in ML models. In this work, we present NEO, a model agnostic framework to detect and mitigate such backdoor attacks in image classification ML models. For a given image classification model, our approach analyses the inputs it receives and determines if the model is backdoored. In addition to this feature, we also mitigate these attacks by determining the correct predictions of the poisoned images. An appealing feature of NEO is that it can, for the first time, isolate and reconstruct the backdoor trigger. NEO is also the first defence methodology, to the best of our knowledge that is completely blackbox. We have implemented NEO and evaluated it against three state of the art poisoned models. These models include highly critical applications such as traffic sign detection (USTS) and facial detection. In our evaluation, we show that NEO can detect $\approx$88% of the poisoned inputs on average and it is as fast as 4.4 ms per input image. We also reconstruct the poisoned input for the user to effectively test their systems.

• The paper introduces Neo, a novel model-agnostic framework designed to detect and mitigate backdoor attacks in machine learning, capable of operating without internal model knowledge.
• Neo identifies potential backdoor triggers through exploration and blocking, mitigates effects by inferring correct outputs, and can reconstruct detected triggers.
• Evaluations show Neo effectively detects backdoors with high rates and low false positives, restoring clean accuracy and providing a practical blackbox defence.

Insightful Overview of "Model Agnostic Defence against Backdoor Attacks in Machine Learning"

The paper "Model Agnostic Defence against Backdoor Attacks in Machine Learning" presents a novel framework titled Neo that is designed to detect and mitigate backdoor attacks in machine learning, specifically targeting image classification models. This research addresses a critical vulnerability in machine learning systems that are susceptible to backdoor attacks, where an adversary can implant hidden backdoors within a model. These backdoors can be activated by specific triggers introduced during training, allowing attackers to manipulate the model's outputs without impacting its performance on clean data.

Neo is developed as a model-agnostic approach, meaning it does not rely on knowledge of the model's internal architecture, making it highly versatile for deployment across various ML systems. This framework operates by identifying backdoor triggers within poisoned inputs and neutralizing their effects, aligning the system's behavior with the correct predictions akin to clean images.

Methodology and Performance Evaluation

The research highlights Neo’s methodology in several key components:

1. Detection of Backdoor Triggers: Neo identifies possible backdoor patterns in images through a combination of random exploration and the application of a trigger blocker, which covers suspected areas of the image that are likely to contain the backdoor trigger. The detection algorithm evaluates the model’s predictions with and without obfuscated regions to identify deviations signaling a backdoor.
2. Mitigation Process: Once potential backdoor triggers are identified, Neo uses the blocked images to infer the correct outputs by ensuring the removal of malicious manipulations from triggered inputs through automated patching.
3. Reconstruction of Triggers: Neo is uniquely capable of reconstructing backdoor triggers to inform users about potential threats, facilitating further testing and enhancement of model robustness.

Neo's efficiency was validated against several state-of-the-art backdoored models, including versions poisoned by the TrojanNN and BadNets methodologies. The evaluations show Neo's capability in accurately detecting backdoor inputs with detection rates of 76% for USTS, 86% for VGG Face, and 100% for MNIST, while maintaining false positive rates as low as 0%. Moreover, Neo significantly restores the model's clean accuracy by effectively neutralizing these backdoor influences.

Comparative Analysis and Discussion

In contrast to existing approaches like Neural Cleanse and Fine Pruning, Neo offers a blackbox technique that does not require prior knowledge of the model structure or the poisoned data. Despite its blackbox nature, Neo outperforms these methodologies in attack success scenarios. This attribute gives Neo an edge in real-world deployments where access to full model details may not be feasible.

This work also addresses potential adaptive attack strategies that modify the patterns or nature of backdoor triggers. The robustness of Neo against these more sophisticated attack vectors further demonstrates its practical applicability in safeguarding machine learning systems.

Implications and Future Directions

The implications of Neo extend to enhancing the trustworthiness and security of ML models deployed in sensitive applications, where undetected backdoors could lead to significant vulnerabilities. The ability to detect, mitigate, and expose backdoor triggers proactively empowers practitioners to deploy models with greater assurance of reliability.

Future research can build on Neo by expanding its defensive capabilities against more complex attacks, such as distributed backdoor triggers that do not conform to localized modifications. Furthermore, the framework could be extended to other domains beyond image classification, such as natural language processing or speech recognition, broadening its practical scope.

In conclusion, the paper makes a notable contribution to the domain of machine learning security by providing an efficient and scalable tool to detect and counteract backdoor attacks. With the increasing integration of ML systems in autonomous and critical applications, robust frameworks like Neo are essential for ensuring the security and integrity of these technologies.