Technical Report: When Does Machine Learning FAIL? Generalized Transferability for Evasion and Poisoning Attacks

Abstract: Recent results suggest that attacks against supervised machine learning systems are quite effective, while defenses are easily bypassed by new attacks. However, the specifications for machine learning systems currently lack precise adversary definitions, and the existing attacks make diverse, potentially unrealistic assumptions about the strength of the adversary who launches them. We propose the FAIL attacker model, which describes the adversary's knowledge and control along four dimensions. Our model allows us to consider a wide range of weaker adversaries who have limited control and incomplete knowledge of the features, learning algorithms and training instances utilized. To evaluate the utility of the FAIL model, we consider the problem of conducting targeted poisoning attacks in a realistic setting: the crafted poison samples must have clean labels, must be individually and collectively inconspicuous, and must exhibit a generalized form of transferability, defined by the FAIL model. By taking these constraints into account, we design StingRay, a targeted poisoning attack that is practical against 4 machine learning applications, which use 3 different learning algorithms, and can bypass 2 existing defenses. Conversely, we show that a prior evasion attack is less effective under generalized transferability. Such attack evaluations, under the FAIL adversary model, may also suggest promising directions for future defenses.

• The paper introduces the FAIL model to systematically characterize adversary capabilities across features, algorithms, instances, and leverage.
• It details the development of StingRay, a targeted poisoning attack that successfully bypasses defenses like RONI and Micromodels with clean-label samples.
• Experimental results show that even weaker adversaries can effectively compromise ML systems, highlighting the urgent need for robust defenses.

Analyzing the Threat of Targeted Poisoning Attacks on Machine Learning Systems

The paper under scrutiny provides a thorough examination of adversarial attacks on supervised machine learning systems, emphasizing the need to understand the capabilities and limitations of attackers. The study introduces the FAIL (Features, Algorithms, Instances, and Leverage) model as a comprehensive framework for analyzing adversarial attacks across varying degrees of adversarial knowledge and control.

Introduction to the FAIL Model

Machine learning systems, deployed in critical domains such as finance, medicine, and cybersecurity, are increasingly becoming targets for adversarial attacks. Recent research outlined in the paper indicates that defenses against these attacks often fall short, highlighting the importance of precisely modeling adversary capabilities. This has led to the development of the FAIL model, which systematically characterizes adversarial knowledge and control over four key dimensions:

1. Features (F): The knowledge of the specific features used within the targeted machine learning model.
2. Algorithms (A): The understanding of the learning algorithm implemented in the model.
3. Instances (I): The accessibility of training data or instances to the adversary.
4. Leverage (L): The degree of influence over the features that the adversary can exercise.

Methodology and Results

The authors build upon this model to design StingRay, a targeted poisoning attack that circumvents existing defenses through cleverly crafted poison samples. These samples are designed under constraints ensuring they possess clean labels, remain inconspicuous, and exhibit transferability. StingRay's efficacy was demonstrated across multiple machine learning applications, utilizing different learning algorithms, thereby showcasing its practical applicability.

Notably, weaker adversaries, as characterized by the FAIL model, can effectively employ StingRay to misdirect learning systems. The study's experimental evaluation compellingly demonstrates StingRay's ability to evade detection by three prominent anti-poisoning defenses: Micromodels, RONI (Reject on Negative Impact), and a targeted RONI variant (tRONI).

Implications and Future Directions

The implications of this study extend both theoretically and practically. The FAIL model provides a valuable framework for future research and defense mechanism development, encouraging more nuanced understanding and modeling of adversarial capabilities. Practically, the success of StingRay across various systems underscores the vulnerability of current machine learning applications to targeted attacks.

The paper points towards future work involving the exploration of more robust defenses that could limit adversarial leverage and fine-tune feature secrecy, as these strategies were identified as particularly effective against sophisticated attacks.

Conclusion

The study critically advances our understanding of adversarial threats in machine learning, highlighting the vulnerabilities of current defenses. By adopting a structured approach to modeling adversarial capabilities and developing targeted attacks like StingRay, the paper provides a foundational step towards enhancing the resilience of machine learning systems against potent adversarial exploits. This research is significant in steering the future of secure machine learning applications, emphasizing the necessity of continuous advancements in both attack techniques and defensive strategies.