Data Poisoning Won't Save You From Facial Recognition (2106.14851v2)

Abstract: Data poisoning has been proposed as a compelling defense against facial recognition models trained on Web-scraped pictures. Users can perturb images they post online, so that models will misclassify future (unperturbed) pictures. We demonstrate that this strategy provides a false sense of security, as it ignores an inherent asymmetry between the parties: users' pictures are perturbed once and for all before being published (at which point they are scraped) and must thereafter fool all future models -- including models trained adaptively against the users' past attacks, or models that use technologies discovered after the attack. We evaluate two systems for poisoning attacks against large-scale facial recognition, Fawkes (500'000+ downloads) and LowKey. We demonstrate how an "oblivious" model trainer can simply wait for future developments in computer vision to nullify the protection of pictures collected in the past. We further show that an adversary with black-box access to the attack can (i) train a robust model that resists the perturbations of collected pictures and (ii) detect poisoned pictures uploaded online. We caution that facial recognition poisoning will not admit an "arms race" between attackers and defenders. Once perturbed pictures are scraped, the attack cannot be changed so any future successful defense irrevocably undermines users' privacy.

• The paper demonstrates that current data poisoning techniques (Fawkes and LowKey) offer only temporary protection as adaptive models bypass these perturbations.
• Experimental evaluations show that even basic adaptive defenses can negate the effects of adversarial perturbations with high accuracy.
• The findings underscore that lasting privacy protection necessitates combining evolving technical defenses with legal measures rather than relying solely on data poisoning.

An Analysis of Data Poisoning in Facial Recognition Systems

Facial recognition technologies have advanced significantly, leveraging large datasets often scraped indiscriminately from the web, thereby raising substantial privacy concerns. To combat unauthorized use of facial images, data poisoning has been proposed as a defense mechanism. The paper "Data Poisoning Won't Save You From Facial Recognition" provides a critical examination of facial recognition poisoning strategies, specifically targeting the systems Fawkes and LowKey. It argues against the long-term effectiveness of these strategies in protecting privacy due to structural and adaptable weaknesses.

Key Arguments and Analysis

The central argument of the paper concerns the asymmetric nature of data poisoning in facial recognition. Users perturb their images prior to online posting, hoping to disrupt any trained model's ability to identify them accurately. However, this appears to be a one-time application of the attack; the perturbations cannot evolve as the models do. Conversely, adversaries can benefit from subsequent technological advancements and modify their models to overcome these perturbations, effectively nullifying the protection afforded by the initial poisoning.

Experimental Evaluation

The authors conducted evaluations on two popular poisoning tools, Fawkes and LowKey, highlighting how these systems can be effectively countered. They demonstrated that even simple adaptive strategies, where model trainers incorporate adversarial perturbations into their training process, can robustly defend against these attacks. These adaptive defenses were shown to be effective in breaking poisoning strategies with high accuracy.

Moreover, the paper revealed an "oblivious" defense strategy—a model trainer postpones their action until a sufficiently robust model technology becomes available, which inherently circumvents existing perturbations. This insight underscores a significant drawback of current poisoning strategies: their reliance on the predictability of adversary advancement is unrealistic given the rapid pace of machine learning research and improvements.

Implications

The paper suggests the insufficiency of technological measures alone to guarantee individual privacy against facial recognition systems. As advances in computer vision render data poisoning ineffective, legislative frameworks may become necessary to safeguard privacy rights. The authors recommend that privacy protection focus more on curbing invasive technologies through legal means than relying solely on adversarial perturbations.

Future Outlook

The findings of this paper imply a need to rethink defensive strategies against facial recognition intrusion. While current data poisoning methods like Fawkes and LowKey provide an immediate albeit limited privacy measure, future investigations could explore more dynamic or legally integrated frameworks. Additionally, there might be potential in research for novel ways of employing synthetic data or exploring the sociotechnical boundaries of facial recognition systems to fortuitously balance accountability and privacy.

Conclusion

This paper contends that data poisoning, as it stands, is not a viable long-term strategy for preserving privacy against facial recognition. The research demonstrates that both technical and legislative measures must be harmonized to achieve sustainable privacy protections in the face of evolving AI technologies. It calls on the community to adopt proactive, forward-looking stances in addressing privacy, cognizant of the need for interventions that extend beyond current adversarial techniques.