Adversarial Semantic and Label Perturbation Attack for Pedestrian Attribute Recognition

Abstract: Pedestrian Attribute Recognition (PAR) is an indispensable task in human-centered research and has made great progress in recent years with the development of deep neural networks. However, the potential vulnerability and anti-interference ability have still not been fully explored. To bridge this gap, this paper proposes the first adversarial attack and defense framework for pedestrian attribute recognition. Specifically, we exploit both global- and patch-level attacks on the pedestrian images, based on the pre-trained CLIP-based PAR framework. It first divides the input pedestrian image into non-overlapping patches and embeds them into feature embeddings using a projection layer. Meanwhile, the attribute set is expanded into sentences using prompts and embedded into attribute features using a pre-trained CLIP text encoder. A multi-modal Transformer is adopted to fuse the obtained vision and text tokens, and a feed-forward network is utilized for attribute recognition. Based on the aforementioned PAR framework, we adopt the adversarial semantic and label-perturbation to generate the adversarial noise, termed ASL-PAR. We also design a semantic offset defense strategy to suppress the influence of adversarial attacks. Extensive experiments conducted on both digital domains (i.e., PETA, PA100K, MSP60K, RAPv2) and physical domains fully validated the effectiveness of our proposed adversarial attack and defense strategies for the pedestrian attribute recognition. The source code of this paper will be released on https://github.com/Event-AHU/OpenPAR.

Adversarial Semantic and Label Perturbation Attack for Pedestrian Attribute Recognition

The paper presents an in-depth analysis of adversarial attacks and defenses within the domain of Pedestrian Attribute Recognition (PAR), marking the first exploration into the security vulnerabilities of PAR models. Given the increasing deployment of pedestrian attribute recognition systems in critical environments such as surveillance and security, addressing their susceptibility to adversarial perturbations is of paramount importance.

Technical Overview

The authors propose a novel framework leveraging adversarial semantic and label perturbations, termed ASL-PAR, to compromise pedestrian attribute recognition systems. The study is centered around the PromptPAR model, a CLIP-based architecture for PAR that exploits vision-language cues via multi-modal Transformers. This model is particularly pertinent, as it uses a foundational CLIP framework, enhancing its applicability in diverse recognition tasks.

Attack Strategy

The adversarial attack strategy is twofold:

1. Label Perturbation: This component involves strategically perturbing pedestrian attributes by reallocating them within localized parts of the pedestrian representation, such as head, upper body, and lower body attributes. The strength of the approach lies in its ability to create plausible variations in attribute values without rendering them detectable through visual inspection.
2. Semantic Perturbation: This aspect of the attack exploits the model’s reliance on the alignment between visual and textual embeddings generated by CLIP. By disaligning these embeddings through adversarial noise, the model's predictive performance is significantly degraded.

Extensive evaluations were performed on widely recognized datasets including PETA, PA100K, MSP60K, and RAPv2, demonstrating the proposed attack strategy's effectiveness. Notably, mA, Accuracy, Precision, Recall, and F1 scores showed substantial degradation, confirming the attack's potency.

Defense Strategy

Complementing the adversarial attack strategy, the authors introduce a semantic offset defense mechanism to mitigate the effects of adversarial noise. This involves the deployment of a visual noise filter coupled with a learnable text prompt tuning method. The defense aims to restore the vision-text alignment in CLIP spaces, countering the adversarial interference effectively.

Implications and Impact

The implications of this research are both practical and theoretical. Practically, the study highlights crucial security challenges that must be addressed to safeguard PAR systems against adversarial threats. Theoretically, it expands the understanding of how multi-modal integration models, like those utilizing CLIP, can be vulnerable to cross-modal adversarial interference.

The findings pave the way for future studies to explore more robust adversarial defense techniques and extend the attack framework to other multi-modal domains. The exploration of cross-dataset and cross-model generalizability of adversarial attacks is another prospective avenue, as it addresses the adaptability of attack strategies beyond specific models and datasets.

Conclusion

In summary, the paper provides crucial insights into the vulnerabilities of pedestrian attribute recognition systems against adversarial attacks. The ASL-PAR framework not only elucidates potential attack vectors but also proposes defensible strategies to enhance model robustness. Going forward, it fosters the advancement of secure and resilient PAR systems for ubiquitous real-world applications, encouraging further research into defending against adversarial attacks in other related domains.