Practical Continual Forgetting for Pre-trained Vision Models (2501.09705v1)

Abstract: For privacy and security concerns, the need to erase unwanted information from pre-trained vision models is becoming evident nowadays. In real-world scenarios, erasure requests originate at any time from both users and model owners, and these requests usually form a sequence. Therefore, under such a setting, selective information is expected to be continuously removed from a pre-trained model while maintaining the rest. We define this problem as continual forgetting and identify three key challenges. (i) For unwanted knowledge, efficient and effective deleting is crucial. (ii) For remaining knowledge, the impact brought by the forgetting procedure should be minimal. (iii) In real-world scenarios, the training samples may be scarce or partially missing during the process of forgetting. To address them, we first propose Group Sparse LoRA (GS-LoRA). Specifically, towards (i), we introduce LoRA modules to fine-tune the FFN layers in Transformer blocks for each forgetting task independently, and towards (ii), a simple group sparse regularization is adopted, enabling automatic selection of specific LoRA groups and zeroing out the others. To further extend GS-LoRA to more practical scenarios, we incorporate prototype information as additional supervision and introduce a more practical approach, GS-LoRA++. For each forgotten class, we move the logits away from its original prototype. For the remaining classes, we pull the logits closer to their respective prototypes. We conduct extensive experiments on face recognition, object detection and image classification and demonstrate that our method manages to forget specific classes with minimal impact on other classes. Codes have been released on https://github.com/bjzhb666/GS-LoRA.

• The paper introduces a continual forgetting framework using GS-LoRA to selectively remove unwanted data from pre-trained vision models.
• It employs group sparse regularization and prototype regularization in GS-LoRA++ to preserve remaining knowledge even with limited training data.
• Experimental results in face recognition, image classification, and object detection show effective forgetting with minimal parameter overhead.

Practical Continual Forgetting for Pre-trained Vision Models

The paper "Practical Continual Forgetting for Pre-trained Vision Models" proposes an innovative approach to address the necessity of selectively erasing information from pre-trained vision models—a requirement driven by growing concerns over privacy and security. This research introduces the concept of "continual forgetting," defined as the ability to sequentially remove unwanted knowledge from a model without affecting its performance on the remaining data.

Key Challenges and Proposed Solutions

The authors identify three primary challenges in implementing continual forgetting:

1. Efficient Deletion of Unwanted Knowledge: The process of forgetting should be lightweight and prompt, as models need to adapt quickly to new forget requests.
2. Minimizing Impact on Remaining Knowledge: The model should avoid catastrophic forgetting for the data that isn't being removed.
3. Data Scarcity in Real-world Scenarios: In practice, the data available to guide the forgetting process may be limited or partially missing.

To tackle these challenges, the paper proposes a novel method called Group Sparse LoRA (GS-LoRA), which utilizes Low-Rank Adaptation (LoRA) to fine-tune the Feed-Forward Network (FFN) layers in Transformer blocks during the forgetting process. Additionally, a group sparse regularization mechanism is implemented to manage network modifications selectively and efficiently.

Incremental Improvements with Prototype Regularization

The research extends the GS-LoRA model to GS-LoRA++, which introduces prototype regularization to enhance performance in more practical scenarios, such as when available training samples are scarce. This adaptation leverages class prototypes to guide the erase procedure for unwanted classes while consolidating knowledge for the remaining classes. This ensures robust performance even in situations where traditional datasets are limited, emphasizing the method's versatility and practical applicability.

Experimental Validation

The paper reports extensive experiments across tasks like face recognition, image classification, and object detection, demonstrating the efficacy of GS-LoRA++ in successfully implementing class-specific forgetting. Models equipped with this method can effectively discard selected classes with minimal influence on performances concerning other classes, even when subjected to scenarios typically characterized by few-shot learning data.

Furthermore, numerical results highlight the model's efficiency, showcasing a minimal tunable parameter ratio compared to existing methods while maintaining substantive improvements in performance metrics for both forgotten and retained data. This underpins the method's practicality for real-world applications where fast and adaptive responses to forgetting requests are critical.

Implications and Future Directions

The proposed solution presents substantial theoretical and practical implications. Theoretically, it advances the understanding of how selective forgetting can be efficiently operationalized in large vision models. Practically, this method aligns closely with real-world demands where privacy regulations and model biases require dynamism in retaining and discarding learned data.

Looking forward, the integration of more complex prototype schemas or the combination of GS-LoRA++ with strategies from other robust machine learning domains could be explored to further enhance performance and adaptability. Additionally, investigating the mechanisms of knowledge storage within the Transformer architecture may provide insights leading to even more efficient continual forgetting solutions.

The paper concludes by suggesting that the introduction of a practical continual forgetting problem, together with the development of the GS-LoRA++ framework, paves the way for new explorations in both continual learning and machine unlearning disciplines.