Lipschitz Regularization for Unlearning

• The paper introduces Lipschitz regularization to enable targeted unlearning by selectively applying penalties and synthetic data generation in models like CLIP.
• Empirical results demonstrate a high removal success rate with minimal impact on retained classes through iterative, layer-selective updating.
• Theoretical guarantees are provided by bounding the local Lipschitz constant, ensuring controlled sensitivity, stability, and privacy in unlearning pipelines.

Lipschitz regularization for unlearning is a structural approach that leverages the Lipschitz continuity property of neural networks and loss landscapes to enable efficient and targeted removal of learned information. In unlearning applications—particularly those motivated by data privacy regulations such as GDPR—the goal is to approximate or reach the effect of retraining a model absent specific data or classes, while maintaining efficiency and utility on other tasks. Recent research has applied Lipschitz-based regularization both in multimodal settings (notably vision-LLMs like CLIP) and convex optimization scenarios, demonstrating wide-ranging implications for unlearning guarantees in deep and classical machine learning (Kravets et al., 2024, Ullah et al., 2023).

1. Mathematical Foundations of Lipschitz Regularization

Lipschitz continuity, formalized as $\|f(x)-f(y)\|_2 \leq L\,\|x-y\|_2$ for an $L$-Lipschitz function $f$, quantifies the sensitivity of neural mappings to input perturbations. In unlearning, this property is operationalized locally, focusing on small perturbations near data points or in targeted regions of the input space.

For models such as CLIP, the unlearning loss is augmented to penalize deviations in embedding spaces for the class intended to be forgotten:

• The CLIP loss combines the original symmetric contrastive objective for visual and textual encoders ($f_v$, $f_t$) with a Lipschitz regularization term applied selectively:

$$\mathcal L = \mathcal L_{\mathrm{CLIP}} + \lambda\,\frac1{|\mathcal D_f|}\!\sum_{(x,c)\in\mathcal D_f}\bigl[R_v(x)+R_t(x,c)\bigr]$$

where the penalties $R_v(x)$ and $R_t(x, c)$ are estimated via Monte Carlo sampling of Gaussian noise perturbations:

$$R_v(x) = \mathbb E_{\epsilon\sim\mathcal N(0,\sigma^2)}\! \frac{\|f_v(x)-f_v(x+\epsilon)\|_2^2}{\|\epsilon\|_2^2}$$

$$R_t(x,c) = \mathbb E_{\epsilon}\! \frac{\|\,f_t(c)-f_v(x+\epsilon)\|_2^2}{\|\epsilon\|_2^2}$$

Only samples belonging to the forget class $\mathcal D_f$ are regularized in this way, confining the effect of forgetting.

In convex settings, Lipschitz regularity is needed for query sensitivity, enabling the use of Gaussian mechanism–based stability and rejection coupling for unlearning, where perturbation levels are calibrated to the Lipschitz constant of the loss (Ullah et al., 2023). This sensitivity ensures the impact of data removal on intermediate queries is tightly controlled.

2. Synthetic Forget Sample Generation

Zero-shot unlearning in domains such as vision-LLMs must address cases where real forget-class data is inaccessible. The process involves generating synthetic samples that maximize the model's confidence in the target class (to be forgotten) using gradient ascent:

$$x_{i+1} = x_i + \eta\,\nabla_{x_i}\log p\bigl(y_{\mathrm{target}}\mid x_i\bigr)$$

where $p(y\mid x)$ is the model's softmax likelihood for class $y$ given synthesized image $x$. The process begins from noise, and synthesis proceeds until the sample is classified as the forget class by the model. Optional steps include clamping and Gaussian smoothing to produce plausible inputs. The resulting synthetic batch $\mathcal D_f$ is used for both Lipschitz-regularized fine-tuning and for tracking unlearning progress (Kravets et al., 2024).

3. Selective, Iterative Unlearning Procedures

Unlearning is not applied globally but in a targeted, iterative manner to minimize utility degradation on retained tasks.

• Iterative schedule: The procedure alternates between evaluating class accuracy on $\mathcal D_f$, adjusting the regularization noise level $\sigma$, and dynamically modifying the number of trainable layers $(k_v, k_t)$ in CLIP’s vision and text encoders.
• Selective layer update: After each backward pass, layers are ranked by the average absolute gradient magnitude:

$$G_\ell = \frac1{|\mathcal D_f|}\sum_{(x,c)\in\mathcal D_f} \Bigl\lVert\,\mathbb E\bigl[\partial\mathcal L/\partial W_\ell\bigr]\Bigr\rVert_1$$

Only the top $k$ layers in each encoder (by $G_\ell$) are updated. Less responsive layers are frozen, reducing over-forgetting and limiting transfer of unlearning to unrelated representations.

• Stopping criterion: The unlearning process terminates when the model’s accuracy on the synthetic forget set $A_{\mathrm{syn}}$ drops below a preset threshold $A_{\mathrm{stop}}$.

The combination of synthetic samples and locally applied, layer-selective Lipschitz penalties enables precise control of the forgetting effect, while iterative adjustment allows the process to tune forgetting strength as needed (Kravets et al., 2024).

4. Theoretical Guarantees and Role of Lipschitzness in General Unlearning

Lipschitz regularization enables well-characterized bounds on sensitivity and risk in both deep and convex frameworks.

• In stochastic convex optimization, Lipschitzness bounds query sensitivity, supporting the use of Gaussian perturbation for TV–stability between models trained on adjacent datasets.
• Exact unlearning is achieved by rejection coupling—comparing noisy query responses between original and modified datasets, with the risk of retraining and additional queries governed by the TV–stability parameter $\rho$ (Ullah et al., 2023).
• Risk bounds for the population loss associated with unlearning are:
  • Smooth losses: $$\tilde O\bigl((G+\beta D)D(\frac1{\sqrt n}+\frac{\sqrt d}{n\rho})\bigr)$$
  • Non-smooth: $$\tilde O\bigl(GD(\frac1{\sqrt n}+(\frac{\sqrt d}{n\rho})^{1/2})\bigr)$$
• Bounded sensitivity (via Lipschitz constant) ensures errors from query noise and coupling are controlled, underpinning both privacy stability and efficient unlearning.

A plausible implication is that in any unlearning pipeline where query or loss sensitivity can be upper-bounded due to Lipschitz regularity, the same Gaussian mechanism and coupling strategies generalize seamlessly to more complex architectures and dynamic data streams.

5. Empirical Results and Comparative Analysis

The Lipschitz regularization framework was empirically validated on CLIP-based zero-shot unlearning tasks across Caltech101, StanfordCars, OxfordFlowers, and StanfordDogs datasets (Kravets et al., 2024). The key metrics measured were:

• Removal success rate: Accuracy on the forgotten class before (BF) and after (AF) unlearning.
• Utility retention: Accuracy on retained classes within the same dataset and on unrelated held-out datasets.

Selected results (ResNet50 backbone):

Method	Target BF→AF	Other BF→AF
Lip (ours)	0.397→0.056	0.558→0.551
Emb	0.397→0.087	0.558→0.536
Amns	0.397→0.357	0.558→0.498
EMMN	0.397→0.000	0.558→0.054
ULip	0.397→0.127	0.558→0.457
AmnsRetain	0.397→0.040	0.558→0.711

Interpretation of results:

• The joint image-text Lipschitz penalty (“Lip”) enables near-complete removal of the forget class with minimal impact on retained or unrelated classes.
• Stronger or less targeted unlearning (e.g., EMMN) yields catastrophic forgetting of other knowledge.
• Limiting unlearning to the vision branch (ULip) underforgets in the text branch and destabilizes the shared embedding space.
• Ablation confirms both branches need regularization, the iterative schedule is necessary, and indiscriminate updating of all layers can cause over-forgetting.

6. Extensions, Generalizations, and Dynamic Settings

Lipschitz-based unlearning generalizes to a broad range of learning scenarios:

• For generalized linear models, embedding input data to a suitably lower dimension via Johnson–Lindenstrauss transforms allows dimension-independent unlearning rates for both smooth and non-smooth Lipschitz losses.
• The adaptive query release viewpoint—viewing optimization as a sequence of TV–stable noisy queries—unifies unlearning across standard convex learning and streaming settings (insertions and deletions) (Ullah et al., 2023).
• In dynamic streams, the binary-tree mechanism accommodates both exact and weak unlearning (model-only), with bounded retraining complexity dependent on the stability parameter $\rho$ and the number of requests.

7. Significance and Limitations

Lipschitz regularization provides a robust foundation for scalable, practical unlearning by controlling the local geometry of loss landscapes and embedding spaces. The selective, local application in both neural and convex frameworks allows for targeted deletion with minimized collateral impact.

A key insight is that the synergy of synthetic sample generation, local Lipschitz penalties, and selective parameter updates results in a highly effective approach for zero-shot class unlearning in multimodal settings. Without Lipschitz (and, where applicable, smoothness) assumptions, neither the TV–stable coupling methodology nor the associated convergence/error bounds would hold.

However, limitations include the calibration of synthetic sample quality, choice of layer updates, and sensitivity to model architectural idiosyncrasies. Further, the approach’s reliance on bounding the local Lipschitz constant may present challenges in architectures or loss surfaces that are highly irregular or in datasets lacking sufficiently discriminative representations.

References: (Kravets et al., 2024, Ullah et al., 2023)