Targeted Contrastive Unlearning (TCU)

• TCU is a machine unlearning technique that uses contrastive losses to adjust embedding spaces and revoke the influence of designated training samples.
• It employs methods like InfoNCE loss, triplet-based unlearning, and alignment calibration to ensure efficient removal with minimal utility loss.
• TCU is applied in privacy compliance, debiasing, and scalable learning across vision, language, and graph models, offering significant speedups over retraining.

Targeted Contrastive Unlearning (TCU) refers to a class of machine unlearning techniques that leverage contrastive objectives—explicitly manipulating embedding-space relationships—to remove the influence of specific training samples or knowledge from learned models while preserving utility on the retained data. TCU is applicable across supervised, self-supervised, and generative paradigms, and is instantiated for vision, language, and graph models. Recent literature develops both general TCU frameworks and domain-specific algorithms with formally defined objectives, empirical validations, and rigorous auditing protocols (Lee et al., 19 Jan 2024, Wang et al., 5 Jun 2024, S et al., 1 Dec 2025, Wang et al., 12 May 2024, Tong et al., 3 Aug 2025, He et al., 19 Mar 2025, Tran et al., 6 Aug 2025, Hu et al., 3 Feb 2025, Lee et al., 4 Mar 2025, Tang et al., 25 Jul 2024, Chen et al., 18 Aug 2024, Suriyakumar et al., 12 Jun 2025).

1. Formal Problem Setting and Theoretical Motivation

TCU addresses the task of revoking the learned influence of a training subset $\mathcal{U} \subset \mathcal{D}_{\mathrm{train}}$ from model parameters while maintaining both model utility and data minimization requirements. For contrastive learning models, this entails producing new parameters $w_u$ such that the InfoNCE loss or its analog computed over the retained dataset $$\mathcal{R} = \mathcal{D}_{\mathrm{train}} \setminus \mathcal{U}$$ is comparable to retraining, and adversarial auditing (membership inference, representation similarity) fails to distinguish members of $\mathcal{U}$ from non-members (Wang et al., 5 Jun 2024, Lee et al., 19 Jan 2024). For generative and classification models, the goal extends to manipulating the output distributions (likelihood, embedding geometry) such that target outputs, behaviors, or biases corresponding to $\mathcal{U}$ are eliminated.

Theoretical motivation centers on the geometric properties of representation learning: class clusters are formed in embedding space, and information about training samples is encoded in relative proximities. TCU manipulates this geometry directly, pushing unlearning sample embeddings away from their original class clusters and toward the domain of other classes, or toward maximal uncertainty (Lee et al., 19 Jan 2024, He et al., 19 Mar 2025). For self-supervised and generative models, auxiliary alignment and repulsion terms are constructed in logit or embedding space, and unlearning can be formalized in terms of embedding disentanglement, representation degradation, or reduced mutual information between unlearned inputs and their prior outputs (Lee et al., 4 Mar 2025, Hu et al., 3 Feb 2025).

2. Core Algorithms and Loss Functions

2.1. Embedding-space Contrastive Objectives

General TCU methods define contrastive losses of the form:

$$\mathcal{L}_{\mathrm{TCU}} = \lambda_{\mathrm{UL}} \cdot \mathcal{L}_{\mathrm{UL}} + \lambda_{\mathrm{CE}} \cdot \mathcal{L}_{\mathrm{CE}}$$

where $\mathcal{L}_{\mathrm{UL}}$ operates over the unlearning set and typically involves InfoNCE-style or triplet margin losses repelling anchor (unlearn) embeddings from their positive class and attracting them to negatives (other classes or background) (Lee et al., 19 Jan 2024, S et al., 1 Dec 2025). For single-class unlearning, $\mathcal{L}_{\mathrm{UL}}$ reduces to a negative-pair-only term.

In generative LMs, TCU may employ contrastive losses over prompt–response pairs, e.g., by contrasting harmful prompt–harmful response behaviors with harmful prompt–neutral response pairs (using weighted positional pooling and $n$-pair losses) (Chen et al., 18 Aug 2024). For vision transformers, attention-aware masking generates positive (de-biased, masked) and negative (original) anchors in the InfoNCE contrastive term (Tong et al., 3 Aug 2025).

2.2. Retention and Utility Preservation

To avoid catastrophic forgetting, all practical TCU approaches add auxiliary loss components preserving performance on the retained data—either as standard classification losses, CE on $D_r$, or an explicit embedding-alignment term with the original model (Lee et al., 19 Jan 2024, He et al., 19 Mar 2025, Wang et al., 5 Jun 2024). Hyperparameters $\gamma, \lambda_{\mathrm{CE}}$ trade off unlearning aggression against utility loss.

2.3. Specialized Mechanisms

• Triplet-based TCU for structured bias unlearning: negatives are drawn exclusively from background or nuisance classes to target a specific confounder (e.g., background in sonar images) (S et al., 1 Dec 2025).
• Alignment Calibration: For contrastive learners, alignment calibration terms directly minimize positive-pair similarity on $\mathcal{U}$ and maximize negative (cross-pair) dissimilarity, plus auxiliary terms for auditing (alignment matrix, AGM) (Wang et al., 5 Jun 2024).
• Graph TCU: For graph neural networks, Node-level CUL alternates node-level embedding repulsion/attraction and neighborhood reconstruction to counteract the spread of unlearning via graph propagation (Lee et al., 4 Mar 2025).
• Inference-time TCU: In large LMs, inference-time approaches modify generation logits via the difference of forget- and retain-tuned auxiliary models (contrastive decoding), requiring no parameter updates on the main model (Suriyakumar et al., 12 Jun 2025).

3. Domain-specific Instantiations and Architectures

Vision and Contrastive Learning

TCU operates on feature extractor backbones (ResNet, EfficientNet, ViT) and is seamlessly integrated with self-supervised paradigms such as SimCLR, MoCo, and CLIP. The embedding-level TCU objective is agnostic to the projection head but depends on the specific pairing strategy (InfoNCE, triplet) and negatives sampling (Wang et al., 5 Jun 2024, S et al., 1 Dec 2025, Tong et al., 3 Aug 2025).

LLMs and Generative Networks

In transformer-based LMs, TCU is instantiated by shaping the geometry of the hidden-layer embedding space via focused contrastive losses (e.g., DeepCUT), by parameter-space interventions via task vectors or partitioned contrastive gradient updates (PCGU), or by deploying small auxiliary models to perform logit-space contrastive decoding at inference (He et al., 19 Mar 2025, Hu et al., 3 Feb 2025, Dige et al., 19 Jun 2024, Suriyakumar et al., 12 Jun 2025). Fine-grained activation manipulation (FALCON) combines layer-wise selection via information-theoretic measures (mutual information, PCA-KDE) and orthogonal projection of gradient directions to resolve forgetting-retention conflicts (Hu et al., 3 Feb 2025).

Graph-structured Data

Node-level CUL for GNNs introduces contrastive objectives to decouple the embeddings of unlearning nodes from their class and neighborhood, while a two-loop (representation unlearning and neighborhood reconstruction) schedule restores the local geometry of $k$-hop neighbors (Lee et al., 4 Mar 2025).

4. Auditing, Evaluation Metrics, and Practical Considerations

Unlearning effectiveness is assessed via accuracy drop or embedding alignment on the unlearned set ($\mathcal{U}$), preservation of utility (accuracy, F1, perplexity) on retained/test sets ($\mathcal{R},\mathcal{D}_{\mathrm{test}}$), and membership inference robustness (MIA AUC, positive-prediction rates) (Lee et al., 19 Jan 2024, Wang et al., 5 Jun 2024). For contrastive learners, novel metrics such as the Alignment Matrix (AM), Alignment Gap Matrix (AGM), and Forgetting Score (FS) provide encoder-level, per-sample, and visual auditability (Wang et al., 5 Jun 2024). Fine-grained explainability is enabled (e.g., via LIME) to visualize which features or contexts have been “forgotten” (S et al., 1 Dec 2025). For generative models, extraction likelihood, memorization accuracy, and response shift under adversarial prompting are reported (Tang et al., 25 Jul 2024, Hu et al., 3 Feb 2025, Suriyakumar et al., 12 Jun 2025).

In practice, TCU methods are highly efficient—experimental results show a 10$\times$–50$\times$ speedup over naive retraining while closing the accuracy gap to zero in most settings (Lee et al., 19 Jan 2024, Tran et al., 6 Aug 2025, Wang et al., 5 Jun 2024). Parameter selection (which layers to unlearn, which weights to update) is crucial. Saliency-guided masks (WSS-CL) or block-wise gradient contrastiveness (PCGU) focus updates for minimal utility loss (Tran et al., 6 Aug 2025, Dige et al., 19 Jun 2024). For inference-time TCU, the main tradeoff is the need for auxiliary models and increased compute at decode (Suriyakumar et al., 12 Jun 2025).

5. Comparative Empirical Results and Ablations

Method	Utility Retention	Forgetting Quality	MIA Resistance	Runtime / Cost
Exact Retraining	Optimal	Optimal	Optimal	High (full retrain)
TCU (general) (Lee et al., 19 Jan 2024)	≈Retrain	≈Retrain	Largest Gap	$10$–$50\times$ faster
Alignment Calibration (Wang et al., 5 Jun 2024)	≈Retrain	≈Retrain	High	$1.9$ min vs $109$ min
LetheViT (ViT) (Tong et al., 3 Aug 2025)	≈Retrain	Best AG (2.79%)	Improved	25 min vs 60 min
Triplet-TCU (S et al., 1 Dec 2025)	0.99 accuracy	Superior bias drop	N/A	—
Node-CUL (GNN) (Lee et al., 4 Mar 2025)	≈Retrain	Unlearn-score ≤3%	AUC ≈ 0.50	—
DeepCUT (LM) (He et al., 19 Mar 2025)	≈Retrain	Min. F1 on $\mathcal{U}$	Best	20–50% faster
WSS-CL (Vision) (Tran et al., 6 Aug 2025)	$>99\%$ RA	Gap to retrain 1.54	$16.8$ MIA	2–3 min vs 40+ min
UCD (Inference) (Suriyakumar et al., 12 Jun 2025)	≈Retrain	≈Retrain	Near-match	No full retrain

Empirical studies consistently show that TCU achieves strong unlearning (accuracy or F1 on $\mathcal{U}$ drops to random-guess) without meaningful degradation on $\mathcal{R}$, and overwhelmingly outperforms naive fine-tuning, gradient ascent, or influence-based baselines both in efficacy and compute (Lee et al., 19 Jan 2024, Wang et al., 5 Jun 2024, Tran et al., 6 Aug 2025).

Ablations confirm that removal of contrastive terms or masking mechanisms degrades both unlearning and utility, while using “soft” (sigmoid-scaled) saliency masks yields the best stability and performance (Tran et al., 6 Aug 2025). For attention-guided unlearning (LetheViT), masking more than 5% of high-attention patches destroys utility (Tong et al., 3 Aug 2025). For graph domains, omitting neighborhood reconstruction lowers generalization by $3$–$4\%$ (Lee et al., 4 Mar 2025).

6. Domain-Specific Applications and Limitations

• Debiasing and background removal: Triplet-based TCU explicitly disentangles object signals from confounding backgrounds (e.g., seafloor in sonar), resulting in explainable, interpretable activation changes corroborated by t-SNE and LIME analyses (S et al., 1 Dec 2025).
• Privacy and compliance: TCU operationalizes the “right to be forgotten” by provably breaking membership inference and erasing latent influence of forgotten samples (Lee et al., 19 Jan 2024, Wang et al., 5 Jun 2024, Wang et al., 12 May 2024, Lee et al., 4 Mar 2025).
• Scalability: Both parameter-efficient and inference-time TCU methods scale to LLMs and vision transformers, avoiding the prohibitive cost of exact retraining (Hu et al., 3 Feb 2025, Suriyakumar et al., 12 Jun 2025).
• Practical limit: Current formulations assume explicit identification of the unlearning set and may require access to model weights for training-based TCU. Most approaches provide no formal differential privacy or theoretical certification of unlearning beyond empirical attack resistance (Wang et al., 12 May 2024, Tang et al., 25 Jul 2024).

7. Open Problems and Future Directions

Outstanding challenges include deriving formal unlearning guarantees (differential privacy, certified removal bounds) for TCU, automating layer and parameter selection, handling continual or federated unlearning scenarios, and extending TCU to non-Euclidean data without explicit graph structure (Wang et al., 12 May 2024, Hu et al., 3 Feb 2025, Lee et al., 4 Mar 2025). There is active investigation into hybrid strategies (e.g., combining contrastive unlearning and parameter negation, or leveraging auxiliary model bootstrapping in inference-time TCU) to balance resource efficiency with robust, certifiable unlearning (Dige et al., 19 Jun 2024, Suriyakumar et al., 12 Jun 2025, Tang et al., 25 Jul 2024).

TCU remains a rapidly evolving methodology, central to privacy-preserving, auditable, and robust machine learning across modalities and architectures.