LetheViT: Selective Unlearning for ViTs

• LetheViT is a selective machine unlearning method for Vision Transformers that uses attention-guided masking to erase sample-specific details while retaining class-level features.
• It employs a two-phase fine-tuning process combining contrastive loss for forgotten samples and cross-entropy on retained data to balance privacy compliance with performance.
• Empirical evaluations show competitive test accuracy and reduced membership-inference attack success rates with only a modest runtime overhead compared to retraining.

Searching arXiv for the specified paper to ground the article. {"queries":[{"term":"(Tong et al., 3 Aug 2025) LetheViT","authors":[],"start_date":"2025-01-01","end_date":"2026-12-31","categories":["cs.CV","cs.LG"]}]} Found the paper on arXiv and will use it as the primary source for the encyclopedia entry. LetheViT is a selective machine unlearning method for Vision Transformers (ViTs) that targets the random-data-forgetting setting, in which specific samples must be forgotten while other samples from the same class are retained. It is formulated as an approximate unlearning approach motivated by privacy regimes such as GDPR and CCPA, which require not only deletion of user data but removal of that data’s influence from trained models. The method combines attention-guided masking with a contrastive objective so that sample-specific details are suppressed while class-level structure is preserved, and it is presented as the first method to leverage ViT self-attention for selective unlearning at the sample level (Tong et al., 3 Aug 2025).

1. Problem setting and motivation

LetheViT is situated in machine unlearning for ViTs, where the central requirement is to forget withdrawn data without sacrificing the utility of the model on retained data. The paper distinguishes between exact unlearning and selective approximate unlearning. Exact unlearning corresponds to retraining the model from scratch on the remaining data; for large ViTs, this is described as prohibitively expensive. Selective approximate unlearning instead aims to remove the model’s memory of only the withdrawn samples while preserving performance on the rest (Tong et al., 3 Aug 2025).

A central distinction in the formulation is between class-wise forgetting and random-data forgetting. In class-wise forgetting, an entire class is removed, such as all images of a given category. In random-data forgetting, only a random subset of samples is removed, potentially from within one or more classes, while the model must continue to recognize other samples from those same classes. The latter is identified as particularly challenging for ViTs because samples within a class are often visually very similar, making it difficult to erase a few instances without collateral degradation. This suggests that unlearning at the sample level requires a mechanism that can decouple class-level content from instance-level memorization.

2. Empirical basis: selective masking in ViTs

The empirical premise of LetheViT comes from selective-masking experiments on DeiT-T with CIFAR-100. In those experiments, 10% of training images are randomly forgotten, and at test time the top $k\%$ of patches ranked by attention score are masked. Two quantities are then measured: Test Accuracy (TA), which tracks class recognition capability, and MIA success rate, which tracks memorization via membership-inference attack.

Mask ratio	Zero-mask: TA / MIA	Gaussian-noise: TA / MIA
0%	81.24% / 24.49%	81.24% / 24.49%
5%	81.25% $(+0.01)$ / 10.16% $(-14.33)$	83.59% $(+2.35)$ / 14.06% $(-10.43)$

These results are interpreted in the paper as evidence that masking high-attention regions strips away sample-specific details while preserving class-level outlines. The notable point is that masking the highest-attention patches does not substantially reduce TA, but it does sharply reduce MIA success. A common intuition would be that suppressing highly attended regions should impair recognition first; the reported behavior indicates the opposite pattern in this setting, namely that memorization is more vulnerable than recognition under this intervention (Tong et al., 3 Aug 2025).

3. Information-theoretic formulation

The method is given an information-theoretic interpretation in terms of class-level content and sample-specific information. Let $\mathcal{C}$ denote class-level content, $\mathcal{S}$ denote sample-specific details, $\mathcal{Z}_n = f_{\theta_0}(x)$ denote the representation of the original image, and $\mathcal{Z}_p = f_{\theta_0}(x_m)$ denote the representation of the masked image.

The key assumption is the attention-masking assumption: $$I(\mathcal{Z}_p;\,\mathcal{S}\mid \mathcal{C}) \approx 0 \quad\text{and}\quad I(\mathcal{Z}_n;\,\mathcal{S}\mid \mathcal{C}) > 0.$$ This states that after masking, little sample-specific information remains, whereas the unmasked representation still contains such information.

Under this assumption, the paper states an InfoNCE-bound result for the contrastive loss

$(+0.01)$0

Minimizing this loss is described as simultaneously maximizing $(+0.01)$1 and minimizing $(+0.01)$2. The stated consequence is

$(+0.01)$3

which formalizes the intended behavior: forgetting sample-specific details while keeping class-level features. In the paper’s framing, the masked branch acts as a proxy for class-only information, and the original unmasked branch acts as a carrier of both class and sample information.

4. Attention-guided masking and model integration

LetheViT builds on a standard pre-trained ViT, with examples including DeiT and Swin, and performs selective fine-tuning in two phases. The masking mechanism is derived from the last-layer attention tensor

$(+0.01)$4

For each patch $(+0.01)$5, the class-token-to-patch attention score is computed as

$(+0.01)$6

The top

$(+0.01)$7

patches are then selected by descending $(+0.01)$8, and the masked image $(+0.01)$9 is produced by zeroing those patches’ pixels (Tong et al., 3 Aug 2025).

The model parameters are organized around an original pre-trained parameter set $(-14.33)$0 and an unlearned model initialized as $(-14.33)$1. During unlearning, $(-14.33)$2 is updated using contrastive loss on the forget set and cross-entropy on the retain set. This two-phase design is meant to separate the forgetting operation from subsequent retention recovery, while still remaining within a fine-tuning regime rather than a full retraining regime.

5. Contrastive objective and algorithmic workflow

For each forget image $(-14.33)$3, LetheViT constructs three representations. The anchor is

$(-14.33)$4

computed by the current model. The positive is

$(-14.33)$5

computed by the original model on the masked image. The negative is

$(-14.33)$6

computed by the original model on the original image. The same contrastive objective is then applied, with cosine similarity

$(-14.33)$7

and temperature $(-14.33)$8. The stated effect is to pull $(-14.33)$9 toward $(+2.35)$0 and push it away from $(+2.35)$1, thereby forgetting instance details while keeping class contour.

The algorithmic workflow is presented in two phases. In the first phase, unlearning is performed on the forget set $(+2.35)$2 for $(+2.35)$3 epochs: attention is computed, the top-$(+2.35)$4 patches are masked to produce $(+2.35)$5, the three representations are formed, the contrastive loss is evaluated, and $(+2.35)$6 is updated by gradient descent with learning rate $(+2.35)$7. In the second phase, retention is performed on the retain set $(+2.35)$8 for $(+2.35)$9 epochs using cross-entropy loss on labeled samples $(-10.43)$0. The output is the unlearned model $(-10.43)$1.

This design makes the original model $(-10.43)$2 serve as a fixed reference during unlearning. A plausible implication is that the method depends not only on the current model state but also on preserving access to the pre-unlearning representation geometry.

6. Experimental evaluation and reported performance

The empirical study covers CIFAR-10/100, SVHN, and Tiny-ImageNet, using ViT-T/S/B, DeiT-T/S/B, and Swin-T/S. Baselines include Retrain, Fine-Tune (FT), Gradient Ascent (GA), Influence Unlearning (IU), Random Labels (RL), $(-10.43)$3-sparse pruning, and SalUn. Evaluation uses five metrics: FA, the accuracy on forgotten samples; RA, the accuracy on retained samples; TA, the test accuracy on a hold-out set; MIA, membership attack success; and AG, the average gap from Retrain across the four metrics, where lower is better (Tong et al., 3 Aug 2025).

For Tiny-ImageNet with 10% forget, the paper reports that LetheViT achieves the lowest AG across ViT-T/S/B. The excerpted result is:

Method	FA / RA / TA / MIA	AG
Retrain (gold)	78.89 / 95.77 / 79.58 / 35.78	0.00
LetheViT	80.09 / 91.55 / 80.26 / 36.81	1.78

The paper gives a specific example on ViT-T, where LetheViT’s AG is $(-10.43)$4 versus SalUn’s $(-10.43)$5, and states that similar gains hold across ViT-S/B, DeiT-T/S/B, and Swin-T/S. Ablation results report that a 5% mask ratio yields the best AG, zero-mask outperforms Gaussian noise injection, and LetheViT adds only approximately 20% runtime overhead to FT, remaining far below retraining.

7. Interpretation, limitations, and research directions

The paper attributes LetheViT’s performance to three components. First, attention-guided masking is said to identify informative patches that carry sample details while leaving class cues sufficiently intact. Second, the contrastive objective explicitly aligns the unlearned representation with class-only features and repels it from full features. Third, the two-phase training procedure is intended to balance forgetting and retention in a single fine-tuning pipeline (Tong et al., 3 Aug 2025).

The stated limitations are also precise. The method assumes access to the original ViT’s attention maps and logits. Certified guarantees remain an open question. Proposed future directions include extension to other architectures such as hybrid CNN-Transformer systems, federated scenarios, and stronger privacy metrics.

Within the paper’s own positioning, LetheViT addresses a specific weakness of approximate unlearning methods in ViTs: the difficulty of removing individual samples without eroding intra-class generalization. Its core claim is not that forgetting can be made exact, but that attention-guided contrastive learning yields a more favorable balance between privacy compliance and model utility in the random-data-forgetting regime.