Machine Unlearning: Features & Labels

• Machine unlearning of features and labels is defined as a methodology for precisely removing specific data influences from models to mimic full retraining outcomes.
• It employs influence functions and independence regularizers to adjust model parameters, addressing distribution shifts from selective feature or label removal.
• Practical applications include data privacy compliance and error correction, while challenges remain in scaling unlearning for deep non-convex architectures.

Machine Unlearning of Features and Labels encompasses a family of algorithmic frameworks and mathematical techniques for efficiently erasing the influence of specific feature dimensions or label information from machine learning models. Motivated by regulatory imperatives (e.g., GDPR’s right to erasure) and the technical demands of handling data leaks or distributional shifts, recent work operationalizes unlearning via parameter-space corrections, optimization-based fine-tuning, and dependence-aware regularization. As distinct from naive retraining, state-of-the-art approaches deliver robust removal of both explicit and implicit information encoded by training instances, at a fraction of conventional computational cost and with provable accuracy retention under adversarial or distributional drift.

1. Problem Definition and Motivations

Machine unlearning of features and labels formalizes the task of editing model behavior such that predictions become statistically indistinguishable from those of a model retrained on a suitably “forgotten” dataset. Let the training set be $\mathcal{D} = \{(X_i, Y_i)\}_{i=1}^n$, with $X_i$ feature vectors (potentially high-dimensional) and $Y_i$ their associated labels. An unlearning request specifies a subset $\Delta\mathcal{D} \subset \mathcal{D}$, which may target:

• Entire data points (instance-level unlearning),
• Particular feature dimensions across points (feature-level unlearning),
• Labels or associations (label-level unlearning).

Feature removal generally entails erasing or zeroing out targeted dimensions, while label removal may involve label reassignment, discarding, or otherwise severing input–output associations. Due to non-uniformity of requests (e.g., selective feature deletion), such operations induce distributional shifts in both the marginal $\mathcal{F}(\mathcal{D})$ and the conditional $P(Y|X)$, which must be robustly addressed to avoid model degradation (Han et al., 2024).

2. Mathematical Formulations and Influence-Based Methods

The dominant formalism for unlearning features and labels rests on influence functions and first-order (gradient) or second-order (Hessian) approximations. Denoting the original model parameters as $\hat{\theta}$ (minimizing empirical risk on $\mathcal{D}$), and its Hessian by $H_{\hat{\theta}}$, the effect of removing or altering a batch $\Delta\mathcal{D}$ can be written:

$X_i$0

where $X_i$1 is the data loss, and $X_i$2 is the independence criterion regularizer (discussed below). This update simultaneously subtracts the direct contribution of the forgotten data and corrects for distributional changes (Han et al., 2024, Warnecke et al., 2021). In particular, (Warnecke et al., 2021) demonstrates that for strongly convex settings, the corrected parameter $X_i$3 is first-order equivalent to retraining on the restricted set, with residual error $X_i$4.

3. Independence Criteria and Distributional Shift Compensation

Standard influence-based unlearning fails under substantial distributional drift—especially when feature/label removal causes drastic changes to $X_i$5. To monitor and penalize unwanted dependence, recent frameworks introduce independence regularizers:

• Mutual Information (MI): $X_i$6
• Hilbert–Schmidt Independence Criterion (HSIC): $X_i$7, with $X_i$8 a centered kernel matrix

The unlearning objective becomes:

$X_i$9

with $Y_i$0 denoting (e.g.) Euclidean distance (Han et al., 2024). Penalizing divergence from pre-unlearning dependence better preserves model calibration and generalization under shifting data regimes (e.g., top-$Y_i$1 feature deletions).

4. Algorithms and Implementation Considerations

An influence–independence hybrid algorithm proceeds as follows:

1. Compute Hessian $Y_i$2 loss on pre-unlearning data
2. For each $Y_i$3 in $Y_i$4, calculate (a) loss gradients, (b) independence-shift gradients ($Y_i$5)
3. Aggregate and form the update direction
4. Solve $Y_i$6gradient (e.g., via conjugate gradients / Hessian-vector products)
5. Return Model with $Y_i$7

These steps ensure computational cost remains 10–20× lower than full retraining while maintaining post-unlearning performance on retained data (Han et al., 2024).

Summary Table: Empirical Performance Under Top-$Y_i$8 Feature/Label Removals

Unlearn Ratio	Method	Cora/GIN F1	Runtime (s)	MNIST/CNN F1	Runtime (s)
5%	Retrain	0.8057	8.31	0.9587	198.9
5%	IF	0.7738	0.48	0.8978	4.38
5%	DUI	0.7868	0.99	0.9433	8.75

DUI denotes the dependence-penalized unlearning framework (Han et al., 2024). DUI approaches or matches retraining accuracy and substantially outperforms standard IF as distributional shift grows.

5. Theoretical Guarantees and Limitations

Approximate equivalence to retraining can be certified under standard smoothness and strong convexity assumptions. Analyses in (Warnecke et al., 2021, Han et al., 2024) provide tight bounds on the residual parameter error and gradient norm. However, non-convexity (as in deep networks) may invalidate these guarantees; in practice, Hessian approximations or diagonalizations are necessary for scalability. Additional limitations include sensitivity to kernel and hyperparameter choices in independence measures, and challenges in guaranteeing information removal in highly entangled or adversarial data subsets.

6. Extensions, Practical Applications, and Future Directions

• Extensions: Incorporation of differential privacy for certified unlearning, continual (incremental) unlearning algorithms, and exploration of alternative independence criteria (e.g., distance correlation, adversarial discriminators) (Han et al., 2024). Layer-wise partial unlearning and sharpness-aware parameter selection represent additional refinements for scalability and targeted forgetting (Gogineni et al., 2024, Malekmohammadi et al., 8 Apr 2025).
• Applications: Regulatory compliance (e.g., GDPR, CCPA), correction of training set errors, memorization/unmemorization of sensitive features (credit card numbers, faces), and model maintenance under streaming unlearning requests.
• Open Challenges: Efficient Hessian computation in large non-convex nets, formal removal certificates for deep architectures, adaptive kernel/regularization selection, and robust auditing of lingering feature/label information post-unlearning.

7. Empirical Validation and Evaluation Metrics

Comprehensive experiments utilize MNIST, Cora, Citeseer, and other benchmarks; metrics include classification/F1 accuracy on retained and forgotten subsets, runtime, membership inference attack (MIA) resistance, and Brier scores for probability calibration. Under both random and adversarial removal scenarios, dependence-penalized and influence-based methods consistently achieve F1 and accuracy gaps to retraining of less than 2% at orders-of-magnitude speedup, confirming both practical efficiency and statistical reliability (Han et al., 2024, Warnecke et al., 2021).