---
title: 'ManiF-SMC: Manifold Forgetting with SMC'
url: https://www.emergentmind.com/topics/manif-smc
type: topic
---

# ManiF-SMC: Manifold Forgetting with SMC

Searching arXiv for ManiF-SMC and related machine unlearning papers.
ManiF-SMC, short for **Manifold Forgetting with Self Mode Connectivity**, is a label-agnostic method for **approximate machine unlearning** that operates entirely in a model’s learned representation space. Its central premise is that a model retrained only on retained data tends to classify erased samples according to their **semantic similarity** to the retained set; ManiF-SMC therefore reformulates forgetting as a geometric operation that pushes each erased sample away from its original representation while pulling it toward the centroid of its nearest retained semantic neighbors. The method realizes this objective with a **margin-based triplet loss** whose margin is not fixed in advance, but is adaptively estimated by a **self mode connectivity** module that reconstructs the local retained manifold through a low-loss path in weight space [2605.22871].

## 1. Problem setting and geometric motivation

The method is defined on a training set $D$ partitioned into an **erased set** $D_e$ and a **retained set** $D_r$, with $D = D_e \cup D_r$ and $D_e \cap D_r = \varnothing$. A trained encoder $f_\theta$ maps an input $x$ to a representation $z = f_\theta(x) \in \mathbb{R}^d$. The paper denotes the original model parameters by $\theta_o$ and the post-unlearning parameters by $\theta_u$, with corresponding embeddings
$$
z_{i,o} = f_{\theta_o}(x_i), \qquad z_{i,u} = f_{\theta_u}(x_i).
$$
For each erased sample $x_i \in D_e$, ManiF-SMC identifies a set $S_k^i \subseteq D_r$ of the **top-$k$ most similar retained samples**, obtained by $k$-NN search in the original representation space. The local retained-neighbor centroid under the unlearned model is defined as
$$
c_{i,u} = \frac{1}{|S_k^i|}\sum_{x_k^i \in S_k^i} f_{\theta_u}(x_k^i),
$$
with an analogous centroid $c_{i,o}$ under the original model [2605.22871].

The method is motivated by the empirical observation that, after retraining from scratch on $D_r$, erased samples tend to be embedded near the manifold regions supported by **similar retained samples**, and their assignment follows the nearest semantic cluster. This motivates a representation-space reinterpretation of unlearning: instead of reversing task gradients or manipulating labels, the model should alter the geometry of erased embeddings so that they no longer occupy their original manifold region and instead align with local retained structure. This suggests a retraining-aligned notion of forgetting in which erased representations are **absorbed into nearby retained semantics** rather than merely perturbed.

## 2. Representation-space reformulation and triplet objective

ManiF-SMC decomposes approximate unlearning into two forces. The first is a **push term**, which moves the erased sample away from its original embedding:
$$
\max_{\theta_u}\;\sum_{x_i \in D_e}\big\| f_{\theta_u}(x_i)-z_{i,o}\big\|^2.
$$
The second is a **pull term**, which brings the erased sample toward the centroid of its nearest retained neighbors:
$$
\min_{\theta_u}\;\sum_{x_i \in D_e}\big\| f_{\theta_u}(x_i)-c_{i,u}\big\|^2.
$$
These two objectives are encapsulated in a **margin-based triplet loss** that treats each erased sample as the anchor, the retained-neighbor centroid as the positive, and the original representation as the negative:
$$
\mathcal{L}_{\text{triplet}}(\theta_u)
=
\sum_{x_i\in S_u}
\Bigl[
\mathrm{dist}\bigl(f_{\theta_u}(x_i),c_{i,u}\bigr)
-
\mathrm{dist}\bigl(f_{\theta_u}(x_i),z_{i,o}\bigr)
+
\alpha
\Bigr]_+,
$$
where $[\cdot]_+ = \max\{\cdot,0\}$ and $\mathrm{dist}(\cdot,\cdot)$ is typically Euclidean distance [2605.22871].

Minimizing this loss enforces the constraint
$$
\mathrm{dist}\bigl(f_{\theta_u}(x_i),c_{i,u}\bigr)+\alpha
\le
\mathrm{dist}\bigl(f_{\theta_u}(x_i),z_{i,o}\bigr),
$$
so the erased embedding becomes at least $\alpha$ closer to the retained-neighbor centroid than to its original location. In this formulation, the margin $\alpha$ is a lower bound on the amount of representation relocation induced by unlearning.

A notable aspect of the formulation is what it does **not** include. The paper does not introduce an explicit retention penalty over the full retained set, such as a term of the form $\sum_{x\in D_r}\|f_{\theta_u}(x)-f_{\theta_o}(x)\|^2$. Instead, retained manifold preservation is delegated to the self mode connectivity mechanism. This is important because ManiF-SMC aims to remain **label-agnostic** and to avoid direct dependence on task-specific gradients or full retraining on $D_r$ [2605.22871].

## 3. Self mode connectivity and adaptive margin generation

The **SMC** component is the mechanism that supplies retained-manifold structure to the unlearning process. It constructs a low-loss path in weight space between the current unlearning model $\theta_u$ and the original model $\theta_o$ using only retained-neighborhood data. The path is parameterized as a **quadratic Bézier curve**
$$
\phi_w(t) = (1-t)^2\theta_u + 2t(1-t)w + t^2\theta_o,\qquad 0\le t\le 1,
$$
where $w$ is a learnable control point [2605.22871].

The control point is optimized by minimizing the retained loss along the path:
$$
\min_w\ \mathbb{E}_{t\sim \mathcal{U}[0,1]}\big[\mathcal{L}_{S_k}(\phi_w(t))\big],
$$
where $S_k = \bigcup_{x_i\in D_e} S_k^i$ is the union of retained neighborhoods relevant to the erased set. A surrogate model is then sampled at a fixed location $t^\star$, empirically chosen as $t^\star = 0.5$:
$$
\tilde{\theta}=\phi_w(t^\star).
$$

This surrogate model serves two roles. First, it estimates the retained-neighbor centroid used during unlearning:
$$
c_{i,\tilde{\theta}} = \frac{1}{|S_k^i|}\sum_{x_j\in S_k^i} f_{\tilde{\theta}}(x_j).
$$
Second, it produces a **sample-wise adaptive margin**
$$
\alpha_{\tilde{\theta}}(x_i,c_{i,\tilde{\theta}},z_{i,o})
=
\Big[
\mathrm{dist}\big(f_{\tilde{\theta}}(x_i),z_{i,o}\big)
-
\mathrm{dist}\big(f_{\tilde{\theta}}(x_i),c_{i,\tilde{\theta}}\big)
\Big]_+.
$$
The triplet loss is then updated to
$$
\mathcal{L}_{\text{triplet}}(\theta_u)
=
\sum_{x_i\in S_u}
\Big[
\mathrm{dist}\big(f_{\theta_u}(x_i),c_{i,\tilde{\theta}}\big)
-
\mathrm{dist}\big(f_{\theta_u}(x_i),z_{i,o}\big)
+
\alpha_{\tilde{\theta}}(x_i,c_{i,\tilde{\theta}},z_{i,o})
\Big]_+.
$$

This adaptive construction replaces a globally fixed margin with one that reflects the **local geometry** of each erased sample relative to its retained neighborhood. The paper further gives a smoothness result: if $g(\theta,x)$ is $L_x$-Lipschitz in $\theta$, then for $\tilde{\theta}=\phi_w(t^\star)$,
$$
\bigl|g(\tilde{\theta},x)-g(\theta_o,x)\bigr|
\le
L_x\Big((1-t^\star)^2\|\theta_u-\theta_o\|_2 + 2t^\star(1-t^\star)\|w-\theta_o\|_2\Big).
$$
This supports the claim that retained predictions and neighbor representations vary smoothly along the SMC path, making $\tilde{\theta}$ a stable estimator of local retained geometry [2605.22871].

## 4. Optimization procedure and implementation profile

The algorithm proceeds in two alternating stages. First, it learns the SMC control point $w$ using mini-batches drawn from $S_k$ and the retained training loss. Second, it updates $\theta_u$ on mini-batches from the erased set using the adaptive triplet objective. The paper’s procedure is:

1. Initialize $\theta_u \leftarrow \theta_o$ and initialize the Bézier control point $w$.
2. Cache the original erased-sample embeddings $z_{i,o} = f_{\theta_o}(x_i)$.
3. Train the SMC path by sampling $t \sim \mathcal{U}[0,1]$, forming $\theta_t = \phi_w(t)$, computing the retained loss on batches from $S_k$, and updating $w$.
4. Sample the surrogate model $\tilde{\theta} = \phi_w(t^\star)$ with $t^\star = 0.5$.
5. For each erased mini-batch, compute $c_{i,\tilde{\theta}}$, compute $\alpha_i$, accumulate the hinge losses, and update $\theta_u$ by backpropagation.
6. Return $\theta_u$ [2605.22871].

Several implementation properties define the method’s scope. The default distance is **Euclidean**, though cosine similarity and NT-Xent-style variants are also evaluated. Neighbor sets are computed once in the original representation space, and the paper notes that **approximate nearest neighbors** can be used for scalability. The method updates the encoder parameters directly and requires **no labels** and **no task-specific gradients**. Complexity is governed by the number of erased samples $|D_e|$, the neighbor count $k$, the neighborhood union size $|S_k|$, and the cost of SMC path training; in practice, this is far below the cost of full retraining [2605.22871].

The paper evaluates ManiF-SMC with representation dimensions matched to dataset complexity: $D_M=5$ for MNIST, $D_M=64$ for CIFAR10 and CelebA, and $D_M=256$ for Tiny-ImageNet. Minibatch sizes are $16$ for MNIST, CIFAR10, and CelebA, and $200$ for Tiny-ImageNet. Experiments are run on **NVIDIA Quadro RTX 6000 GPUs** [2605.22871].

## 5. Empirical behavior, ablations, and applications

The evaluation covers **MNIST**, **CIFAR10**, **CelebA**, and **Tiny-ImageNet**, using **Membership Inference Attack (MIA)** success rate on $D_e$, **Remaining Accuracy (RA)** on $D_r$, **Test Accuracy (TA)** on the test set, **Remaining MSE (R-MSE)** and **Test MSE (T-MSE)** for generative models, and **Running Time (RT)**. Baselines are **Retraining**, **Gradient Ascent (GA)**, **Variational Bayesian Unlearning (VBU)**, **Representation Forgetting Unlearning (RFU)**, and **SalUn** [2605.22871].

On CIFAR10 with **1% (500) erased samples**, the paper reports that ManiF-SMC preserves compact class structure and moves erased samples toward their most semantically similar retained clusters, resembling retraining and SalUn while requiring **no labels and no retained fine-tuning**. Across unlearning sample sizes, MIA rises for all methods, but ManiF-SMC preserves RA and TA better than VBU while remaining label-agnostic. SalUn preserves utility best at higher unlearning sample sizes because it includes retained fine-tuning; the paper notes that ManiF-SMC can also add retained fine-tuning, but doing so reintroduces label dependence [2605.22871].

Runtime is a major differentiator. Retraining is reported as the slowest method by **2–3 orders of magnitude**, in the range of $10^2$–$10^4$ seconds depending on dataset, whereas ManiF-SMC, VBU, and SalUn complete under approximately **$1$–$12$ seconds** across the tested settings. For example, with **USS=200** and MMCRs on MNIST, ManiF-SMC obtains **MIA 62.00**, **RA 99.59**, **TA 99.15**, and **RT 1.482 s**; on CIFAR10 with the same setting, it obtains **MIA 59.00**, **RA 99.04**, **TA 81.75**, and **RT 2.372 s** [2605.22871].

The ablation on **adaptive SMC margins versus fixed margins** shows consistent gains in both unlearning effectiveness and utility, particularly at larger unlearning sample sizes. On MNIST with **USS=1000**, the adaptive version changes **MIA 59.60→63.39**, **RA 98.93→99.43**, **TA 98.87→99.11**, and **RT 6.53→7.56 s**. On CIFAR10 with **USS=1000**, it changes **MIA 65.33→69.47**, **RA 89.80→94.91**, **TA 72.34→77.00**, and **RT 10.25→11.83 s** [2605.22871].

The method is also applied to **generative models (VAE unlearning)**. On MNIST and CIFAR10, forgetting effectiveness increases with unlearning sample size while **R-MSE** and **T-MSE** degrade only modestly. For instance, on CIFAR10, **MIA 53.83% (orig) → 54.00%, 58.52%, 57.59%, 61.15%, 66.99%** as USS increases from $200$ to $1000$, while **R-MSE 0.00192→0.00192,0.00194,0.00197,0.00213,0.00223** and **T-MSE 0.00192→0.00192,0.00195,0.00199,0.00216,0.00224** [2605.22871].

A further application concerns **limited label access** in semantic communication encoders. On CelebA, only ManiF-SMC remains applicable when the decoder and task labels are not accessible, and it achieves the **same performance whether or not the decoder is available**: **MIA 54.50**, **RA 96.34**, **TA 95.93** [2605.22871].

The main hyperparameter trends are explicit. **Smaller $k$** strengthens forgetting but may reduce utility; **larger $k$** improves RA and TA but reduces MIA. On CelebA, increasing $k$ from $5$ to $10$ changes **MIA 54.50→53.00**, **RA 96.34→96.48**, and **TA 95.93→96.19**. **Euclidean distance** yields stronger unlearning than cosine similarity: on MNIST with **USS=200**, **L2** gives **MIA 62.00**, **RA 99.59**, **TA 99.15**, **RT 1.482 s**, whereas **Cosine (NT-Xent)** gives **MIA 60.00**, **RA 99.38**, **TA 99.04**, **RT 2.082 s**. For the SMC sampling position, **$t^\star = 0.5$** gives the best MIA in the reported ablation, with nearby values trading off forgetting against utility [2605.22871].

## 6. Scope, limitations, and terminology

ManiF-SMC is an **approximate** unlearning method, not an exact substitute for retraining. The paper is explicit that it can achieve a strong approximation to retraining behavior at a fraction of the computational cost, but it does not provide the full fidelity guarantees of exact retraining. Verification and auditability are identified as open problems [2605.22871].

Its limitations are tied to the geometry of the learned representation space. If manifolds are **highly entangled** or exhibit high centroid correlation, nearest-neighbor centroids may not reflect retraining behavior, and the triplet constraints may become less effective. The method also assumes that nearest semantic neighbors in $D_r$ are meaningful under the current representation; large **distribution shifts** can invalidate this assumption. The paper notes that further study is needed for **text, graphs, or other modalities** beyond the evaluated visual settings [2605.22871].

A separate limitation is that retention is enforced only **implicitly**. The method does not include an explicit penalty over the retained set; instead, it relies on SMC-guided centroid and margin estimation to preserve local retained manifold structure. This design keeps the method label-agnostic, but it also means utility preservation depends on the quality of the surrogate geometry reconstructed from $S_k$. The paper notes that adding explicit retained fine-tuning can mitigate RA and TA degradation, though this slightly reduces MIA and reintroduces label dependence [2605.22871].

The acronym **SMC** in ManiF-SMC denotes **Self Mode Connectivity**. It is distinct from the unrelated **SMC/SMC++** abbreviation used for **Semantic-Mining-then-Compression** in unsupervised video semantic compression, where “SMC++” refers to a masked-video-modeling-based codec rather than a machine-unlearning method [2406.04765]. This distinction matters because the two lines of work share the letters “SMC” but address different problems, use different objectives, and define the acronym differently.

In the form presented in the paper, ManiF-SMC is best understood as a representation-space unlearning framework whose defining contribution is the coupling of **manifold-based triplet forgetting** with **self-mode-connectivity-based adaptive margin estimation**. Its empirical profile is characterized by label agnosticism, strong runtime advantages over retraining, and unlearning effectiveness comparable to state-of-the-art approximate methods, with performance governed by the separability of the underlying learned manifold and the fidelity of the retained-neighborhood surrogate [2605.22871].

Source: https://www.emergentmind.com/topics/manif-smc