---
title: SISA Training for Efficient Machine Unlearning
url: https://www.emergentmind.com/topics/sisa-training
type: topic
---

# SISA Training for Efficient Machine Unlearning

SISA training refers to the Sharded, Isolated, Sliced, and Aggregated framework for efficient machine unlearning—enabling the precise and computationally tractable removal of individual data points from stateful models, especially those trained via stochastic gradient descent. SISA explicitly structures the training process so that the influence of any training example is confined to a limited, highly local segment of model state. SISA and its variants provide unlearning guarantees and superior efficiency relative to naïve retraining in diverse data domains including vision, speech, natural language, and multimodal mobility data [1912.03817][2212.09573][2506.02230][2508.19554].

## 1. Foundations and Motivation

Machine unlearning is mandated by privacy statutes such as the GDPR’s “right to be forgotten,” which require that, upon a user's request, all traces of their data be removed not only from storage but also from any trained models. Traditional deep learning pipelines are fundamentally stateful: every gradient step is globally entangled with prior data. Consequently, unlearning by retraining from scratch is computationally prohibitive at real-world scale, especially for models with $|D| = 10^8$ and frequent deletion requests [1912.03817].

SISA’s architectural innovation injects retraining-locality by partitioning the dataset into several *shards*, training each submodel *in isolation*, and implementing staged *slicing* within each shard to facilitate fine-grained, checkpointed state. Aggregation at inference combines the constituent models, retaining predictive performance while confining data influence.

## 2. Mathematical Formulation and Core Algorithms

Given a dataset $D = \{(x_i, y_i)\}_{i=1}^N$, choose
- $S$: the number of shards
- $R$: the number of slices per shard

Partition $D$ into $S$ disjoint shards $D_1, \ldots, D_S$, each of size $N/S$. Within each shard $D_k$, partition further into $R$ slices $D_{k,1}, \ldots, D_{k,R}$ with $|D_{k,r}| \approx N/(S R)$ [1912.03817][2212.09573][2508.19554].

### Training

For each shard $k$:
- Initialize model parameters $\theta_k^{(0)}$.
- For $r = 1, \dots, R$:
    - Train on slices $D_{k,1}, \ldots, D_{k,r}$; checkpoint $\theta_k^{(r)}$.
- The final constituent model is $M_k = \theta_k^{(R)}$.

The per-shard empirical risk is
$$
L_k(\theta_k) = \frac{1}{|D_k|} \sum_{(x,y) \in D_k} \ell(M_k(x;\theta_k), y)
$$
where $\ell(\cdot, \cdot)$ is the appropriate loss.

### Inference

Aggregate constituent models’ predictions:
- **Classification:** Majority vote or uniform average of softmax probabilities.
- **Regression:** Uniform mean over constituent outputs:
  $$
  \hat{y} = \frac{1}{S} \sum_{k=1}^S M_k(x; \theta_k)
  $$

### Unlearning

When a record $(x^*, y^*)$ is to be deleted:
- Identify shard and slice $(k^*, r^*)$ containing $x^*$.
- Reload $\theta_{k^*}^{(r^*-1)}$.
- Remove $x^*$ from $D_{k^*,r^*}$.
- Retrain from slice $r^*$ through $R$ (on $k^*$ only), replaying with $x^*$ excluded, and replace $M_{k^*}$.
- All other shards/models remain unchanged [1912.03817][2508.19554][2212.09573].

This procedure achieves *exact unlearning*: the resulting model state matches that which would have been obtained had $x^*$ never been in the training set.

## 3. SISA Variants and Extensions

Several SISA-based and inspired strategies exist for optimization under storage, memory, and performance constraints:

- **SISA-FC** ([SISA with Fully-Connected head]): Stores only FC-head parameters per checkpoint, yielding up to 95% memory reduction, but may incur substantial accuracy loss (20–30 points on GLUE tasks) [2212.09573].
- **SISA-A** ([SISA with Adapters]): Uses lightweight trainable adapters within a frozen backbone, balancing memory savings and retaining accuracy within 1–6 points [2212.09573].
- **MobText-SISA**: Employs similarity-aware sharding on latent embeddings for complex spatio-temporal and text mobility logs, using GMM clustering and round-robin assignment to preserve inter-shard diversity and ensure efficient rollback [2508.19554].
- **SISA++**: Advances SISA by replacing output-ensemble aggregation with weight-averaging of constituent model parameters at inference, empirically reducing post-unlearning performance drop by landing in a “flat” low-loss region of parameter space [2506.02230].

## 4. Empirical Results and Performance Analysis

Multiple empirical studies demonstrate SISA’s efficacy:

| Data Domain        | Task           | Shards × Slices | Model                               | Speed-up | ΔAccuracy (unlearning) | Reference        |
|--------------------|----------------|-----------------|-------------------------------------|----------|------------------------|------------------|
| Ecommerce/CV       | Classification | 20 × 50         | MLP, Wide ResNet, ResNet-50         | 4.6×     | –1.3 to –19.5 pp       | [1912.03817]     |
| NLP (GLUE SST, QQP, MNLI) | Classification | 5 × 16           | BERT+Adapter, BERT+FC              | 10–100×  | –1 to –6 pp (Adapter); –20 to –30 pp (FC head) | [2212.09573]     |
| Mobility           | Multimodal cls | 2–16 × 8        | MLP over numerical+BERT embeddings  | –        | +5–10% RMSE if random sharding vs similarity sharding | [2508.19554]     |
| Speech (CREMA-D, E-DAIC) | SER/DD         | 4–8 (K)          | TRILLsson+Transformer, x-vector, XLS-R, WavLM | –        | ΔAcc –2.3 (SISA++), –5.3 (SISA) | [2506.02230]     |

SISA consistently achieves 1.3–4.6× speed-ups in unlearning over retraining from scratch (higher when S and R are large), with only modest drops in accuracy (often recoverable by proper aggregation and downstream model selection). In speech and NLP tasks, SISA-A and SISA++ further reduce degradation post-unlearning while preserving computational efficiency [2212.09573][2506.02230].

## 5. Practical Guidelines and Trade-Offs

Best-practice recommendations include:

- **Shard/slice allocation:** Increasing $S$ and $R$ (smaller shards/slices) accelerates unlearning at the cost of possible accuracy loss. Empirically, $S = 5$ and $R = 8$–$16$ balance speedup and performance in NLP; for speech, $8 \leq K \leq 16$ is optimal [2212.09573][2506.02230].
- **Feature representations:** Use robust high-level embeddings (e.g., TRILLsson for SER/DD) for post-unlearning stability; avoid purely hand-crafted features (e.g. MFCC) when possible [2506.02230].
- **Downstream architectures:** Small Transformer encoders preserve global structure more stably than CNN or SVM for SISA-based post-unlearning scenarios [2506.02230].
- **Checkpoint management:** Storing $S \cdot R$ model or adapter checkpoints is feasible given their small size. Adapter-based or head-only checkpointing strongly reduces memory demand [2212.09573].
- **Aggregation at inference:** Averaging softmax probabilities across shard/slice models recovers accuracy lost to small shard sizes. Weight averaging (SISA++) further stabilizes performance post-unlearning [2506.02230].
- **Distribution-aware partitioning:** Probabilistically cluster high-deletion-probability points into small shards (“distribution-aware sharding”) to further concentrate retraining effort and reduce expected cost [1912.03817][2508.19554].

## 6. Applications and Domain-Specific Adaptations

SISA has been applied and/or extended in:

- **Computer Vision:** Canonical benchmarks (e.g., MNIST, SVHN, ImageNet) [1912.03817].
- **NLP:** GLUE tasks (SST, QQP, MNLI), leveraging either FC-head or adapter checkpoints [2212.09573].
- **Speech Processing:** SER (CREMA-D), Depression Detection (E-DAIC), with distinct embedding and architecture choices [2506.02230].
- **Mobility Analytics:** Urban-scale GPS and text logs with similarity-based sharding [2508.19554].

Each domain presents task-specific tradeoffs among checkpoint size, aggregation strategy, and feature robustness. The ability to apply SISA in a model-agnostic fashion (via checkpointing and consensus aggregation) underpins its extensibility.

## 7. Limitations and Theoretical Guarantees

SISA achieves *provable* exact unlearning: after removing $x^*$, the model distribution matches the counterfactual in which $x^*$ was never part of training, conditional on independence among shards and strict isolation during training and unlearning [1912.03817][2508.19554]. However, very small shards may induce model overfitting or excessive statistical inefficiency. Excessively granular checkpointing inflates disk storage, though this is mitigated by using lightweight adapters or FC-heads. Empirically, most performance loss from SISA arises when per-shard data is heavily subsampled relative to model capacity [2212.09573], but this can be controlled by task-specific hyperparameter tuning and proper inference aggregation.

In summary, SISA training—alongside its adapter- and weight-averaging-based variants—constitutes the dominant methodological framework for machine unlearning at scale across diverse modern ML domains, providing both operational guarantees and practical deployment efficiency [1912.03817][2212.09573][2506.02230][2508.19554].

Source: https://www.emergentmind.com/topics/sisa-training