Conformal Unlearning in Machine Learning

• Conformal Unlearning is a framework that employs conformal prediction to provide statistical guarantees for excluding the influence of specified forgotten data.
• It leverages risk-optimized paradigms like FROC and conformal loss formulations to balance forgetting precision with the retention of model utility.
• Empirical studies show reduced privacy risks and improved regulatory compliance, making this approach vital for safety-critical deployments.

Conformal Unlearning refers to a body of machine unlearning methodologies that incorporate conformal prediction as a foundation for principled, uncertainty-aware, and risk-controlled removal of specific data influences from machine learning models. These approaches provide statistical guarantees regarding the exclusion of forgotten data while maintaining model utility on retained data, offering a solution to the insufficiencies of traditional unlearning metrics and heuristics, especially for regulatory and safety-critical deployments of large-scale models.

1. Foundational Principles of Conformal Unlearning

Conformal unlearning explicitly reconceptualizes the unlearning task in terms of coverage and risk guarantees derived from conformal prediction theory. Rather than focusing only on pointwise metrics such as unlearning accuracy (UA) or canonic membership inference attack (MIA) rates, conformal unlearning asks: with what probability is a forgotten point's true label excluded from the conformal prediction set of the (post-unlearning) model, and with what probability does a retained point's label remain covered?

In the setting where a model $p_\theta$ is trained on dataset $\mathcal T$ and a forget set $\mathcal D_f \subset \mathcal T$ is specified, conformal unlearning seeks to modify or post-process the model into $p_{\theta'}$ such that, with high probability:

• For $(x,y) \in$ forget set, $y \notin C_{\theta'}(x)$ (the conformal prediction set at specified risk $\alpha$).
• For $(x,y) \in$ retain set, $y \in C_{\theta'}(x)$ with at least $1-\alpha$ probability (Alkhatib et al., 5 Aug 2025).

Such coverage-based guarantees align the model's predictive uncertainty with the unlearning target, offering explicit, verifiable forgetting behavior and utility preservation.

2. Formal Definitions and Conformal Metrics

Conformal unlearning is formalized via the following statistical definitions:

• $\mathcal T$0-Conformal Unlearning: An update $\mathcal T$1 achieves $\mathcal T$2-conformal unlearning if

$\mathcal T$3

where $\mathcal T$4 is the conformal set at level $\mathcal T$5 (Alkhatib et al., 5 Aug 2025).

• Conformal Ratio (CR): For any set $\mathcal T$6, $\mathcal T$7, penalizing high residual coverage on the forget set. Lower CR on $\mathcal T$8 corresponds to stronger forgetting (Shi et al., 31 Jan 2025).
• MIA Conformal Ratio (MIACR): In MIA, $\mathcal T$9, quantifying the fraction of forgotten points confidently marked as non-members.
• Efficiently Covered/Uncovered Frequency: For retain/forget test points whose CP-set size is at most $\mathcal D_f \subset \mathcal T$0, $\mathcal D_f \subset \mathcal T$1 and $\mathcal D_f \subset \mathcal T$2 respectively estimate the achievable $\mathcal D_f \subset \mathcal T$3-conformal unlearning rates (Alkhatib et al., 5 Aug 2025).
• Conformal Unlearning Risk (CUR): A data-driven, distribution-free upper bound, $\mathcal D_f \subset \mathcal T$4, calibrated (via large deviation or binomial tail inequalities) so that

$\mathcal D_f \subset \mathcal T$5

for a specified empirical risk $\mathcal D_f \subset \mathcal T$6 and risk budget $\mathcal D_f \subset \mathcal T$7 (Goh et al., 15 Dec 2025).

3. Algorithms and Paradigms

Conformal unlearning frameworks have diverged into three main algorithmic paradigms:

a) Risk-Optimized Conformal Unlearning (FROC)

The FROC framework for LLMs establishes a continuous risk score $\mathcal D_f \subset \mathcal T$8 unifying forgetting deficiency and utility degradation, then calibrates this with conformal risk analysis to enforce a probability-based constraint:

$\mathcal D_f \subset \mathcal T$9

Hyperparameters $p_{\theta'}$0 are selected by minimizing the Conformal Unlearning Risk (CUR), systematically balancing memory erasure and utility preservation under user-specified risk budgets. FROC precomputes a grid of configurations, calibrates risk via empirical sampling, and admits optional Bonferroni correction for simultaneous parameter control (Goh et al., 15 Dec 2025).

b) Conformal Prediction-Driven Loss Formulations

By integrating split conformal calibration into the objective, e.g., via a Carlini & Wagner–inspired loss function, the model is optimized to push the forget set labels outside the conformal sets. The total loss is:

$p_{\theta'}$1

where $p_{\theta'}$2 is the non-conformity score, and $p_{\theta'}$3 is the conformal quantile. This enforces $p_{\theta'}$4, so $p_{\theta'}$5 is excluded from $p_{\theta'}$6. This approach enables flexible augmentation of most training-based unlearning methods (Shi et al., 31 Jan 2025).

c) Inference-Time Conformal Unlearning

For generative models, inference-time conformal unlearning circumvents parameter updates entirely. Instead, it iteratively samples outputs, applies an application-specific verifier $p_{\theta'}$7, and only returns outputs passing $p_{\theta'}$8 within a conformally-determined number of trials $p_{\theta'}$9. The conformal threshold $(x,y) \in$0 is determined using a held-out calibration set, guaranteeing

$(x,y) \in$1

This approach enables distribution-free coverage guarantees for on-the-fly unlearning without retraining, particularly suited to LLMs (Chowdhury et al., 3 Feb 2026).

4. Theoretical Guarantees and Risk Calibration

The central theoretical property underpinning conformal unlearning is its coverage guarantee: for any i.i.d. test point (relative to the calibration set), the probability that the forget data is still covered by the prediction set does not exceed $(x,y) \in$2, and that retained data is not covered is at most $(x,y) \in$3 (Alkhatib et al., 5 Aug 2025, Shi et al., 31 Jan 2025).

Key results include:

• Split CP validity: $(x,y) \in$4 for arbitrary $(x,y) \in$5, when calibration and test are exchangeable.
• Unlearning trade-off bound: $(x,y) \in$6, i.e., effective forget set mass limits possible coverage for forgetting.
• Distribution-shift awareness: FROC tracks monotonic risk increases as the Hellinger distance between calibration and test increases, allowing operators to decide on conservativeness under covariate shift (Goh et al., 15 Dec 2025).
• High-probability bounds: FROC’s CUR and inference-time conformal approaches provide explicit $(x,y) \in$7 confidence levels, meaning unlearning errors exceed the threshold on no more than a $(x,y) \in$8 fraction of future data (Goh et al., 15 Dec 2025, Chowdhury et al., 3 Feb 2026).

A noisy verifier with error rate $(x,y) \in$9 yields theoretical coverage at $y \notin C_{\theta'}(x)$0 in inference-time unlearning (Chowdhury et al., 3 Feb 2026).

5. Empirical Findings and Benchmarks

Empirical results across diverse domains—classification (CIFAR-10, Tiny ImageNet, CIFAR-100) and open-ended LLM knowledge—demonstrate that conformal unlearning frameworks:

• Reveal residual privacy risk: Traditional UA or MIA metrics consistently overestimate “forgetting”; a large fraction (over 50%) of forget points remains included in conformal sets even when UA exceeds 90% (Shi et al., 31 Jan 2025).
• Achieve stricter exclusion: Incorporation of conformal-driven loss terms or conformal risk constraints decreases CR on the forget set (e.g., from 0.98 to 0.75 with minimal utility loss), outperforms retrain and fine-tune baselines, and increases MIACR, directly measuring successful exclusion (Shi et al., 31 Jan 2025).
• Facilitate risk-utility trade-off control: FROC's risk parameter allows tuning, with monotonic degradation in retain accuracy commensurate with increases in forgetting power (Goh et al., 15 Dec 2025).
• Retain distribution-free calibration: Inference-time conformal unlearning achieves error rates tracking the target $y \notin C_{\theta'}(x)$1 (e.g., empirical unlearning error ≈0.04 for $y \notin C_{\theta'}(x)$2), with up to 93% reduction in unlearning error compared to best parameter-based baselines (Chowdhury et al., 3 Feb 2026).
• Model- and method-specialized insights: No single method dominates all architectures; e.g., different LLMs require distinct optimal strategies, and calibration set sizes and covariate shift parameters influence risk bounds (Goh et al., 15 Dec 2025).

6. Practical Implications and Deployment Considerations

Conformal unlearning introduces several operational and research advantages:

• Regulatory compliance and transparency: By permitting specification of explicit $y \notin C_{\theta'}(x)$3 risk budgets, conformal unlearning aligns with “right to be forgotten” mandates and provides clear, quantitative guarantees (Goh et al., 15 Dec 2025).
• Inter-method comparability and benchmarking: Unified risk axes allow direct, method-agnostic comparison between unlearning strategies (Goh et al., 15 Dec 2025, Alkhatib et al., 5 Aug 2025).
• Adaptivity to data shift: The sensitivity of risk to reference distribution (via $y \notin C_{\theta'}(x)$4 or similar divergences) facilitates principled adaptivity as deployment domains evolve (Goh et al., 15 Dec 2025).
• Resource and computational trade-offs: While advanced frameworks (e.g., CPMU) may incur higher memory overhead due to calibration set management, wall-clock unlearning times are on par with strong baselines (Alkhatib et al., 5 Aug 2025).

Inference-time conformal unlearning eliminates parameter update costs altogether, providing fast, risk-aware unlearning, though at the cost of increased inference latency as iteration count grows.

7. Challenges, Limitations, and Future Directions

Challenges remain in extending conformal unlearning to highly non-exchangeable settings (e.g., continual domain drift, adversarial forget requests), scaling calibration to extremely high-dimensional LLMs, and efficiently estimating tight risk bounds under distribution shift or verifier noise. Further work is ongoing to develop adaptive calibration, online conformal unlearning, unified loss landscapes across model classes, and robust empirical risk estimators for practical deployment (Goh et al., 15 Dec 2025, Chowdhury et al., 3 Feb 2026).

A plausible implication is that as more privacy and safety regulations require auditable guarantees for machine unlearning, conformal unlearning frameworks will become the preferred foundation for both research and industry-unlearning pipelines, thanks to their statistical rigor and operational transparency.