Unlearning Offline Stochastic Multi-Armed Bandits

Abstract: Machine unlearning aims to unlearn data points from a learned model, offering a principled way to process data-deletion requests and mitigate privacy risks without full retraining. Prior work has mainly studied unsupervised / supervised machine unlearning, leaving unlearning for sequential decision-making systems far less understood. We initiate the first study of a foundational sequential decision-making problem: offline stochastic multi-armed bandits (MAB). We formalize the privacy constraint for offline MAB and measure utility by the post-unlearning decision quality. We conduct a systematic study of both single- and multi-source unlearning scenarios under two data-generation models, the fixed-sample model and the distribution model. For these settings, our algorithmic design is built on two canonical base algorithms: Gaussian mechanism and rollback, and we propose adaptive algorithms that switch between them according to the data regime and privacy constraint. We further introduce a mixing procedure that elucidates the rationale behind these baselines. We provide performance guarantees across the above settings and establish lower bounds under both dataset models. Experiments validate the predicted tradeoffs and demonstrate the effectiveness of the proposed methods.

• The paper formalizes (ε, δ)-unlearning for offline stochastic bandits, ensuring privacy while preserving decision optimality.
• The proposed adaptive algorithm dynamically selects between Gaussian noise addition and rollback mechanisms to manage the impact of data deletions.
• The study establishes matching upper and lower bounds that quantify the tradeoff between privacy and utility in sequential decision-making.

Unlearning in Offline Stochastic Multi-Armed Bandits: Formalism, Algorithms, and Theory

Introduction and Formalization

This paper addresses the formally underexplored problem of machine unlearning within sequential decision-making, specifically focusing on the offline stochastic multi-armed bandit (MAB) setting (2605.00638). In the context of data deletion for privacy or legal compliance, machine unlearning concerns the systematic removal of the influence of specified data points from a trained model, without retraining from scratch. While much of prior art concentrates on supervised or unsupervised settings, this work initiates a systematic study of unlearning for offline stochastic MABs, capturing the privacy-utility tradeoff when deleting samples affects the discovery of the optimal arm.

The authors formalize $(\epsilon, \delta)$-unlearning for offline bandits, extending earlier machine unlearning and differential privacy (DP) frameworks. The utility is measured as the expected sub-optimality (the difference in mean reward between the best arm $a^*$ and the selected arm $\hat{a}$ post-unlearning), and the privacy constraint enforces indistinguishability between outputs reflecting presence/absence of deleted data as in DP. The unlearning problem is instantiated in two standard data regimes: the fixed-sample model (per-arm counts specified in advance) and the distribution model (data generated i.i.d. from an arm-sampling policy).

Algorithmic Design and Theoretical Guarantees

Adaptive Unlearning Strategy

The proposed approach develops an adaptive unlearning algorithm for both single-source and multi-source deletion scenarios. The algorithm combines two core mechanisms:

• Gaussian Mechanism: Adds calibrated Gaussian noise to arm statistics (e.g., Lower Confidence Bounds) to mask the influence of deleted samples, ensuring $(\epsilon, \delta)$-unlearning. The variance is set according to the sensitivity of the statistic and the privacy requirement.
• Rollback: Efficiently reverts the statistics as if the deleted samples in $U$ were never included, effectively retraining on $D \setminus U$ where feasible.

The adaptive rule dynamically selects between these mechanisms depending on the regime:

• If the deletion request targets the arm currently identified as optimal ($U$ from $a^*$) and the privacy constraint is sufficiently loose, the Gaussian mechanism is preferred.
• In all other cases, rollback yields better expected sub-optimality, particularly under tighter privacy requirements.

Mixing and Limitation

A hybrid mixing procedure interpolates between these extremes, rolling back a subset and adding noise to the remainder. However, rigorous analysis and experiments show that the optimal tradeoff is always achieved at the endpoints—either pure Gaussian or pure rollback—aligning with theoretical predictions.

Upper and Lower Bounds

For both offline data models and deletion types, the paper derives explicit upper and lower bounds for achievable sub-optimality under $(\epsilon, \delta)$-unlearning. These bounds characterize the dependence on:

• Per-arm sample counts $N(a)$,
• Number of deletions $a^*$0,
• Privacy parameter $a^*$1,
• Data imbalance ($a^*$2, the concentrability coefficient in the distributional model).

Key results include:

• In the fixed-sample model, the bound is $a^*$3 in specific regimes, where $a^*$4 is the number of points for the deletion arm and $a^*$5 for the optimal arm.
• In the distributional model with data imbalance $a^*$6, similar forms are derived, but accounting for coverage randomness.
• Lower bounds based on information-theoretic considerations (Le Cam’s method) match the upper bounds up to logarithmic factors in the low-$a^*$7 limit.
• For the particularly favorable regime where $a^*$8, an imitation learning-based approach plus rollback yields sharper $a^*$9 dependence, tightly matching the lower bound.

Empirical evaluation on synthetic offline bandit datasets verifies these theoretical behaviors and the adaptivity of the proposed algorithm.

Implications, Limitations, and Future Directions

This work provides a rigorous operationalization of unlearning for offline MABs, connecting privacy constraints to deletion-aware sequential decision-making. The derived bounds quantify the inherent tradeoff between privacy (stringency of $\hat{a}$0-unlearning), utility (decision optimality), and data regime (coverage and imbalance). The results have significant practical implications for privacy-law compliance in data-driven decision-making systems, especially in domains like recommender systems or healthcare, and clarify the cost of unlearning under rigorous guarantees.

Limitations arise from the focus on offline bandit settings; extension to contextual or linear bandits, as well as to general offline reinforcement learning (RL), is highlighted as a critical direction. Another open problem is the extension of sharp upper/lower bounds for multi-source deletions in the distributional model and the exploration of additional storage and computational constraints.

From a theoretical perspective, these results close gaps in understanding the privacy-utility frontier in sequential unlearning. The explicit adaptivity to the deletion source suggests similar strategies could benefit a broader class of sequential models.

Conclusion

The paper inaugurates a systematic, theoretically grounded framework for machine unlearning in offline stochastic multi-armed bandits. It presents adaptive algorithms combining noise-adding and rollback strategies, quantifies the optimality gap imposed by unlearning, and demonstrates near-matching upper and lower bounds. These contributions emphasize that unlearning can be achieved much more efficiently than trivial DP approaches by exploiting the structure and statistics of arm-wise data. The formalism and results set the stage for privacy-conscious sequential learning in complex, data-driven environments.