Shape of Memory: a Geometric Analysis of Machine Unlearning in Second-Order Optimizers

Abstract: We argue that current definitions of machine unlearning are underspecified for second-order optimizers. We compare first-order and second-order learners for their ability to handle the data deletion task with varying degrees of eigendecomposition to mimic the loss model memory. While both first and second-order methods realign with the ideal counterfactul in terms of performance and gradient, the second-order optimizer shows significant volatility in the optimizer state. This indicates residual information, supposedly deleted, that isn't detectable by first-order analysis. Various eigendecay treatments show that stability and information loss is regained only under controlled state pertubation where geometric information (or memory) is erased.

• The paper argues that deleting information in second-order optimizers leaves persistent residual memory detectable through second-order diagnostics.
• First-order recovery time per metric indicates full recovery, yet second-order diagnostics show persistent misalignment and volatility.
• Spectral interventions can address this misalignment by explicitly altering the spectrum of eigenvalues $A_t$ of the optimizer state, impacting performance based on intervention strength.

Motivation and central claim

Machine unlearning research has overwhelmingly evaluated deletion success by comparing post-deletion parameters, predictions, or regret to those of a counterfactual model retrained without the deleted data. This paper argues that such criteria are underspecified for stateful second-order optimizers, whose auxiliary state — in the case of Online Newton Step (ONS), the matrix $A_t = \lambda I + \sum_s g_s g_s^\top$ — encodes a compressed record of the entire optimization trajectory. The author's central claim is that deleted information persists as volatility and misalignment in this second-order state even when first-order diagnostics (regret, tracking error) indicate full recovery, and that only explicit spectral perturbation of the state reduces this residual geometric memory (2604.23046).

The paper positions itself deliberately: it does not propose a certified unlearning mechanism. Instead, it reframes unlearning for stateful learners as a state alignment problem rather than a parameter-matching problem, and poses the construction of certificates over higher-order information states as an open challenge.

Problem formulation

The setting is online convex optimization under adversarial deletions. The learner state is decomposed as $\mathcal{S}_t = (w_t, \mathcal{A}_t)$, separating parameters from optimizer state. Two learners are compared:

• OGD: update $w_{t+1} = w_t - \eta \nabla \ell_t(w_t)$, with an empty auxiliary state; deletions affect only the parameter trajectory.
• ONS: steepest descent under the metric induced by $A_t$, i.e., $w_{t+1} = w_t - \eta A_t^{-1} g_t$. The matrix $A_t$ defines a time-varying inner product on parameter space and constitutes persistent curvature memory.

Deletion requests at time $\tau$ remove the influence of past gradients without assuming access to the full training history or the budget to recompute $A_t$ exactly. The paper notes that ONS is invariant to affine changes of the loss surface, so linear discrepancies between observed and counterfactual models do not alter descent trajectories.

Experimental design

Experiments use a controlled linear convex problem with horizon $T = 400$ rounds in dimension $d=2$, a deletion event at $\mathcal{S}_t = (w_t, \mathcal{A}_t)$0 removing the 10 most recent observations, both stationary and drifting streams, and 20 random seeds per condition. ONS interventions act directly on the spectrum of $\mathcal{S}_t = (w_t, \mathcal{A}_t)$1: Partial Reset subtracts $\mathcal{S}_t = (w_t, \mathcal{A}_t)$2 from eigenvalues (with positivity thresholding), and Eigenvalue Decay rescales eigenvalues by $\mathcal{S}_t = (w_t, \mathcal{A}_t)$3. Metrics include cumulative regret against the best fixed comparator, tracking error $\mathcal{S}_t = (w_t, \mathcal{A}_t)$4, instantaneous parameter shock $\mathcal{S}_t = (w_t, \mathcal{A}_t)$5, recovery time, overshoot, and spectral diagnostics (trace, condition number, cosine alignment of update directions).

Results

Three patterns recur across conditions.

First-order recovery is finite but environment-dependent. For OGD in the stationary setting, deletion produces a small transient: mean parameter shock $\mathcal{S}_t = (w_t, \mathcal{A}_t)$6, overshoot $\mathcal{S}_t = (w_t, \mathcal{A}_t)$7, recovery in roughly 22.75 rounds. Under drift, disruption grows substantially — shock $\mathcal{S}_t = (w_t, \mathcal{A}_t)$8, overshoot $\mathcal{S}_t = (w_t, \mathcal{A}_t)$9, recovery $w_{t+1} = w_t - \eta \nabla \ell_t(w_t)$0 rounds. Deletion difficulty therefore cannot be attributed to deletion alone; ambient nonstationarity dominates recovery cost.

Second-order regret recovery masks state misalignment. Across all ONS intervention settings, regret-based recovery time is essentially immediate ($w_{t+1} = w_t - \eta \nabla \ell_t(w_t)$1 rounds on average), yet parameter shock remains approximately $w_{t+1} = w_t - \eta \nabla \ell_t(w_t)$2 and post-deletion overshoot ranges from 1.03 to 1.43. This is the paper's most consequential observation: scalar performance summaries return to nominal scale while the realized trajectory remains displaced from the counterfactual path. Rapid regret recovery does not certify agreement with the counterfactual state.

Model	Environment	Intervention	Recovery (rounds)	Overshoot	Param. shock
OGD	Stationary	Baseline	22.75 ± 59.04	0.025 ± 0.005	0.014 ± 0.008
OGD	Drifting	Baseline	78.35 ± 18.65	0.734 ± 0.102	0.049 ± 0.010
ONS	Drifting	All settings	0.05 ± 0.22	1.03–1.43	0.065 ± 0.030

Spectral interventions rewrite geometry, not performance. The mild partial reset ($w_{t+1} = w_t - \eta \nabla \ell_t(w_t)$3) yields the smallest overshoot (1.026) and lowest final regret (5.402), while aggressive decay ($w_{t+1} = w_t - \eta \nabla \ell_t(w_t)$4) yields the largest overshoot (1.425) and final regret (5.747). Notably, mean parameter shock is identical across all ONS treatments, indicating that separation between treatments emerges during post-deletion evolution under a modified metric rather than at the deletion instant itself. Stronger suppression of stored curvature does not reconstruct the counterfactual geometry; it introduces a larger geometric discontinuity while leaving the learner broadly functional.

Volatility persists in the iterate. Spectral diagnostics show that the trace of $w_{t+1} = w_t - \eta \nabla \ell_t(w_t)$5 evolves smoothly after deletion while the condition number and cosine-alignment diagnostics exhibit discontinuities and elevated volatility proportional to intervention strength. Counterintuitively, lighter interventions produce heavier cosine dissimilarity, whereas near-total eigendecay minimizes divergence from the counterfactual geometry. The paper interprets this combination as geometric hysteresis: deletion perturbs the shape and orientation of the second-order state more than its aggregate scale, leaving residual information invisible to first-order analysis.

Discussion

The evidence supports the hypothesis that unlearned information persists in second-order optimizer state. The practical implication is direct: unlearning procedures that measure only parameter distance or predictive performance will systematically fail to detect residual leakage in stateful optimizers, since the deleted data's influence survives in the metric under which future updates are computed. This exposes a structural trade-off — richer optimizer state buys faster adaptation at the price of greater path dependence under deletions — and implies that effective unlearning requires principled control over optimizer memory rather than parameter updates alone.

Limitations and open questions

Several constraints bound the strength of these conclusions. The experiments are confined to a two-dimensional linear convex problem with a single deletion event; whether the hysteresis phenomenon scales to high-dimensional quasi-Newton methods such as online L-BFGS (Mokhtari et al., 2014) is untested. The spectral interventions are heuristic — subtraction and scaling of eigenvalues chosen to span light-to-aggressive distortion — and no treatment achieves genuine convergence to the counterfactual state; stronger interventions increase downstream deviation rather than eliminating it. The paper also concedes that it does not solve certified unlearning and explicitly frames the certification question as unresolved. Two open problems are stated: extending the state-alignment formulation to general higher-order information states within an MDP/regret-control framework, and designing online algorithms that simultaneously provide tight regret bounds and aligned internal state under compute and memory constraints suitable for resource-limited deployments.

Conclusion

This paper demonstrates that conventional, parameter-centric definitions of machine unlearning are insufficient for second-order optimizers. In controlled online convex experiments, ONS recovers counterfactual-level regret almost immediately after deletion while its preconditioner retains persistent misalignment and volatility relative to the ideal unlearned state, and only aggressive spectral erasure of stored geometry reduces this residual. The work's contribution is primarily diagnostic and conceptual: it establishes optimizer-state alignment as a measurable failure mode of unlearning and identifies the design of certified, state-aware deletion mechanisms as the outstanding problem.