---
title: Inverse Hessian Attack (IHA)
url: https://www.emergentmind.com/topics/inverse-hessian-attack-iha
type: topic
---

# Inverse Hessian Attack (IHA)

Searching arXiv for the cited IHA paper and closely related work on iHVPs, membership inference, and transfer-learning evaluations.
Inverse Hessian Attack (IHA) is a white-box membership inference attack for stochastic gradient descent (SGD)-trained models that explicitly uses model parameters, curvature, and inverse-Hessian vector products (iHVPs) to infer whether a record was used in training [2406.11544]. In contrast to black-box attacks that rely primarily on the loss or model outputs, IHA is derived from discrete-time SGD dynamics under assumptions that the loss surface is approximately quadratic near local minima, the training dynamics reach a stationary state, and the Hessian and loss value at the optimum change little by removing a single training data point [2406.11544]. The method is presented as evidence that white-box access can reveal more about membership than loss-based black-box attacks in common practical settings, while also functioning as a privacy auditing tool whose applicability is constrained by computational cost and unusually strong data-access assumptions [2406.11544].

## 1. Origin and threat model

IHA was introduced in "Do Parameters Reveal More than Loss for Membership Inference?" [2406.11544]. The paper studies membership inference attacks (MIAs), which try to determine if a specific data point was present in a model's training set, and argues that prior claims around black-box access being sufficient for optimal membership inference do not hold for stochastic gradient descent [2406.11544]. In the formulation given there, optimal membership inference indeed requires white-box access, and this motivates IHA as an explicit white-box inference attack that takes advantage of computing inverse-Hessian vector products [2406.11544].

The attack assumes access to the final trained parameters of the target model, knowledge of training hyperparameters, and knowledge of all training records except possibly the target record [2406.11544]. The transfer-learning evaluation later characterizes this as breaking the “standard MIA threat model,” since standard attacks typically do not assume access to model weights together with all but the target record in the training set [2510.05753]. The same evaluation states that IHA requires access to all but the target record in a training set and the final trained parameters of the target model, and describes the attack as a white-box membership inference attack specifically designed to leverage deeper access to a target model compared to standard MIAs [2510.05753].

This threat model makes IHA realistic for a privacy auditor, but not directly feasible for a real adversary in the ordinary sense described in the source material [2406.11544]. A plausible implication is that IHA is best interpreted as a high-information or worst-case audit mechanism rather than as a baseline for routine black-box privacy risk estimation.

## 2. Theoretical derivation from SGD dynamics

The derivation of IHA in [2406.11544] departs from earlier theory that modeled the parameter distribution after training with a Boltzmann distribution,
\[
\mathbb{P}(\mathbf{w} | \text{train data}) \propto \exp\left(-\frac{1}{T} \sum_i m_i \ell(\mathbf{w}, \mathbf{z}_i)\right),
\]
where \( m_i \) indicates membership and \( \ell \) is the loss. The paper argues that these assumptions do not hold for typical, finite-step stochastic gradient descent training used in practice [2406.11544].

Instead, the authors leverage recent advances in the theoretical understanding of discrete-time SGD dynamics and make three assumptions: the loss surface is approximately quadratic near local minima, the training dynamics reach a stationary state, and the Hessian and loss value at the optimum change little by removing a single training data point [2406.11544]. Under these assumptions, the paper derives an explicit formula for the optimal membership inference score when white-box access and knowledge of training hyperparameters are available [2406.11544].

The resulting score, presented in the source as Equation 7, is
\[
\text{IHA}(\mathbf{z}_1) := \frac{\ell(\mathbf{w}, \mathbf{z}_1)}{1 + \mu} - \frac{1}{\lambda}\left( \frac{1}{n} \| \mathbf{H}_*^{-1} \nabla \ell(\mathbf{w}, \mathbf{z}_1) \|^2 + 2 (\mathbf{H}_*^{-1} \nabla L_0(\mathbf{w}))^\top (\mathbf{H}_*^{-1}\nabla \ell(\mathbf{w}, \mathbf{z}_1)) \right).
\]
Here, the quantities identified in the source are \( \mathbf{H}_* \) as the Hessian at the local minimum, \( \ell(\mathbf{w}, \mathbf{z}_1) \) as the loss on the record in question, \( \nabla \ell(\mathbf{w}, \mathbf{z}_1) \) as the gradient with respect to the model parameters, \( L_0(\mathbf{w}) \) as the mean loss excluding \(\mathbf{z}_1\), \( n \) as the total number of training records, \( \lambda \) as the learning rate, \( \mu \) as momentum, and \( S \) as batch size [2406.11544].

The transfer-learning evaluation states the mathematical intuition differently, emphasizing a local similarity assumption: models trained with or without a single data point converge to nearby local minima, and thus the Hessian matrices at these minima are expected to be similar [2510.05753]. It summarizes this as
\[
\mathbf{H}_* = \mathbf{H}_0(\bm{w}_0^*) = \mathbf{H}_1(\bm{w}_1^*),
\]
with \(\bm{w}_0^*\) and \(\bm{w}_1^*\) denoting parameters with and without the target sample [2510.05753]. This suggests that the attack’s second-order terms are intended to encode the sensitivity of the learned optimum to inclusion or exclusion of an individual record.

## 3. Structure of the score and relation to prior attacks

The source material explicitly positions IHA relative to several prior attack families [2406.11544]. A traditional loss attack uses only the loss, that is, \(-\ell(\mathbf{w}, \mathbf{z}_1)\) [2406.11544]. Reference or calibration attacks such as those associated with Sablayrolles et al. and Carlini et al. use multiple reference models to calibrate the loss or perform likelihood-ratio tests [2406.11544]. Self-influence uses \({\bm{g}_i}^\top \mathbf{H}^{-1}\bm{g}_i\) as a membership signal and is described as closely related to a term in the IHA formula [2406.11544].

Within this comparison, IHA is described as a more principled, theoretically optimal score under SGD-based training because it combines the loss and second-order or statistical terms reflecting the expected model change from removing the queried record [2406.11544]. The use of the inverse Hessian is stated to capture how a data point influences the entire parameter space, correcting for “easier” or “harder” points [2406.11544]. That framing links IHA to a broader line of research on inverse Hessian-vector products and influence-style reasoning, even though the membership objective differs from training data attribution.

A closely related strand of work revisits iHVP computation for influence functions and explains the role of the inverse Hessian as reweighting gradient directions by curvature, de-emphasizing directions with high curvature and amplifying low-curvature directions [2409.17357]. That paper gives the influence formula
\[
\mathcal{I}(z_{\text{train}, z_{\text{test}) = - \nabla f(z_{\text{test}, \theta)^{T} (H + \lambda I)^{-1} \nabla \ell(z_{\text{train}; \theta)
\]
and argues that the inverse Hessian acts to "whiten" gradient directions [2409.17357]. Although this is not itself IHA, it clarifies why the second-order terms in IHA are not reducible to loss alone.

## 4. Practical implementation and computational bottlenecks

The practical implementation described in [2406.11544] requires computing two iHVPs:
\[
\mathbf{H}_*^{-1} \nabla \ell(\mathbf{w}, \mathbf{z}_1)
\quad \text{and} \quad
\mathbf{H}_*^{-1} \nabla L_0(\mathbf{w}).
\]
In the experiments reported there, on small models such as linear models and 2-layer nets, the authors compute the full Hessian and its inverse directly, with damping for near-zero or negative eigenvalues to address ill-conditioning [2406.11544]. For larger models, the paper states that this would not scale, and mentions techniques like LiSSA or conjugate gradient as potential but error-prone alternatives for approximate iHVPs [2406.11544].

The same paper also notes that no reference models are needed, unlike strong black-box attacks, and that the score is used directly as the membership score, with higher values indicating that a point is more likely to be a member [2406.11544]. However, the need to compute \( \nabla L_0(\mathbf{w}) \) by summing the gradients over all other training data further increases the attack’s operational burden [2406.11544].

Subsequent work on iHVPs provides additional context for this bottleneck. "Revisiting inverse Hessian vector products for calculating influence functions" presents LiSSA as a stochastic iterative approach for approximating iHVPs, with recursion
\[
u^t = g + (I - \eta (\tilde{H}^t + \lambda))u^{t-1},
\]
and argues that its hyperparameters can be selected using spectral properties of the Hessian, particularly the trace and largest eigenvalue [2409.17357]. More recently, "Better Training Data Attribution via Better Inverse Hessian-Vector Products" introduces ASTRA, which uses the EKFAC-preconditioner on Neumann series iterations to obtain accurate iHVP approximations for training data attribution [2507.14740]. ASTRA is described as easy to tune, requiring fewer iterations than Neumann series iterations, and being more accurate than EKFAC-based approximations [2507.14740]. The source explicitly notes interpretability and security implications, including that more accurate iHVPs can make it easier to identify training points that, if manipulated, have maximal or targeted influence on model predictions, while also enabling more robust detection of malicious or noisy training data [2507.14740].

This suggests that the computational limitations identified in the original IHA work are not fixed properties of the attack itself; rather, they are tightly coupled to the state of iHVP approximation methods.

## 5. Empirical findings

The original empirical evaluation in [2406.11544] tests IHA on Purchase-100, MNIST-odd/even, and FashionMNIST. The metrics reported are attack AUC and TPR at low FPR, specifically at \(1\%\) and \(0.1\%\) false positive rate [2406.11544]. According to the source, IHA outperforms all baselines on Purchase-100 and FashionMNIST, and on Purchase-100 the reported AUC values are .709 for IHA, .645 for LiRA, and .531 for LOSS [2406.11544]. The same source also states that IHA does not dominate in every setting, citing MNIST-odd as an example, but always achieves at least competitive or better performance [2406.11544].

A later empirical comparison in deep transfer learning extends the evaluation regime and states that there is no one MIA which captures all privacy risks in models trained with transfer learning [2510.05753]. In that study, LiRA demonstrates superior performance across most experimental scenarios, while IHA proves to be more effective against models fine-tuned on PatchCamelyon in high data regime [2510.05753]. The paper evaluates CIFAR-10, CIFAR-100, and PatchCamelyon using ViT-B/16 and BiT-M-R50x1 backbones pre-trained on ImageNet-21k, with head-only, ALL, and FiLM parameterization modes, and averages results over 10 experimental repeats [2510.05753].

The transfer-learning paper highlights a specific asymmetry across data regimes. On PatchCamelyon with high shots, IHA’s efficacy remains stable or increases as data scale grows, whereas standard MIAs typically decrease in efficacy [2510.05753]. It further reports that for PatchCamelyon, ViT-B/16, head-only, \(S=16384, 32768, 65536\), LiRA has TPR@FPR\(=0.001 \approx 0.00\), while IHA has TPR@FPR\(=0.001\) in the range \(0.01\)–\(0.04\) [2510.05753]. By contrast, on CIFAR-10 at similar scales, IHA does not excel in the same way, and its advantage is described as most pronounced in PatchCamelyon [2510.05753].

| Setting | Reported outcome | Source |
|---|---|---|
| Purchase-100 | IHA AUC .709 vs. .645 LiRA, .531 LOSS | [2406.11544] |
| FashionMNIST | IHA outperforms all baselines | [2406.11544] |
| MNIST-odd/even | IHA does not dominate in every setting, but is at least competitive or better | [2406.11544] |
| PatchCamelyon, high data regime | IHA more effective than LiRA in reported low-FPR evaluations | [2510.05753] |

Taken together, these results support a narrow but important conclusion: IHA is not uniformly strongest across all tasks, but it can expose privacy leakage that score-based MIAs do not capture, especially when its stronger threat model is plausible [2510.05753].

## 6. Limitations, scope, and controversies

The limitations of IHA are explicit in the source material. The original paper states that the attack requires full access to model parameters, all-but-one training data, and training hyperparameters [2406.11544]. It also requires computing the full Hessian and its inverse in the reported experiments, which is computationally expensive and only tractable for small models in practice [2406.11544]. The theory assumes a locally quadratic loss surface and that removing one record does not significantly affect the Hessian or minimum; these assumptions are described as approximately valid for large datasets and relatively smooth models, but less so for highly overparameterized or hardly regularized ones [2406.11544]. The attack also assumes that SGD stationary-state conditions are met and that model training follows the usual SGD dynamics, so unusual optimizers or aggressive learning rate schedules may invalidate the model [2406.11544].

The transfer-learning comparison reiterates that IHA’s required knowledge is rarely realistic outside small-scale or managed or insider threat settings [2510.05753]. It also notes fluctuating performance with increasing dataset size, possibly due to instability in Hessian approximations and sensitivity to model initialization or optimization [2510.05753]. Because of this, the paper advises combining black-box methods such as LiRA with white-box methods such as IHA for practical auditing [2510.05753].

A recurrent misconception in the surrounding literature is that black-box access is sufficient for optimal membership inference. The central contribution of [2406.11544] is precisely to dispute that claim for SGD-trained models. Another potential source of confusion is terminological: a separate adversarial-learning paper uses the phrase “IHA framework” to refer to inverse Hessian-based adversarial perturbations of the form
\[
\delta_{\text{IHA} = \mathbf{H}^{-1} \mathbf{g}
\]
in the context of second-order adversarial examples [2207.01396]. That usage concerns input-space adversarial example generation rather than membership inference, even though both rely on inverse-Hessian structure.

## 7. Relation to inverse-Hessian methods beyond membership inference

IHA belongs to a wider methodological family in which inverse Hessians or iHVPs are used to characterize sensitivity. In influence functions and training data attribution, iHVPs arise when estimating how changes to a training sample affect a function of a test point’s prediction [2507.14740]. The source gives the influence-function quantity
\[
T_{\mathrm{IF}(z_m, z_q, D) = \nabla f_{z_q}(\theta^*)^\top (G + \lambda I)^{-1} \nabla \ell(\theta^*, z_m)
\]
and emphasizes that direct computation is infeasible for large models, motivating approximate solvers such as Neumann-series methods, EKFAC, LiSSA, and ASTRA [2507.14740].

The relevance of this broader literature to IHA is twofold. First, IHA’s feasibility is directly determined by how well iHVPs can be approximated under realistic compute budgets [2406.11544]. Second, the security implications of improved iHVP computation are dual-use: more accurate second-order analysis can improve auditing and interpretability, but may also facilitate stronger attacks or more targeted manipulation of training data [2507.14740]. A plausible implication is that advances in scalable iHVP approximation may shift IHA from a small-model auditing technique toward a more general white-box privacy analysis tool, provided the strong access assumptions remain acceptable for the intended evaluation setting.

Source: https://www.emergentmind.com/topics/inverse-hessian-attack-iha