---
title: 'Squisher: Zero-Cost Fisher Approximation'
url: https://www.emergentmind.com/topics/squisher
type: topic
---

# Squisher: Zero-Cost Fisher Approximation

Squisher is a zero-cost approximation to the diagonal empirical Fisher obtained by reusing the squared-gradient accumulator already maintained by adaptive optimizers such as Adam. In the formulation introduced by Li et al., the term denotes the bias-corrected second-moment estimate $\hat v_T$ at the end of training, interpreted as a surrogate for parameter sensitivity. The method is motivated by the observation that both the empirical Fisher diagonal and Adam’s second-moment state are averages of squared gradients, albeit with different weighting and sampling structure. Across six Fisher-based applications, Squisher is reported to perform similarly to the Fisher diagonal while outperforming Fisher-free baselines [2507.18807].

## 1. Formal definition and relation to the Fisher diagonal

Let $\theta\in\mathbb{R}^D$ denote the parameters of a probabilistic model $p(y|x,\theta)$. The Fisher Information Matrix is

$$
F(\theta)=\mathbb{E}_{x,y\sim p(\cdot|\theta)}\left[\nabla_\theta\log p(y|x,\theta)\nabla_\theta\log p(y|x,\theta)^\top\right],
$$

and its diagonal is frequently used as a measure of parameter sensitivity. In deep-learning practice, one typically replaces the model distribution by the empirical distribution over training examples $\{(x_i,y_i)\}_{i=1}^N$ and works with the empirical-Fisher diagonal estimator

$$
\operatorname{diag}F(\theta)\approx \frac{1}{N}\sum_{i=1}^N g_i(\theta)\odot g_i(\theta),
$$

where

$$
g_i(\theta)=\nabla_\theta\log p(y_i|x_i,\theta),
$$

and $\odot$ denotes elementwise square.

Squisher replaces this post hoc estimation procedure with a quantity already accumulated during optimization. In Adam, the batch-gradient second moment is updated as

$$
v_t=\beta v_{t-1}+(1-\beta)\,g_t\odot g_t,\qquad
\hat v_t=\frac{v_t}{1-\beta^t},
$$

where $\beta\in[0,1)$, often $\beta=0.999$, and $g_t=\nabla_\theta L_{\text{batch}}(\theta_{t-1})$ is the gradient of the average loss over the $t$th mini-batch. The Squisher is defined as $\hat v_T$ at the end of training.

## 2. Construction from the optimizer state

The method requires no additional backward passes. During standard Adam or AdamW training, one stores the second-moment state and, after the final update, reinterprets the bias-corrected accumulator $\hat v_T$ as a parameter-importance estimate.

Operationally, the update consists of four steps. A mini-batch $B_t$ of size $B$ is sampled. The batch gradient is computed as

$$
g_t=\frac{1}{B}\sum_{i\in B_t}\nabla_\theta[-\log p(y_i|x_i,\theta_{t-1})].
$$

The second moment is then updated through the exponential moving average

$$
v_t=\beta v_{t-1}+(1-\beta)\,g_t\odot g_t,
$$

followed by bias correction,

$$
\hat v_t=\frac{v_t}{1-\beta^t}.
$$

Parameter updates proceed as in standard Adam. The final output is the pair $(\theta_T,\hat v_T)$.

When absolute magnitude matters, $\hat v_T$ may be rescaled. The exposition explicitly notes that one may multiply $\hat v_T$ by $B$ or by $N$ depending on context, because the batch-gradient square and the per-example sum of squares are not on the same scale.

## 3. Approximation mechanism and exact points of divergence

The central approximation is that the empirical Fisher diagonal and Adam’s second-moment state are both aggregations of squared gradients, but over different index sets and with different weights [2507.18807].

The empirical Fisher uses a uniform average over examples,

$$
\frac{1}{N}\sum_{i=1}^N g_i\odot g_i,
$$

whereas the Squisher is essentially

$$
\hat v_T\approx (1-\beta)\sum_{t=1}^T \beta^{T-t}(g_t\odot g_t).
$$

Thus Squisher substitutes exponentially decaying temporal weights $(1-\beta)\beta^{T-t}$ for the uniform weight $1/N$.

A second discrepancy arises from the use of batch gradients. In practice,

$$
g_t=\frac{1}{B}\sum_{i\in B_t} g_i,
$$

so squaring occurs after averaging. This yields

$$
g_t\odot g_t=\frac{1}{B^2}\left[\sum_{i\in B_t} g_i\right]\odot\left[\sum_{i\in B_t} g_i\right],
$$

which corresponds to the diagonal of a joint Fisher over the mini-batch rather than the average of per-example squares. The exposition therefore recommends optional rescaling by $B$ when magnitude fidelity is required, as in Elastic Weight Consolidation.

A third discrepancy is temporal. The empirical Fisher squares gradients at the final parameters $\theta_T$, whereas the Squisher aggregates squared gradients collected at successive $\theta_t$. The stated justification is that if training converges, the $\theta_t$ stabilize and the accumulator concentrates on a narrow band of similar gradients. Under such mild stationarity assumptions and with sufficiently many steps covering all data patterns, the exponential moving average of squared batch gradients provides a close surrogate for the diagonal empirical Fisher.

## 4. Computational profile, bias structure, and limitations

The principal practical distinction is computational cost [2507.18807]. A true Fisher diagonal requires either per-example gradient computations, such as $N$ backward passes or a specialized library such as BackPACK, or at least a loop over $O(1000)$ examples. The reported wall-clock cost ranges from seconds to hours depending on model size and dataset. By contrast, Squisher requires no extra backward passes and only reuses Adam’s stored $v$-state; the stated overhead is less than $1$ s in all studied scenarios.

The paper isolates three sources of bias. The first is the exponential weighting itself: recent gradients are emphasized more heavily, on a timescale of approximately $1/(1-\beta)$ steps. The second is parameter drift: if training is cut short, the accumulator may not reflect the curvature at the final $\theta_T$. The third is magnitude mismatch due to the noncommutativity of averaging and squaring, expressed in the statement that $(\text{average}\to\text{square})\neq(\text{square}\to\text{average})$.

These distinctions delimit a common misconception. Squisher is not identical to the Fisher diagonal; it is an approximation whose accuracy depends on training duration, optimizer decay, and whether downstream use depends on relative importance or absolute scale. A plausible implication is that applications based primarily on ranking or comparative weighting are less sensitive to scale mismatch than applications that insert the estimate directly into a regularizer.

## 5. Empirical behavior across Fisher-based applications

Li et al. evaluated Squisher in six applications: two forms of model merging, pruning, sparse “FISH” masking, Task2Vec embeddings, and EWC continual learning. The recurring pattern is that Squisher matches the Fisher to within experimental noise in several settings and outperforms Fisher-free baselines in all reported cases [2507.18807].

| Application | Metric | Reported result |
|---|---|---|
| Model Merging (Fisher-Weighted Average of T5-Large) | Average accuracy on eight fine-tuning tasks | Squisher $\approx 58\%$, Fisher $\approx 54\%$, unweighted average $\approx 48\%$ |
| UBGM Merging (RoBERTa) | Mean accuracy over 5 classification tasks | Squisher $\approx$ Fisher $\approx 94.0\%$, parameter averaging $\approx 93.9\%$ |
| Fisher Pruning (VGG-13 on CIFAR-100) | Test accuracy after pruning $25/50/75\%$ of weights | Squisher only $\sim 0.9$–$1.0\%$ below Fisher, far above random pruning |
| FISH Mask (BERT on GLUE) | GLUE average score with $50\%$ parameters masked | Squisher $\approx 82.3$, Fisher $\approx 82.4$, random mask $\approx 73.2$ |
| Task2Vec Embeddings | MRR and NDCG across 21 tasks | Squisher MRR $=0.468$, Fisher MRR $=0.421$, dataset-size baseline MRR $=0.136$ |
| EWC Continual Learning | Final test accuracy after sequential-task training | Squisher matched or slightly outperformed Fisher in all task-incremental protocols |

The same section reports that Fisher-diagonal computation required tens to thousands of GPU-seconds in these scenarios, whereas Squisher introduced less than $1$ s of overhead regardless of model size. This suggests that the method is primarily valuable when Fisher-style parameter sensitivity is needed repeatedly or at scale.

## 6. Practical use and methodological scope

The paper gives several explicit recommendations for practitioners. The optimizer state, especially $v_t$, should be saved and shared if parameter-importance measures may later be needed. Training should run for enough steps that the exponential moving average has covered a representative gradient distribution. If the absolute scale of the Fisher diagonal matters, $\hat v_T$ should be rescaled by the appropriate factor, such as batch size $B$ or dataset size $N$, to match the sum-of-squares Fisher. The default Adam decay $\beta\approx 0.999$ is reported to work well, but reducing $\beta$ toward $0.99$ or $0.9$ may better align the estimate with a more uniform Fisher on shorter timescales or in cases where recent curvature is underestimated. In extremely low-data or few-step regimes, the paper recommends supplementing Squisher with a small number of fresh per-example Fisher estimates or lowering $\beta$ [2507.18807].

Within that scope, Squisher functions as a drop-in proxy for the diagonal empirical Fisher in settings where Fisher-based parameter importance is used for weighting, ranking, masking, pruning, merging, embedding construction, or continual-learning regularization. Its significance lies less in introducing a new curvature object than in showing that an optimizer artifact already present in standard training can substitute for an otherwise nontrivial estimation procedure.

## 7. Conceptual significance

Squisher reframes second-moment optimizer state as a statistical object rather than merely an optimization aid. The empirical contribution is that Adam’s squared-gradient accumulator can stand in for the Fisher diagonal across a diverse set of downstream procedures, with only minor and controllable biases. The theoretical contribution is the explicit clarification of where the approximation is exact in spirit and where it is not: exponential versus uniform weighting, batch-level versus per-example squaring, and accumulation along a trajectory $\theta_1,\dots,\theta_T$ rather than evaluation solely at $\theta_T$.

This places Squisher at the intersection of optimizer-state reuse, curvature approximation, and parameter-sensitivity estimation. A plausible implication is that future Fisher-based workflows may treat optimizer checkpoints as sufficient statistics for many downstream analyses, provided that the intended use tolerates the temporal and scaling biases identified in the original study.

Source: https://www.emergentmind.com/topics/squisher