Squisher: Zero-Cost Fisher Approximation

• Squisher is a zero-cost approximation to the Fisher diagonal that reuses Adam’s bias-corrected squared-gradient accumulator to gauge parameter sensitivity.
• It computes an exponentially weighted average of batch gradients during training, eliminating the need for additional backward passes while maintaining efficiency.
• Empirical evaluations across tasks like model merging, pruning, and continual learning show Squisher performs comparably to the Fisher diagonal with minimal overhead and controllable biases.

Squisher is a zero-cost approximation to the diagonal empirical Fisher obtained by reusing the squared-gradient accumulator already maintained by adaptive optimizers such as Adam. In the formulation introduced by Li et al., the term denotes the bias-corrected second-moment estimate $\hat v_T$ at the end of training, interpreted as a surrogate for parameter sensitivity. The method is motivated by the observation that both the empirical Fisher diagonal and Adam’s second-moment state are averages of squared gradients, albeit with different weighting and sampling structure. Across six Fisher-based applications, Squisher is reported to perform similarly to the Fisher diagonal while outperforming Fisher-free baselines (Li et al., 24 Jul 2025).

1. Formal definition and relation to the Fisher diagonal

Let $\theta\in\mathbb{R}^D$ denote the parameters of a probabilistic model $p(y|x,\theta)$. The Fisher Information Matrix is

$$F(\theta)=\mathbb{E}_{x,y\sim p(\cdot|\theta)}\left[\nabla_\theta\log p(y|x,\theta)\nabla_\theta\log p(y|x,\theta)^\top\right],$$

and its diagonal is frequently used as a measure of parameter sensitivity. In deep-learning practice, one typically replaces the model distribution by the empirical distribution over training examples $\{(x_i,y_i)\}_{i=1}^N$ and works with the empirical-Fisher diagonal estimator

$$\operatorname{diag}F(\theta)\approx \frac{1}{N}\sum_{i=1}^N g_i(\theta)\odot g_i(\theta),$$

where

$g_i(\theta)=\nabla_\theta\log p(y_i|x_i,\theta),$

and $\odot$ denotes elementwise square.

Squisher replaces this post hoc estimation procedure with a quantity already accumulated during optimization. In Adam, the batch-gradient second moment is updated as

$$v_t=\beta v_{t-1}+(1-\beta)\,g_t\odot g_t,\qquad \hat v_t=\frac{v_t}{1-\beta^t},$$

where $\beta\in[0,1)$, often $\theta\in\mathbb{R}^D$0, and $\theta\in\mathbb{R}^D$1 is the gradient of the average loss over the $\theta\in\mathbb{R}^D$2th mini-batch. The Squisher is defined as $\theta\in\mathbb{R}^D$3 at the end of training.

2. Construction from the optimizer state

The method requires no additional backward passes. During standard Adam or AdamW training, one stores the second-moment state and, after the final update, reinterprets the bias-corrected accumulator $\theta\in\mathbb{R}^D$4 as a parameter-importance estimate.

Operationally, the update consists of four steps. A mini-batch $\theta\in\mathbb{R}^D$5 of size $\theta\in\mathbb{R}^D$6 is sampled. The batch gradient is computed as

$\theta\in\mathbb{R}^D$7

The second moment is then updated through the exponential moving average

$\theta\in\mathbb{R}^D$8

followed by bias correction,

$\theta\in\mathbb{R}^D$9

Parameter updates proceed as in standard Adam. The final output is the pair $p(y|x,\theta)$0.

When absolute magnitude matters, $p(y|x,\theta)$1 may be rescaled. The exposition explicitly notes that one may multiply $p(y|x,\theta)$2 by $p(y|x,\theta)$3 or by $p(y|x,\theta)$4 depending on context, because the batch-gradient square and the per-example sum of squares are not on the same scale.

3. Approximation mechanism and exact points of divergence

The central approximation is that the empirical Fisher diagonal and Adam’s second-moment state are both aggregations of squared gradients, but over different index sets and with different weights (Li et al., 24 Jul 2025).

The empirical Fisher uses a uniform average over examples,

$p(y|x,\theta)$5

whereas the Squisher is essentially

$p(y|x,\theta)$6

Thus Squisher substitutes exponentially decaying temporal weights $p(y|x,\theta)$7 for the uniform weight $p(y|x,\theta)$8.

A second discrepancy arises from the use of batch gradients. In practice,

$p(y|x,\theta)$9

so squaring occurs after averaging. This yields

$$F(\theta)=\mathbb{E}_{x,y\sim p(\cdot|\theta)}\left[\nabla_\theta\log p(y|x,\theta)\nabla_\theta\log p(y|x,\theta)^\top\right],$$0

which corresponds to the diagonal of a joint Fisher over the mini-batch rather than the average of per-example squares. The exposition therefore recommends optional rescaling by $$F(\theta)=\mathbb{E}_{x,y\sim p(\cdot|\theta)}\left[\nabla_\theta\log p(y|x,\theta)\nabla_\theta\log p(y|x,\theta)^\top\right],$$1 when magnitude fidelity is required, as in Elastic Weight Consolidation.

A third discrepancy is temporal. The empirical Fisher squares gradients at the final parameters $$F(\theta)=\mathbb{E}_{x,y\sim p(\cdot|\theta)}\left[\nabla_\theta\log p(y|x,\theta)\nabla_\theta\log p(y|x,\theta)^\top\right],$$2, whereas the Squisher aggregates squared gradients collected at successive $$F(\theta)=\mathbb{E}_{x,y\sim p(\cdot|\theta)}\left[\nabla_\theta\log p(y|x,\theta)\nabla_\theta\log p(y|x,\theta)^\top\right],$$3. The stated justification is that if training converges, the $$F(\theta)=\mathbb{E}_{x,y\sim p(\cdot|\theta)}\left[\nabla_\theta\log p(y|x,\theta)\nabla_\theta\log p(y|x,\theta)^\top\right],$$4 stabilize and the accumulator concentrates on a narrow band of similar gradients. Under such mild stationarity assumptions and with sufficiently many steps covering all data patterns, the exponential moving average of squared batch gradients provides a close surrogate for the diagonal empirical Fisher.

4. Computational profile, bias structure, and limitations

The principal practical distinction is computational cost (Li et al., 24 Jul 2025). A true Fisher diagonal requires either per-example gradient computations, such as $$F(\theta)=\mathbb{E}_{x,y\sim p(\cdot|\theta)}\left[\nabla_\theta\log p(y|x,\theta)\nabla_\theta\log p(y|x,\theta)^\top\right],$$5 backward passes or a specialized library such as BackPACK, or at least a loop over $$F(\theta)=\mathbb{E}_{x,y\sim p(\cdot|\theta)}\left[\nabla_\theta\log p(y|x,\theta)\nabla_\theta\log p(y|x,\theta)^\top\right],$$6 examples. The reported wall-clock cost ranges from seconds to hours depending on model size and dataset. By contrast, Squisher requires no extra backward passes and only reuses Adam’s stored $$F(\theta)=\mathbb{E}_{x,y\sim p(\cdot|\theta)}\left[\nabla_\theta\log p(y|x,\theta)\nabla_\theta\log p(y|x,\theta)^\top\right],$$7-state; the stated overhead is less than $$F(\theta)=\mathbb{E}_{x,y\sim p(\cdot|\theta)}\left[\nabla_\theta\log p(y|x,\theta)\nabla_\theta\log p(y|x,\theta)^\top\right],$$8 s in all studied scenarios.

The paper isolates three sources of bias. The first is the exponential weighting itself: recent gradients are emphasized more heavily, on a timescale of approximately $$F(\theta)=\mathbb{E}_{x,y\sim p(\cdot|\theta)}\left[\nabla_\theta\log p(y|x,\theta)\nabla_\theta\log p(y|x,\theta)^\top\right],$$9 steps. The second is parameter drift: if training is cut short, the accumulator may not reflect the curvature at the final $\{(x_i,y_i)\}_{i=1}^N$0. The third is magnitude mismatch due to the noncommutativity of averaging and squaring, expressed in the statement that $\{(x_i,y_i)\}_{i=1}^N$1.

These distinctions delimit a common misconception. Squisher is not identical to the Fisher diagonal; it is an approximation whose accuracy depends on training duration, optimizer decay, and whether downstream use depends on relative importance or absolute scale. A plausible implication is that applications based primarily on ranking or comparative weighting are less sensitive to scale mismatch than applications that insert the estimate directly into a regularizer.

5. Empirical behavior across Fisher-based applications

Li et al. evaluated Squisher in six applications: two forms of model merging, pruning, sparse “FISH” masking, Task2Vec embeddings, and EWC continual learning. The recurring pattern is that Squisher matches the Fisher to within experimental noise in several settings and outperforms Fisher-free baselines in all reported cases (Li et al., 24 Jul 2025).

Application	Metric	Reported result
Model Merging (Fisher-Weighted Average of T5-Large)	Average accuracy on eight fine-tuning tasks	Squisher $\{(x_i,y_i)\}_{i=1}^N$2, Fisher $\{(x_i,y_i)\}_{i=1}^N$3, unweighted average $\{(x_i,y_i)\}_{i=1}^N$4
UBGM Merging (RoBERTa)	Mean accuracy over 5 classification tasks	Squisher $\{(x_i,y_i)\}_{i=1}^N$5 Fisher $\{(x_i,y_i)\}_{i=1}^N$6, parameter averaging $\{(x_i,y_i)\}_{i=1}^N$7
Fisher Pruning (VGG-13 on CIFAR-100)	Test accuracy after pruning $\{(x_i,y_i)\}_{i=1}^N$8 of weights	Squisher only $\{(x_i,y_i)\}_{i=1}^N$9–$$\operatorname{diag}F(\theta)\approx \frac{1}{N}\sum_{i=1}^N g_i(\theta)\odot g_i(\theta),$$0 below Fisher, far above random pruning
FISH Mask (BERT on GLUE)	GLUE average score with $$\operatorname{diag}F(\theta)\approx \frac{1}{N}\sum_{i=1}^N g_i(\theta)\odot g_i(\theta),$$1 parameters masked	Squisher $$\operatorname{diag}F(\theta)\approx \frac{1}{N}\sum_{i=1}^N g_i(\theta)\odot g_i(\theta),$$2, Fisher $$\operatorname{diag}F(\theta)\approx \frac{1}{N}\sum_{i=1}^N g_i(\theta)\odot g_i(\theta),$$3, random mask $$\operatorname{diag}F(\theta)\approx \frac{1}{N}\sum_{i=1}^N g_i(\theta)\odot g_i(\theta),$$4
Task2Vec Embeddings	MRR and NDCG across 21 tasks	Squisher MRR $$\operatorname{diag}F(\theta)\approx \frac{1}{N}\sum_{i=1}^N g_i(\theta)\odot g_i(\theta),$$5, Fisher MRR $$\operatorname{diag}F(\theta)\approx \frac{1}{N}\sum_{i=1}^N g_i(\theta)\odot g_i(\theta),$$6, dataset-size baseline MRR $$\operatorname{diag}F(\theta)\approx \frac{1}{N}\sum_{i=1}^N g_i(\theta)\odot g_i(\theta),$$7
EWC Continual Learning	Final test accuracy after sequential-task training	Squisher matched or slightly outperformed Fisher in all task-incremental protocols

The same section reports that Fisher-diagonal computation required tens to thousands of GPU-seconds in these scenarios, whereas Squisher introduced less than $$\operatorname{diag}F(\theta)\approx \frac{1}{N}\sum_{i=1}^N g_i(\theta)\odot g_i(\theta),$$8 s of overhead regardless of model size. This suggests that the method is primarily valuable when Fisher-style parameter sensitivity is needed repeatedly or at scale.

6. Practical use and methodological scope

The paper gives several explicit recommendations for practitioners. The optimizer state, especially $$\operatorname{diag}F(\theta)\approx \frac{1}{N}\sum_{i=1}^N g_i(\theta)\odot g_i(\theta),$$9, should be saved and shared if parameter-importance measures may later be needed. Training should run for enough steps that the exponential moving average has covered a representative gradient distribution. If the absolute scale of the Fisher diagonal matters, $g_i(\theta)=\nabla_\theta\log p(y_i|x_i,\theta),$0 should be rescaled by the appropriate factor, such as batch size $g_i(\theta)=\nabla_\theta\log p(y_i|x_i,\theta),$1 or dataset size $g_i(\theta)=\nabla_\theta\log p(y_i|x_i,\theta),$2, to match the sum-of-squares Fisher. The default Adam decay $g_i(\theta)=\nabla_\theta\log p(y_i|x_i,\theta),$3 is reported to work well, but reducing $g_i(\theta)=\nabla_\theta\log p(y_i|x_i,\theta),$4 toward $g_i(\theta)=\nabla_\theta\log p(y_i|x_i,\theta),$5 or $g_i(\theta)=\nabla_\theta\log p(y_i|x_i,\theta),$6 may better align the estimate with a more uniform Fisher on shorter timescales or in cases where recent curvature is underestimated. In extremely low-data or few-step regimes, the paper recommends supplementing Squisher with a small number of fresh per-example Fisher estimates or lowering $g_i(\theta)=\nabla_\theta\log p(y_i|x_i,\theta),$7 (Li et al., 24 Jul 2025).

Within that scope, Squisher functions as a drop-in proxy for the diagonal empirical Fisher in settings where Fisher-based parameter importance is used for weighting, ranking, masking, pruning, merging, embedding construction, or continual-learning regularization. Its significance lies less in introducing a new curvature object than in showing that an optimizer artifact already present in standard training can substitute for an otherwise nontrivial estimation procedure.

7. Conceptual significance

Squisher reframes second-moment optimizer state as a statistical object rather than merely an optimization aid. The empirical contribution is that Adam’s squared-gradient accumulator can stand in for the Fisher diagonal across a diverse set of downstream procedures, with only minor and controllable biases. The theoretical contribution is the explicit clarification of where the approximation is exact in spirit and where it is not: exponential versus uniform weighting, batch-level versus per-example squaring, and accumulation along a trajectory $g_i(\theta)=\nabla_\theta\log p(y_i|x_i,\theta),$8 rather than evaluation solely at $g_i(\theta)=\nabla_\theta\log p(y_i|x_i,\theta),$9.

This places Squisher at the intersection of optimizer-state reuse, curvature approximation, and parameter-sensitivity estimation. A plausible implication is that future Fisher-based workflows may treat optimizer checkpoints as sufficient statistics for many downstream analyses, provided that the intended use tolerates the temporal and scaling biases identified in the original study.