Fisher Information Ranking for Masking

• The paper demonstrates how Fisher information ranking effectively identifies sensitive parameters for selective masking, thereby improving efficiency and robustness in learning systems.
• The methodology employs a diagonal approximation of the Fisher Information Matrix to rank parameters and features, facilitating adaptive masking strategies in active, continual, unlearning, and adversarial scenarios.
• Empirical results show notable gains such as 2–5% accuracy improvements in active learning and enhanced stability in continual learning, underscoring the practical impact of FI-based masking.

Fisher Information Ranking for Masking characterizes a family of techniques that leverage the Fisher Information (FI) to identify and rank the most critical dimensions, parameters, or features in statistical models and learning systems. Masking, in this context, refers to the selective zeroing, freezing, or perturbation of high‐ or low‐FI entries to improve efficiency, robustness, unlearning, privacy, or continual learning performance. The FI ranking principle provides a mathematically principled measure of importance: entries with large FI denote high sensitivity of the model's output or loss to their changes, and thus guide masking decisions across applications.

1. Fisher Information: Definition and Diagonal Approximation

The Fisher Information Matrix (FIM) quantifies the expected local curvature of the log-likelihood function with respect to model parameters. For models parameterized by $\theta \in \mathbb{R}^d$ and observations $(x, y)$ with density $p_\theta(y|x)$, the FIM is

$$F(\theta) = \mathbb{E}_{(x,y)} \left[ \nabla_\theta \log p_\theta(y|x) \; \nabla_\theta \log p_\theta(y|x)^\top \right].$$

In large-scale neural nets, the full FIM is intractable. All contemporary masking strategies, including FisherMask and FGGM, employ the diagonal approximation: $$\mathrm{FIM}_i = \frac{1}{|S|} \sum_{t=1}^{|S|} \mathbb{E}_{y \sim p_\theta(\cdot | x_t)} \left[ \frac{\partial}{\partial \theta_i} \log p_\theta(y|x_t) \right]^2.$$ In practice, the FI per parameter is accumulated as the average of squared gradients evaluated at model predictions or observed labels over a dataset, computed efficiently via standard automatic differentiation (Gul et al., 2024, Tan et al., 26 Jan 2026, Liu et al., 2023).

2. Ranking and Masking Schemes Based on Fisher Information

The FI ranking procedure sorts components (parameters, features, actions) by their importance as measured by the FI score. Several masking strategies emerge, common to all applications:

• Threshold/Top-k Masking: Identify the $k$ highest-FI entries and retain (or mask) them, setting the mask $M_i = 1$ if $\mathrm{FIM}_i$ is above a quantile threshold or within the top $k$, $M_i = 0$ otherwise (Gul et al., 2024, Tan et al., 26 Jan 2026).
• Adaptive Quantile Thresholding: In continual learning, the threshold is chosen such that a fixed fraction $\alpha$ of parameters with largest $\mathrm{FIM}_i$ are preserved, dynamically adjusting across tasks (Tan et al., 26 Jan 2026).
• Feature-Wise Masking: For input features, FI is computed per input dimension by differentiating Fisher-based sensitivity scores with respect to each feature, yielding a per-input feature ranking (Martin et al., 2019).
• Task-Specific Aggregation: For structured parameters such as weight matrices, FI values may be aggregated over input dimensions to yield output-unit rankings (Tan et al., 26 Jan 2026).

The canonical pipeline computes FI scores, sorts by rank, selects a subset under constraints (e.g., fixed budget or plasticity ratio), and constructs a binary mask used to modify model updates or inputs.

3. Application Domains and Algorithmic Workflows

Active Learning (FisherMask)

FisherMask (Gul et al., 2024) employs per-parameter FI ranking to select the $k$ most sensitive parameters, building a sparse mask that focuses further active learning sample selection on these directions. The algorithm iteratively:

1. Computes diagonal FI over the current unlabeled dataset pool.
2. Ranks and masks the top $k$ parameters.
3. Builds a reduced Fisher object restricted to masked parameters.
4. Selects samples by maximizing expected reduction in FI uncertainty along the masked subspace, using an explicit trace-based criterion.
5. Updates the sample set and retrains, repeating the cycle for multiple rounds. The pseudocode is explicitly given in [(Gul et al., 2024), Algorithm 1], supporting full reproducibility.

Continual Learning (FGGM)

FGGM (Tan et al., 26 Jan 2026) applies FI ranking to mitigate catastrophic forgetting in LLMs under task sequences:

• Diagonal Fisher is estimated per task, potentially with aggregation for noisy matrix structures.
• Parameters are ranked and an adaptive quantile threshold yields a binary mask freezing high-FI weights.
• Gradients are masked elementwise, confining weight updates and achieving a controllable trade-off between task stability and adaptation.

Privacy, Unlearning, and Robustness

Fisher masking for unlearning (Liu et al., 2023) computes the difference between FI on the forget and remain sets, $s_j = F_{f,jj} - F_{r,jj}$, masking parameters chiefly responsible for encoding the target data. This accelerates and stabilizes subsequent fine-tuning, facilitating efficient and complete unlearning.

For adversarial analysis (Martin et al., 2019), FI-based feature scores rank input features by their effect on the model curvature; masking the top features leads to effective detection and defense strategies.

Control and Sensing Masking

In control contexts (Jain et al., 2024), FI determinant is incorporated as a regularizer or constraint in MDP optimization (not for parameter ranking as in learning, but directly ranking entire policies or kernels by their ability to mask model identifiability from an adversary). Candidate strategies are ranked in terms of $\det \mathcal{I}$, enabling formal trade-offs between performance and privacy.

4. Mathematical Formulations and Pseudocode

Detailed FI masking algorithms are specified explicitly:

Per-Parameter FI Computation and Masking (generic structure):

for each parameter i: FIM_i = (1/|S|) * sum_x_in_S (grad_theta_i log p_theta(y|x))^2 threshold = quantile(FIM, 1 - alpha) mask = (FIM > threshold).astype(int)

(Pseudocode reflects (Gul et al., 2024, Tan et al., 26 Jan 2026, Liu et al., 2023).)

Sample Selection in AL with FisherMask (Gul et al., 2024):

• For each candidate $x$, compute: $$\mathrm{score}(x) = \operatorname{tr}(V_x^T M^{-1}_i F_{\theta_r} M^{-1}_i V_x A^{-1}),$$ where $A = F_{\theta_r} + V_x^T M^{-1}_i V_x$. Samples maximizing this score are selected and labels queried.

Block-Coordinate Descent for FI-Constrained MDPs (Jain et al., 2024):

• Alternate between updating cost $c$, policy occupancy $\pi$, or kernel $P$ to minimize expected cost plus (regularized) FI determinant, with explicit constraints.

The table below compares the core masking operations:

Domain	Ranking Object	Mask Type	Selection Principle
Active Learning	Param FI ($F_i$)	Top-k	Sensitivity to new data
Continual Learning	Param FI ($F_i$)	Quantile ($\alpha$)	Balance stability/plasticity
Unlearning	$F_{f} - F_{r}$	Top-R	Remove parameters encoding $D_f$
Adversarial/Feature	Feature FI ($s_i$)	Top-k	Impact on model decision
Control/MDP	$\det \mathcal{I}$	Policy-level	Minimize identifiability vs. cost

5. Theoretical Motivation and Properties

FI provides a second-order (curvature-based) importance measure. Large FI entries identify:

• Sensitive parameters—the loss landscape is steep in these directions.
• Directions critical for retention of information (for stability) vs. directions available for adaptation (plasticity).
• In unlearning, parameters whose Fisher difference is large encode information unique to forgotten data.

Mathematically, masking high-FI parameters minimizes the risk of abrupt loss increases (stability); freezing low-FI parameters confers adaptability. In the MDP setting, minimizing the FI determinant raises the Cramér–Rao bound for estimator variance, increasing inference uncertainty for adversaries (Jain et al., 2024). Theoretical bounds on KL divergence after masking further quantify retention/unlearning trade-offs (Liu et al., 2023).

6. Empirical Evidence and Impact

• Active Learning: FisherMask yields 2–5% higher accuracy under severe label sparsity and class imbalance compared to entropy, margin, and k-center baselines, across CIFAR-10, FashionMNIST (Gul et al., 2024).
• Continual Learning: FGGM demonstrates up to 4.4% absolute improvement over magnitude-based masking on TRACE-OP benchmarks and stable scaling to 7B LLMs (Tan et al., 26 Jan 2026). Input-aggregation in FI computation is found crucial.
• Unlearning: Fisher masking nearly completely erases forget-class accuracy while preserving original retain-class performance, with quick fine-tune recovery. Effects generalize to backdoor and noisy-label scenarios (Liu et al., 2023).
• Adversarial Examples: FI-based feature sensitivity scores detect adversarial samples with high AUC and localize salient regions (Martin et al., 2019).
• Control: FI-constrained radar controllers achieve significant masking of transition parameters with minor cost perturbations; the Fisher-based criterion exceeds maximum-entropy randomization in confounding adversaries at fixed cost (Jain et al., 2024).

7. Contextualization and Ongoing Directions

Fisher information ranking for masking unifies a range of algorithmic advances across supervised, unsupervised, and reinforcement learning, with clear theoretical grounding and empirical support. Connections to Elastic Weight Consolidation (EWC) and parameter importance estimation in continual learning are direct, though Fisher masking hardens these ideas into explicit masking rather than regularization (Tan et al., 26 Jan 2026). In adversarial robustness and privacy, FI-based ranks provide interpretable and adaptive masking rules.

Limitations of current approaches include computational cost of large-batch FI estimation, possible loss of generality beyond diagonal FIM approximations, and applications primarily to scenarios with manageable mask densities or task decomposability. Extensions under study involve full-matrix schemes, learning adaptive masks, and broadening application domains. The robustness of FI-masked systems under worst-case distributional and task shift settings remains an active research area.