---
title: Fisher-Orthogonal Projection (FOP)
url: https://www.emergentmind.com/topics/fisher-orthogonal-projection-fop
type: topic
---

# Fisher-Orthogonal Projection (FOP)

Fisher-Orthogonal Projection (FOP) denotes a family of information-geometric projection procedures in which gradients or parameter updates are constrained to be orthogonal, under a Fisher-induced metric, to directions deemed redundant, protected, or task-preserving. In the literature provided here, the term appears in at least three closely related roles: as the core mechanism in the large-batch natural-gradient method "Fisher-Orthogonal Projection Methods for Natural Gradient Descent with Large Batches" [2508.13898], as the central projection principle in the continual-learning optimizer "Fisher-Orthogonal Projected Natural Gradient Descent for Continual Learning" [2601.12816], and as the preservation mechanism used by PhysGuard for sim-to-real neural PDE surrogates [2606.16602]. Across these settings, FOP is used to preserve useful structure while still permitting adaptation or descent.

## 1. Conceptual scope and unifying idea

The common principle behind FOP is that not all gradient components should be treated equally. In standard first-order updates, directions are separated only by their Euclidean coordinates. FOP instead uses the Fisher information matrix to define which directions are meaningful, redundant, or dangerous to move along. In the large-batch setting, the projected object is a gradient-difference term extracted from two sub-batches; in continual learning, the projected object is the update for the current task relative to gradients associated with previous tasks; in PhysGuard, the protected directions are top Fisher eigendirections estimated from simulation data and removed from fine-tuning updates [2508.13898] [2601.12816] [2606.16602].

| Setting | Projected quantity | Reference or protected directions |
|---|---|---|
| Large-batch natural gradient | $g_{\rm diff}$ | $g_{\rm avg}$ under the Fisher metric |
| Continual learning | Update $v$ or projected gradient | Previous-task gradients $G$ under $F_{\rm old}$ |
| PhysGuard | Fine-tuning gradient $g$ | Top Fisher eigendirections $U$ |

This shared geometry is important because each paper frames Euclidean orthogonality as insufficient for the relevant invariances. The continual-learning formulation explicitly contrasts Fisher-orthogonal constraints with methods that operate in Euclidean parameter space. The large-batch formulation uses Fisher-orthogonality to isolate intra-batch variability that is not already represented by the average gradient. PhysGuard uses Fisher-derived sensitive directions to preserve low-frequency physical structure during sim-to-real adaptation.

## 2. Information-geometric formulation

In the large-batch formulation, model parameters are $\theta \in \mathbb{R}^P$, the Fisher information matrix is
\[
F \;=\; \mathbb E_{x\sim p_\text{data}}\Big[ \nabla_\theta\log p_\theta(x)\,\nabla_\theta\log p_\theta(x)^\top\Big],
\]
and the Fisher metric is
\[
\langle u,v\rangle_F \;=\; u^\top F\,v.
\]
A large mini-batch $\mathcal B$ is split into two disjoint sub-batches $\mathcal B_1,\mathcal B_2$, yielding
\[
g_1 \;=\;\nabla_\theta \mathcal L(\theta;\mathcal B_1),\quad
g_2 \;=\;\nabla_\theta \mathcal L(\theta;\mathcal B_2),
\]
with
\[
g_{\rm avg} \;=\;\tfrac12(g_1+g_2), \qquad
g_{\rm diff}\;=\;g_1 - g_2.
\]
The Fisher-orthogonal projection removes from $g_{\rm diff}$ any component already present in $g_{\rm avg}$:
\[
s \;=\;
\frac{g_{\rm diff}^\top\,F\,g_{\rm avg}}
     {g_{\rm avg}^\top\,F\,g_{\rm avg} \;+\;\epsilon},
\qquad
g_{\rm diff}^\perp
\;=\;
g_{\rm diff}
\;-\;
s\,g_{\rm avg},
\]
so that $\langle g_{\rm avg},\,g_{\rm diff}^\perp\rangle_F = 0$. The corrected gradient is then
\[
g_{\rm comb}
\;=\;
g_{\rm avg}
\;+\;
\beta\,g_{\rm diff}^\perp,
\]
with the adaptive weight
\[
\beta^*
\;=\;
\frac{
g_{\rm avg}^\top\,F^{-1}\,g_{\rm diff}^\perp
}{
(g_{\rm diff}^\perp)^\top\,F^{-1}\,g_{\rm diff}^\perp
}.
\]
Using damping $\lambda$, the final update is
\[
\theta_{t+1}
\;=\;
\theta_t
\;-\;
\eta\,\bigl(F + \lambda I\bigr)^{-1}\,g_{\rm comb}.
\]
This construction is presented as a variance-aware update direction that leverages gradients from two sub-batches and enhances the average gradient with a component of the gradient difference that is orthogonal to the average under the Fisher metric [2508.13898].

In continual learning, the formulation is explicitly task-sequential. Let $\theta\in\mathbb{R}^p$, let $L_t(\theta)$ denote the loss of task $t$, let $g_t=\nabla_\theta L_t(\theta)$, and let
\[
F_t = \mathbb E_{x,y}\big[\nabla_\theta \log p_\theta(y|x)\,\nabla_\theta \log p_\theta(y|x)^\top\big].
\]
After tasks $1,\dots,k-1$, collect final gradients $g_1,\dots,g_{k-1}$ as columns of $G\in\mathbb{R}^{p\times m}$ and denote by $F_{\rm old}$ the Fisher matrix estimated on previous-task data. The Fisher-orthogonal complement is the set of all parameter increments $v$ satisfying
\[
G^\top F_{\rm old} v = 0.
\]
To project an arbitrary vector $u\in\mathbb{R}^p$, define
\[
M = F_{\rm new}^{-1/2} F_{\rm old} G,
\qquad
P = I_p - M(M^\top M)^{-1}M^\top,
\]
compute
\[
u_{\rm whitened} = F_{\rm new}^{1/2}u,
\qquad
u_{\rm proj} = F_{\rm new}^{-1/2}P\,u_{\rm whitened},
\]
and obtain a projected vector satisfying $G^\top F_{\rm old}u_{\rm proj}=0$. The resulting projected natural-gradient update solves a constrained problem that maximizes progress on the new loss, stays within a trust-region in the new-task Fisher metric, and is Fisher-orthogonal to old tasks:
\[
v^*
= \epsilon\,
\frac{F_{\rm new}^{-1/2}PF_{\rm new}^{-1/2}g}
{\sqrt{g^\top F_{\rm new}^{-1/2}P\,F_{\rm new}^{-1}P\,F_{\rm new}^{-1/2}g}}
=
\epsilon\,
\frac{F_{\rm new}^{-1/2}PF_{\rm new}^{-1/2}g}
{\|F_{\rm new}^{-1/2}PF_{\rm new}^{-1/2}g\|_2}.
\]
If the orthogonality constraint is dropped, $P\to I$ and the method recovers ordinary natural gradient; if $F_{\rm new}\to I$ and $F_{\rm old}\to I$, it recovers Euclidean orthogonal-gradient descent [2601.12816].

## 3. Geometric properties and interpretation

The continual-learning formulation states three geometric properties. First, the update $v^*$ is reparameterization-invariant because all norms and projections are defined via the Fisher information. Second, it guarantees descent in the Fisher metric, with $g^\top v^*\ge 0$. Third, by enforcing $G^\top F_{\rm old}v^*=0$, the second-order change in KL divergence on each previous task is zero. The exposition further states that this ensures that moving by such a $v$ causes, to second order, no change in the model’s output distributions on prior tasks [2601.12816].

The large-batch formulation emphasizes a different, though related, geometric role. It is motivated by the claim that very large batches suppress gradient noise and that high damping can wash out the curvature information in KFAC. FOP is introduced specifically to restore useful curvature at large batch sizes by injecting a variance-aware correction into the natural-gradient step, while projecting out redundant directions under the Fisher metric so that only genuinely new curvature information is added [2508.13898].

A recurrent misconception is to equate FOP with ordinary Euclidean orthogonalization. The primary formulations above do not do that: they define orthogonality with respect to the Fisher metric. PhysGuard presents a useful contrast. There, the protected subspace is first identified from the empirical Fisher eigenspectrum, but the online projector is written in Euclidean form,
\[
P = I - UU^\top,
\qquad
g_{\rm proj} = Pg = g - U(U^\top g).
\]
The paper explicitly notes that one may view this under the Fisher-induced inner product similarly, but in practice $UU^\top$ under $\ell_2$ is equivalent once the columns of $U$ are Fisher eigenvectors scaled to unit $\ell_2$ norm [2606.16602]. This indicates that the decisive ingredient is often not the final algebraic appearance of the projector, but the Fisher-derived choice of the protected subspace.

## 4. Algorithms and computational profile

The continual-learning algorithm is presented with diagonal-Fisher approximations for both $F_{\rm old}$ and $F_{\rm new}$. The procedure is: initialize $\theta$; train on task $1$ normally; estimate diagonal Fisher $F_{\rm old}\leftarrow \operatorname{diag}\{E[(\nabla \log p)^2]\}$; store last-task gradients $G\leftarrow[g_1,\dots,g_k]$; for each subsequent task, compute a diagonal $F_{\rm new}$ on a small random batch; for each minibatch compute $g=\nabla_\theta L_t(\theta)$, build
\[
u \leftarrow F_{\rm new}^{-1/2}g,
\qquad
M \leftarrow F_{\rm new}^{-1/2}F_{\rm old}G,
\]
solve $(M^\top M+\lambda I)^{-1}$ via $m\times m$ inversion, project
\[
u_{\rm proj}\leftarrow (I-M(M^\top M+\lambda I)^{-1}M^\top)u,
\]
set
\[
v^*\leftarrow \epsilon F_{\rm new}^{-1/2}u_{\rm proj},
\]
and update $\theta\leftarrow \theta-\eta v^*$. After each task, update
\[
F_{\rm old}\leftarrow (1-\alpha)F_{\rm old}+\alpha F_{\rm new},
\]
and append new gradients to $G$ while keeping at most $m$ columns. Here $\lambda$ is a small ridge for numerical stability, $\alpha\in[0,1]$ averages the old-task Fisher, and $\epsilon$ is folded into the learning rate $\eta$. The cost of diagonal Fisher storage and multiplication is $O(p)$, storing $m$ old gradients costs $O(pm)$, forming $M$ costs $O(pm)$, inverting $M^\top M$ costs $O(m^3)$, and the total per-batch extra cost is $O(pm+m^3)$. The exposition states that in practice $m\approx 50$–$100$ so $m^3$ is negligible, and that a PreFisher variant stores $\tilde g_j=F_jg_j$ once per task to eliminate $F_{\rm old}$ in the inner loop [2601.12816].

The large-batch algorithm is correspondingly direct if a KFAC pipeline already exists. Each full batch is split into two equal halves, two gradients are computed, the Fisher-metric projection is applied, and the combined gradient is passed through the existing KFAC preconditioner instead of the mean gradient. The pseudocode states that the method requires two forward-backward passes per update, so approximately $2\times$ the cost of a single gradient step. KFAC factors cost $O(p_\ell^2)$ per layer, inversions cost $O(p_\ell^3)$ but are typically done every few steps, memory stores two gradient vectors plus KFAC factors and inverses, and distributed training uses two global gradients in parallel over disjoint GPU groups with dual all-reduces. The paper further states that on modern multi-GPU nodes such as $8\times$MI300X, the method can sustain batches of $50\,\text{k}+$ with no extra memory beyond standard KFAC [2508.13898].

These algorithmic descriptions show that FOP is not a monolithic implementation. It is a projection principle that can be instantiated with diagonal Fisher approximations, Kronecker-factored preconditioners, layer-wise subspace estimates, or offline Gram-matrix decompositions, depending on the regime.

## 5. Reported empirical behavior

The continual-learning and large-batch papers report distinct evaluation protocols, but both present FOP as competitive precisely where unprojected updates lose either memory or curvature information [2601.12816] [2508.13898].

| Setting | Benchmark | Reported outcome |
|---|---|---|
| Continual learning | Split-MNIST & Rotated-MNIST | FOPNG achieved $\sim 95\%$ vs OGD $\sim 90\%$ and EWC $\sim 92\%$ |
| Continual learning | Split-CIFAR10 | FOPNG $\sim 75\%$ vs OGD $\sim 65\%$, EWC $\sim 70\%$ |
| Continual learning | Split-CIFAR100 | FOPNG $\sim 45\%$ vs OGD $\sim 40\%$, EWC $\sim 43\%$ |
| Continual learning | Permuted-MNIST | FOPNG performed slightly worse than EWC |
| Large-batch training | CIFAR-10, BS=$2\,048$ | SGD needed $58 / 743$ s, KFAC $37 / 589$ s, FOP $29 / 475$ s |
| Large-batch training | CIFAR-10, BS=$32\,768$ | only FOP reached $91\%$ in $60 / 90.6$ s |
| Large-batch training | CIFAR-10, BS=$50\,000$ | FOP in $82 / 84.3$ s |
| Large-batch training | ImageNet-100, BS=$4\,096$ | KFAC $87 / 2\,715$ s, FOP $49 / 1\,670$ s |
| Large-batch training | ImageNet-1K, BS=$8\,192$ | only FOP hit $75.9\%$ in $40 / 335$ min |

For continual learning, the paper also states that performance is robust to the extra hyperparameters $\lambda$ and $\alpha$, with a good default of $\lambda\in[10^{-4},10^{-3}]$ and $\alpha=0.5$, while wall-clock overhead is $40$–$80\%$ above EWC/OGD and is dominated by extra Fisher-vector multiplies [2601.12816]. The same section notes an important caveat: on Permuted-MNIST, the method is slightly worse than EWC, likely because the random permutations create highly OOD tasks; on more realistic tasks with gradual shifts it consistently outperforms.

For large-batch training, the evaluation spans CIFAR-10 with ResNet-18, ImageNet-100 with T2T-ViT, ImageNet-1K with ResNet-50, and long-tailed CIFAR with ResNet-32. The paper further reports that on ImageNet-100 at BS=$512$, AdamW required $97 / 17\,499$ s, KFAC $42 / 8\,316$ s, and FOP $44 / 9\,536$ s, whereas at BS=$4\,096$ KFAC required $87 / 2\,715$ s and FOP $49 / 1\,670$ s. On ImageNet-1K at BS=$1\,024$, SGD required $71 / 2\,511$ min, KFAC $35 / 1\,337$ min, and FOP $32 / 1\,306$ min. On long-tailed CIFAR, FOP reduces Top-1 error by $2.3$–$3.3\%$ relative to strong baselines and by $\sim 2\%$ vs KFAC [2508.13898].

## 6. Extensions and broader significance

PhysGuard extends the projection idea beyond optimization stability and continual learning into sim-to-real adaptation for neural operators. Starting from pretrained parameters $\theta^*$ and per-sample simulation gradients
\[
g_i \coloneqq \nabla_\theta \,\ell(f_{\theta^*}(x_i),y_i)\in\mathbb{R}^d,
\]
it forms the empirical Fisher
\[
F \coloneqq \frac{1}{N}G^\top G,
\]
where $G$ stacks the sample gradients row-wise. Because direct eigendecomposition of the $d\times d$ Fisher is infeasible when $d$ is large, the paper uses the Gram matrix
\[
K \coloneqq GG^\top \in \mathbb{R}^{N\times N},
\]
whose nonzero eigenvalues coincide with those of $F$. In practice this is done per layer: build $G^{(m)}$, form $K^{(m)}=G^{(m)}(G^{(m)})^\top$, eigendecompose it, and recover Fisher eigenvectors as
\[
u_j^{(m)} = (G^{(m)})^\top v_j^{(m)}.
\]
Only a compact leading subspace is protected: choose the minimal $k$ such that
\[
\rho(k)=\frac{\sum_{j=1}^k \lambda_j}{\sum_{j=1}^N \lambda_j}\ge \tau,
\]
with $\tau=0.9$ in all experiments, then define $U=[u_1,\dots,u_k]$ and project each fine-tuning gradient as
\[
g_{\rm proj}=g-U(U^\top g).
\]
The reported complexity is $O(N\cdot d_{\rm total})$ for gradient collection, $O(N^2\cdot d_m)$ for Gram construction, $O(N^3)$ for the SVD of $K$, $O(N\cdot k\cdot d_m)$ for recovering $U$, and $O(d_m\cdot k)$ per layer for each online projection. The paper states that all steps fit on a single GPU and take minutes offline for $N\approx 200$, with only a few milliseconds of per-step overhead during fine-tuning [2606.16602].

PhysGuard also gives a domain-specific interpretation of Fisher-sensitive directions. Its spectral probe experiment states that the dominant Fisher directions are strongly associated with low-frequency output structures, and that projecting away from them preserves large-scale physics while still permitting adaptation in the remaining nullspace. Under severe domain shift, the paper reports that it reduces low-frequency error by up to $32\%$ compared to standard fine-tuning while maintaining adaptability [2606.16602].

Taken together, these works suggest that FOP is best understood not as one optimizer but as a reusable information-geometric design pattern. In one regime it revives natural-gradient curvature at extreme batch size; in another it preserves prior task outputs during sequential learning; in another it protects physics-critical subspaces during sim-to-real transfer. A plausible implication is that the central research question is not whether projection should be used, but which Fisher-defined subspace should be protected or amplified for a given learning regime.

Source: https://www.emergentmind.com/topics/fisher-orthogonal-projection-fop