Effectiveness of stronger curvature estimators under expanded calibration

Determine whether stronger shrinkage, structured Fisher factors, or substantially more calibration data can reverse the observed performance disadvantage of dense K-FAC relative to diagonal whitening for Fisher-whitened cross-covariance adaptation.

Background

FCCA constructs frozen left and right bases by whitening the signed input–error cross-covariance with input- and output-side Fisher moments before applying a rank constraint. The default implementation uses diagonal second-moment estimates because dense covariance estimation is potentially unreliable with only 128–256 calibration examples.

In the reported ablation, diagonal whitening outperformed full dense K-FAC on SVAMP, ARC-C, and GSM8K. The paper attributes this result plausibly to finite-sample conditioning but does not establish whether the disadvantage is intrinsic to dense curvature or instead results from insufficient shrinkage, an unsuitable structured parameterization, or too little calibration data. Resolving this would clarify when richer curvature models improve frozen-subspace selection.

References

This supports diagonal whitening in the tested low-calibration regime, while leaving open whether stronger shrinkage, structured factors, or substantially more calibration could reverse the result.

— Frozen Cores Need Task Signal: Fisher-Whitened Cross-Covariance for Low-Resource LLM Adaptation  (2609.00762 - Ye et al., 1 Sep 2026) in Section 5.5, “Span selection and coordinate conditioning are distinct,” discussion following Table 4; Appendix, Section “Ablations and Robustness,” subsection “QR coordinates and dense K-FAC”