FIM-LoRA: Empirical Fisher-Informed Rank Allocation

• The paper presents FIM-LoRA, a calibration-time method that uses diagonal eFIM estimates to guide per-layer rank allocation in LoRA.
• It employs a two-phase water-filling strategy to redistribute a fixed rank budget, ensuring no additional parameters or runtime overhead compared to uniform-rank LoRA.
• Empirical evaluations demonstrate that FIM-LoRA yields interpretable, task-driven rank patterns, particularly benefiting transformer models by focusing adaptation on informative layers.

Empirical Fisher-Informed Rank Allocation (FIM-LoRA) is a calibration-time methodology for per-layer rank selection in Low-Rank Adaptation (LoRA). It leverages empirical Fisher Information Matrix (eFIM) estimates—specifically, diagonal approximations based on gradient variance—collected over a limited number of calibration mini-batches to identify the most “task-informative” LoRA adapter matrices. The principal outcome is a standard LoRA parameterization with an uneven, information-driven allocation of the total rank budget across modules, incurring neither new parameters, training, nor inference overhead relative to uniform-rank LoRA. FIM-LoRA provides interpretable, task-driven rank patterns, especially beneficial for large transformers where different modules contribute unequally to adaptation performance (Sathyavageeswaran, 16 May 2026).

1. Motivation and Conceptual Foundation

Conventional LoRA assigns a uniform rank to every adapted weight matrix, disregarding the empirical reality that transformer layers and projections differ in their contributions to task adaptation. FIM-LoRA addresses this by allocating higher parameter capacity—via greater rank—to modules exhibiting greater loss sensitivity, as quantified by the variance of their gradients during a brief calibration phase. The eFIM diagonal of each LoRA-B parameter, at initialization, acts as a direct proxy for this layer informativeness. Empirically, this allocation scheme produces interpretable rank maps that concentrate adaptation capacity in early-to-middle layers and in value-projection modules, aligning with established transformer semantics (Sathyavageeswaran, 16 May 2026).

2. Calibration Phase: Gradient-Variance Estimation

During calibration, the base model is frozen and LoRA adapters are inserted at each adapted projection with a uniform initial rank $r$. Over $T$ mini-batches (typically $T=8$), the following procedure is executed:

• For each mini-batch, a forward and backward pass is run, but only the gradients $\partial \mathcal{L}_t/\partial B_\ell$ for adapter $\ell$ are retained.
• For each element in $B_\ell$, the squared gradients are accumulated in $F_\ell$ (an array shaped $d_{\text{out}} \times r$).
• Gradients of $A_\ell$ are not used: at initialization, $B_\ell = 0$ implies $T$0.

This restricted Fisher estimation yields approximately $T$1 memory savings compared to full-model Fisher evaluation for typical parameter regimes ($T$2, $T$3) (Sathyavageeswaran, 16 May 2026). Only the diagonal eFIMs per adapter are computed and stored.

3. Mathematical Formulation

The informativeness of each parameter in $T$4 is quantified as the mean squared gradient across calibration batches:

$T$5

where $T$6 for parameter $T$7 in $T$8. Mean-centering (i.e., using empirical variance) is optional, but in practice the mean gradients are near zero at initialization, rendering raw squared gradients adequate. To aggregate the per-element eFIM into a per-layer importance score,

$T$9

is used, representing the average gradient variance across all LoRA-B parameters in module $T=8$0. Higher scores correspond to greater expected adaptation utility.

4. Budget-Constrained Proportional Rank Allocation

Given a total rank budget $T=8$1 across $T=8$2 adapted modules and a minimum rank $T=8$3 (optionally also $T=8$4), FIM-LoRA redistributes $T=8$5 via a two-phase “water-filling” procedure:

• Phase 1: Proportionally assign ranks $T=8$6 to each layer. If any $T=8$7, cap and fix at $T=8$8, remove from $T=8$9, and subtract from $\partial \mathcal{L}_t/\partial B_\ell$0.
• Phase 2: For remaining layers, use largest-remainder rounding on $\partial \mathcal{L}_t/\partial B_\ell$1. Enforce $\partial \mathcal{L}_t/\partial B_\ell$2, borrowing rank from least-informative modules where needed so the sum is exactly $\partial \mathcal{L}_t/\partial B_\ell$3.

These integer per-layer ranks define the final adapter configuration. The allocation is strictly biasing rank to high-signal modules while respecting the prescribed budget and optional per-layer caps.

5. In-Place Adapter Resizing and Integration

After allocation, LoRA adapters are resized in place for each layer:

• For $\partial \mathcal{L}_t/\partial B_\ell$4, the first $\partial \mathcal{L}_t/\partial B_\ell$5 rows are kept, new rows (if created) are randomly initialized (e.g., via Kaiming).
• For $\partial \mathcal{L}_t/\partial B_\ell$6, columns are zero-padded or truncated as necessary.
• The scaling factor $\partial \mathcal{L}_t/\partial B_\ell$7 is updated to $\partial \mathcal{L}_t/\partial B_\ell$8.

This produces a standard LoRA adapter with an explicit per-layer rank pattern (conforming to the “rank_pattern” field in the PEFT library). Fine-tuning and deployment require no changes in code infrastructure, and there are no additional parameters or runtime overhead (Sathyavageeswaran, 16 May 2026).

6. Quantitative Evaluation and Rank Pattern Analysis

On GLUE with DeBERTa-v3-base, FIM-LoRA (with $\partial \mathcal{L}_t/\partial B_\ell$9, $\ell$0) achieves an average score of 88.60 vs. 88.67 for uniform LoRA and 88.54 for a random-rank control at the same rank budget. On seven commonsense reasoning tasks with LLaMA-3-8B ($\ell$1), FIM-LoRA with $\ell$2 achieves 68.47 (LoRA: 68.74; FIM-LoRA with $\ell$3 underperforms due to over-concentration of rank). For per-layer analysis, value projections consistently receive the highest rank (mean $\ell$429.7 for $\ell$5), query/key/gate projections remain near the minimum (mean $\ell$68), and early-to-middle layers (0–7) are assigned approximately 3$\ell$7 the rank of late layers (24–31). This assignment pattern agrees with prior findings: early layers and value projections are the loci of meaningful task-specific adaptation (Sathyavageeswaran, 16 May 2026).

7. Extensions and Relationship to Geometry-Aware LoRA

FIM-LoRA represents a light-weight, calibration-only alternative to ongoing geometry-aware methods. For example, GRIT combines eFIM-based dynamic rank adaptation with K-FAC natural-gradient preconditioning and periodic Fisher-guided subspace reprojection. GRIT adaptively selects effective rank $\ell$8 at each reprojection step using cumulative Fisher “energy” criteria and enforces stability via bounds and hysteresis. While GRIT operates dynamically during fine-tuning and incorporates curvature alignment in gradient updates, FIM-LoRA confines all information-theoretic decisions to the pre-tuning calibration window, aiming for maximal marketplace compatibility and zero runtime overhead (Sathyavageeswaran, 16 May 2026, Saha et al., 1 Jan 2026).

Method	Rank Adaptation Phase	Fisher Usage	Overhead
FIM-LoRA	Calibration (pre-training)	eFIM diagonal	$\ell$9 backward passes
GRIT	Throughout fine-tuning	Full Fisher in rank space	+6–10% step time

A plausible implication is that FIM-LoRA is preferable in settings where serving infrastructure or deployment constraints prohibit algorithmic deviation from standard LoRA, while dynamic approaches such as GRIT provide further efficiency gains in tasks and hardware contexts tolerant of moderate additional computation.

8. Practical Workflow

A high-level workflow of FIM-LoRA is as follows (Sathyavageeswaran, 16 May 2026):

1. Insert uniform-rank LoRA adapters into the frozen base model.
2. Initialize per-module eFIM accumulators.
3. For $B_\ell$0 calibration batches:
   • Compute forward loss, backward gradients; accumulate squared gradients into per-layer eFIM diagonals.
4. Aggregate mean gradient variances into per-layer scores.
5. Run budget-constrained allocation to assign integer ranks to each module.
6. Resize adapters in place as per the allocation.
7. Train as with standard LoRA; all subsequent steps are unchanged.

The procedure can be summarized in pseudocode explicitly provided in the reference (Sathyavageeswaran, 16 May 2026), with no deviation from standard LoRA APIs or hyperparameter logic after calibration.

9. Empirical and Theoretical Implications

FIM-LoRA demonstrates that with as few as eight backward passes over initial LoRA-B gradients, it is possible to quantitatively map task informativeness and allocate adaptation capacity in a way that matches or closely approaches the empirical performance of best-case uniform LoRA. The resulting rank maps exhibit strong agreement with prior mechanistic transformer studies, highlighting the utility of information-theoretic metrics in parameter-efficient fine-tuning. A plausible implication is that further refinements—for example, combining with dynamic methods such as K-FAC preconditioning or Fisher-aligned subspace tracking—can yield additional parameter savings and performance robustness in highly resource-constrained deployments (Saha et al., 1 Jan 2026).