Budgeted Rank Allocator for Adaptive LoRA

• Budgeted Rank Allocator is a method that assigns non-uniform low-rank capacity across model layers based on calibration-phase Fisher information estimates.
• It uses a two-phase water-filling allocation process that enforces a fixed total rank budget while preserving the standard LoRA model structure.
• Empirical results show that applying a minimum floor prevents rank over-concentration and maintains competitive performance across multiple tasks.

Searching arXiv for the primary paper and closely related adaptive rank-allocation work. Budgeted Rank Allocator is a fixed-budget procedure for assigning non-uniform low-rank adaptation capacity across model components instead of using a uniform LoRA rank in every adapted matrix. In the formulation introduced with FIM-LoRA, the allocator uses a small calibration phase before fine-tuning, estimates layer informativeness from the empirical Fisher Information Matrix diagonal restricted to LoRA-B parameters, and redistributes a total rank budget proportionally across adapted modules, while preserving the standard PEFT LoRA serving interface (Sathyavageeswaran, 16 May 2026). The same general phrase also appears in other literatures, including treatment allocation, ranking-and-selection, learned indexes, and activation steering, but in parameter-efficient fine-tuning it denotes a budgeted assignment of per-layer ranks under a global rank constraint rather than a change to the downstream model architecture itself (Vanderschueren et al., 2024).

1. Definition and problem setting

In the FIM-LoRA setting, Low-Rank Adaptation assigns a uniform rank to every adapted weight matrix, whereas the Budgeted Rank Allocator begins from the premise that different layers contribute unequally to task adaptation (Sathyavageeswaran, 16 May 2026). The total budget is defined by a base uniform LoRA configuration: if $L$ is the number of adapted modules and $r$ is the uniform base rank, then the total budget is $\mathcal{B}=L\cdot r$. The allocator replaces the uniform choice by a layerwise pattern $\{r_\ell\}$ such that $\sum_\ell r_\ell=\mathcal{B}$.

The central operational constraint is that the resulting model remains a standard LoRA model. In the reported implementation, no extra parameters or new operator kernels are introduced; the result is a standard PEFT LoRA model with a rank_pattern field, and inference and merging into the base model are unchanged (Sathyavageeswaran, 16 May 2026). This makes the method a budget reallocation mechanism rather than a new adapter family.

A related misconception is that adaptive rank allocation necessarily implies a more complex deployment path. The FIM-LoRA formulation explicitly does not: the allocator modifies only how a fixed total rank budget is split across layers, and leaves serving infrastructure unchanged (Sathyavageeswaran, 16 May 2026). By contrast, other adaptive-rank frameworks may incorporate dynamic pruning, online incremental search, or input-dependent masking, which changes the algorithmic setting even when the phrase “rank allocation” is shared across papers (Liu et al., 2024, Zhou et al., 2024, Garcia et al., 23 Mar 2025).

2. Calibration-time estimation of layer informativeness

The Budgeted Rank Allocator in FIM-LoRA begins with a calibration phase before fine-tuning. It draws $T$ mini-batches $\{B_t\}_{t=1}^T$ from the fine-tuning training set, with the default $T=8$, and performs one backward pass per batch through the model with LoRA adapters inserted at zero initialization (Sathyavageeswaran, 16 May 2026). During this phase, only gradients on the LoRA-B matrices are accumulated.

The restriction to LoRA-B is not heuristic but structural. At initialization, $B=0$ but $A\neq0$, so only $r$0 is nonzero; LoRA-A gradients are zero at initialization (Sathyavageeswaran, 16 May 2026). No forward-pass score is stored; only per-parameter squared gradients are needed. This design is the basis for the method’s low overhead.

The informativeness proxy is the empirical Fisher Information diagonal. For each LoRA-B parameter $r$1 in layer $r$2, the estimator is

$r$3

where

$r$4

This is precisely the diagonal entry of the empirical Fisher Information Matrix,

$r$5

The method therefore uses empirical Fisher diagonal entries as a proxy for per-parameter gradient variance (Sathyavageeswaran, 16 May 2026).

This calibration-time design differs from approaches that estimate importance during training or through iterative reallocation. ALoRA prunes and reallocates ranks over repeated fine-tuning stages using AB-LoRA importance scores (Liu et al., 2024). RankAdaptor learns a performance model with offline meta-learning and online incremental learning over candidate rank vectors (Zhou et al., 2024). SubLoRA formulates rank determination through a second-order combinatorial objective with Hessian-based submodular maximization (Gao et al., 2 Jul 2025). FIM-LoRA instead performs a one-shot pretraining-free calibration on task data and then freezes the rank pattern (Sathyavageeswaran, 16 May 2026).

3. Budgeted allocation rule and implementation mechanics

After calibration, each layer’s matrix of empirical Fisher diagonal entries is reduced to a scalar score. For layer $r$6 with LoRA-B shape $r$7, the score is

$r$8

A simple proportional allocation would then compute

$r$9

In practice, the Budgeted Rank Allocator uses a two-phase “water-filling” procedure with a minimum floor $\mathcal{B}=L\cdot r$0, an optional ceiling $\mathcal{B}=L\cdot r$1, and largest-remainder rounding to exactly enforce $\mathcal{B}=L\cdot r$2 (Sathyavageeswaran, 16 May 2026). The floor is used to avoid starving layers.

Once the ranks are chosen, each LoRA adapter is resized in place. The first $\mathcal{B}=L\cdot r$3 rows of $\mathcal{B}=L\cdot r$4 are kept and new rows are Kaiming-init if needed; $\mathcal{B}=L\cdot r$5 is zero-padded or truncated to shape $\mathcal{B}=L\cdot r$6; and the scale $\mathcal{B}=L\cdot r$7 is updated to $\mathcal{B}=L\cdot r$8 (Sathyavageeswaran, 16 May 2026). The resulting adapter is still a standard LoRA module.

A useful contrast is with proportional rank allocation under GRPO. There, ranks are assigned by

$\mathcal{B}=L\cdot r$9

using per-layer gradient-magnitude profiling, then clipped into $\{r_\ell\}$0 and adjusted to satisfy the exact budget (Sawant, 8 May 2026). The structural similarity suggests a shared allocation pattern under a fixed total budget, but the training regime and the empirical behavior differ substantially. This suggests that the quality of a Budgeted Rank Allocator depends not only on the allocation rule but also on whether the informativeness signal remains stable under the optimization dynamics.

4. Efficiency and memory accounting

A defining feature of the FIM-LoRA Budgeted Rank Allocator is that it estimates the empirical Fisher diagonal only for LoRA-B parameters rather than for the full model (Sathyavageeswaran, 16 May 2026). A full-model eFIM would require storing gradients for every base-model weight $\{r_\ell\}$1. Restricting the estimate to LoRA-B stores only the $\{r_\ell\}$2 entries of each LoRA-B.

For typical transformer heads with $\{r_\ell\}$3 and $\{r_\ell\}$4, this yields a $\{r_\ell\}$5 memory saving over full-model Fisher estimation (Sathyavageeswaran, 16 May 2026). The method is therefore framed as a lightweight engineering solution rather than a heavy second-order procedure.

The same paper also states three system-level consequences: no new parameters, no training overhead, and no changes to serving infrastructure (Sathyavageeswaran, 16 May 2026). These claims should be read in the narrow sense used there: the allocator adds a short calibration stage of eight backward passes, but the resulting fine-tuning and serving model remain standard LoRA.

This efficiency profile differs from several related rank-allocation methods. ALoRA incurs occasional importance-score evaluations and extra fine-tuning segments, with total training time on E2E reported as 3.81 h versus 2.68 h for LoRA (Liu et al., 2024). RankAdaptor requires repeated fine-tuning evaluations to train and update its surrogate performance model (Zhou et al., 2024). SubLoRA computes gradients and Hessians with respect to concatenated singular values and alternates pruning with LoRA updates (Gao et al., 2 Jul 2025). FIM-LoRA’s allocator is comparatively narrow in scope and deliberately engineered around calibration-time cost control (Sathyavageeswaran, 16 May 2026).

5. Empirical behavior and interpretability

On GLUE with DeBERTa-v3-base and ranks $\{r_\ell\}$6 over 8 tasks, the reported average at $\{r_\ell\}$7 is 88.67 for LoRA, 88.57 for EVA, 88.54 for Random-rank, and 88.60 for FIM-LoRA, with $\{r_\ell\}$8 (Sathyavageeswaran, 16 May 2026). On commonsense reasoning with LLaMA-3-8B, zero-shot on 7 tasks at $\{r_\ell\}$9, LoRA reaches 68.74, FIM-LoRA with $\sum_\ell r_\ell=\mathcal{B}$0 reaches 67.01, and FIM-LoRA with $\sum_\ell r_\ell=\mathcal{B}$1 reaches 68.47 (Sathyavageeswaran, 16 May 2026).

Two empirical points are especially important. First, at equal parameter budget, FIM-LoRA matches or slightly outperforms uniform LoRA on GLUE and remains close on LLaMA-3-8B when a sufficient floor is used (Sathyavageeswaran, 16 May 2026). Second, the allocator is sensitive to the minimum floor. The paper reports that greedy allocation with $\sum_\ell r_\ell=\mathcal{B}$2 over-concentrates rank and loses approximately 1.7 percentage points, whereas a half-rank floor $\sum_\ell r_\ell=\mathcal{B}$3 recovers to within 0.27 percentage points of uniform LoRA (Sathyavageeswaran, 16 May 2026).

The learned rank patterns are also reported as interpretable. Averaging over 3 seeds on LLaMA-3-8B with $\sum_\ell r_\ell=\mathcal{B}$4, value-projection heads receive nearly maximal rank with mean approximately 29.7; query, key, and gate heads hover near the floor with mean approximately 8; and early-to-middle layers, layers 0–15, get approximately $\sum_\ell r_\ell=\mathcal{B}$5 more rank than late layers 24–31 (Sathyavageeswaran, 16 May 2026). These observations align with the reported interpretation that early layers build core representations and value projections carry content, whereas query and key are stable routing mechanisms.

A plausible implication is that the Budgeted Rank Allocator can serve a diagnostic role in addition to a compression role. The paper explicitly describes the resulting rank maps as interpretable diagnostics of which layers matter most (Sathyavageeswaran, 16 May 2026).

6. Related methods, limitations, and points of controversy

Budgeted rank allocation is not a unitary method class. In LoRA fine-tuning alone, the literature includes calibration-time eFIM allocation, iterative pruning and reallocation, surrogate-model-guided search, stable-rank priors, second-order submodular selection, and activation-aware factorization under a global budget (Sathyavageeswaran, 16 May 2026, Liu et al., 2024, Zhou et al., 2024, Zhang et al., 30 Jun 2025, Gao et al., 2 Jul 2025, Martínez, 29 Sep 2025). What they share is a fixed-resource constraint; what differs is the source of importance information and whether the allocation is one-shot, iterative, dynamic, or input-dependent.

A common misconception is that any adaptive rank signal that works under supervised fine-tuning should transfer to reinforcement-learning-style alignment. The GRPO study reports the opposite: on Qwen 2.5 1.5B with GSM8K, proportional rank allocation degrades accuracy by 4.5 points compared to uniform allocation, from 74.5% to 70.0%, despite identical parameter budgets (Sawant, 8 May 2026). The paper attributes this to a flatter gradient landscape under GRPO, with max-to-min layer importance ratio only $\sum_\ell r_\ell=\mathcal{B}$6, and to a gradient amplification effect in which non-uniform allocation widens the spread to approximately $\sum_\ell r_\ell=\mathcal{B}$7 (Sawant, 8 May 2026). This establishes an important limitation: a Budgeted Rank Allocator is not regime-agnostic.

Another limitation concerns the floor parameter. FIM-LoRA shows that unconstrained or weakly constrained proportional allocation can over-concentrate capacity, while the GRPO study reports that low-rank silencing can emerge through positive feedback (Sathyavageeswaran, 16 May 2026, Sawant, 8 May 2026). This makes the minimum floor more than an implementation detail; it is part of the method’s stability conditions.

Several extensions are explicitly proposed in FIM-LoRA. The paper states that the approach can be extended to other low-rank methods by using eFIM-based scores to seed EVA’s PCA initialization, periodically re-running calibration mid-training for dynamic reallocation, and allocating budget across all submodules $\sum_\ell r_\ell=\mathcal{B}$8 jointly rather than uniformly within a layer (Sathyavageeswaran, 16 May 2026). These are proposed generalizations rather than validated results in the reported experiments.

Taken together, these results suggest that “Budgeted Rank Allocator” is best understood as a design pattern: estimate per-component utility under a global constraint, then solve a small integer allocation problem. In FIM-LoRA, the utility surrogate is calibration-time gradient variance via the eFIM diagonal on LoRA-B, and the outcome is a standard LoRA model with a per-layer rank pattern and unchanged inference path (Sathyavageeswaran, 16 May 2026).