InRank: Incremental Low-Rank Learning

• Incremental Low-Rank Learning (InRank) is a framework that incrementally updates model weights via low-rank factorizations, balancing expressivity with efficient resource use.
• It employs dynamic rank scheduling with SVD evaluations to adaptively increase rank only when necessary, optimizing the trade-off between plasticity and stability.
• InRank significantly reduces memory and computation in applications like federated learning, vision, and reinforcement learning, ensuring scalable and robust performance.

Incremental Low-Rank Learning (InRank) is a parameter-efficient paradigm for continual, adaptive, and scalable training of machine learning models wherein model weight updates are explicitly performed and represented through a sequence of low-rank factorizations that grow in rank only as needed. InRank encompasses a theoretical framework, algorithmic design, and applied variants for domains such as vision, natural language, federated, and reinforcement learning; it provides substantial benefits in stability, memory, and computational efficiency through the use of adaptive low-rank update regimes.

1. Theoretical Basis and Foundations

The conceptual foundation for InRank traces to the greedy low-rank learning (GLRL) theory, which demonstrates that stochastic gradient–based training of deep (especially linear) models naturally induces trajectories in which weight updates remain low-rank and incrementally increase rank as training progresses. For a sequence of SGD updates $D_t=W_t-W_0=\sum_{i=1}^t\Delta W_i$, it is shown that $D_t$ is approximately low-rank at every stage, with rank growth reflecting the order of target singular values $s_\alpha$ of the data (Zhao et al., 2023). In particular, each singular mode $u_\alpha,v_\alpha$ is learned in a sigmoidal “step-in-rank” fashion, and the cumulative update trajectory is characterized by sequential accretion of rank-1 components, providing an implicit regularization bias.

This behavior generalizes beyond deep linear networks: empirical evidence demonstrates incremental spectrum growth of layer-wise updates in nonlinear transformers, vision models, and RL agents (Zhao et al., 2023, Erden, 17 Dec 2025). Fine-grained analyses of matrix sensing further show that gradient descent, under small initialization, sequentially builds up the recovered matrix $X^\ast$ rank-by-rank, tracking an incremental greedy solution schedule (Jin et al., 2023). This insight motivates explicit InRank algorithms that directly parameterize weight changes in low-rank forms, adaptively increasing rank only as required.

2. Algorithmic Framework and Practical Implementations

Core InRank algorithms operate by parameterizing each weight update as a low-rank product $U^lV^l$, with $W^l=W^l_0 + U^lV^l$ per layer $l$ (Zhao et al., 2023). At each training iteration:

1. Gradients with respect to $U^l, V^l$ are computed and parameters updated.
2. The current reconstructed update $D^l=U^lV^l$ is subjected to an SVD; the explained-variance ratio of leading singular values is checked.
3. If the explained ratio falls below a threshold $D_t$0, the rank $D_t$1 is incremented, expanding $D_t$2 with new singular vectors (initialized with small norm).

This incremental expansion is performed per layer as needed, yielding an efficient balance between representational capacity and resource usage.

Variants of InRank adapt this core recipe to diverse domains:

• Incremental LoRA: Adapter-style InRank for transformers, where low-rank ($D_t$3) adapters are merged incrementally into aggregate adapters via data-dependent convex combinations and periodically truncated by SVD (Monga et al., 24 Feb 2026).
• Federated InRank (Fed-TaLoRA): In federated, class-incremental settings, InRank employs global aggregation of low-rank factors paired with a residual weight update (ResWU) to correct for aggregation bias in non-IID contexts (Yu et al., 18 May 2025).
• Dynamic Rank Scheduling: In online and RL scenarios (e.g., exoskeleton control), InRank includes data-driven mechanisms for dynamically selecting the rank $D_t$4 at inference time depending on sensed complexity, with adaptation managed via policy gradients (Liu et al., 3 Jun 2026, Erden, 17 Dec 2025).
• OSS-Constrained InRank: For class-incremental vision-LLMs (e.g., CLIP), InRank restricts update directions to data-dependent approximate-orthogonal “safe” subspaces, ensuring old-task features remain unperturbed (Li et al., 14 Nov 2025).

3. Rank Adaptation, Stability, and Plasticity Trade-offs

A principal theme in InRank methodology is the tunable tradeoff between plasticity (ability to learn new tasks) and stability (resistance to catastrophic forgetting), governed primarily by the choice and evolution of the update rank $D_t$5.

• Continuum of adaptation: Lower ranks restrict updates to a narrow subspace, yielding strong retention of old knowledge but limited expressivity; higher ranks afford greater adaptability at the cost of increased forgetting (Soutif--Cormerais et al., 2024, Monga et al., 24 Feb 2026).
• Automatic rank growth: By monitoring the spectrum of current updates, InRank can automatically detect when additional modes are needed; this adaptive mechanism ensures capacity is allocated only as empirically justified (Zhao et al., 2023, Erden, 17 Dec 2025).
• Empirical trade-off: Experiments across vision and language tasks demonstrate a sharp rise in forgetting with increasing rank (e.g., on ImageNet, average forgetting rises from $D_t$6 at $D_t$7 to $D_t$8 at $D_t$9), with performance curves supporting the necessity of principled rank selection (Soutif--Cormerais et al., 2024).

In mechanisms such as BOFA or specialized InRank adapters, additional structural constraints—e.g., subspace orthogonality— further calibrate this trade-off, often obviating the need for task rehearsal or data replay (Li et al., 14 Nov 2025).

4. Memory, Communication, and Computational Efficiency

The low-rank structure underpinning InRank yields marked improvements in both training and inference resource use:

• Parameter scaling: For a layer with dimensions $s_\alpha$0 and rank $s_\alpha$1, memory and compute scale as $s_\alpha$2, a substantial reduction vs. $s_\alpha$3 for dense weights (Zhao et al., 2023).
• Federated learning: In federated settings, sharing only the low-rank factors (rather than full model weights) yields up to a $s_\alpha$4 reduction in communication cost compared to comparable approaches, with task-agnostic aggregation further consolidating these gains (Yu et al., 18 May 2025).
• Inference cost: Dynamic scheduling in online exoskeleton control restricts matrix operations to only the necessary rank, delivering $s_\alpha$5–$s_\alpha$6 reductions in FLOPs at inference time with sub-10ms end-to-end latency (Liu et al., 3 Jun 2026).
• Training time and footprint: Large-scale language modeling (GPT-2, WikiText-103) achieves up to a $s_\alpha$7 reduction in total training time and $s_\alpha$8 in model size while incurring negligible loss in accuracy relative to full-rank baselines (Zhao et al., 2023).

5. Applications and Domain-Specific Variants

A wide spectrum of domains leverages InRank-style learning:

• Continual and Incremental Learning: Both class- and task-incremental paradigms benefit from layered InRank where prior-task adapters are frozen or aggregated, and new tasks invoke fresh low-rank components (Wistuba et al., 2023, Hyder et al., 2022).
• Vision Transformers and Foundation Models: Merged LoRA adapters consistently maintain a balance between adaptation and forgetting, with hyperparameters such as rank $s_\alpha$9 and scale $u_\alpha,v_\alpha$0 providing fine-grained control over update magnitude (Soutif--Cormerais et al., 2024, Monga et al., 24 Feb 2026).
• Federated Class-Incremental Learning: Task-agnostic adapters and residual weight correction (ResWU) improve state-of-the-art (e.g., Final Average Accuracy on CIFAR-100), and require no storage of per-task adapters or exemplars (Yu et al., 18 May 2025).
• Reinforcement Learning and Online Control: Gated, context-conditional low-rank updates enable rapid and robust personalization in sequential, real-time control settings (e.g., adaptive exoskeletons benefit from terrain-aware dynamic rank scaling) (Liu et al., 3 Jun 2026).
• Orthogonal Safe Subspace (OSS) Adaptation: In multi-modal and vision-language scenarios, constraining updates to the orthogonal complement of prior feature subspaces sharply reduces interference and catastrophic forgetting without data replay (Li et al., 14 Nov 2025).

6. Limitations and Open Challenges

Several structural and empirical limitations are intrinsic to current InRank instantiations:

• Limited expressivity at low rank: If target updates cannot be spanned by the allocatable rank, the ultimate approximation is suboptimal (Zi et al., 2023).
• Memory growth in task-specific approaches: For methods that store one adapter per task, total memory scales linearly with the task count (Hyder et al., 2022, Wistuba et al., 2023); task-agnostic or aggregated merging variants ameliorate but do not always eliminate this effect.
• Task or domain identification at inference: Some implementations require task-ID or domain classifiers to route inputs to the correct adapter subset (Wistuba et al., 2023).
• Dynamic rank management overhead: Frequent SVD calculations or dynamic rank adjustment introduce algorithmic complexity, although amortization strategies exist (Zhao et al., 2023, Erden, 17 Dec 2025).
• Lack of convergence to global optimum when constraints are restrictive: Under OSS or strong low-rank projection constraints, the update space is limited and may not capture all needed solution directions (Li et al., 14 Nov 2025).

7. Outlook and Emerging Directions

Open questions in the design, analysis, and application of InRank methods include:

• Learned or context-sensitive rank schedules: Dynamic, data-driven adaptation of rank across layers, inputs, or training phases is emerging as a key avenue for capacity optimization (Liu et al., 3 Jun 2026, Erden, 17 Dec 2025).
• Structured composition and compression: Hybrid methods combining low-rank, sparse, and masked updates may yield further efficiency and retention improvements (Hyder et al., 2022).
• Zero-forgetting with scalability: Aggregated or merged-adapter approaches aim to combine the forgetting resistance of per-task methods with the resource efficiency of task-agnostic updates (Monga et al., 24 Feb 2026, Yu et al., 18 May 2025).
• Generalization to nonlinear or non-Euclidean parameter spaces: While most current InRank theory is developed in the context of linear operators and convex ERMs, extension to broader model classes remains an open area (Hanada et al., 2023).

InRank encapsulates a mathematically grounded framework for scalable, parameter-efficient adaptation across a range of incremental, federated, and online learning scenarios. It harmonizes theory and practice in continual learning by leveraging the incremental nature of low-rank update trajectories and providing class-leading trade-offs among memory, speed, and adaptability (Zhao et al., 2023, Soutif--Cormerais et al., 2024, Monga et al., 24 Feb 2026).