Iterative LoRA Training (ILT)

• Iterative LoRA Training (ILT) is a framework that iteratively adapts low-rank modules to enhance parameter-efficiency in fine-tuning large models.
• ILT employs a multi-stage process—Focus, Feed Back, and Fix—to mitigate overfitting and optimize multilingual ASR performance.
• ILT integrates rank annealing and post-hoc compression to decouple training expressivity from deployment constraints, ensuring robust adaptation.

Iterative LoRA Training (ILT) is a framework designed to enhance parameter-efficient fine-tuning in large-scale neural architectures by structuring the update of low-rank adaptation (LoRA) modules into sequential, knowledge-oriented stages, or by post-hoc and in-tuning rank adaptation via compression. ILT addresses challenges such as overfitting in supervised fine-tuning and the inflexibility of fixed-rank LoRA approaches, improving both multilingual speech recognition and general few-shot adaptation in text and vision-language domains (Meng et al., 11 Jul 2025, Vulić et al., 11 Feb 2026).

1. Conceptual Foundations and Motivation

ILT arises from limitations observed in standard LoRA fine-tuning, especially when applied to large pretrained models in low-resource, multilingual, or data-scarce scenarios. In classical LoRA, the weight update for a frozen pretrained matrix $$W_0\in\mathbb{R}^{d_\mathrm{out}\times d_\mathrm{in}}$$ is parameterized as

$$W = W_0 + \Delta W,\quad \Delta W = \frac{\alpha}{r}A B,$$

with $A\in\mathbb{R}^{d_\mathrm{out}\times r}$, $B\in\mathbb{R}^{r\times d_\mathrm{in}}$, and $r\ll \min(d_\mathrm{out}, d_\mathrm{in})$. Only $A,B$ are optimized, reducing parameter overhead.

This parameter-efficiency, however, does not immunize LoRA to overfitting, particularly in supervised fine-tuning of large LLMs and ASR models. The model tends to forget general multilingual features and over-specializes, with poor generalization to unseen or underrepresented dialects and languages (Meng et al., 11 Jul 2025). Moreover, standard practice involves a fixed-rank setting for $r$, which may not yield optimal size–performance trade-offs and complicates hyperparameter selection (Vulić et al., 11 Feb 2026).

ILT mitigates these shortcomings by (1) breaking down the adaptation process into discrete, iterative phases—each with distinct objectives and data distributions (Meng et al., 11 Jul 2025), and/or by (2) dynamically compressing LoRA ranks either post-hoc or via staged, in-tuning annealing, decoupling training rank from deployment rank (Vulić et al., 11 Feb 2026).

2. ILT for Multilingual Speech Recognition: Three-Stage Procedure

In the context of multilingual automatic speech recognition (ASR), ILT organizes model adaptation into three stages: Focus, Feed Back, and Fix. Each stage targets different knowledge domains and data distributions (Meng et al., 11 Jul 2025):

1. Focus Training: Rapid specialization to the official downstream data (e.g., MLC-SLM), avoiding catastrophic forgetting of pretraining knowledge.
   • Loss: Standard autoregressive LoRA loss

$$\mathcal{L}_{\mathrm{lora}} = -\sum_{t=1}^T \log P(y_t | y_{<t}, x; A, B)$$

• Hyperparameters: LoRA rank $r_1=16$, scale $\alpha_1=32$
• Epochs: 6

1. Feed Back Training: Domain and language expansion using a broader, mixed-lingual dataset, including low-resource and previously underrepresented languages.
   • Dataset: 40,000 hours (official + external)
   • Hyperparameters: $$W = W_0 + \Delta W,\quad \Delta W = \frac{\alpha}{r}A B,$$0, $$W = W_0 + \Delta W,\quad \Delta W = \frac{\alpha}{r}A B,$$1
   • Epochs: 10
2. Fix Training: Final correction using high-confidence pseudo labels and curated datasets to address residual errors.
   • Data: 2791 hours curated + generated pseudo labels
   • Hyperparameters: $$W = W_0 + \Delta W,\quad \Delta W = \frac{\alpha}{r}A B,$$2, $$W = W_0 + \Delta W,\quad \Delta W = \frac{\alpha}{r}A B,$$3
   • Epochs: 6

At each stage, the updated LoRA modules are merged into the model before proceeding. Pseudo labeling leverages ensemble model checkpoints with hard-voting for robust annotation, which are iteratively integrated into subsequent training sets.

3. Iterative Pseudo Labeling and Knowledge Task Construction

Pseudo label generation in ILT employs an ensemble hard-voting protocol. For an unlabeled input $$W = W_0 + \Delta W,\quad \Delta W = \frac{\alpha}{r}A B,$$4, $$W = W_0 + \Delta W,\quad \Delta W = \frac{\alpha}{r}A B,$$5 model checkpoints $$W = W_0 + \Delta W,\quad \Delta W = \frac{\alpha}{r}A B,$$6 produce hypotheses $$W = W_0 + \Delta W,\quad \Delta W = \frac{\alpha}{r}A B,$$7. The pseudo label is the majority token at each position:

$$W = W_0 + \Delta W,\quad \Delta W = \frac{\alpha}{r}A B,$$8

Generated pseudo labels are incrementally added to the downstream knowledge task set $$W = W_0 + \Delta W,\quad \Delta W = \frac{\alpha}{r}A B,$$9, reinforcing supervision and reducing reliance on manually annotated data. This iterative pseudo labeling strategy is particularly suited to multilingual ASR, where labeled data in many languages is scarce or unevenly distributed (Meng et al., 11 Jul 2025).

4. ILT with Rank Annealing and Post-hoc Compression

Beyond ASR, the ILT paradigm encompasses systematic adaptation of LoRA ranks through "Post-Squeeze" and "In-Squeeze" procedures (Vulić et al., 11 Feb 2026):

• Post-Squeeze: Fine-tune a high-rank LoRA adapter (source rank $A\in\mathbb{R}^{d_\mathrm{out}\times r}$0), then compress it post-hoc to a lower, deployment-ready target rank $A\in\mathbb{R}^{d_\mathrm{out}\times r}$1 using Randomized SVD (RSVD). Optionally, a continued fine-tuning phase ("Cont-Squeeze") recovers any accuracy lost due to aggressive compression.
• In-Squeeze: Implement a scheduled, iterative reduction in rank—$A\in\mathbb{R}^{d_\mathrm{out}\times r}$2 with $A\in\mathbb{R}^{d_\mathrm{out}\times r}$3—each followed by an RSVD compression and brief fine-tuning, distributed using "standard" or "minimum-steps" hyperparameter allocation.

These strategies decouple the expressivity and convergence properties of high-rank LoRA training from the efficiency constraints of deployment, streamline hyperparameter optimization, and improve the performance of fixed low-rank adapters (Vulić et al., 11 Feb 2026).

5. Theoretical Insights and Convergence Properties

The iterative, multi-stage nature of ILT can be formalized as a sequence of subproblem minimizations:

$A\in\mathbb{R}^{d_\mathrm{out}\times r}$4

where each stage $A\in\mathbb{R}^{d_\mathrm{out}\times r}$5 addresses a specific knowledge tuple $A\in\mathbb{R}^{d_\mathrm{out}\times r}$6. The cumulative effect $A\in\mathbb{R}^{d_\mathrm{out}\times r}$7 enlarges the hypothesis space explored by low-rank updates. Under smoothness and contraction conditions on $A\in\mathbb{R}^{d_\mathrm{out}\times r}$8, this leads to convergence of $A\in\mathbb{R}^{d_\mathrm{out}\times r}$9 to a point with reduced loss on the union of all tasks, achieving a strictly better loss on aggregate compared to any single-stage fine-tuning (Meng et al., 11 Jul 2025). This suggests that multi-stage ILT enables approximation of the global minimizer in compositional task spaces otherwise inaccessible to monolithic adaptation.

6. Empirical Evaluation and Comparative Performance

ILT has shown consistent gains in large-scale experiments. In multilingual ASR with Whisper-large-v3 and Qwen2-Audio backbones, applying the Focus–Feed Back–Fix ILT increased average WER performance from 15.36% to 9.20% (Whisper) and from 19.82% to 9.89% (Qwen2-Audio) across 16 languages (Meng et al., 11 Jul 2025). Ablation studies indicate that combining all three stages yields the best results:

Training Protocol	Whisper WER (%)	Qwen2-Audio WER (%)
Focus only	10.38	19.92
Feed Back only	11.59	14.53
Fix only	10.07	18.99
Full ILT (all stages)	9.20	9.89

In text and vision-language adaptation, compressing from higher rank ($B\in\mathbb{R}^{r\times d_\mathrm{in}}$0) and continued fine-tuning outperformed direct low-rank training both in accuracy (by 0.2–0.5 points) and robustness, with best trade-offs observed using minimum-steps annealing schedules in the In-Squeeze variant (Vulić et al., 11 Feb 2026). Random seed sensitivity was minimal (SEM $B\in\mathbb{R}^{r\times d_\mathrm{in}}$1 0.1%).

7. Industrial Applications and Broader Implications

ILT has seen competitive deployment in the Interspeech 2025 Multilingual Conversational Speech Language Modeling Challenge (MLC-SLM), with the MegaAIS team achieving 4th in Multilingual ASR and 1st in Speech Separation and Recognition (Meng et al., 11 Jul 2025). The approach is practical for industrial, multilingual ASR and instruction-tuning due to efficient hardware footprint (e.g., 8$B\in\mathbb{R}^{r\times d_\mathrm{in}}$2A800 GPUs), flexibility for dynamic rank selection, and the absence of architectural changes to the base model.

Empirical findings in (Vulić et al., 11 Feb 2026) further demonstrate that ILT (in both Post-Squeeze and In-Squeeze forms) decouples the choice of LoRA rank from both accuracy convergence and deployment needs, simplifies hyperparameter selection, and preserves architectural homogeneity.

A plausible implication is that ILT frameworks—by orchestrating adaptivity both in the knowledge domain and in the optimizer's expressivity via rank scheduling—represent a universal strategy for advancing parameter-efficient fine-tuning across modality and task boundaries.