Gap-Init: Bridging Initialization Gaps

• Gap-Init is a family of initialization techniques that bridge the gap between coarse model constraints and true problem structure, ensuring stable optimization.
• It aligns update directions with empirically estimated gap vectors in diverse domains like PEFT, VLSI placement, and protoplanetary disc modeling to improve convergence.
• Empirical results and theoretical models validate that Gap-Init enhances performance metrics while reducing training variance across multiple tasks.

Gap-Init refers to a family of geometry- or physics-guided initialization techniques that address instability or suboptimal convergence when naively initializing optimization or simulation procedures in the presence of a structural capacity or modeling gap. Although independently introduced in several distinct domains, the unifying principle is to bridge the initialization gap between the modeling constraints (e.g., extreme low-rank for PEFT, zero-area point modeling for global placement, or absence of shock-induced torque in protoplanetary discs) and the true problem structure, thereby enhancing stability and performance across tasks.

1. Gap-Init in Parameter-Efficient Fine-Tuning (PEFT) with LoRA

In modern multimodal PEFT, particularly with Low-Rank Adaptation (LoRA) modules at minimal rank ($r=1$), training can be unstable, with the optimization highly sensitive to the initialization direction. Pretrained representations from different modalities (e.g., vision and language) occupy anisotropic cones in the shared feature space, with their means separated by a dominant translation vector $g = \mu_t - \mu_v$. Under random rank-1 initialization, the update direction $b$ is typically nearly orthogonal to $g$ in high dimension, causing the gradient component along $g$ to be attenuated by $O(1/\sqrt{d})$, often leading to weak gradients and collapse during early training. Gap-Init aligns the rank-1 LoRA update direction with an empirically estimated modality gap vector, overcoming this orthogonality bottleneck and stabilizing training, without increasing parameter count (Zhao et al., 2 Feb 2026).

2. Mathematical and Algorithmic Formulation of Gap-Init for LoRA

Let $$\mathcal{D}_{\text{cal}} = \{(x_i^{\text{img}}, x_i^{\text{txt}})\}_{i=1}^n$$ represent a calibration set of paired examples. For each transformer layer $l$, the sample-level gap vectors $g_i^{(l)} = h_i^{(t)} - h_i^{(v)}$ (where $h_i^{(t)}$ and $h_i^{(v)}$ are text and image hidden states respectively) are averaged to produce a per-layer global gap $g^{(l)} = \frac{1}{n} \sum_i g_i^{(l)}$. The rank-1 LoRA matrices are then initialized such that $B^{(l)} = g^{(l)}/\|g^{(l)}\|^2$ (the unique update direction) and $A^{(l)} = 0$, ensuring the initial adapter update is zero but the allowed update direction is well-aligned. For $r>1$, only the first column is aligned, with the rest randomly initialized as in standard LoRA (Zhao et al., 2 Feb 2026).

3. Theoretical Justification and Properties in High-Dimensional Regimes

Gap-Init's effectiveness is theoretically justified by a Gaussian translation model, where the optimal rank-1 update direction is exactly $g$. Under isotropic random initialization, alignment with $g$ is exponentially unlikely as dimension $d$ grows, rendering the optimization ineffective—the so-called "orthogonality catastrophe." With Gap-Init, $b \parallel g$, and early training gradients are no longer suppressed. Proposition 3.1 in (Zhao et al., 2 Feb 2026) provides the concentration bounds for random directions, quantifying the negligible probability of substantial overlap without explicit alignment.

4. Empirical Performance Across Tasks and Model Variants

Gap-Init has been empirically validated on COCO captioning, Flickr30k transfer, VQA (VQAv2), and cross-backbone settings. Notably, with only $r=1$, Gap-Init not only stabilizes training but frequently matches or exceeds baseline performance from $r=8$ LoRA adapters:

Task	Standard LoRA (r=1)	LoRA (r=8)	Gap-Init (r=1)
COCO CIDEr	98.08	138.49	140.59
COCO BLEU-4	24.48	40.63	41.87
Flickr30k CIDEr	63.00	–	79.60
VQA2 Acc.	15.58	–	57.23

On multi-seed runs, Gap-Init narrows variance substantially ($140.57 \pm 1.44$ vs. $135.37 \pm 7.10$). On new backbones (Qwen2-VL-7B, Gemma3-4B), it yields consistent performance gains. In all cases, Gap-Init achieves similar or better results relative to much larger ($8\times$) parameter adapters (Zhao et al., 2 Feb 2026).

5. Gap-Init in VLSI Global Placement

In placement optimization for VLSI designs, there exists a critical initialization gap between computationally cheap but uninformed point-based initializers, and slow but realistic area-aware initializers. Gap-Init bridges this difference by combining two strategies:

1. Area-Hint Refinement: Encoding area information into a signed-graph Laplacian via virtual nodes and signed edges, yielding a spectrally-filtered, area-aware initial placement at near GSP filtering speed.
2. Macro-Scheduled Placement: Progressively restoring hard area constraints by modeling macros as time-varying charge distributions over global placement iterations, allowing smooth evolution from point- to area-aware objectives (Ren et al., 13 Nov 2025).

Gap-Init achieves up to 2.2% improvement in HPWL over fast point-initializers, with runtimes $\sim$100$\times$ faster than full area-aware QCQP schemes, and demonstrates high robustness across a variety of benchmarks, all while closely matching the placement quality of the most sophisticated initializations.

6. Analytic Gap-Init Model in Protoplanetary Disc Evolution

In the context of planetary gap-opening in protoplanetary discs, Gap-Init denotes a linear analytic solution for the very early, self-similar stages of gap formation. The model incorporates a previously overlooked $\partial_t l$ term in the angular momentum equation, representing time-variability in specific angular momentum due to radial pressure gradients as the surface density $\Sigma$ is perturbed. For shallow gaps ($|\delta\Sigma|/\Sigma_0 \lesssim 0.2$), the solution predicts linear-in-time growth of the gap and explicit coorbital region evacuation, even in the absence of direct wave torque deposition. The radial profile and gap depth evolution can be written in terms of planetary structure parameters, and agree quantitatively with 2D simulations. The self-similar regime extends up to $t \lesssim 0.07\, t_{\text{gap}}$, beyond which viscosity and nonlinearity intervene (Cordwell et al., 2024).

7. Practical Considerations, Limitations, and Recommendations

• LoRA PEFT: Gap-Init is most reliable in the extreme rank-1 setting and relies on the presence of a strong, translation-dominant gap in pretrained representations. The calibration set should be in-domain; out-of-domain data degrade alignment fidelity. A calibration set of $n \approx 256$ paired samples is generally sufficient; performance saturates beyond this point.
• VLSI Placement: Area hints must be credible; incorrect modeling of macro footprints or bin densities can undermine placement quality. The macro-scheduled strategy smooths, but does not eliminate, abrupt constraint imposition.
• Protoplanetary Discs: The analytic Gap-Init regime is limited to shallow gaps and short timescales relative to $t_{\text{gap}}$. The model does not fully capture viscous or highly nonlinear evolution.

These limitations define the operational domain where Gap-Init strategies are effective, emphasizing careful calibration or domain modeling for robust alignment (Zhao et al., 2 Feb 2026, Ren et al., 13 Nov 2025, Cordwell et al., 2024).