Scaling behavior of MiCA across model and data regimes

Determine the scaling behavior of Minor Component Adaptation (MiCA) with respect to model parameter count, training dataset size, and downstream task complexity, beyond the demonstrated results on 7B-parameter models, to assess how MiCA’s performance and benefits change across these dimensions.

Background

The paper introduces MiCA, a parameter-efficient fine-tuning method that constrains updates to minor singular directions of pre-trained weight matrices, showing improvements over LoRA on 7B-parameter models across several datasets.

While empirical gains are demonstrated at this scale, the authors explicitly note that it is not yet established how MiCA’s effectiveness varies with larger or smaller model sizes, different amounts of training data, or tasks of varying complexity. Understanding this scaling behavior is essential to generalize MiCA’s applicability and to inform practical deployment across diverse settings.