MiCA Learns More Knowledge Than LoRA and Full Fine-Tuning

Abstract: Minor Component Adaptation (MiCA) is a novel parameter-efficient fine-tuning method for LLMs that focuses on adapting underutilized subspaces of model representations. Unlike conventional methods such as Low-Rank Adaptation (LoRA), which target dominant subspaces, MiCA leverages Singular Value Decomposition to identify subspaces related to minor singular vectors associated with the least significant singular values and constrains the update of parameters during fine-tuning to those directions. This strategy leads to up to 5.9x improvement in knowledge acquisition under optimized training hyperparameters and a minimal parameter footprint of 6-60% compared to LoRA. These results suggest that constraining adaptation to minor singular directions provides a more efficient and stable mechanism for integrating new knowledge into pre-trained LLMs.

• The paper introduces MiCA, a method that leverages singular value decomposition to adapt fine-tuning along minor singular directions, achieving 3–5% accuracy gains over LoRA.
• It demonstrates enhanced knowledge retention and mitigated catastrophic forgetting on domain-specific benchmarks like BLOGS-MC and HISTORY-MC.
• MiCA attains improvements using only 6–60% of the parameter cost of full fine-tuning, offering a resource-efficient solution for specialized LLM deployment.

MiCA: Parameter-Efficient Fine-Tuning via Minor Component Adaptation

Introduction and Motivation

Minor Component Adaptation (MiCA) is introduced as a principled advancement for parameter-efficient fine-tuning (PeFT) of LLMs, designed to address the limitations of conventional low-rank adaptation strategies such as LoRA. Standard PeFT methods, while reducing computational overhead by optimizing a small subset of parameters, do not explicitly account for the spectral structure of pre-trained model weights and frequently adapt along dominant singular directions, potentially interfering with core capabilities. MiCA targets the underutilized minor singular directions—those associated with the smallest singular values—to maximize knowledge acquisition with minimal parameter updates and to minimize catastrophic forgetting.

Methodology: Spectrally Constrained Adaptation

MiCA augments LoRA by incorporating singular value decomposition (SVD) of pre-trained weights to systematically inject adaptation capacity into low-variance subspaces. Given a pre-trained weight matrix $W \in \mathbb{R}^{d \times d}$ in a transformer block, SVD yields $W = U \Sigma V^\top$. MiCA constrains the adaptation matrix $B$ to the last $r$ columns of $U$ (minor left singular vectors), while initializing $A$ to zero. During training, only $A$ is updated. The model weights are thus modified according to

$$W_{\text{final}} = W + \Delta W,\qquad \Delta W = \frac{\alpha}{r}BA,$$

where $B = U_{[:, -r:]}$ is fixed. This explicit spectral targeting ensures that the information injected through fine-tuning occupies a subspace with maximum orthogonality to dominant, task-agnostic features, mitigating parameter interference.

(Figure 1)

Figure 1: Diagrammatic comparison showing LoRA’s unconstrained adaptation (left) and MiCA's update restriction to minor singular directions (right).

Experimental Design

Empirical validation covers continued pre-training on Llama-2-7B and Qwen2.5-7B, with experiments focusing on two principal axes:

1. Knowledge Injection & Retention: Fine-tuning on BLOGS (OpenAI blog posts unseen during pre-training) and a German history book, with downstream evaluation on corresponding multiple-choice question sets (BLOGS-MC, HISTORY-MC).
2. Catastrophic Forgetting & Generalization: Evaluation on TruthfulQA and HellaSwag, benchmarks orthogonal to the injected domain, to test for retention of general pretrained knowledge.

All methods (MiCA, LoRA, full fine-tuning) undergo independent hyperparameter optimization in rank $r$, learning rate, and epochs, with controlled random seeds and identical data pipelines.

Results

Knowledge Integration Efficacy

MiCA delivers 3–5 percentage points improvement in accuracy over LoRA on BLOGS-MC and HIStory-MC, consistently outperforming both full fine-tuning and LoRA at 6–60% the parameter cost.

Figure 2: Baseline performance for Llama-2-7B, demonstrating the impact of continued pre-training with different PeFT methods.

This effect is persistent across both the BLOGS and HISTORY book corpora. On the BLOGS-MC set, MiCA with Llama-2-7B-chat achieves 61.33% accuracy (vs. 58.28% with LoRA and 56.18% baseline), using just 4M trainable parameters compared to LoRA's 67M. For Qwen2.5-7B-Instruct, MiCA achieves 75.63% vs. 73.87% (LoRA) and 72.91% (baseline). On the much larger HISTORY-MC, MiCA reaches 39.2% (vs. LoRA’s 29.4% and full fine-tuning's 30.4%).

Figure 3: Domain knowledge retention curves for the history book corpus, highlighting MiCA's stable superiority in precision and catastrophic forgetting mitigation.

Catastrophic Forgetting and Generalization

Notably, MiCA exhibits minimal performance degradation on general benchmarks (HellaSwag) while significantly outperforming LoRA and full fine-tuning on new knowledge. This validates its ability to localize representational change, preserving generic skills in subspaces orthogonal to the adaptation.

Ablation on Spectral Subspaces

Control experiments confirm that MiCA’s efficacy is not a generic consequence of restricting adaptation to fixed low-rank subspaces: adapting along randomly chosen singular directions or dominant singular vectors yields systematically lower accuracy than minor directions. This substantiates the central MiCA hypothesis regarding the high plasticity and informativeness of minor singular directions.

Practical and Theoretical Implications

MiCA’s results indicate that targeted adaptation in minor singular directions offers a Pareto improvement over traditional LoRA/PeFT methods in the critical regime of domain adaptation with strong catastrophic forgetting risk. This spectral localization enables parameter-efficient deployment—relevant for federated learning, on-device personalization, and scenarios where communication or computation is at a premium. The explicit SVD-based design introduces an interpretable and stable mechanism for adapter injection, rather than the empirically drifting subspaces in standard LoRA.

On the theory side, MiCA provides evidence that the pre-trained weight spectrum encodes structurally distinct subspaces with separable roles: dominant directions preserve generality, while minor ones maximize adaptability and knowledge integration. This aligns with, and extends, the perspective of minor component analysis (MCA) in neural coding.

Limitations and Future Directions

MiCA is not directly suited for tasks necessitating instruction-following or other non-content-specific adaptation, as it only modifies minor components in the original non-instructional model. Further work can extend spectral adaptation techniques to instruction deltas or explore their synergy with reinforcement learning-based fine-tuning, where localized updates are critical for reward-aligned behavior.

Scaling SVD computations to extremely large models presents preprocessing overhead, though this cost is amortized over repeated adaptation. The behavior of MiCA on even larger (e.g., 70B+) models and across more diverse task classes warrants future investigation.

Conclusion

MiCA substantiates spectrally grounded, minor component adaptation as a superior strategy for parameter-efficient domain specialization in LLMs. By localizing fine-tuning to directions least represented in pre-trained weights, MiCA achieves greater knowledge acquisition, significant reductions in catastrophic forgetting, and strong efficiency improvements over conventional PeFT techniques. This framework opens a pathway to robust, lightweight, and interpretable adaptation for specialized AI applications.

Reference: "MiCA Learns More Knowledge Than LoRA and Full Fine-Tuning" (2604.01694)