The Fine-Tuning Trap: Evaluating Negative Transfer and the Role of PEFT in Sub-1B Mathematical Reasoning

Abstract: Deploying Small LLMs (SLMs) on edge devices requires efficient fine-tuning strategies that adapt models to new tasks without degrading their general capabilities. In this study, we benchmark five sub-1B models (135M-1B) on mathematical reasoning tasks and uncover a critical vulnerability: Full Fine-Tuning (Full FT) actively harms performance in models under 300M parameters, often dropping accuracy below zero-shot baselines. This "negative transfer" makes Parameter-Efficient Fine-Tuning (PEFT) not just an efficiency preference, but a stability requirement. We find that while Low-Rank Adaptation (LoRA) and Weight-Decomposed LoRA (DoRA) perform comparably, their strengths vary by task; DoRA excels in complex reasoning (GSM8K), while LoRA dominates pattern matching (OrcaMath). In particular, Full FT is outperformed by LoRA on aligned models (Qwen2.5-0.5B) and even by simple 5-shot In-Context Learning on the smallest architectures (SmolLM2-135M). Based on these findings, we recommend defaulting to PEFT for all aligned sub-1B models and caution against Full FT for any architecture smaller than 500M parameters to prevent catastrophic forgetting. Reproduction of this work can be found at https://github.com/gulguluu/tiny-slm-finetune-compare.

• The paper demonstrates that full fine-tuning of sub-1B models leads to negative transfer and catastrophic forgetting, especially in models under 500M parameters.
• The paper shows that Parameter-Efficient Fine-Tuning methods such as LoRA and DoRA stabilize training by constraining updates and preserving alignment.
• The paper highlights that using PEFT enables safe and cost-effective on-device deployment while maintaining performance in mathematical reasoning tasks.

The Fine-Tuning Trap: Negative Transfer and the Role of PEFT in Sub-1B Mathematical Reasoning

Introduction

This paper rigorously interrogates the optimization and generalization limits of Small LLMs (SLMs), specifically those with sub-1B parameter counts, for mathematical reasoning tasks. The central finding is that, in contrast to LLMs, Full Fine-Tuning (Full FT) introduces a pronounced risk of negative transfer and catastrophic forgetting in SLMs—particularly under 500M parameters—undermining both zero-shot and few-shot baselines. The empirical and theoretical analyses robustly demonstrate that Parameter-Efficient Fine-Tuning (PEFT) methods, specifically Low-Rank Adaptation (LoRA) and Weight-Decomposed LoRA (DoRA), emerge as not only memory-efficient alternatives but also as essential regularizers to guard against optimization instability and alignment collapse.

Model Capacity, Cost-Accuracy Trade-Off, and Stability

The sub-1B SLM design space is driven by cost-efficiency and the need for deployability on edge devices (SRAM/DRAM or NPUs). However, the "cost vs. accuracy" Pareto frontier is highly non-linear in this regime, as demonstrated by the high variance in model accuracy for similar-sized architectures. Notably, highly optimized architectures, e.g., Qwen2.5-0.5B, approach the performance of >1B parameter models at a fraction of compute cost.

Figure 1: Sub-1B SLMs (blue) offer high cost efficiency but exhibit high accuracy variance; Qwen2.5-0.5B exemplifies the outlier performance possible with architectural and alignment improvements.

Despite such potential, systematic evaluation reveals a "stability cliff" where diminishing capacity amplifies negative transfer during Full FT. Sub-500M models are acutely prone to catastrophic degradation, whereas well-aligned architectures and PEFT methods can stabilize training and maintain baseline competencies.

Optimization Dynamics: Theoretical Framework

Three core phenomena drive the observed instability with Full FT in SLMs:

• Intrinsic Dimensionality and Intruder Dimensions: SLMs operate near their memory and parameter capacity saturation, with little redundancy available to absorb new gradients. Full FT executes unconstrained updates in high-curvature, orthogonal subspaces ("Intruder dimensions"), violating the pre-trained weight manifold and promoting divergence.
• Architectural Conditioning: "Deep and Thin" SLM architectures exacerbate optimization sensitivity; as depth increases and width decreases, the condition number of layer-to-output Jacobians grows exponentially, amplifying gradient vanishing and explosion. Full FT, updating all layers, is especially vulnerable; PEFT (via LoRA/DoRA) stabilizes convergence by constraining updates to low-rank adapters at selected sites.
• Catastrophic Forgetting Boundaries: The displacement of model weights from the pre-trained manifold is formally bounded in PEFT by the adapter scaling parameters and rank but unbounded in Full FT, leading to potential elimination of essential linguistic or reasoning functions.

Empirical Results

Systematic benchmarking of five architectures (135M to 2B parameters) across multiple mathematical reasoning datasets (OrcaMath, GSM8K, MATH, SVAMP) exposes the phase transition in stability and transfer dynamics:

• For 135M/360M SLMs, Full FT frequently depresses accuracy below zero-shot baselines and exhibits volatile loss and unbounded gradient norms.
• LoRA and DoRA not only stabilize training but consistently outperform Full FT.
• Notably, simple 5-shot in-context learning surpasses Full FT in the smallest models.

Figure 2: Full FT on SmolLM shows plateaued, high loss; LoRA and DoRA achieve lower loss and better convergence stability.

Figure 3: In Qwen2.5-0.5B, Full FT produces high-variance, unstable gradients; LoRA and DoRA maintain tightly bounded convergence.

Figure 4: Reference models (>1B, such as OLMo) demonstrate the recoverability of Full FT, with stable loss decay outperforming PEFT.

The instability in Full FT is further quantified via gradient norm telemetry. For Qwen2.5-0.5B, the Full FT gradient norm is consistently an order of magnitude above that of LoRA/DoRA, directly correlating with catastrophic failures in out-of-distribution robustness (SVAMP accuracy: 0.0%). PEFT trajectories remain near-constant and well-bounded.

Figure 5: Gradient norm for Qwen2.5-0.5B: Full FT is unstable (red), while PEFT trajectories (green/blue) are tightly bounded.

Implications for Model Alignment and Safety

A critical observation is the fragility of "alignment vectors"—safety and robustness features induced by post-training (e.g., RLHF, DPO). Full FT with narrow supervised datasets (e.g., mathematical reasoning) overwrites these alignment directions, leading to catastrophic forgetting and safety violations. The experiments establish that PEFT, by freezing the backbone and introducing low-capacity adapters, inherently preserves alignment and should be the default for regulated or robust-critical deployments.

Hardware and Economic Trade-Offs

Contrary to earlier beliefs, PEFT is not always computationally superior on high-end clusters due to the adapter kernel and memory access overhead. However, its value proposition lies in democratizing training: memory savings and parameter isolation enable fine-tuning on commodity hardware—crucial for privacy, accessibility, and widespread deployment. The cost comparison further highlights the democratizing effect of PEFT, reducing barriers for consumer-level adaptation.

DoRA versus LoRA: Task-Specific Efficacy

DoRA provides modest but consistent gains over LoRA on complex multi-step reasoning tasks (e.g., GSM8K), likely due to its decoupling of update magnitude and direction. This separation enables selective feature amplification critical for mathematical logic, while maintaining the regularization effect of low-rank adaptation.

Deployment: Toward On-Device Reasoning

The study reinforces that, given proper fine-tuning, sub-1B SLMs—especially those fine-tuned via PEFT—are viable for always-on, edge/Mobile deployment, including resource-constrained NPUs and SoC architectures. The combination of DoRA fine-tuning and aggressive quantization compresses the deployment footprint below 300MB, making concurrent inference on consumer devices feasible.

Conclusion

This work systematically delineates the stability-capacity trade-off for sub-1B LLMs in mathematical reasoning fine-tuning. Full FT is contraindicated for models under 500M parameters due to its strong tendency toward negative transfer and alignment destruction. PEFT not only stabilizes training but also systematically preserves zero-shot capabilities, alignment, and safety. For practitioners, defaulting to PEFT (specifically DoRA) on aligned SLMs is both a practical and theoretical necessity. Full FT is only advisable above the 1B parameter threshold, when sufficient intrinsic capacity is present to absorb full-rank updates without risking catastrophic forgetting. The implications for guideline construction, safety-critical deployment, and future SLM research are immediate and substantial.