Transfer profile of full-parameter fine-tuning

Characterize how full-parameter fine-tuning transfers across base-model upgrades, including whether its higher effective update rank produces a transfer profile different from that of rank-16 QLoRA specialists.

Background

The study evaluates specialists adapted with QLoRA at rank 16 and therefore measures only a low-rank parameter-efficient fine-tuning regime. Full-parameter fine-tuning perturbs model weights at a substantially higher effective rank, which may alter portability across independent pretraining runs and continued-pretraining checkpoints. The authors explicitly state that they cannot determine this transfer profile from the present experiments.

References

Specialists are capped at 8B by the single-GPU constraint and are adapted only with QLoRA at rank 16; full-parameter fine-tuning perturbs weights at a much higher effective rank \citep{biderman2024lora} and may well have a different transfer profile, which we cannot speak to.

— UpgradeBench: A Decision-Centric Benchmark for Upgrading Fine-Tuned LLM Specialists  (2608.20918 - Chen et al., 21 Aug 2026) in Limitations section

We have not tested this directly, and layer-wise ablation of what each adapter changes is future work.

— UpgradeBench: A Decision-Centric Benchmark for Upgrading Fine-Tuned LLM Specialists  (2608.20918 - Chen et al., 21 Aug 2026) in Section 5.3, “Copying and the continuity criterion”