Full-Model Tuning (FMT)

• Full-Model Tuning (FMT) is the process of updating all parameters in a pre-trained model to fully adapt its representation for specific downstream tasks.
• The methodology uses complete gradient-based optimization with techniques like AdamW and precision adjustments to maximize task accuracy in various settings.
• While FMT often achieves higher performance in domain adaptation and low-resource scenarios, it incurs significant computational, memory, and time costs compared to parameter-efficient methods.

Full-Model Tuning (FMT) refers to the process of adapting a pre-trained deep neural network for a downstream task by updating every parameter in the model during supervised or self-supervised fine-tuning. In contrast to parameter-efficient tuning (PET) strategies such as Low-Rank Adaptation (LoRA), Adapters, or prompt-based methods, FMT allows complete representational flexibility at the cost of significant computational and memory overhead, as every weight matrix and bias vector is subject to gradient-based updating throughout fine-tuning. FMT is standard for maximizing downstream accuracy, enabling thorough domain or task adaptation, and is foundational to rigorous analysis of deep model adaptation dynamics.

1. FMT: Definition, Formulation, and Contrast with PET

FMT entails optimizing all parameters $\theta$ of a pre-trained model $f_{\theta_0}$ over a downstream dataset $D$ by minimizing an objective such as cross-entropy:

$$\theta^* = \arg\min_{\theta} \mathcal{L}(\theta; D)$$

During FMT, no layers or parameters are frozen: gradient flow and optimization apply to all Transformer blocks, embedding tables, and (for multimodal models) encoders and projection heads (Yao et al., 2024). This full-unfreezing distinguishes FMT from PET, in which only a constrained low-dimensional subspace is trained—e.g., LoRA adds a low-rank factorization $\Delta W = BA$ per weight matrix with $B \in \mathbb{R}^{m\times r}$, $A \in \mathbb{R}^{r\times n}$, and $r \ll \min(m,n)$, resulting in greatly reduced parameter count and smaller compute/memory footprint (Sun et al., 2023, Shuttleworth et al., 2024).

By updating all model weights, FMT unlocks the full representational capacity of the model, enabling arbitrary changes to singular values and singular vectors of each weight matrix, whereas LoRA can only induce changes within an $r$-dimensional subspace of the parameter space (Shuttleworth et al., 2024). Therefore, FMT is the gold standard for task/domain adaptation performance analysis, theoretical exploration of transfer, and as a reference for PETL (parameter-efficient transfer learning) methods.

2. Methodology and Scaling Properties

The FMT workflow consists of:

• Initializing the model with all pre-trained weights.
• Defining loss $\mathcal{L}$ over the downstream dataset.
• Applying batched stochastic optimization (commonly AdamW or Lion) to all parameters.
• Employing appropriate precision (fp32, bf16, or int8 in special cases) and hyperparameters (learning rate, scheduler, batch size).

Typical model/experiment configurations include LLaMA-7B to 70B (Sun et al., 2023, Liu et al., 20 May 2026), RoBERTa-large (Liu et al., 20 May 2026), or mDeBERTa-v3 (278M parameters) (Oji et al., 10 Jan 2025). For large models, training setups require GPU clusters (e.g., 8×A100-40GB) and extended training times (e.g., 17–31 hours per epoch for LLaMA-7B on 0.6–2M Chinese instruction pairs) (Sun et al., 2023). Hyperparameters for state-of-the-art FMT on LLaMA-7B include bf16 precision, batch size 32, 3 epochs, peak LR $f_{\theta_0}$0, and cosine scheduling.

Power-law scaling holds robustly for FMT:

$f_{\theta_0}$1

For En→De MT, $f_{\theta_0}$2, $f_{\theta_0}$3; for pretraining size scaling, $f_{\theta_0}$4 (Zhang et al., 2024). FMT exhibits higher scaling exponents ($f_{\theta_0}$5) with respect to finetuning data, indicating stronger benefits from increased downstream data than PET methods ((Zhang et al., 2024), Table 5).

3. Empirical Results: Performance vs. Cost and Competing Methods

FMT systematically outperforms LoRA and other PET variants in raw downstream metrics when run from scratch, with gains of up to +0.10 in average GPT-score for Chinese instruction-following (LLaMA-7B, 2M examples: FMT 0.710 vs. LoRA 0.609) (Sun et al., 2023). For commonsense QA tasks at comparable parameter budgets, FMT approaches or matches maximal accuracy (e.g., LLaMA-7B: LoFT 76.08%, FMT $f_{\theta_0}$677%, LoRA 73.57%) (Tastan et al., 27 May 2025).

However, FMT is expensive: 3–5x the wall-clock time and GPU-memory per epoch vs. LoRA for the same base model and data size. For LLaMA-7B, FMT(2M, 3 epochs) consumes $f_{\theta_0}$7744 GPU-hours versus $f_{\theta_0}$8200 GPU-hours for LoRA (Sun et al., 2023). Peak per-GPU memory for dense AdamW is $f_{\theta_0}$940GB, often making FMT infeasible on smaller devices.

Empirical data from mDeBERTa-V3 on Germanic languages demonstrates FMT superiority for named entity recognition in low-resource settings (improving F1 by $D$0–$D$1 points over PET), but PET matches/exceeds FMT for high-resource extractive QA (Oji et al., 10 Jan 2025). Thus, task and resource regime critically mediate FMT’s relative value.

4. Algorithmic Advancements: Memory and Communication-Efficient FMT

Recent advances have made FMT more practical for limited hardware.

• ChunkFT (Liu et al., 20 May 2026): Partitions all weights into $D$2 byte-balanced chunks, sequentially fitting only one chunk’s fp32 master weights, gradients, and optimizer states in GPU memory at a time. Forward pass uses the full fp16 model, and backward computes gradients only for the active chunk. This enables large models (LLaMA3-8B/70B) to be fully tuned with $D$314GB (7B, $D$4) on commodity GPUs. Accuracy is maintained: ChunkFT achieves $D$5 on MT-Bench for Llama 3-8B, exceeding full-model AdamW ($D$6).
• QFT (Li et al., 2023): Quantizes all training states (weights, gradients, optimizer moments) to INT8 and uses the Lion optimizer for robust update steps, enabling LLaMA-7B FMT with $D$730GB memory—21% of the standard FP32 cost. QFT matches or nearly matches full-precision FMT for headline metrics (e.g., 57.4% vs. 58.0% few-shot accuracy on MMLU/ARC-c/HellaSwag/TruthfulQA).
• FedKSeed (Qin et al., 2023): Makes FMT feasible in federated learning by using zeroth-order gradient estimates communicated via seeds and scalars (not full gradients), reducing per-round communication to $D$818 KB. This achieves accuracy gains ($D$97.3% over LoRA-based FL) with minimal memory and communication overhead.

These methods have lowered the effective memory floor and communication barrier, supporting FMT of multi-billion parameter models on single mid-range GPUs or distributed client settings.

5. Theoretical and Structural Consequences

Unlike LoRA/adapter approaches, FMT exerts unconstrained control over both the singular values and singular vectors of each weight matrix. Spectral analysis establishes that FMT preserves the orientation and scales of existing pre-trained subspaces; it does not introduce “intruder dimensions”—new, orthogonal high-variance axes—that are a hallmark of LoRA adaptation (Shuttleworth et al., 2024). The SVD alignment between pre-trained and FMT-tuned weights shows nearly perfect diagonality (cosine similarity $$\theta^* = \arg\min_{\theta} \mathcal{L}(\theta; D)$$0 for top-$$\theta^* = \arg\min_{\theta} \mathcal{L}(\theta; D)$$1 singular vectors), in contrast to LoRA, which accumulates high-magnitude outliers after each task.

Empirical evidence demonstrates that such intruder dimensions in LoRA are localized drivers of catastrophic forgetting and incremental error during sequential/continual learning, whereas FMT retains earlier-task accuracy more robustly (Shuttleworth et al., 2024). However, FMT also tends to overwrite more pre-trained knowledge, manifesting as greater “pseudo-perplexity drift” on the original pre-training corpus—a direct result of large-scale parameter changes.

6. Trade-offs and Practical Guidelines

Selecting FMT vs. PET depends on task, data, model size, and compute/memory budget:

• Use FMT when:
  • End-task accuracy is paramount and resource constraints are secondary.
  • Initial adaptation of a fresh, pre-trained model to instruction-following, domain, or language.
  • The task or language is low-resource and requires full representational re-specialization (e.g., NER in Icelandic) (Oji et al., 10 Jan 2025).
• PET/LoRA is preferable for:
  • Rapid prototyping, multi-domain adapters, modular incremental updates.
  • Deployment under strict resource, memory, or communication constraints.
  • Scenarios demanding retention of pre-trained zero/few-shot performance.

Critical thresholds for transitioning from PET to FMT depend on both model size and data volume. For example, in MT, FMT outperforms LoRA only above $$\theta^* = \arg\min_{\theta} \mathcal{L}(\theta; D)$$2200K finetuning examples for a 1B model, with the threshold dropping for larger models (Zhang et al., 2024). Task and transfer properties further modulate the practical crossover point.

7. Future Directions and Open Challenges

Outstanding research challenges include:

• Precise explanation of LoRA’s initial underperformance relative to FMT when applied to vanilla pre-trained models, and the potential for hybrid regimes (e.g., seed with FMT on small data, finish with PET on large data) (Sun et al., 2023).
• Validation and refinement of scaling laws for FMT in creative or open-ended generation (Zhang et al., 2024).
• Extending memory- and communication-efficient FMT to 33B/65B+ models, low-resource edge devices, and extremely large federated networks (Liu et al., 20 May 2026, Li et al., 2023, Qin et al., 2023).
• Investigation of optimizer variants (Lion, AdamW) and their interactions with quantization for robust, accurate INT8-full tuning (Li et al., 2023).
• Application of reinforcement learning from human feedback (RLHF) layered on FMT vs. PET regimes (Sun et al., 2023).
• Deeper spectral and continual learning analysis to minimize catastrophic forgetting in full-tuned and partially adapted models (Shuttleworth et al., 2024).

Empirical, theoretical, and methodological advances continue to scale FMT across language, modality, and deployment regime, cementing its role as a reference and target for adaptation research in deep networks.