---
title: Full-Model Tuning (FMT)
url: https://www.emergentmind.com/topics/full-model-tuning-fmt
type: topic
---

# Full-Model Tuning (FMT)

Full-Model Tuning (FMT) refers to the process of adapting a pre-trained deep neural network for a downstream task by updating every parameter in the model during supervised or self-supervised fine-tuning. In contrast to parameter-efficient tuning (PET) strategies such as Low-Rank Adaptation (LoRA), Adapters, or prompt-based methods, FMT allows complete representational flexibility at the cost of significant computational and memory overhead, as every weight matrix and bias vector is subject to gradient-based updating throughout fine-tuning. FMT is standard for maximizing downstream accuracy, enabling thorough domain or task adaptation, and is foundational to rigorous analysis of deep model adaptation dynamics.

## 1. FMT: Definition, Formulation, and Contrast with PET

FMT entails optimizing all parameters $\theta$ of a pre-trained model $f_{\theta_0}$ over a downstream dataset $D$ by minimizing an objective such as cross-entropy:

\[
\theta^* = \arg\min_{\theta} \mathcal{L}(\theta; D)
\]

During FMT, no layers or parameters are frozen: gradient flow and optimization apply to all Transformer blocks, embedding tables, and (for multimodal models) encoders and projection heads [2412.15652]. This full-unfreezing distinguishes FMT from PET, in which only a constrained low-dimensional subspace is trained—e.g., LoRA adds a low-rank factorization $\Delta W = BA$ per weight matrix with $B \in \mathbb{R}^{m\times r}$, $A \in \mathbb{R}^{r\times n}$, and $r \ll \min(m,n)$, resulting in greatly reduced parameter count and smaller compute/memory footprint [2304.08109, 2410.21228]. 

By updating all model weights, FMT unlocks the full representational capacity of the model, enabling arbitrary changes to singular values and singular vectors of each weight matrix, whereas LoRA can only induce changes within an $r$-dimensional subspace of the parameter space [2410.21228]. Therefore, FMT is the gold standard for task/domain adaptation performance analysis, theoretical exploration of transfer, and as a reference for PETL (parameter-efficient transfer learning) methods.

## 2. Methodology and Scaling Properties

The FMT workflow consists of:

- Initializing the model with all pre-trained weights.
- Defining loss $\mathcal{L}$ over the downstream dataset.
- Applying batched stochastic optimization (commonly AdamW or Lion) to all parameters.
- Employing appropriate precision (fp32, bf16, or int8 in special cases) and hyperparameters (learning rate, scheduler, batch size).

Typical model/experiment configurations include LLaMA-7B to 70B [2304.08109, 2605.21177], RoBERTa-large [2605.21177], or mDeBERTa-v3 (278M parameters) [2501.06025]. For large models, training setups require GPU clusters (e.g., 8×A100-40GB) and extended training times (e.g., 17–31 hours per epoch for LLaMA-7B on 0.6–2M Chinese instruction pairs) [2304.08109]. Hyperparameters for state-of-the-art FMT on LLaMA-7B include bf16 precision, batch size 32, 3 epochs, peak LR $5\times10^{-6}$, and cosine scheduling.

Power-law scaling holds robustly for FMT:

\[
\hat{\mathcal{L}}(N, D_{\rm ft}) = \frac{A_m}{N^{\alpha_m} D_{\rm ft}^{\beta}} + E
\]

For En→De MT, $\alpha_m=0.52$, $\beta=0.15$; for pretraining size scaling, $\alpha_p=0.21$ [2402.17193]. FMT exhibits higher scaling exponents ($\beta$) with respect to finetuning data, indicating stronger benefits from increased downstream data than PET methods ([2402.17193], Table 5).

## 3. Empirical Results: Performance vs. Cost and Competing Methods

FMT systematically outperforms LoRA and other PET variants in raw downstream metrics when run from scratch, with gains of up to +0.10 in average GPT-score for Chinese instruction-following (LLaMA-7B, 2M examples: FMT 0.710 vs. LoRA 0.609) [2304.08109]. For commonsense QA tasks at comparable parameter budgets, FMT approaches or matches maximal accuracy (e.g., LLaMA-7B: LoFT 76.08%, FMT $\approx$77%, LoRA 73.57%) [2505.21289].

However, FMT is expensive: 3–5x the wall-clock time and GPU-memory per epoch vs. LoRA for the same base model and data size. For LLaMA-7B, FMT(2M, 3 epochs) consumes $\approx$744 GPU-hours versus $<$200 GPU-hours for LoRA [2304.08109]. Peak per-GPU memory for dense AdamW is $\approx$40GB, often making FMT infeasible on smaller devices.

Empirical data from mDeBERTa-V3 on Germanic languages demonstrates FMT superiority for named entity recognition in low-resource settings (improving F1 by $1$–$2$ points over PET), but PET matches/exceeds FMT for high-resource extractive QA [2501.06025]. Thus, task and resource regime critically mediate FMT’s relative value.

## 4. Algorithmic Advancements: Memory and Communication-Efficient FMT

Recent advances have made FMT more practical for limited hardware.

- **ChunkFT** [2605.21177]: Partitions all weights into $K$ byte-balanced chunks, sequentially fitting only one chunk’s fp32 master weights, gradients, and optimizer states in GPU memory at a time. Forward pass uses the full fp16 model, and backward computes gradients only for the active chunk. This enables large models (LLaMA3-8B/70B) to be fully tuned with $<$14GB (7B, $K=16$) on commodity GPUs. Accuracy is maintained: ChunkFT achieves $6.8$ on MT-Bench for Llama 3-8B, exceeding full-model AdamW ($6.3$).
- **QFT** [2310.07147]: Quantizes all training states (weights, gradients, optimizer moments) to INT8 and uses the Lion optimizer for robust update steps, enabling LLaMA-7B FMT with $<$30GB memory—21% of the standard FP32 cost. QFT matches or nearly matches full-precision FMT for headline metrics (e.g., 57.4% vs. 58.0% few-shot accuracy on MMLU/ARC-c/HellaSwag/TruthfulQA).
- **FedKSeed** [2312.06353]: Makes FMT feasible in federated learning by using zeroth-order gradient estimates communicated via seeds and scalars (not full gradients), reducing per-round communication to $<$18 KB. This achieves accuracy gains ($\approx$7.3% over LoRA-based FL) with minimal memory and communication overhead.

These methods have lowered the effective memory floor and communication barrier, supporting FMT of multi-billion parameter models on single mid-range GPUs or distributed client settings.

## 5. Theoretical and Structural Consequences

Unlike LoRA/adapter approaches, FMT exerts unconstrained control over both the singular values and singular vectors of each weight matrix. Spectral analysis establishes that FMT preserves the orientation and scales of existing pre-trained subspaces; it does not introduce “intruder dimensions”—new, orthogonal high-variance axes—that are a hallmark of LoRA adaptation [2410.21228]. The SVD alignment between pre-trained and FMT-tuned weights shows nearly perfect diagonality (cosine similarity $>0.9$ for top-$k$ singular vectors), in contrast to LoRA, which accumulates high-magnitude outliers after each task. 

Empirical evidence demonstrates that such intruder dimensions in LoRA are localized drivers of catastrophic forgetting and incremental error during sequential/continual learning, whereas FMT retains earlier-task accuracy more robustly [2410.21228]. However, FMT also tends to overwrite more pre-trained knowledge, manifesting as greater “pseudo-perplexity drift” on the original pre-training corpus—a direct result of large-scale parameter changes.

## 6. Trade-offs and Practical Guidelines

Selecting FMT vs. PET depends on task, data, model size, and compute/memory budget:

- **Use FMT** when:
  - End-task accuracy is paramount and resource constraints are secondary.
  - Initial adaptation of a fresh, pre-trained model to instruction-following, domain, or language.
  - The task or language is low-resource and requires full representational re-specialization (e.g., NER in Icelandic) [2501.06025].
- **PET/LoRA** is preferable for:
  - Rapid prototyping, multi-domain adapters, modular incremental updates.
  - Deployment under strict resource, memory, or communication constraints.
  - Scenarios demanding retention of pre-trained zero/few-shot performance.

Critical thresholds for transitioning from PET to FMT depend on both model size and data volume. For example, in MT, FMT outperforms LoRA only above $\sim$200K finetuning examples for a 1B model, with the threshold dropping for larger models [2402.17193]. Task and transfer properties further modulate the practical crossover point.

## 7. Future Directions and Open Challenges

Outstanding research challenges include:

- Precise explanation of LoRA’s initial underperformance relative to FMT when applied to vanilla pre-trained models, and the potential for hybrid regimes (e.g., seed with FMT on small data, finish with PET on large data) [2304.08109].
- Validation and refinement of scaling laws for FMT in creative or open-ended generation [2402.17193].
- Extending memory- and communication-efficient FMT to 33B/65B+ models, low-resource edge devices, and extremely large federated networks [2605.21177, 2310.07147, 2312.06353].
- Investigation of optimizer variants (Lion, AdamW) and their interactions with quantization for robust, accurate INT8-full tuning [2310.07147].
- Application of reinforcement learning from human feedback (RLHF) layered on FMT vs. PET regimes [2304.08109].
- Deeper spectral and continual learning analysis to minimize catastrophic forgetting in full-tuned and partially adapted models [2410.21228].

Empirical, theoretical, and methodological advances continue to scale FMT across language, modality, and deployment regime, cementing its role as a reference and target for adaptation research in deep networks.

Source: https://www.emergentmind.com/topics/full-model-tuning-fmt