---
title: 'BitFit/BEFT: Bias-Only Fine-Tuning'
url: https://www.emergentmind.com/topics/bias-only-fine-tuning-bitfit-beft
type: topic
---

# BitFit/BEFT: Bias-Only Fine-Tuning

Bias-Only Fine-Tuning (BitFit/BEFT) is a family of parameter-efficient adaptation strategies for large pretrained models, wherein only the additive bias vectors in network layers are updated during downstream training. All other parameters—including the potentially vast weight matrices in self-attention, MLP, or convolutional modules—are frozen. This selective fine-tuning regime enables rapid, memory- and storage-efficient transfer to new tasks while retaining the core modeling capacity of the originating foundation model. Recent research further refines these methods by recognizing that only specific subsets of bias terms may drive adaptation performance, leading to bias-efficient fine-tuning (BEFT). These approaches have demonstrated competitive results across natural language processing, vision, medical imaging, and time-series domains [2106.10199, 2409.11302, 2509.15974, 2210.00036, 2401.04051, 2506.18434].

## 1. Theoretical and Algorithmic Foundations

BitFit, initially introduced for transformer-based masked language models, consists of partitioning model parameters as $\theta = (W, b)$, where $W$ are all weight matrices and $b$ are the bias vectors. During fine-tuning, $W$ is strictly frozen while $b$ is updated. The downstream training problem is then
$$
\min_{b,\phi} \; \mathbb{E}_{(x, y) \sim D}\bigl[ L(f(x; W_{\mathrm{fixed}}, b, \phi), y) \bigr],
$$
where $\phi$ is a typically small task-specific head. In BEFT, a further selection is made: only a single family of bias vectors (such as all $b_v$ in the value projections) is updated, minimizing the same objective but over a much smaller subspace [2509.15974].

For each linear layer $l$, the forward computation is modified from $x \mapsto Wx + b$ to $x \mapsto Wx + (b + \Delta b)$, with $\Delta b$ initialized to zero and treated as the optimization variable. The gradient update is
$$
\Delta b \leftarrow \Delta b - \eta \frac{\partial \mathcal{L}}{\partial (b + \Delta b)},
$$
with all other parameters held fixed.

## 2. Implementation Details and Variants

BitFit applies to all bias parameters in attention, feed-forward, and normalization layers—e.g., query/key/value/output projections, MLPs, and LayerNorms in transformers; convolutional, linear, and normalization biases in CNNs [2503.14836, 2506.18434]. Initial values are typically inherited from pretraining. Adam or SGD optimizers are common, with learning rates for bias-only steps set 5–10× higher than typical full fine-tuning rates (e.g., $1$--$5 \times 10^{-3}$ for non-private settings; lower for high-data or larger models) [2106.10199, 2210.00036].

In differentially private settings (DP-BiTFiT), only the gradients with respect to $b$ are clipped and perturbed using DP-SGD; this eliminates the overhead of per-sample activation storage and weight gradient computation, greatly reducing time and memory complexity [2210.00036].

BEFT introduces a bias selection mechanism based on a "projection-ratio" importance score, computed after a brief "pilot" epoch of all-bias fine-tuning. This score quantifies the geometric divergence between pre- and post-pilot bias vectors. Only the single bias type with greatest change (typically value, $b_v$) is then retained for full fine-tuning, resulting in further parameter and runtime reductions [2509.15974].

## 3. Empirical Performance and Comparative Analysis

### NLP and Foundation Models
BitFit (or BEFT) consistently matches or closely approaches full fine-tuning (which updates all parameters) and other PEFT methods (adapters, LoRA, prefix-tuning) on sentence-level and token-level tasks for both small and medium data regimes [2106.10199, 2401.04051]. On GLUE, BERT-Large achieves 84.2 (BitFit, 0.08% params) vs 84.8 (full FT, 100%). BitFit may outperform full FT for small datasets (e.g., RTE 73.2 vs 71.9) and shows comparable or improved generalization gaps.

In low-data settings, BEFT further outperforms all-bias BitFit and LoRA/Prefix, with <0.01% parameter updates. For BERT-Base on GLUE (low/medium/high data), updating only $b_v$ yields 64.5/70.8/71.7 accuracy, exceeding other bias types and matching or surpassing full BitFit [2509.15974].

### Vision and Medical Imaging
In ViT-Base (86M parameters), BitFit adapts just ~0.13% of parameters [2503.14836]. Robustness-accuracy trade-offs reveal BitFit excels on simple vision tasks (CIFAR-10/100; up to 81.5% above average Pareto AUC) but becomes suboptimal for fine-grained problems (Caltech-256, CUB-200) where more informative or mid-level adaptations are required.

In chest X-ray prognosis prediction, BitFit (0.01% params) achieves moderate MCCs in balanced full-data regimes but is outperformed by LoRA and related methods in highly imbalanced or few-shot settings. BitFit is preferable when training and memory budgets are extremely limited and data are not severely scarce or imbalanced [2506.18434].

### Time Series Foundation Models
In Chronos TSFM (8.3M–201M parameters), BitFit tunes only 200–700 parameters but delivers MeanBP forecasting MSEs that closely match or even surpass LoRA and full fine-tuning in large models (MSE=19.77 for BitFit vs 20.12 LoRA, 20.8 FT in Chronos Base) [2409.11302].

| Model          | Params Tuned       | MeanBP MSE (Tiny/Base) |
|----------------|--------------------|------------------------|
| Full FT        | 8.3M / 201M        | 19.90 / 20.80          |
| LoRA           | 0.049M / 0.442M    | 19.79 / 20.12          |
| BitFit         | 0.0002M / 0.0007M  | 20.68 / 19.77          |
| FourierFT      | 0.0024M / 0.007M   | 19.51 / 20.98          |

## 4. Parameter Efficiency and Computational Benefits

The parameter savings of BitFit are substantial. Only ~0.05–0.2% of model parameters (sometimes an order of magnitude fewer for BEFT, down to 0.01‰) are updated per task [2210.00036, 2509.15974]. This drastically reduces storage requirements for multi-task or streaming adaptation setups. BitFit also accelerates wall-clock fine-tuning and, in DP settings (DP-BiTFiT), reduces memory and compute by 2–30× compared to full DP fine-tuning [2210.00036].

BitFit/BEFT training is inherently stateless for weights; pre-trained matrices can be deployed in hardware or immutably shared, while only per-task biases and heads must be managed. This property is especially attractive in resource-constrained environments, federated learning, and for privacy-sensitive deployment.

## 5. Robustness, Generalization, and Analysis

BitFit preserves much of the upstream model's robustness on simple or highly overlapping downstream tasks. On CIFAR-10/100, its adversarial and OOD robustness closely tracks accuracy and remains superior to other PEFT methods for base-level or low-class-count tasks [2503.14836]. For more complex/fine-grained tasks, or with severe distributional shifts, BitFit's expressivity is restrictive and robustness-accuracy trade-offs become unfavorable compared to adapters or mid-layer updates.

Generalization gap analyses reveal that restricting adaptation to biases acts as implicit regularization. This yields lower overfitting in small- or medium-data regimes, but full fine-tuning can realize higher peak accuracy when abundant labeled data is available [2106.10199, 2401.04051].

The surprising effectiveness of bias-only adaptation supports the hypothesis that pretraining induces representations where only threshold and shift parameters need to be task-adapted, exposing rather than (re-)learning new knowledge [2106.10199]. BEFT's geometric projection-ratio scoring further clarifies which biases, such as value projections, most support rapid transfer, especially under low data [2509.15974].

## 6. Best Practices, Limitations, and Extensions

### Usage Recommendations

- Start with BitFit (or BEFT) when data, compute, or memory budgets are tight, the domain shift is moderate, or rapid/streaming task adaptation is necessary.
- Select a bias learning rate 5–10× higher than standard FT; adapt epochs/steps as convergence is typically faster.
- In differentially private applications, BitFit's gradient isolation for biases minimizes DP costs [2210.00036].
- Use BEFT's pilot procedure to select the most effective bias subset, especially for large models and low-data tasks.

### Limitations and Open Questions

- For tasks requiring deep feature modulation (fine-grained vision, extreme distributional shift, few-shot settings), BitFit can underperform richer PEFT approaches (LoRA, adapters, Compacter) [2506.18434, 2503.14836, 2509.15974].
- Hybrid or two-phase approaches (brief full or LoRA fine-tuning followed by bias-only updates) partially mitigate these limitations.
- The theoretical underpinnings of why bias shifts suffice in many cases—particularly in overparameterized, privacy-constrained, or multitask settings—remain a subject of active investigation [2210.00036, 2509.15974].

## 7. Extensions: Layer and Term Selection (BEFT)

Selective bias adaptation is formalized in BEFT, which leverages geometric measures to rank and identify the most informative bias subfamilies for target tasks. This dynamic, data-driven selection outperforms static magnitude or Fisher-based heuristics and pushes the resource gains of BitFit even further without loss in accuracy [2509.15974].

In practical terms, BEFT performs a brief (1–2 epoch) pilot fine-tuning on all bias types with a large learning rate, computes projection-ratio scores, resets, and then performs downstream fine-tuning on the single best bias family (e.g., $b_v$). This results in parameter and runtime savings by an additional order of magnitude, with empirical superiority across NLP and autoregressive LLM benchmarks in both low-data and standard regimes.

---

BitFit and BEFT constitute parameter-minimal, hyperparameter-robust, and “model-agnostic” tools for fine-tuning foundation models. Their appeal rests on empirical competitiveness across modalities and data regimes, extreme deployment efficiency, and theoretical insight into the structural sufficiency of bias terms for model adaptation [2106.10199, 2409.11302, 2509.15974, 2210.00036, 2503.14836, 2506.18434, 2401.04051].

Source: https://www.emergentmind.com/topics/bias-only-fine-tuning-bitfit-beft