---
title: 'Tied-LoRA: Efficient Low-Rank Adaptation'
url: https://www.emergentmind.com/topics/tied-lora
type: topic
---

# Tied-LoRA: Efficient Low-Rank Adaptation

Tied-LoRA is a parameter-efficient extension of Low-Rank Adaptation (LoRA) that addresses the storage and computational challenges inherent to large-scale, multi-task fine-tuning scenarios. By introducing layerwise weight tying and selective parameter training, Tied-LoRA compresses adaptation overhead while maintaining competitive model performance. This paradigm has been empirically validated on multiple tasks and architectures, demonstrating its efficacy in real-world large language model (LLM) customization workflows.

## 1. Conceptual Foundation and Motivation

Large-scale LLMs require significant resources to fine-tune and store per-task or per-user customizations. Parameter-Efficient Fine-Tuning (PEFT) methods, such as LoRA, mitigate this by injecting low-rank trainable updates into frozen pretrained weights. However, as tasks and variants proliferate, even LoRA's per-layer adaptation overhead compounds.

Tied-LoRA targets this bottleneck by:
- **Weight tying**: Sharing a single set of LoRA’s low-rank matrices across all transformer layers rather than parameterizing unique adaptations per layer.
- **Selective training**: Explicitly controlling which components in the low-rank update (projection matrices, scaling vectors) are trainable versus frozen, defining a spectrum of configuration granularities.

These design choices seek to retain most of LoRA's adaptation capability while using a dramatically reduced number of trainable parameters—crucial for scalable and on-device deployment scenarios.

## 2. Technical Formulation and Parameterization Spectrum

### 2.1 LoRA and Tied-LoRA Generalization

Standard LoRA extends a frozen pretrained linear weight $W$ via:
\[
z = Wx + \Delta Wx = Wx + \alpha \cdot BAx
\]
with $A \in \mathbb{R}^{d \times r}$, $B \in \mathbb{R}^{r \times d'}$, and $\alpha$ a scalar. Each transformer module typically receives its own uniquely parameterized $A,B$.

Tied-LoRA generalizes this to:
\[
z = Wx + \alpha \cdot A_v B A_u x
\]
where $A$ and $B$ are low-rank matrices (potentially tied across layers); $A_u$ (input-side) and $A_v$ (output-side) are diagonal scaling matrices or vectors per layer. Selective tying (sharing across layers) and training (frozen or trainable) on any of these components yields a continuum between LoRA, Tied-LoRA, and even more aggressive parameter reductions (see for example VeRA).

### 2.2 Configuration Table

A representative survey of configurations is shown below (letting $L$ denote number of layers, $d$ hidden size, $r$ rank):

| Method           | Trainable Elements                      | Trainable Parameters                           |
|------------------|----------------------------------------|------------------------------------------------|
| LoRA (vBuA)      | All, per-layer                         | $4Ldr$                                         |
| TL5 (vBg uAg)    | A,B tied; u,v frozen                   | $4dr$                                          |
| TL6 (vBg uAg)    | A,B tied; u,v trainable, per-layer     | $4dr + L(r+3d)$                                |
| VeRA             | A,B tied & frozen; u,v trainable       | $L(r+3d)$                                      |

The greatest savings arise from tying the $A,B$ matrices across all layers. Selectively training per-layer scaling vectors ($u,v$) in TL6 increases flexibility at a moderate parameter cost.

## 3. Experimental Evaluation Across Diverse Tasks

### 3.1 Datasets and Tasks

Empirical validation is reported over five distinct benchmarks:
- **Extractive QA** (SQuAD v1; EM accuracy)
- **Summarization** (DialogSum; ROUGE-L)
- **Commonsense NLI** (HellaSwag; MC accuracy)
- **Machine Translation** (IWSLT 2017 De→En; BLEU)
- **Mathematical Reasoning** (GSM8K; EM accuracy)

### 3.2 Base Models and Training Regime

- **Backbones**: LLaMA-2 7B and GPT-2B-001.
- **Training**: NVIDIA NeMo; AdamW; cosine learning rate annealing; early stopping; $r \in \{2, 4, 8, ..., 128\}$; no extensive hyperparameter tuning.

## 4. Quantitative Results and Trade-offs

### 4.1 Performance vs Parameter Curve

Empirical findings highlight:
- **TL6**: Delivers performance within $1\text{–}2\%$ of classic LoRA across tasks while using only $12.5\%$ (or less) of its parameters at higher ranks. For machine translation, TL6 slightly outperformed LoRA.
- **TL5**: Achieves even greater compression (e.g., $3\%$ of LoRA's parameters for LLaMA-2 7B rank 8), with small accuracy drops relative to both TL6 and LoRA.
- Performance is robust across a wide rank range, except for some extreme compression configurations or heavily capacity-limited tasks (e.g., GSM8K).
- Layer-sharing (tying $A,B$ across layers) is markedly superior to simply reducing LoRA to a single-layer adaptation at fixed parameter counts.

| Method   | Translation (BLEU) | QA (EM) | NLI (Acc) | Summarization (ROUGE-L) | Params (% LoRA) |
|----------|--------------------|---------|-----------|-------------------------|-----------------|
| LoRA     | 41.3               | 88.5    | 91.97     | 40.76                   | 100             |
| TL6      | 41.33              | 87.97   | 91.15     | 39.24                   | 12.5            |
| TL5      | 41.01              | 87.11   | 91.75     | 40.62                   | 3.1             |
| VeRA     | 40.41              | 87.69   | 90.47     | 40.07                   | 9.4             |

### 4.2 Task Dependence and Sensitivity

- Tasks requiring substantial adaptation capacity (such as mathematical reasoning) exhibit a more pronounced gap between LoRA and highly compressed Tied-LoRA variants.
- Tasks closely matched to the base model’s pretraining distribution (NLI, QA, summarization) are less sensitive to adaptation parameter reduction.

## 5. Methodological Nuances and Implications

Tied-LoRA enables a "drop-in" replacement for LoRA: the adaptation can be merged with base weights for inference, with no additional computational overhead. The approach is particularly advantageous for:
- Scenarios with deep transformers or many concurrent task/user customizations.
- Use cases where device or network bandwidth limits adapter deployment (due to significant reduction in adapter size).
- Situations demanding rapid deployment or storage-efficient sharing of fine-tuned variants.

A key finding is that tying (sharing) low-rank matrices across layers, combined with judiciously chosen scaling vector parameterization, can serve a vast majority of LoRA's adaptation role with only a modest subset of its original parameter footprint.

## 6. Comparison to Other Parameter-Efficient Fine-Tuning Approaches

Tied-LoRA exists on a spectrum with other PEFT methods:
- **Classic LoRA**: Most flexible, parameter-expensive.
- **VeRA**: Even more parameter-efficient (freezes tied matrices, trains only per-layer/scaling vectors); can be less performant than Tied-LoRA in some tasks.
- **Uni-LoRA** [2506.00799]: Extends these ideas, unifying LoRA, Tied-LoRA, and related approaches as instances of subspace projection with explicit global parameter sharing. Tied-LoRA corresponds to a block-diagonal, layer-local projection in this framework, while Uni-LoRA employs a global, isometric projection maximizing sharing and efficiency.

A distinct feature of Tied-LoRA is its flexibility—by selecting which matrices/vectors to tie and/or train, one can interpolate between expressiveness and efficiency.

## 7. Formalism Summary

- **Generalized Tied-LoRA update**:
  \[
  z = Wx + \alpha \cdot A_v B A_u x
  \]
  - $A, B$: Low-rank matrices, either tied across layers or uniquely parameterized.
  - $A_u, A_v$: Diagonal scaling matrices/vectors, potentially per-layer and trainable.
- **Canonical configurations**:
  - **TL5**: $A,B$ tied, $u,v$ frozen; all adaptation via global low-rank matrices.
  - **TL6**: $A,B$ tied, $u,v$ per-layer and trainable; balances global adaptation flexibility with moderate per-layer adjustment.

Tied-LoRA thus provides a parameterization regime that covers both the classic LoRA and more aggressive PEFT methods—enabling practitioners to tailor adaptation footprints to resource constraints and targeted task complexity.

Source: https://www.emergentmind.com/topics/tied-lora