---
title: Tensorized Clustered LoRA (TC-LoRA)
url: https://www.emergentmind.com/topics/tensorized-clustered-lora-tc-lora
type: topic
---

# Tensorized Clustered LoRA (TC-LoRA)

Tensorized Clustered LoRA (TC-LoRA) is a parameter-efficient fine-tuning (PEFT) method designed to address multi-task interference in large language model (LLM) adaptation by combining input-level clustering with a joint tensor decomposition of LoRA adapters. TC-LoRA achieves improved downstream performance in out-of-domain, zero-shot, and skill-compositional settings through a dual-level approach: leveraging text-level clustering to isolate input-format similarities, and parameter-level merged factorization to disentangle and preserve shared versus task-specific knowledge across LoRA adapters [2508.03999].

## 1. Motivation and Conceptual Overview

Multi-task PEFT schemes commonly train a separate LoRA adapter $\Delta_i = A_i B_i^\top$ for each task $i$ and merge adapters via summation or averaging during inference. However, naively merged adapters from heterogeneous tasks exhibit "task interference," resulting in degraded accuracy and compositionality. Prior approaches—including SVD-based merging (e.g., TSV, Ties, DARE)—mitigate redundancy within a single $\Delta_i$ but do not resolve cross-adapter conflicts.

TC-LoRA introduces two levels of interference reduction:

- **Text-level (C-LoRA):** Using input-format clustering in embedding space, then training a specialized LoRA adapter per cluster, reducing conflicts at the example level.
- **Parameter-level (CP merging):** Aggregating all cluster-specific adapters into a tensor and jointly decomposing via Canonical Polyadic (CP) factorization, thereby disentangling shared and cluster-specific knowledge while mitigating parameter-level interference.

This dual approach exploits latent structure at both the data and parameter granularity.

## 2. Text-Level Clustering: C-LoRA

Let $\{I_i\}$ denote the instruction texts from the multi-task corpus and $E(\cdot)$ a fixed pretrained sentence encoder (implemented with sentence-t5-xxl). The C-LoRA module proceeds in two substeps:

### 2.1 Embedding and Clustering

- For each instruction $I_i$, produce embedding $e_i = E(I_i) \in \mathbb{R}^d$.
- Extract $K$ clusters $\{C_1, \dots, C_K\}$ via $k$-means in embedding space, fitting on a 20% sample for efficiency, then assigning the remainder.
- Cluster centroids: 
  $$
  \mu_k = \frac{1}{|C_k|}\sum_{i\in C_k} e_i,\quad k = 1 \ldots K.
  $$
- For each $C_k$, train a LoRA adapter using data from that cluster.

### 2.2 Routing for Inference

Given a new input $x$, assign it to a cluster adapter via:
- **Hard assignment:** $k^* = \arg\min_k \|e_x - \mu_k\|_2$
- **Soft assignment (optional):** 
  $$
  p(k\mid x) = \frac{\exp(-\|e_x - \mu_k\|^2/\tau)}{\sum_{j=1}^K \exp(-\|e_x - \mu_j\|^2/\tau)}
  $$
where $\tau$ is a temperature parameter.

Routing determines which specialized adapter is employed per input in both training and inference.

## 3. Parameter-Level CP Decomposition and Adapter Merging

Cluster adapters $\Delta_i \in \mathbb{R}^{d_{\mathrm{in}}\times d_{\mathrm{out}}}$ are aggregated into a tensor $\mathcal{T} \in \mathbb{R}^{d_{\mathrm{in}}\times d_{\mathrm{out}}\times K}$ where $\mathcal{T}_{:,:,i} = \Delta_i$ for each cluster $i$. TC-LoRA applies a global CP decomposition:

$$
\mathcal{T} \approx \sum_{r=1}^R \mathbf{b}_r \otimes \mathbf{c}_r \otimes \mathbf{a}_r
$$

- $\mathbf{b}_r \in \mathbb{R}^{d_{\mathrm{in}}}$: row-mode factors (shared across clusters)
- $\mathbf{c}_r \in \mathbb{R}^{d_{\mathrm{out}}}$: column-mode factors (shared)
- $\mathbf{a}_r \in \mathbb{R}^K$: cluster-mode factors (cluster-specific weighting)
- $R$: CP rank (set to $R=20$ by default for best performance on GSM8K-hard)

Each cluster slice is approximated as:
$$
\Delta_i \approx \sum_{r=1}^R a_{r,i} \mathbf{b}_r \mathbf{c}_r^\top,\quad (a_{r,i} = [\mathbf{a}_r]_i)
$$

### Merging All Adapters

A single weight update for deployment is synthesized by:
$$
\Delta_{\mathrm{merged}} \approx \sum_{r=1}^R \left(\sum_{i=1}^K a_{r,i}\right) \mathbf{b}_r \mathbf{c}_r^\top
$$
and added to the base LLM:
$$
W_{\mathrm{MT}} = W_0 + \alpha \sum_{r=1}^R \left(\sum_{i=1}^K a_{r,i}\right) \mathbf{b}_r \mathbf{c}_r^\top
$$
where $W_0$ is the base model weights and $\alpha$ is a scaling hyperparameter.

### Optimization

The factors $\{A, B, C\}$ are fit by minimizing Frobenius norm reconstruction error with $\ell_2$ regularization:
$$
\min_{A,B,C} \left\| \mathcal{T} - \sum_{r=1}^R \mathbf{b}_r \otimes \mathbf{c}_r \otimes \mathbf{a}_r \right\|_F^2 + \lambda ( \|A\|_F^2 + \|B\|_F^2 + \|C\|_F^2 )
$$
where $A$ stacks the cluster-mode factors, and $B$, $C$ stack the row/column factors. Alternating Least Squares (ALS) is employed in practice.

## 4. Algorithmic Pipeline

The following summarizes the principal workflow:

```text
Algorithm 1: TC-LoRA Building and Merging

Input: Multi-task datasets D_1…D_T, base LLM weights W_0
Hyperparameters: #clusters K, CP rank R
Output: Merged multi-task weights W_MT

Step 1: Text-level clustering (C-LoRA)
  For each example x in ⋃_{i=1}^T D_i:
    e ← E(x)
  {C_1…C_K} ← k-means(E)
  For k=1…K:
    Train LoRA(A_k,B_k) on cluster C_k
    Δ_k ← A_k B_k^T

Step 2: Parameter-level CP merging
  Stack Δ_1…Δ_K into tensor  𝒯 ∈ ℝ^{d_in×d_out×K}
  Solve CP:  𝒯 ≈ ∑_{r=1}^R b_r⊗c_r⊗a_r
  Compute merged Δ:  Δ_merged ← ∑_{r=1}^R (∑_{k=1}^K [a_r]_k) (b_r c_r^T)
  W_MT ← W_0 + α·Δ_merged
Return W_MT
```

## 5. Empirical Results and Practical Considerations

### Experimental Benchmarks

TC-LoRA is evaluated on prominent LLMs—Phi-3 (3B) and Mistral-7B—against competitive baselines (single-task LoRA, multi-task LoRA, uniform library, Ties, DARE, Task-Arithmetic, TSV merging). Key results:

- **Phi-3 Zero-Shot Average Accuracy:** BASE=66.6%, Multi=65.4%, Uniform=67.2%, TSV=67.7%, C-LoRA=68.9%, TC-LoRA=69.1% ($\Delta$TC-LoRA=+1.4% over base/TSV).
- **Mistral-7B Zero-Shot Average Accuracy:** BASE=59.5%, Multi=63.1%, Uniform=61.9%, TSV=62.7%, C-LoRA=64.9%, TC-LoRA=65.0% ($\Delta$TC-LoRA=+2.3% over base/TSV).
- **Skill Composition (GSM8K-hard):** BASE=4.3% (524 invalid codes), LoRA(math)=11.8%, LoRA(code)=8.0%, Multi=13.5% (413), Uniform=12.96% (217), TSV=13.49% (224), CP merging=15.69% (201).

### Ablative and Hyperparameter Insights

- Performance increases with CP rank $R$, peaking at $R\approx20$.
- Phi-3 performance peaks at $K=10$ clusters; Mistral-7B exhibits robustness about $K\sim10$.
- Interference (CP-STI metric) drops in deeper network layers, correlating with effective merging.

### Implementation and Complexity

Default settings: LoRA rank $r=4$, clustering $K=10$, CP rank $R=20$. Memory use for merged TC-LoRA adapters is highly efficient (e.g., 1.88MB for Phi-3 at $R=1$), approaching the overhead of a single LoRA and considerably less than a full library (C-LoRA, $K=10$: 75.0MB; TSV-merging: 2.37MB; LoRA Lib $K=256$: 2304MB).

Inference involves a sentence embedding, cluster lookup, and a single LoRA adapter, matching the computational complexity of standard LoRA.

## 6. Strengths, Limitations, and Future Directions

### Strengths

- **Dual-level mitigation:** Example- and parameter-level interference addressed by data-driven clustering and global joint factorization.
- **Parameter efficiency:** Merged adapter has nearly the footprint of a single low-rank LoRA; avoids excessive memory costs.
- **Zero-training merge:** No task re-tuning required after CP decomposition.
- **Superior transfer:** Joint decomposition exploits cross-cluster structure and outperforms independent SVD-based strategies.

### Limitations

- Clustering sensitivity: Efficacy depends on the quality of sentence embeddings; hyperparameters $K$ and encoder type may need retuning for new domains or tasks.
- CP computational overhead: For large feature, output, or cluster dimensions, ALS-based CP factorization increases cost.
- Uniform CP rank: Current approach assumes shared CP rank $R$ across all layers, which may limit adaptation for heterogeneously informative layers.

### Prospective Research

- Adaptive or hierarchical clustering; soft input routing functions.
- Exploring alternative tensor decompositions (Tucker, Tensor-Train) for potentially richer cross-cluster interactions.
- Per-layer or cross-layer CP rank/factorization strategies.
- Extending TC-LoRA to additional PEFT paradigms such as prefix-tuning and prompt tuning.

## 7. Positioning within Multi-Task LLM Adaptation

TC-LoRA demonstrates state-of-the-art empirical accuracy for out-of-domain, zero-shot, and skill-compositional benchmarks among PEFT merging techniques, specifically outperforming strong SVD-based approaches (TSV, Ties, DARE) by up to +2.3% on Mistral-7B with minimal parameter overhead. By jointly modeling input similarity and parameter sharing, it provides a robust framework for scalable multi-task LLM adaptation with effective mitigation of negative task interference [2508.03999].

Source: https://www.emergentmind.com/topics/tensorized-clustered-lora-tc-lora