Tensorized Clustered LoRA (TC-LoRA)

• Tensorized Clustered LoRA (TC-LoRA) is a parameter-efficient fine-tuning method that mitigates multi-task interference by integrating text-level clustering with CP decomposition.
• It uses k-means clustering on sentence embeddings to segregate input data, enabling the training of specialized LoRA adapters per cluster.
• By aggregating cluster-specific adapters into a tensor and applying CP factorization, TC-LoRA disentangles shared from task-specific knowledge for improved performance.

Tensorized Clustered LoRA (TC-LoRA) is a parameter-efficient fine-tuning (PEFT) method designed to address multi-task interference in LLM adaptation by combining input-level clustering with a joint tensor decomposition of LoRA adapters. TC-LoRA achieves improved downstream performance in out-of-domain, zero-shot, and skill-compositional settings through a dual-level approach: leveraging text-level clustering to isolate input-format similarities, and parameter-level merged factorization to disentangle and preserve shared versus task-specific knowledge across LoRA adapters (Su et al., 6 Aug 2025).

1. Motivation and Conceptual Overview

Multi-task PEFT schemes commonly train a separate LoRA adapter $\Delta_i = A_i B_i^\top$ for each task $i$ and merge adapters via summation or averaging during inference. However, naively merged adapters from heterogeneous tasks exhibit "task interference," resulting in degraded accuracy and compositionality. Prior approaches—including SVD-based merging (e.g., TSV, Ties, DARE)—mitigate redundancy within a single $\Delta_i$ but do not resolve cross-adapter conflicts.

TC-LoRA introduces two levels of interference reduction:

• Text-level (C-LoRA): Using input-format clustering in embedding space, then training a specialized LoRA adapter per cluster, reducing conflicts at the example level.
• Parameter-level (CP merging): Aggregating all cluster-specific adapters into a tensor and jointly decomposing via Canonical Polyadic (CP) factorization, thereby disentangling shared and cluster-specific knowledge while mitigating parameter-level interference.

This dual approach exploits latent structure at both the data and parameter granularity.

2. Text-Level Clustering: C-LoRA

Let $\{I_i\}$ denote the instruction texts from the multi-task corpus and $E(\cdot)$ a fixed pretrained sentence encoder (implemented with sentence-t5-xxl). The C-LoRA module proceeds in two substeps:

2.1 Embedding and Clustering

• For each instruction $I_i$, produce embedding $e_i = E(I_i) \in \mathbb{R}^d$.
• Extract $K$ clusters $\{C_1, \dots, C_K\}$ via $k$-means in embedding space, fitting on a 20% sample for efficiency, then assigning the remainder.
• Cluster centroids:

$i$0

• For each $i$1, train a LoRA adapter using data from that cluster.

2.2 Routing for Inference

Given a new input $i$2, assign it to a cluster adapter via:

• Hard assignment: $i$3
• Soft assignment (optional):

$i$4

where $i$5 is a temperature parameter.

Routing determines which specialized adapter is employed per input in both training and inference.

3. Parameter-Level CP Decomposition and Adapter Merging

Cluster adapters $i$6 are aggregated into a tensor $i$7 where $i$8 for each cluster $i$9. TC-LoRA applies a global CP decomposition:

$\Delta_i$0

• $\Delta_i$1: row-mode factors (shared across clusters)
• $\Delta_i$2: column-mode factors (shared)
• $\Delta_i$3: cluster-mode factors (cluster-specific weighting)
• $\Delta_i$4: CP rank (set to $\Delta_i$5 by default for best performance on GSM8K-hard)

Each cluster slice is approximated as:

$\Delta_i$6

Merging All Adapters

A single weight update for deployment is synthesized by:

$\Delta_i$7

and added to the base LLM:

$\Delta_i$8

where $\Delta_i$9 is the base model weights and $\{I_i\}$0 is a scaling hyperparameter.

Optimization

The factors $\{I_i\}$1 are fit by minimizing Frobenius norm reconstruction error with $\{I_i\}$2 regularization:

$\{I_i\}$3

where $\{I_i\}$4 stacks the cluster-mode factors, and $\{I_i\}$5, $\{I_i\}$6 stack the row/column factors. Alternating Least Squares (ALS) is employed in practice.

4. Algorithmic Pipeline

The following summarizes the principal workflow:

$I_i$1

5. Empirical Results and Practical Considerations

Experimental Benchmarks

TC-LoRA is evaluated on prominent LLMs—Phi-3 (3B) and Mistral-7B—against competitive baselines (single-task LoRA, multi-task LoRA, uniform library, Ties, DARE, Task-Arithmetic, TSV merging). Key results:

• Phi-3 Zero-Shot Average Accuracy: BASE=66.6%, Multi=65.4%, Uniform=67.2%, TSV=67.7%, C-LoRA=68.9%, TC-LoRA=69.1% ($\{I_i\}$7TC-LoRA=+1.4% over base/TSV).
• Mistral-7B Zero-Shot Average Accuracy: BASE=59.5%, Multi=63.1%, Uniform=61.9%, TSV=62.7%, C-LoRA=64.9%, TC-LoRA=65.0% ($\{I_i\}$8TC-LoRA=+2.3% over base/TSV).
• Skill Composition (GSM8K-hard): BASE=4.3% (524 invalid codes), LoRA(math)=11.8%, LoRA(code)=8.0%, Multi=13.5% (413), Uniform=12.96% (217), TSV=13.49% (224), CP merging=15.69% (201).

Ablative and Hyperparameter Insights

• Performance increases with CP rank $\{I_i\}$9, peaking at $E(\cdot)$0.
• Phi-3 performance peaks at $E(\cdot)$1 clusters; Mistral-7B exhibits robustness about $E(\cdot)$2.
• Interference (CP-STI metric) drops in deeper network layers, correlating with effective merging.

Implementation and Complexity

Default settings: LoRA rank $E(\cdot)$3, clustering $E(\cdot)$4, CP rank $E(\cdot)$5. Memory use for merged TC-LoRA adapters is highly efficient (e.g., 1.88MB for Phi-3 at $E(\cdot)$6), approaching the overhead of a single LoRA and considerably less than a full library (C-LoRA, $E(\cdot)$7: 75.0MB; TSV-merging: 2.37MB; LoRA Lib $E(\cdot)$8: 2304MB).

Inference involves a sentence embedding, cluster lookup, and a single LoRA adapter, matching the computational complexity of standard LoRA.

6. Strengths, Limitations, and Future Directions

Strengths

• Dual-level mitigation: Example- and parameter-level interference addressed by data-driven clustering and global joint factorization.
• Parameter efficiency: Merged adapter has nearly the footprint of a single low-rank LoRA; avoids excessive memory costs.
• Zero-training merge: No task re-tuning required after CP decomposition.
• Superior transfer: Joint decomposition exploits cross-cluster structure and outperforms independent SVD-based strategies.

Limitations

• Clustering sensitivity: Efficacy depends on the quality of sentence embeddings; hyperparameters $E(\cdot)$9 and encoder type may need retuning for new domains or tasks.
• CP computational overhead: For large feature, output, or cluster dimensions, ALS-based CP factorization increases cost.
• Uniform CP rank: Current approach assumes shared CP rank $I_i$0 across all layers, which may limit adaptation for heterogeneously informative layers.

Prospective Research

• Adaptive or hierarchical clustering; soft input routing functions.
• Exploring alternative tensor decompositions (Tucker, Tensor-Train) for potentially richer cross-cluster interactions.
• Per-layer or cross-layer CP rank/factorization strategies.
• Extending TC-LoRA to additional PEFT paradigms such as prefix-tuning and prompt tuning.

7. Positioning within Multi-Task LLM Adaptation

TC-LoRA demonstrates state-of-the-art empirical accuracy for out-of-domain, zero-shot, and skill-compositional benchmarks among PEFT merging techniques, specifically outperforming strong SVD-based approaches (TSV, Ties, DARE) by up to +2.3% on Mistral-7B with minimal parameter overhead. By jointly modeling input similarity and parameter sharing, it provides a robust framework for scalable multi-task LLM adaptation with effective mitigation of negative task interference (Su et al., 6 Aug 2025).