Joint Tensor-Train Parameterization for Efficient and Expressive Low-Rank Adaptation (2506.16456v1)

Abstract: Low-Rank Adaptation (LoRA) is widely recognized for its parameter-efficient fine-tuning of large-scale neural models. However, standard LoRA independently optimizes low-rank matrices, which inherently limits its expressivity and generalization capabilities. While classical tensor-train (TT) decomposition can be separately employed on individual LoRA matrices, this work demonstrates that the classical TT-based approach neither significantly improves parameter efficiency nor achieves substantial performance gains. This paper proposes TensorGuide, a novel tensor-train-guided adaptation framework to overcome these limitations. TensorGuide generates two correlated low-rank LoRA matrices through a unified TT structure driven by controlled Gaussian noise. The resulting joint TT representation inherently provides structured, low-rank adaptations, significantly enhancing expressivity, generalization, and parameter efficiency without increasing the number of trainable parameters. Theoretically, we justify these improvements through neural tangent kernel analyses, demonstrating superior optimization dynamics and enhanced generalization. Extensive experiments on quantum dot classification and GPT-2 fine-tuning benchmarks demonstrate that TensorGuide-based LoRA consistently outperforms standard LoRA and TT-LoRA, achieving improved accuracy and scalability with fewer parameters.

• The paper demonstrates that TensorGuide’s joint TT parameterization significantly improves model expressivity and conditioning compared to standard LoRA methods.
• It unifies adaptation matrices via a single TT network, enabling efficient fine-tuning without additional trainable parameters.
• Empirical evaluations reveal higher accuracy and lower loss in both quantum dot classification and GPT-2 fine-tuning on WikiText-2.

Joint Tensor-Train Parameterization for Efficient and Expressive Low-Rank Adaptation

The paper presents TensorGuide, a new framework for parameter-efficient adaptation of large-scale neural networks via joint tensor-train (TT) parameterization. TensorGuide addresses key limitations of widely-used Low-Rank Adaptation (LoRA) and its TT-based variant (TT-LoRA), particularly regarding model expressivity and intrinsic inefficiency arising from independently parameterized low-rank matrices. By constructing both adaptation matrices within a unified TT architecture, the method facilitates structured, correlated parameter sharing and delivers substantial empirical gains in generalization, expressiveness, and scalability without additional trainable parameter overhead.

Limitations of Standard LoRA and TT-LoRA

Classic LoRA introduces task-specific low-rank matrices as adaptors to a frozen pre-trained model, with the principal advantage being a considerable reduction in fine-tuning parameters. However, by optimizing each low-rank matrix independently, LoRA is inherently limited in expressivity and conditioning—particularly in high-dimensional settings—because adaptation rank $r$ strongly constrains capacity and may induce poor optimization landscapes. TT-LoRA, which separately factorizes each adapter matrix into a TT format, offers only negligible extra parameter compression and does not introduce the inter-matrix correlation necessary for improved generalization or expressivity. Empirical and theoretical analyses (e.g., neural tangent kernel and eigenvalue characterizations) support this view, showing that these approaches tend toward ill-conditioning and diminished gradient representations, especially as model width increases.

The TensorGuide Framework

TensorGuide fundamentally differs in leveraging a joint TT parameterization to generate both LoRA matrices from a single TT network, which is stochastically driven by structured Gaussian noise. This design enforces beneficial correlation structures across adaptation weights, yielding enhanced model expressivity and improved conditioning—an effect that TT-LoRA cannot achieve due to independent decompositions. A key property of TensorGuide is its decoupling of hidden network width from parameter count: Hidden dimensions of the task-specific MLP head can scale arbitrarily, with only a minimal linear increase in TT parameters. Computationally, this permits efficient scaling and adaptation in resource-constrained settings.

Algorithmically, training in TensorGuide involves optimizing the TT cores, with all backbone model weights kept frozen. Adaptation is realized fully through TT-core updates, ensuring a compact parameter footprint while maintaining flexibility akin to full fine-tuning.

Theoretical Analysis

A rigorous neural tangent kernel (NTK) and eigenvalue-based analysis underpins TensorGuide’s reported advantages in expressivity, generalization, and optimization behavior. Let $M$ denote the width of the MLP hidden layer and $\epsilon_{\text{tt}}$ the TT approximation error. The approximation error for the full operator can be upper-bounded by

$$\epsilon_{\rm app} \le \frac{L_{\rm ce} C_{1}}{\sqrt{M}} + 2 C_{2}L_{\rm ce} L_{\sigma}\epsilon_{\rm tt}$$

implying that capacity scales with $M$—but parameter count increases only due to TT cores.

Further, NTK eigenvalue analysis demonstrates that the minimum eigenvalue $\lambda_{\text{min}}(\mathcal{T}_{\text{tg}})$ of TensorGuide’s kernel substantially exceeds that of LoRA, improving convergence dynamics and optimization stability without the adverse parameter scaling trade-offs.

Empirical Evaluation

Extensive experiments substantiate the theoretical claims across two distinct domains:

1. Quantum Dot Charge Classification:

Using simulated and noisy real-world single/double quantum dot images, TensorGuide significantly outperforms both LoRA and TT-LoRA, achieving higher accuracy (up to 99.3%) and much lower loss, while employing fewer parameters (as low as 4276 compared to 5192 for LoRA). The experimental protocol fixates on equal parameter budgets and highlights the effect of expanding MLP width; even as hidden width quadruples (1024→4096), parameter increase is marginal ($<100$ extra parameters), with performance consistently improving.

2. GPT-2 Fine-Tuning on WikiText-2:

For large-scale LLMing, TensorGuide replaces the last GPT-2 projection layer with a TT-parameterized MLP and fine-tunes only TT-related parameters. TensorGuide achieves lower perplexity than LoRA at similar (or even markedly lower) parameter budgets, with performance improving as TT rank increases.

Computational Analysis:

The appendix provides explicit pseudocode for TensorGuide training and details computational and parameter complexity. Compared to LoRA and TT-LoRA, TensorGuide reduces both forward cost and memory usage by eliminating redundant, separately-decomposed matrices and by exploiting TT’s ability to scale hidden dimensions with near-constant parameter count.

Method	Trainable Params	Forward Cost
LoRA	$\mathcal{O}(r(D + Q))$	$\mathcal{O}(r(D + Q))$
TT-LoRA	$\mathcal{O}(2K r_{\text{tt}}^2 H)$	$\mathcal{O}(2K r_{\text{tt}}^2 H)$
TensorGuide	$\mathcal{O}(K r_{\text{tt}}^2 H)$	$\mathcal{O}(K r_{\text{tt}}^2 H + r(D + Q))$

Implementation Details

TensorGuide can be implemented as follows:

• Parameterization: Replace LoRA adaptation heads by a TT-generating module with Gaussian latent inputs, with the TT network output split and reshaped into $\hat{W}_{1}$ and $\hat{W}_{2}$.
• Training: Freeze backbone weights and update only TT-core parameters via Adam or SGD. Latent Gaussian vectors are optionally sampled per batch to inject regularizing noise.
• TT Core Design: Careful selection of the number of cores, core shapes, and TT-ranks is important for balancing expressivity and parameter count.
• Integration: TensorGuide is drop-in compatible with any architecture supporting LoRA/TT-LoRA, including transformer models and ResNets.

Pseudocode for inner-loop training:

z = torch.randn(tt_in_dim) # Gaussian latent input lora_vec = TT(z, {G_k for k in range(K)}) W1, W2 = split_and_reshape(lora_vec) h = activation(x @ W1) y_hat = h @ W2 loss = cross_entropy(y_hat, y) loss.backward() optimizer.step()

Limitations

• Implementation Complexity: TT layers and joint decomposition require more intricate model plumbing and hyperparameter tuning than classic LoRA.
• Computational Overhead: Despite parameter savings, TT contraction incurs nontrivial computation, especially with deep TT core hierarchies.
• Scalability: The gains in expressivity and efficiency must be validated empirically for very large architectures beyond the scope of current experiments.
• Adoption Barrier: Enhanced flexibility and reduced parameter need must be weighed against increased implementation complexity and sensitivity to TT rank configurations.

Implications and Future Directions

TensorGuide represents a compelling approach for energy-efficient, scalable fine-tuning. It offers:

• Resource Savings: By dramatically reducing the number of adaptation parameters, it enables fine-tuning on modest hardware and supports deployment on edge devices.
• Expressivity-Parameter Decoupling: Large hidden dimensions in adapters become feasible without bandwidth/memory cost, facilitating richer adaptation.
• Integration with Other PEFT Methods: TensorGuide can be combined with dynamic rank allocation strategies (e.g., DoRA) for even greater flexibility.
• Potential for Wider Applicability: Beyond transformers and vision backbones, TT-based joint parameterization could benefit any context demanding scalable, structured low-rank adaptation.

The architecture sets a new paradigm for parameter-efficient adaptation, opening avenues for more advanced structured reparameterizations in PEFT, and invites further exploration of hybrid TT-based and adaptive low-rank routing strategies. Adoption in production systems requires careful evaluation of computational budgets and TT implementation efficiency, but the practical performance improvements demonstrated here strongly motivate continued research and application.