Hyper-shared LoRA (HoRA)

• HoRA is a hypernetwork-based variant of low-rank adaptation that couples updates across attention heads for enhanced parameter sharing.
• It replaces independent per-head LoRA adapters with a shared generator conditioned on head embeddings, achieving improved sample efficiency and reduced redundancy.
• Empirical results on vision and language tasks demonstrate HoRA’s effectiveness, especially in low-data regimes with minimal trainable parameter increases.

Hyper-shared Low-Rank Adaptation (HoRA) is a hypernetwork-based variant of Low-Rank Adaptation (LoRA) for multi-head self-attention (MHA) that replaces independent per-head low-rank updates with a shared generator conditioned on head embeddings. In the formulation introduced in “HoRA: Cross-Head Low-Rank Adaptation with Joint Hypernetworks,” low-rank matrices for query and value projections are generated jointly across attention heads, so that heads are coupled through shared meta-parameters rather than optimized as isolated adapters. The stated objective is to exploit cross-head information sharing, reduce redundancy, and improve sample efficiency while increasing the trainable-parameter budget only marginally (Diep et al., 5 Oct 2025).

1. Problem setting and design objective

HoRA is motivated by a specific limitation of LoRA inside MHA: when LoRA is applied per head, each head effectively receives its own independent adapter parameters. The paper argues that this is suboptimal because attention heads are not truly independent functional units and because independent per-head adaptation prevents statistical sharing, especially in low-data regimes (Diep et al., 5 Oct 2025).

The underlying MHA formulation is written as

$$\mathrm{MHA}(W_Q, W_K, W_V) = \mathrm{Concat}(\boldsymbol{h}_1,\cdots,\boldsymbol{h}_H)W_O,$$

with head-wise attention

$$\boldsymbol{h}_i = \mathrm{Attention}(W_{Q,i},W_{K,i},W_{V,i}) = \mathrm{softmax}\left(\frac{W_{Q,i}W_{K,i}^\top}{\sqrt{d_k}}\right)W_{V,i}.$$

The projection shapes are

$$W_{Q,i},W_{K,i}\in\mathbb{R}^{d\times d_k},\qquad W_{V,i}\in\mathbb{R}^{d\times d_v},\qquad W_O\in\mathbb{R}^{Hd_v\times d},$$

typically with $d_k=d_v=d/H$.

In the main experimental setup, HoRA is applied to query and value matrices at each layer. The key and output projections are not adapted in the main formulation. This places HoRA in a narrower but more targeted regime than generic PEFT schemes that attach low-rank modules to every linear map in an attention block (Diep et al., 5 Oct 2025).

2. Joint hypernetworks and generated low-rank factors

The LoRA baseline used in the paper starts from a frozen pretrained weight

$W_0\in\mathbb{R}^{m\times n}$

and a low-rank update

$$\Delta W = BA,\qquad B\in\mathbb{R}^{m\times r},\qquad A\in\mathbb{R}^{r\times n},\qquad r\ll \min(m,n),$$

giving

$\widehat{\boldsymbol{y}} = XW_0 + XBA.$

For MHA, the per-head LoRA form is

$$\tilde{\boldsymbol{h}}_i = \mathrm{Attention}\bigl( W_{Q,i}+B_{Q,i}A_{Q,i}, \,W_{K,i}, \,W_{V,i}+B_{V,i}A_{V,i} \bigr),$$

with

$$A_{Q,i}\in\mathbb{R}^{r\times d_k},\quad B_{Q,i}\in\mathbb{R}^{d\times r},\quad A_{V,i}\in\mathbb{R}^{r\times d_v},\quad B_{V,i}\in\mathbb{R}^{d\times r}.$$

HoRA replaces direct learning of those per-head factors with a shared hypernetwork parameterization. In the practical method section, the generated/shared factors are written as

$$A_Q=\sigma_1(W_{Q,A}),\qquad A_V=\sigma_1(W_{V,A}),$$

and the head-specific factors are generated by

$$\boldsymbol{h}_i = \mathrm{Attention}(W_{Q,i},W_{K,i},W_{V,i}) = \mathrm{softmax}\left(\frac{W_{Q,i}W_{K,i}^\top}{\sqrt{d_k}}\right)W_{V,i}.$$0

Here each head has a learnable embedding $$\boldsymbol{h}_i = \mathrm{Attention}(W_{Q,i},W_{K,i},W_{V,i}) = \mathrm{softmax}\left(\frac{W_{Q,i}W_{K,i}^\top}{\sqrt{d_k}}\right)W_{V,i}.$$1, with normalization

$$\boldsymbol{h}_i = \mathrm{Attention}(W_{Q,i},W_{K,i},W_{V,i}) = \mathrm{softmax}\left(\frac{W_{Q,i}W_{K,i}^\top}{\sqrt{d_k}}\right)W_{V,i}.$$2

The paper gives

$$\boldsymbol{h}_i = \mathrm{Attention}(W_{Q,i},W_{K,i},W_{V,i}) = \mathrm{softmax}\left(\frac{W_{Q,i}W_{K,i}^\top}{\sqrt{d_k}}\right)W_{V,i}.$$3

and analogously for the value branch. It also states that, for parameter efficiency, some $$\boldsymbol{h}_i = \mathrm{Attention}(W_{Q,i},W_{K,i},W_{V,i}) = \mathrm{softmax}\left(\frac{W_{Q,i}W_{K,i}^\top}{\sqrt{d_k}}\right)W_{V,i}.$$4 matrices are implemented as diagonal matrices (Diep et al., 5 Oct 2025).

Operationally, HoRA is “hyper-shared” because the low-rank matrices are not independently stored for each head. A shared generator maps head embeddings to head-specific adapters, so gradient information from all heads updates the same generator. The paper presents two expositions—a theory-side shared parameterization and a practical head-embedding construction—and the notation is somewhat inconsistent across them. The central architectural claim, however, is stable: shared hypernetworks couple low-rank adaptation across heads while retaining head-specific specialization (Diep et al., 5 Oct 2025).

3. Hierarchical mixture-of-experts view and sample-efficiency theory

A major part of HoRA’s contribution is theoretical. The paper reinterprets MHA as a hierarchical mixture of experts (HMoE) by vectorizing the input,

$$\boldsymbol{h}_i = \mathrm{Attention}(W_{Q,i},W_{K,i},W_{V,i}) = \mathrm{softmax}\left(\frac{W_{Q,i}W_{K,i}^\top}{\sqrt{d_k}}\right)W_{V,i}.$$5

and expressing each output row as a hierarchy over head index $$\boldsymbol{h}_i = \mathrm{Attention}(W_{Q,i},W_{K,i},W_{V,i}) = \mathrm{softmax}\left(\frac{W_{Q,i}W_{K,i}^\top}{\sqrt{d_k}}\right)W_{V,i}.$$6 and token/expert index $$\boldsymbol{h}_i = \mathrm{Attention}(W_{Q,i},W_{K,i},W_{V,i}) = \mathrm{softmax}\left(\frac{W_{Q,i}W_{K,i}^\top}{\sqrt{d_k}}\right)W_{V,i}.$$7. In this view, attention scores become gates and value projections become expert functions. LoRA then perturbs both the routing and expert components of that HMoE (Diep et al., 5 Oct 2025).

For the non-shared case, the paper studies a regression function $$\boldsymbol{h}_i = \mathrm{Attention}(W_{Q,i},W_{K,i},W_{V,i}) = \mathrm{softmax}\left(\frac{W_{Q,i}W_{K,i}^\top}{\sqrt{d_k}}\right)W_{V,i}.$$8 and proves a lower bound

$$\boldsymbol{h}_i = \mathrm{Attention}(W_{Q,i},W_{K,i},W_{V,i}) = \mathrm{softmax}\left(\frac{W_{Q,i}W_{K,i}^\top}{\sqrt{d_k}}\right)W_{V,i}.$$9

The authors interpret the non-shared parameterization as statistically inefficient because independent low-rank factorizations induce a weak-identifiability pathology. The argument is tied to a PDE-style degeneracy in the independently factorized low-rank parameters, represented in the paper by second derivatives such as

$$W_{Q,i},W_{K,i}\in\mathbb{R}^{d\times d_k},\qquad W_{V,i}\in\mathbb{R}^{d\times d_v},\qquad W_O\in\mathbb{R}^{Hd_v\times d},$$0

for

$$W_{Q,i},W_{K,i}\in\mathbb{R}^{d\times d_k},\qquad W_{V,i}\in\mathbb{R}^{d\times d_v},\qquad W_O\in\mathbb{R}^{Hd_v\times d},$$1

For the shared setting, HoRA uses a shared nonlinear parameterization and proves

$$W_{Q,i},W_{K,i}\in\mathbb{R}^{d\times d_k},\qquad W_{V,i}\in\mathbb{R}^{d\times d_v},\qquad W_O\in\mathbb{R}^{Hd_v\times d},$$2

The paper interprets this as a shift from an effectively exponential sample-complexity regime in the non-shared setting to polynomial rates such as $$W_{Q,i},W_{K,i}\in\mathbb{R}^{d\times d_k},\qquad W_{V,i}\in\mathbb{R}^{d\times d_v},\qquad W_O\in\mathbb{R}^{Hd_v\times d},$$3 or $$W_{Q,i},W_{K,i}\in\mathbb{R}^{d\times d_k},\qquad W_{V,i}\in\mathbb{R}^{d\times d_v},\qquad W_O\in\mathbb{R}^{Hd_v\times d},$$4 in the shared setting. The proof relies on assumptions including algebraic independence, uniform Lipschitz control, and strong identifiability of the shared parameterization (Diep et al., 5 Oct 2025).

This suggests that HoRA’s benefit is not merely parameter sharing in a storage sense. The paper’s claim is stronger: cross-head hyper-sharing yields a better-conditioned estimation problem.

4. Empirical performance across vision and language

HoRA is evaluated on ViT-B/16 for VTAB-1K and FGVC, and on LLaMA-7B and LLaMA-13B for commonsense reasoning. Across these settings, the trainable-parameter increase over LoRA is small: VTAB-1K moves from $$W_{Q,i},W_{K,i}\in\mathbb{R}^{d\times d_k},\qquad W_{V,i}\in\mathbb{R}^{d\times d_v},\qquad W_O\in\mathbb{R}^{Hd_v\times d},$$5 to $$W_{Q,i},W_{K,i}\in\mathbb{R}^{d\times d_k},\qquad W_{V,i}\in\mathbb{R}^{d\times d_v},\qquad W_O\in\mathbb{R}^{Hd_v\times d},$$6, FGVC from $$W_{Q,i},W_{K,i}\in\mathbb{R}^{d\times d_k},\qquad W_{V,i}\in\mathbb{R}^{d\times d_v},\qquad W_O\in\mathbb{R}^{Hd_v\times d},$$7 to $$W_{Q,i},W_{K,i}\in\mathbb{R}^{d\times d_k},\qquad W_{V,i}\in\mathbb{R}^{d\times d_v},\qquad W_O\in\mathbb{R}^{Hd_v\times d},$$8, LLaMA-7B from $$W_{Q,i},W_{K,i}\in\mathbb{R}^{d\times d_k},\qquad W_{V,i}\in\mathbb{R}^{d\times d_v},\qquad W_O\in\mathbb{R}^{Hd_v\times d},$$9 to $d_k=d_v=d/H$0, and LLaMA-13B from $d_k=d_v=d/H$1 to $d_k=d_v=d/H$2 (Diep et al., 5 Oct 2025).

Setting	LoRA	HoRA
VTAB-1K average	72.2	74.4
FGVC average	84.78	89.96
Commonsense, LLaMA-7B	74.09	76.64
Commonsense, LLaMA-13B	80.18	80.82

On VTAB-1K, HoRA improves Natural from $d_k=d_v=d/H$3 to $d_k=d_v=d/H$4, Specialized from $d_k=d_v=d/H$5 to $d_k=d_v=d/H$6, Structured from $d_k=d_v=d/H$7 to $d_k=d_v=d/H$8, and the overall average from $d_k=d_v=d/H$9 to $W_0\in\mathbb{R}^{m\times n}$0. On FGVC, the average improvement over LoRA is $W_0\in\mathbb{R}^{m\times n}$1. On LLaMA-7B commonsense reasoning, HoRA improves the average from $W_0\in\mathbb{R}^{m\times n}$2 to $W_0\in\mathbb{R}^{m\times n}$3; on LLaMA-13B, it improves from $W_0\in\mathbb{R}^{m\times n}$4 to $W_0\in\mathbb{R}^{m\times n}$5 (Diep et al., 5 Oct 2025).

The paper’s strongest empirical support for the sample-efficiency claim is a data-fraction experiment on LLaMA-7B. Using fractions $W_0\in\mathbb{R}^{m\times n}$6, HoRA outperforms LoRA at all fractions, and in the $W_0\in\mathbb{R}^{m\times n}$7 data regime the performance gap is reported to exceed $W_0\in\mathbb{R}^{m\times n}$8. An additional ablation on LLaMA-13B applies HoRA to Q/V and ordinary low-rank matrices to proj_up and proj_down; in that setting HoRA reaches $W_0\in\mathbb{R}^{m\times n}$9 against LoRA’s $$\Delta W = BA,\qquad B\in\mathbb{R}^{m\times r},\qquad A\in\mathbb{R}^{r\times n},\qquad r\ll \min(m,n),$$0 at the same $$\Delta W = BA,\qquad B\in\mathbb{R}^{m\times r},\qquad A\in\mathbb{R}^{r\times n},\qquad r\ll \min(m,n),$$1 trainable-parameter budget (Diep et al., 5 Oct 2025).

5. Position within shared low-rank adaptation research

HoRA belongs to a broader family of methods that try to overcome the rigidity of fully layer-local LoRA by reusing low-rank structure. A plausible characterization is that HoRA is a cross-head hyper-sharing method, whereas several related methods explore cross-layer direct sharing.

ShareLoRA directly ties LoRA factors across layers, most successfully by sharing the down-projection matrix $$\Delta W = BA,\qquad B\in\mathbb{R}^{m\times r},\qquad A\in\mathbb{R}^{r\times n},\qquad r\ll \min(m,n),$$2 while keeping the up-projection $$\Delta W = BA,\qquad B\in\mathbb{R}^{m\times r},\qquad A\in\mathbb{R}^{r\times n},\qquad r\ll \min(m,n),$$3 layer-specific. Its main successful form is static and not hypernetwork-generated (Song et al., 2024). ASLoRA likewise shares one low-rank factor globally across layers, then adaptively merges the other factor during training through similarity-based hard merges (Hu et al., 2024). Lily breaks the strict per-layer pairing of LoRA factors by keeping a local projector and routing over globally shared experts, yielding a dynamic cross-layer shared-factor scheme (Zhong et al., 2024). Queryable LoRA moves further toward conditional reuse by introducing a global memory bank of rank-space atoms and blockwise routing over retrieved low-rank operators (Vaidya et al., 8 May 2026).

Against this background, HoRA is distinguished by the axis along which sharing occurs. It does not primarily share across depth, nor does it retrieve a blockwise operator from a global atom bank. Instead, it couples attention heads within MHA through a shared generator. This suggests that “hyper-shared” low-rank adaptation is not a single architectural pattern but a design space spanning direct factor tying, adaptive merging, routed experts, retrieved atoms, and hypernetwork-generated adapters.

6. Training setup, practical details, and limitations

The vision experiments use 100 epochs, warmup steps 100, batch size 64, rank $$\Delta W = BA,\qquad B\in\mathbb{R}^{m\times r},\qquad A\in\mathbb{R}^{r\times n},\qquad r\ll \min(m,n),$$4, LoRA alpha $$\Delta W = BA,\qquad B\in\mathbb{R}^{m\times r},\qquad A\in\mathbb{R}^{r\times n},\qquad r\ll \min(m,n),$$5, AdamW, and a cosine scheduler, with learning-rate and weight-decay grids. The language experiments use 3 epochs, warmup steps 100, batch size 32, learning rate $$\Delta W = BA,\qquad B\in\mathbb{R}^{m\times r},\qquad A\in\mathbb{R}^{r\times n},\qquad r\ll \min(m,n),$$6, dropout $$\Delta W = BA,\qquad B\in\mathbb{R}^{m\times r},\qquad A\in\mathbb{R}^{r\times n},\qquad r\ll \min(m,n),$$7, AdamW, a linear scheduler, rank $$\Delta W = BA,\qquad B\in\mathbb{R}^{m\times r},\qquad A\in\mathbb{R}^{r\times n},\qquad r\ll \min(m,n),$$8, and alpha $$\Delta W = BA,\qquad B\in\mathbb{R}^{m\times r},\qquad A\in\mathbb{R}^{r\times n},\qquad r\ll \min(m,n),$$9. The reported implementation uses 1 A100 GPU. For the hypernetwork producing the low-rank matrix $\widehat{\boldsymbol{y}} = XW_0 + XBA.$0, the vision setting uses input dimension 64 and hidden dimension 16, while the language setting uses input dimension 64 and hidden dimension 40 (Diep et al., 5 Oct 2025).

The initialization scheme is LoRA-like but asymmetric. The paper states that $\widehat{\boldsymbol{y}} = XW_0 + XBA.$1 and $\widehat{\boldsymbol{y}} = XW_0 + XBA.$2 are initialized with Kaiming uniform, $\widehat{\boldsymbol{y}} = XW_0 + XBA.$3 and $\widehat{\boldsymbol{y}} = XW_0 + XBA.$4 with zero initialization, and $\widehat{\boldsymbol{y}} = XW_0 + XBA.$5 with Kaiming uniform. It also states that $\widehat{\boldsymbol{y}} = XW_0 + XBA.$6 are sigmoid in experiments, while the appendix implementation details for the hypernetwork used in low-rank matrix $\widehat{\boldsymbol{y}} = XW_0 + XBA.$7 mention leaky ReLU. This is one of several notation and implementation inconsistencies in the paper.

The paper explicitly notes that too much sharing can hurt expressiveness. It also notes that evaluation is limited to Transformer architectures and that more work is needed on adaptive sharing mechanisms that balance specialization and reuse. At the same time, the paper does not provide extensive controlled ablations on number of heads, rank sweeps, degree of sharing, alternate hypernetwork topologies, or head-embedding design. These omissions matter because HoRA’s central claim is structural rather than purely empirical: its performance depends on how much adapter capacity should be shared across heads and how much should remain head-specific (Diep et al., 5 Oct 2025).