---
title: Hyper-shared LoRA (HoRA)
url: https://www.emergentmind.com/topics/hyper-shared-low-rank-adaptation-hora
type: topic
---

# Hyper-shared LoRA (HoRA)

Hyper-shared Low-Rank Adaptation (HoRA) is a hypernetwork-based variant of Low-Rank Adaptation (LoRA) for multi-head self-attention (MHA) that replaces independent per-head low-rank updates with a shared generator conditioned on head embeddings. In the formulation introduced in “HoRA: Cross-Head Low-Rank Adaptation with Joint Hypernetworks,” low-rank matrices for query and value projections are generated jointly across attention heads, so that heads are coupled through shared meta-parameters rather than optimized as isolated adapters. The stated objective is to exploit cross-head information sharing, reduce redundancy, and improve sample efficiency while increasing the trainable-parameter budget only marginally [2510.04295].

## 1. Problem setting and design objective

HoRA is motivated by a specific limitation of LoRA inside MHA: when LoRA is applied per head, each head effectively receives its own independent adapter parameters. The paper argues that this is suboptimal because attention heads are not truly independent functional units and because independent per-head adaptation prevents statistical sharing, especially in low-data regimes [2510.04295].

The underlying MHA formulation is written as
$$
\mathrm{MHA}(W_Q, W_K, W_V) = \mathrm{Concat}(\boldsymbol{h}_1,\cdots,\boldsymbol{h}_H)W_O,
$$
with head-wise attention
$$
\boldsymbol{h}_i = \mathrm{Attention}(W_{Q,i},W_{K,i},W_{V,i}) = \mathrm{softmax}\left(\frac{W_{Q,i}W_{K,i}^\top}{\sqrt{d_k}}\right)W_{V,i}.
$$
The projection shapes are
$$
W_{Q,i},W_{K,i}\in\mathbb{R}^{d\times d_k},\qquad W_{V,i}\in\mathbb{R}^{d\times d_v},\qquad W_O\in\mathbb{R}^{Hd_v\times d},
$$
typically with $d_k=d_v=d/H$.

In the main experimental setup, HoRA is applied to query and value matrices at each layer. The key and output projections are not adapted in the main formulation. This places HoRA in a narrower but more targeted regime than generic PEFT schemes that attach low-rank modules to every linear map in an attention block [2510.04295].

## 2. Joint hypernetworks and generated low-rank factors

The LoRA baseline used in the paper starts from a frozen pretrained weight
$$
W_0\in\mathbb{R}^{m\times n}
$$
and a low-rank update
$$
\Delta W = BA,\qquad B\in\mathbb{R}^{m\times r},\qquad A\in\mathbb{R}^{r\times n},\qquad r\ll \min(m,n),
$$
giving
$$
\widehat{\boldsymbol{y}} = XW_0 + XBA.
$$
For MHA, the per-head LoRA form is
$$
\tilde{\boldsymbol{h}}_i = \mathrm{Attention}\bigl( W_{Q,i}+B_{Q,i}A_{Q,i}, \,W_{K,i}, \,W_{V,i}+B_{V,i}A_{V,i} \bigr),
$$
with
$$
A_{Q,i}\in\mathbb{R}^{r\times d_k},\quad B_{Q,i}\in\mathbb{R}^{d\times r},\quad A_{V,i}\in\mathbb{R}^{r\times d_v},\quad B_{V,i}\in\mathbb{R}^{d\times r}.
$$

HoRA replaces direct learning of those per-head factors with a shared hypernetwork parameterization. In the practical method section, the generated/shared factors are written as
$$
A_Q=\sigma_1(W_{Q,A}),\qquad A_V=\sigma_1(W_{V,A}),
$$
and the head-specific factors are generated by
$$
B_{Q,i}=W_{Q,B,2}\,\sigma_2(W_{B,1}\,\mathrm{LN}(e_i)),\qquad
B_{V,i}=W_{V,B,2}\,\sigma_2(W_{B,1}\,\mathrm{LN}(e_i)).
$$
Here each head has a learnable embedding $e_i\in\mathbb{R}^{d_e}$, with normalization
$$
e_i = \mathrm{LN}(e_i') = \frac{e_i'-\mathbb{E}(e_i')}{\mathrm{Std}(e_i')}.
$$
The paper gives
$$
W_{B,1} \in \mathbb{R}^{d_{\text{hid}}\times d_e},\qquad
W_{Q,B,2} \in \mathbb{R}^{(r\times d)\times d_{\text{hid}}},
$$
and analogously for the value branch. It also states that, for parameter efficiency, some $W$ matrices are implemented as diagonal matrices [2510.04295].

Operationally, HoRA is “hyper-shared” because the low-rank matrices are not independently stored for each head. A shared generator maps head embeddings to head-specific adapters, so gradient information from all heads updates the same generator. The paper presents two expositions—a theory-side shared parameterization and a practical head-embedding construction—and the notation is somewhat inconsistent across them. The central architectural claim, however, is stable: shared hypernetworks couple low-rank adaptation across heads while retaining head-specific specialization [2510.04295].

## 3. Hierarchical mixture-of-experts view and sample-efficiency theory

A major part of HoRA’s contribution is theoretical. The paper reinterprets MHA as a hierarchical mixture of experts (HMoE) by vectorizing the input,
$$
\tilde{\boldsymbol{x}}=\mathrm{Vec}(X)\in\mathbb{R}^{Nd},
$$
and expressing each output row as a hierarchy over head index $h$ and token/expert index $j$. In this view, attention scores become gates and value projections become expert functions. LoRA then perturbs both the routing and expert components of that HMoE [2510.04295].

For the non-shared case, the paper studies a regression function $g_{G_*}$ and proves a lower bound
$$
\sup_{G\in \mathcal{G}_{H,L'}(\widehat{\Theta}) \setminus \mathcal{G}_{H,L-1}(\widehat{\Theta})} \mathbb{E}_{g_G}[\mathcal{D}_{1,r}(\widehat{G}_n,G)] \gtrsim n^{-1/2}.
$$
The authors interpret the non-shared parameterization as statistically inefficient because independent low-rank factorizations induce a weak-identifiability pathology. The argument is tied to a PDE-style degeneracy in the independently factorized low-rank parameters, represented in the paper by second derivatives such as
$$
\frac{\partial^2F}{\partial A_V^{(u_1v_1)}\partial A_V^{(u_2v_2)}} =
\frac{\partial^2F}{\partial B_V^{(u_1v_1)}\partial B_V^{(u_2v_2)}} =0,
$$
for
$$
F(X,A,B):=\exp(X^\top(W_Q+B_QA_Q)W_K)(W_V+B_VA_V).
$$

For the shared setting, HoRA uses a shared nonlinear parameterization and proves
$$
\mathcal{D}_2(\widetilde{G}_n,\widetilde{G}_*) = \mathcal{O}_P\left(\sqrt{\frac{\log(n)}{n}}\right).
$$
The paper interprets this as a shift from an effectively exponential sample-complexity regime in the non-shared setting to polynomial rates such as $\mathcal{O}(\epsilon^{-2})$ or $\mathcal{O}(\epsilon^{-4})$ in the shared setting. The proof relies on assumptions including algebraic independence, uniform Lipschitz control, and strong identifiability of the shared parameterization [2510.04295].

This suggests that HoRA’s benefit is not merely parameter sharing in a storage sense. The paper’s claim is stronger: cross-head hyper-sharing yields a better-conditioned estimation problem.

## 4. Empirical performance across vision and language

HoRA is evaluated on ViT-B/16 for VTAB-1K and FGVC, and on LLaMA-7B and LLaMA-13B for commonsense reasoning. Across these settings, the trainable-parameter increase over LoRA is small: VTAB-1K moves from $0.39\%$ to $0.47\%$, FGVC from $0.55\%$ to $0.64\%$, LLaMA-7B from $0.25\%$ to $0.28\%$, and LLaMA-13B from $0.20\%$ to $0.21\%$ [2510.04295].

| Setting | LoRA | HoRA |
|---|---:|---:|
| VTAB-1K average | 72.2 | 74.4 |
| FGVC average | 84.78 | 89.96 |
| Commonsense, LLaMA-7B | 74.09 | 76.64 |
| Commonsense, LLaMA-13B | 80.18 | 80.82 |

On VTAB-1K, HoRA improves Natural from $79.4$ to $81.67$, Specialized from $84.55$ to $86.68$, Structured from $59.78$ to $61.96$, and the overall average from $72.2$ to $74.4$. On FGVC, the average improvement over LoRA is $89.96 - 84.78 = 5.18$. On LLaMA-7B commonsense reasoning, HoRA improves the average from $74.09$ to $76.64$; on LLaMA-13B, it improves from $80.18$ to $80.82$ [2510.04295].

The paper’s strongest empirical support for the sample-efficiency claim is a data-fraction experiment on LLaMA-7B. Using fractions $f\in\{1\%,10\%,30\%,50\%,100\%\}$, HoRA outperforms LoRA at all fractions, and in the $1\%$ data regime the performance gap is reported to exceed $20\%$. An additional ablation on LLaMA-13B applies HoRA to Q/V and ordinary low-rank matrices to `proj_up` and `proj_down`; in that setting HoRA reaches $81.87$ against LoRA’s $80.4$ at the same $0.57\%$ trainable-parameter budget [2510.04295].

## 5. Position within shared low-rank adaptation research

HoRA belongs to a broader family of methods that try to overcome the rigidity of fully layer-local LoRA by reusing low-rank structure. A plausible characterization is that HoRA is a **cross-head hyper-sharing** method, whereas several related methods explore **cross-layer direct sharing**.

ShareLoRA directly ties LoRA factors across layers, most successfully by sharing the down-projection matrix $A$ while keeping the up-projection $B_i$ layer-specific. Its main successful form is static and not hypernetwork-generated [2406.10785]. ASLoRA likewise shares one low-rank factor globally across layers, then adaptively merges the other factor during training through similarity-based hard merges [2412.10135]. Lily breaks the strict per-layer pairing of LoRA factors by keeping a local projector and routing over globally shared experts, yielding a dynamic cross-layer shared-factor scheme [2407.09946]. Queryable LoRA moves further toward conditional reuse by introducing a global memory bank of rank-space atoms and blockwise routing over retrieved low-rank operators [2605.08423].

Against this background, HoRA is distinguished by the axis along which sharing occurs. It does not primarily share across depth, nor does it retrieve a blockwise operator from a global atom bank. Instead, it couples **attention heads within MHA** through a shared generator. This suggests that “hyper-shared” low-rank adaptation is not a single architectural pattern but a design space spanning direct factor tying, adaptive merging, routed experts, retrieved atoms, and hypernetwork-generated adapters.

## 6. Training setup, practical details, and limitations

The vision experiments use 100 epochs, warmup steps 100, batch size 64, rank $r=8$, LoRA alpha $=8$, AdamW, and a cosine scheduler, with learning-rate and weight-decay grids. The language experiments use 3 epochs, warmup steps 100, batch size 32, learning rate $2\times 10^{-4}$, dropout $0.05$, AdamW, a linear scheduler, rank $r=32$, and alpha $=64$. The reported implementation uses 1 A100 GPU. For the hypernetwork producing the low-rank matrix $B$, the vision setting uses input dimension 64 and hidden dimension 16, while the language setting uses input dimension 64 and hidden dimension 40 [2510.04295].

The initialization scheme is LoRA-like but asymmetric. The paper states that $W_{Q,A,1}$ and $W_{Q,B,2}$ are initialized with Kaiming uniform, $W_{V,A,1}$ and $W_{V,B,2}$ with zero initialization, and $W_{B,1}$ with Kaiming uniform. It also states that $\sigma_1,\sigma_2$ are sigmoid in experiments, while the appendix implementation details for the hypernetwork used in low-rank matrix $B$ mention leaky ReLU. This is one of several notation and implementation inconsistencies in the paper.

The paper explicitly notes that too much sharing can hurt expressiveness. It also notes that evaluation is limited to Transformer architectures and that more work is needed on adaptive sharing mechanisms that balance specialization and reuse. At the same time, the paper does not provide extensive controlled ablations on number of heads, rank sweeps, degree of sharing, alternate hypernetwork topologies, or head-embedding design. These omissions matter because HoRA’s central claim is structural rather than purely empirical: its performance depends on how much adapter capacity should be shared across heads and how much should remain head-specific [2510.04295].

Source: https://www.emergentmind.com/topics/hyper-shared-low-rank-adaptation-hora