Weight Sharing Attention (WSA) Explained

• Weight Sharing Attention (WSA) is a family of techniques that reuse attention maps, logits, or projection parameters across layers, heads, or sources to minimize redundancy in deep learning models.
• WSA includes approaches like cross-layer attention reuse, head-wise parameter sharing, and context-conditioned fusion, each offering trade-offs between speed and performance.
• Empirical studies report that WSA methods can achieve up to 1.8× speed-up and 66.7% parameter reduction while maintaining competitive accuracy on tasks such as machine translation and LLM inference.

Weight Sharing Attention (WSA) denotes a family of mechanisms that reuse attention-related computations or attention-related parameters instead of instantiating a fully independent attention module at every layer, head, or source. In the literature represented here, this includes cross-layer reuse of attention weights or logits in Transformers and LLMs (Xiao et al., 2019, Liao et al., 2024, Mu et al., 2024), sharing of attention projection parameters through head-wise tying or dictionary-based decompositions (Cao et al., 2024, Zhussip et al., 6 Aug 2025), and a context-conditioned fusion module in reinforcement learning in which a shared scorer weights multiple frozen pretrained embeddings (Piccoli et al., 9 Jul 2025). The acronym is not fully standardized: some papers use closely related names such as “Shared Attention” or “Matrix Atom Sharing in Attention,” while others use the same acronym for different modules such as “Weight Self-Attention Module” or “Weight-Sharing Aggregation” (Han et al., 27 Mar 2025, Wang et al., 2023).

1. Terminology and conceptual scope

In current usage, WSA is not a single canonical operator. The most consistent technical core is the idea that some object ordinarily recomputed or stored independently in attention is instead shared under an explicit rule. What varies is the shared object: attention maps, pre-softmax logits, projection matrices, head parameters, or source-scoring networks.

Form	Shared object	Representative papers
Cross-layer attention reuse	Softmax attention weights, pre-softmax logits, or attended context across layers	San (Xiao et al., 2019), SA (Liao et al., 2024), LiSA (Mu et al., 2024)
Parameter-sharing attention	$Q/K/V/O$ projections reconstructed or tied from shared components	Head-wise Shareable Attention (Cao et al., 2024), MASA (Zhussip et al., 6 Aug 2025)
Shared-source fusion	One shared MLP scores multiple pretrained embeddings	RL WSA (Piccoli et al., 9 Jul 2025)
Acronym collisions	Different modules that are not WSA in the compression sense	BiAG WSA (Han et al., 27 Mar 2025), WSAFlowNet (Wang et al., 2023)

This terminological spread matters because superficially similar phrases refer to different intervention points. Some methods share runtime attention maps while keeping layer-specific values; some share parameters but not activations; and some use “WSA” for a module outside Transformer compression altogether. A recurrent misconception is therefore to treat all WSA papers as variants of hard parameter tying. The literature does not support that simplification.

2. Cross-layer reuse of attention maps and logits

A foundational line of work shares attention vertically across depth. In standard scaled dot-product attention, the weight matrix and attended context are

$$S=\textrm{Softmax}\!\left(\frac{Q \cdot K^T}{\sqrt{d_k}}\right), \qquad A=S \cdot V.$$

“Sharing Attention Weights for Fast Transformer” introduced the shared attention network, or San, for autoregressive machine translation decoding. In decoder self-attention, layer $m$ computes $S^m=s(Q^m,K^m)$, and upper layers in the same sharing block reuse that matrix: $S^{m+i} = s(Q^m,K^m), \quad i \in [1,\pi-1].$ For encoder-decoder attention, the reuse is even stronger: $A^{m+i}=A^m, \qquad A^m=S^m \cdot V.$ The paper reports an average of $1.3\times$ speed-up with almost no decrease in BLEU on ten WMT and NIST OpenMT tasks, and $1.8\times$ speed-up when combined with AAN (Xiao et al., 2019).

“Beyond KV Caching: Shared Attention for Efficient LLMs” proposes a closely related inference-time mechanism for pretrained decoder-only LLMs. A contiguous layer span $S$ is selected; the first layer in the span computes

$$A_{l_0}=\mathrm{softmax}\!\left(\frac{Q_{l_0}K_{l_0}^T}{\sqrt{d_k}}\right),$$

and later layers set $$S=\textrm{Softmax}\!\left(\frac{Q \cdot K^T}{\sqrt{d_k}}\right), \qquad A=S \cdot V.$$0 and compute

$$S=\textrm{Softmax}\!\left(\frac{Q \cdot K^T}{\sqrt{d_k}}\right), \qquad A=S \cdot V.$$1

The reused object is explicitly the softmax-normalized attention matrix, not the projection parameters. The paper attributes the viability of this strategy to “isotropic tendencies of attention distributions” in middle-to-late layers of pretrained LLMs and shows that late-layer spans such as $$S=\textrm{Softmax}\!\left(\frac{Q \cdot K^T}{\sqrt{d_k}}\right), \qquad A=S \cdot V.$$2 are much safer than earlier spans such as $$S=\textrm{Softmax}\!\left(\frac{Q \cdot K^T}{\sqrt{d_k}}\right), \qquad A=S \cdot V.$$3 (Liao et al., 2024).

LiSA occupies an intermediate position between exact reuse and full recomputation. Rather than tying parameters, it reuses the adjacent previous layer’s pre-softmax attention score tensor $$S=\textrm{Softmax}\!\left(\frac{Q \cdot K^T}{\sqrt{d_k}}\right), \qquad A=S \cdot V.$$4, then adds two learnable corrections: a head-alignment module and a low-rank difference term

$$S=\textrm{Softmax}\!\left(\frac{Q \cdot K^T}{\sqrt{d_k}}\right), \qquad A=S \cdot V.$$5

The final score is formed by integrating the shared score matrix and the correction term. The motivation is empirical: adjacent layers often have very similar attention distributions, but direct head-indexed reuse is ineffective because heads are not semantically aligned across layers, and shallow layers are sensitive to perturbations. LiSA therefore belongs to WSA as approximate cross-layer attention-map sharing with learned alignment and low-rank residual correction, rather than as strict weight tying (Mu et al., 2024).

3. Sharing attention projection parameters

A second major WSA branch shares the parameters that generate attention rather than the runtime attention map itself. “Head-wise Shareable Attention for LLMs” argues that attention heads are a more natural sharing unit than full layers. For head $$S=\textrm{Softmax}\!\left(\frac{Q \cdot K^T}{\sqrt{d_k}}\right), \qquad A=S \cdot V.$$6,

$$S=\textrm{Softmax}\!\left(\frac{Q \cdot K^T}{\sqrt{d_k}}\right), \qquad A=S \cdot V.$$7

and the paper chooses shareable head pairs using cosine similarity of concatenated query and key weights: $$S=\textrm{Softmax}\!\left(\frac{Q \cdot K^T}{\sqrt{d_k}}\right), \qquad A=S \cdot V.$$8 After selecting pairs, it ties $$S=\textrm{Softmax}\!\left(\frac{Q \cdot K^T}{\sqrt{d_k}}\right), \qquad A=S \cdot V.$$9 together between matched heads. DirectShare performs this reuse without retraining. PostShare first post-trains with the regularizer

$m$0

and then shares. This is a fine-grained WSA scheme: selection is driven by $m$1, but the final sharing applies jointly to $m$2 (Cao et al., 2024).

MASA, introduced in “Share Your Attention: Transformer Weight Sharing via Matrix-based Dictionary Learning,” generalizes cross-layer parameter sharing from exact tying to dictionary-based soft sharing. For a given projection type and layer $m$3,

$m$4

where $m$5 are shared atoms and $m$6 are layer-specific coefficients. The stacked form is

$m$7

with $m$8, $m$9, and $S^m=s(Q^m,K^m)$0. For one projection, the parameter count drops from $S^m=s(Q^m,K^m)$1 to $S^m=s(Q^m,K^m)$2, giving

$S^m=s(Q^m,K^m)$3

Choosing $S^m=s(Q^m,K^m)$4 yields the paper’s central $S^m=s(Q^m,K^m)$5 attention-parameter reduction. The default and empirically best configuration uses separate dictionaries for $S^m=s(Q^m,K^m)$6, $S^m=s(Q^m,K^m)$7, $S^m=s(Q^m,K^m)$8, and $S^m=s(Q^m,K^m)$9, with MASA-QKV and MASA-QKVO as the main variants (Zhussip et al., 6 Aug 2025).

These two approaches mark an important internal distinction within WSA. Head-wise sharing uses discrete pairwise tying among existing projections. MASA instead learns a continuous interpolation between full independence and full sharing. This suggests that, inside parameter-sharing WSA, the main design choice is not merely whether to share but how much layer- or head-specific flexibility to retain.

4. Model-source fusion and adjacent meanings of WSA

In reinforcement learning, WSA is the explicit name of a different mechanism: a context-conditioned fusion module for combining multiple frozen pretrained representations into one state representation. The pipeline computes a context

$S^{m+i} = s(Q^m,K^m), \quad i \in [1,\pi-1].$0

projects each pretrained model output to a common space,

$S^{m+i} = s(Q^m,K^m), \quad i \in [1,\pi-1].$1

scores each embedding with the same shared MLP,

$S^{m+i} = s(Q^m,K^m), \quad i \in [1,\pi-1].$2

normalizes the weights by the $S^{m+i} = s(Q^m,K^m), \quad i \in [1,\pi-1].$3 norm,

$S^{m+i} = s(Q^m,K^m), \quad i \in [1,\pi-1].$4

and forms the fused state representation

$S^{m+i} = s(Q^m,K^m), \quad i \in [1,\pi-1].$5

Here the “weight sharing” is across model sources because the same scorer $S^{m+i} = s(Q^m,K^m), \quad i \in [1,\pi-1].$6 is reused for every embedding; there are no Transformer heads and no cross-layer attention maps (Piccoli et al., 9 Jul 2025).

The acronym also collides with non-equivalent modules. In BiAG for few-shot class-incremental learning, WSA means “Weight Self-Attention Module,” which refines a weight-space query rather than sharing attention across layers or heads (Han et al., 27 Mar 2025). In WSAFlowNet, WSA means “Weight-Sharing Aggregation,” a point-cloud scene-flow upsampling module that reuses the same learned neighbor weights for coordinates, scene flow, and features (Wang et al., 2023). A related but indirect literature studies cross-channel weight sharing around axial-attention backbones via quaternion and PHM layers; there the attention operator itself remains unchanged (Shahadat et al., 2021).

A broader theoretical perspective appears in work on $S^{m+i} = s(Q^m,K^m), \quad i \in [1,\pi-1].$7-equivariant message passing, which formalizes weight sharing as parameter tying over equivalence classes of point-pairs under a group action: $S^{m+i} = s(Q^m,K^m), \quad i \in [1,\pi-1].$8 The message function is then conditioned on a complete invariant identifier $S^{m+i} = s(Q^m,K^m), \quad i \in [1,\pi-1].$9 of the equivalence class $A^{m+i}=A^m, \qquad A^m=S^m \cdot V.$0. Although this paper is not framed as WSA, it provides a rigorous template for symmetry-respecting shared pairwise interactions (Bekkers et al., 2023).

5. Training regimes and implementation patterns

WSA methods differ sharply in how they are introduced into a model. San is primarily an inference-time acceleration mechanism with a learned layer-sharing policy. The policy is based on attention-weight similarity, measured through $A^{m+i}=A^m, \qquad A^m=S^m \cdot V.$1JS divergence, and optimized by alternating between training the machine translation model and recomputing the sharing block structure $A^{m+i}=A^m, \qquad A^m=S^m \cdot V.$2 (Xiao et al., 2019).

LiSA is an uptraining method for pretrained LLMs. The base model is frozen; only the new parameters are trained: tiny head-alignment FFNs and low-rank $A^{m+i}=A^m, \qquad A^m=S^m \cdot V.$3. Supervision combines a feature-level distillation objective on the attention score tensor with the standard language-model loss,

$A^{m+i}=A^m, \qquad A^m=S^m \cdot V.$4

with default $A^{m+i}=A^m, \qquad A^m=S^m \cdot V.$5. The method targets 17, 21, or 27 of the 32 layers in the tested models, corresponding to $A^{m+i}=A^m, \qquad A^m=S^m \cdot V.$6, $A^{m+i}=A^m, \qquad A^m=S^m \cdot V.$7, and $A^{m+i}=A^m, \qquad A^m=S^m \cdot V.$8 of total layers (Mu et al., 2024).

MASA is explicitly designed as a drop-in replacement trained from scratch. It uses no distillation, no auxiliary reconstruction loss, and no architectural widening. For coefficient stability, each block has a trainable embedding and a 3-layer MLP predicts the coefficient vector $A^{m+i}=A^m, \qquad A^m=S^m \cdot V.$9 during training; after training, the embeddings and MLP are discarded and only the learned coefficient matrix $1.3\times$0 is retained. The paper also proposes a separate no-finetuning adaptation path for pretrained LLMs: layer grouping by KL divergence of pseudo-output distributions, shared-basis extraction by Matrix PCA, and optional local low-rank residual refinement after Cholesky whitening (Zhussip et al., 6 Aug 2025).

Head-wise parameter-sharing WSA supports both zero-retraining and light post-training regimes. DirectShare directly rebinds matched head parameters in minutes. PostShare first performs language-model post-training on English Wikipedia with the similarity regularizer $1.3\times$1, then ties the selected head pairs. This split between immediate structural sharing and regularized adaptation is one of the clearest implementation dichotomies in the WSA literature (Cao et al., 2024).

The RL variant again differs. There the pretrained models remain frozen during RL, while adapters, the shared weighting network, and the downstream policy/value head are trained jointly under a standard RL objective. This establishes a fourth regime: shared-attention-style scoring over fixed upstream representations rather than over a pretrained Transformer’s own attention stack (Piccoli et al., 9 Jul 2025).

6. Empirical regularities, trade-offs, and open questions

Several empirical regularities recur across otherwise different WSA formulations. First, redundancy is highly non-uniform across depth. LiSA reports that most JS divergence scores between adjacent attention distributions remain below $1.3\times$2, but the first layer is consistently less similar to the rest; Shared Attention similarly separates layers 0 and 1 and the final output layer from a large highly similar middle region (Mu et al., 2024, Liao et al., 2024). Second, direct hard reuse is often too brittle. In LiSA, naive direct sharing without alignment or difference correction is catastrophic, whereas the full method preserves high response quality; in MASA, soft sharing through atom mixtures outperforms rigid sequential or repeated sharing at comparable budgets (Mu et al., 2024, Zhussip et al., 6 Aug 2025). Third, the shareable object matters. MASA’s ablations indicate that $1.3\times$3 are more compressible than $1.3\times$4, and head-wise sharing shows that $1.3\times$5 is the best signal for selecting shareable heads (Zhussip et al., 6 Aug 2025, Cao et al., 2024).

The quality-efficiency trade-off is substantial but method-dependent. San yields an average of $1.3\times$6 speed-up with almost no decrease in BLEU, and $1.3\times$7 when combined with AAN (Xiao et al., 2019). LiSA achieves a $1.3\times$8 compression of $1.3\times$9 and $1.8\times$0 in its generic formulation, preserves $1.8\times$1 average performance for LLaMA3-8B in the 17-layer setting, and reaches throughput improvements up to $1.8\times$2 for LLaMA3-8B and $1.8\times$3 for LLaMA2-7B (Mu et al., 2024). MASA reduces attention parameters by $1.8\times$4 while remaining on par or competitive across language and vision settings, including experiments from 100M to 700M parameters and ViT extensions (Zhussip et al., 6 Aug 2025). Head-wise DirectShare at $1.8\times$5 sharing retains $1.8\times$6 average performance on five reasoning benchmarks for Llama 2-7B, but reading comprehension and some knowledge tasks are markedly more fragile (Cao et al., 2024).

The RL formulation exhibits a different failure mode. WSA matches or exceeds end-to-end learning on Pong and Ms.Pacman and remains close in mean HNS/Capped-HNS, but default Breakout performance is poor because random-agent pretraining data does not cover later-game states well enough. When mixed random and expert trajectories are used for pretraining, Breakout WSA improves from $1.8\times$7 to $1.8\times$8, approaching the end-to-end baseline $1.8\times$9 (Piccoli et al., 9 Jul 2025). This indicates that shared weighting cannot compensate for systematic deficiencies in frozen upstream encoders.

A final misconception is that WSA is a settled design space. The literature itself leaves several open questions. LiSA explicitly raises the possibility of a global cross-layer shared attention basis, permutation-aware head alignment, and stability under distribution shift (Mu et al., 2024). MASA points to finer grouping and richer coefficient parameterizations beyond scalar atom weights (Zhussip et al., 6 Aug 2025). Head-wise sharing leaves unresolved how to scale beyond pairwise head tying and how to combine memory reduction with stronger inference-speed gains (Cao et al., 2024). This suggests that WSA is less a single method than an active design area organized around one recurring premise: attention modules often contain exploitable redundancy, but the useful form of sharing depends on whether the target is runtime attention maps, projection parameters, or cross-source fusion.