Advanced Attention Sharing in Transformers

• Advanced Attention Sharing (AAS) is a family of methods that reuse or fuse attention outputs across layers in models like Transformers to reduce redundancy and computational cost.
• It incorporates diverse techniques such as cross-layer copying, fusion of attention maps, and control mechanisms to enhance saliency modeling, machine translation, image editing, and collaborative analytics.
• Empirical results demonstrate improved accuracy, speed, and efficiency, although careful design is needed to mitigate challenges like over-sharing and positional misalignment.

Advanced Attention Sharing (AAS) is not a single canonical mechanism but a family of attention-sharing formulations appearing across Transformer optimization, visual saliency modeling, diffusion-based generation and editing, detection, super-resolution, and collaborative immersive analytics. In these formulations, the shared object may be an attention map, an attention-weight matrix, values or weights learned by teacher branches, socially derived “push” maps, or accumulated multi-user attention fields. This suggests a common research program: when attention exhibits redundancy, alignment, or social structure, explicit sharing can replace repeated computation or expose information that isolated attention estimates miss (Gorji et al., 2016, Xiao et al., 2019, Chen et al., 2021, Zhao et al., 2023, Mu et al., 2024, Park et al., 27 Jan 2025, Srinivasan et al., 11 May 2025, Zhang et al., 21 Sep 2025, Chen et al., 16 Dec 2025).

1. Terminological scope and historical development

In the supplied literature, the expression denotes several related but non-identical ideas. In visual attention prediction, it refers to shared-attention modeling that augments bottom-up salience with socially meaningful “Attentional Push” cues such as head or body pose direction, center bias, scene-cut or “bounce” events, and abrupt head-movement onsets. In machine translation, vision transformers, super-resolution, DETR variants, and LLM inference, it denotes cross-layer reuse of attention weights, maps, or downstream products to reduce redundant computation. In diffusion transformers and diffusion-based image editing, it denotes selective cross-image or cross-branch sharing, often constrained by positional-encoding considerations. In collaborative immersive analytics, it denotes capture, aggregation, decay, and revisualization of collective user attention over a shared spatial scene (Gorji et al., 2016, Xiao et al., 2019, Chen et al., 2021, Zhao et al., 2023, Mu et al., 2024, Park et al., 27 Jan 2025, Srinivasan et al., 11 May 2025, Zhang et al., 21 Sep 2025, Chen et al., 16 Dec 2025).

The historical progression in the supplied material begins with the 2016 saliency framework, continues with 2019 decoder acceleration for machine translation, and expands through 2021–2025 into ViT architecture search, detection training, LLM efficiency, super-resolution, DiT style alignment, non-rigid image editing, and WebXR collaboration. This chronology suggests that the term has broadened from a socially grounded attention model to a wider class of mechanisms for sharing attention-related structure across computational units or users.

Setting	Shared object	Immediate purpose
Saliency / fixation prediction	push maps $M(s_i;x)$	augment salience
MT / ViT / LLM / SR	attention weights or maps across layers	reduce redundancy
DETR training	teacher values or maps	improve student attention learning
DiT generation / editing	cross-image or cross-branch attention components	style consistency or faithful editing
WebXR collaboration	accumulated user attention fields	group awareness

2. Core mathematical patterns

In the Transformer-centered variants, AAS begins from standard multi-head self-attention. PSViT writes, for layer $\ell$,

$$Q_\ell = X_\ell W^Q,\quad K_\ell = X_\ell W^K,\quad V_\ell = X_\ell W^V,$$

$$A_\ell=\mathrm{Softmax}\!\bigl(\tfrac{Q_\ell K_\ell^T}{\sqrt{d_h}}\bigr),\qquad O_\ell=A_\ell V_\ell.$$

Its sharing indicator $s_\ell\in\{0,1\}$ yields

$$A_\ell=(1-s_\ell)\,\mathrm{Softmax}\!\bigl(\tfrac{Q_\ell K_\ell^T}{\sqrt{d_h}}\bigr)+s_\ell A_{\ell-1},$$

with a “pure copy” case $A_\ell=A_{\ell-1}$ when $s_\ell=1$. The machine-translation formulation shares within contiguous decoder blocks: for a block starting at layer $m$ of size $\pi$,

$\ell$0

LiSA generalizes this logic by taking a cached $\ell$1, aligning heads with a tiny FFN, and adding a low-rank residual

$\ell$2

before applying softmax. KS-DETR, by contrast, shares teacher-produced values $\ell$3 or maps $\ell$4 with a student path through shared downstream MHA and FFN modules rather than by copying an adjacent-layer map verbatim (Chen et al., 2021, Xiao et al., 2019, Mu et al., 2024, Zhao et al., 2023).

Outside pure cross-layer reuse, AAS often takes a fusion form. The saliency model defines a push map

$\ell$5

and an augmented attention map

$\ell$6

SynPS frames image editing as a balance between positional and semantic attention by using

$\ell$7

where $\ell$8 is derived from an editing measurement $\ell$9. In HeedVision, the shared object is neither a softmax matrix nor a latent attention map, but a voxelized field $$Q_\ell = X_\ell W^Q,\quad K_\ell = X_\ell W^K,\quad V_\ell = X_\ell W^V,$$0 over a shared 3D scene, updated by kernel-weighted accumulation and exponential decay: $$Q_\ell = X_\ell W^Q,\quad K_\ell = X_\ell W^K,\quad V_\ell = X_\ell W^V,$$1 These formulations show that “sharing” may involve copying, constraining, blending, or spatially aggregating attention-related quantities rather than a single universal operator (Gorji et al., 2016, Chen et al., 16 Dec 2025, Srinivasan et al., 11 May 2025).

3. Architectural realizations

PSViT divides the Transformer encoder into three stages separated by token-pooling layers. Within each stage there are $$Q_\ell = X_\ell W^Q,\quad K_\ell = X_\ell W^K,\quad V_\ell = X_\ell W^V,$$2 cells, and in the Tiny/small super-net $$Q_\ell = X_\ell W^Q,\quad K_\ell = X_\ell W^K,\quad V_\ell = X_\ell W^V,$$3. Each cell may be a standard multi-head self-attention layer, a shared-attention layer that copies the attention map from the immediately previous layer, or identity. Token pooling and attention sharing are treated as discrete hyper-parameters, and PSViT casts the joint search as a one-shot Neural Architecture Search problem with uniform sampling in a supernet followed by an evolutionary population-based search over subnet configurations under a FLOPS budget (Chen et al., 2021).

The machine-translation formulation partitions the decoder’s $$Q_\ell = X_\ell W^Q,\quad K_\ell = X_\ell W^K,\quad V_\ell = X_\ell W^V,$$4 layers into contiguous blocks, learns or tunes a threshold $$Q_\ell = X_\ell W^Q,\quad K_\ell = X_\ell W^K,\quad V_\ell = X_\ell W^V,$$5 using Jensen–Shannon divergence, and reuses the same attention weight matrix inside each block. Gradient contributions from all layers sharing the same $$Q_\ell = X_\ell W^Q,\quad K_\ell = X_\ell W^K,\quad V_\ell = X_\ell W^V,$$6 are summed before propagating into the block-start $$Q_\ell = X_\ell W^Q,\quad K_\ell = X_\ell W^K,\quad V_\ell = X_\ell W^V,$$7 and $$Q_\ell = X_\ell W^Q,\quad K_\ell = X_\ell W^K,\quad V_\ell = X_\ell W^V,$$8 projections. LiSA adapts cross-layer sharing to well-trained LLMs by selecting a share set, preserving sensitive shallow layers, training only the head-alignment FFN and low-rank $$Q_\ell = X_\ell W^Q,\quad K_\ell = X_\ell W^K,\quad V_\ell = X_\ell W^V,$$9 projections, and freezing all original parameters (Xiao et al., 2019, Mu et al., 2024).

KS-DETR replaces the last encoder layer’s standard attention with three parallel attentions: Attn$$A_\ell=\mathrm{Softmax}\!\bigl(\tfrac{Q_\ell K_\ell^T}{\sqrt{d_h}}\bigr),\qquad O_\ell=A_\ell V_\ell.$$0 as the plain student, Attn$$A_\ell=\mathrm{Softmax}\!\bigl(\tfrac{Q_\ell K_\ell^T}{\sqrt{d_h}}\bigr),\qquad O_\ell=A_\ell V_\ell.$$1 as a teacher-values branch, and Attn$$A_\ell=\mathrm{Softmax}\!\bigl(\tfrac{Q_\ell K_\ell^T}{\sqrt{d_h}}\bigr),\qquad O_\ell=A_\ell V_\ell.$$2 as a teacher-maps branch. The teacher feature $$A_\ell=\mathrm{Softmax}\!\bigl(\tfrac{Q_\ell K_\ell^T}{\sqrt{d_h}}\bigr),\qquad O_\ell=A_\ell V_\ell.$$3 is produced by a sparse MLP that transforms foreground tokens using a GT Fg-Bg Mask. All three branches share the downstream MHA combine/projection and FFN, and the two teacher attentions are removed during inference, yielding zero extra cost or parameters at test time (Zhao et al., 2023).

ASID places attention sharing inside cascaded Information-Distillation Blocks. The first IDB computes three spatial attention matrices $$A_\ell=\mathrm{Softmax}\!\bigl(\tfrac{Q_\ell K_\ell^T}{\sqrt{d_h}}\bigr),\qquad O_\ell=A_\ell V_\ell.$$4, and subsequent IDBs reuse them rather than recomputing $$A_\ell=\mathrm{Softmax}\!\bigl(\tfrac{Q_\ell K_\ell^T}{\sqrt{d_h}}\bigr),\qquad O_\ell=A_\ell V_\ell.$$5 and softmax. This sharing is coupled to a channel-split distillation ratio $$A_\ell=\mathrm{Softmax}\!\bigl(\tfrac{Q_\ell K_\ell^T}{\sqrt{d_h}}\bigr),\qquad O_\ell=A_\ell V_\ell.$$6, so only refined subsets of channels enter the expensive attention path (Park et al., 27 Jan 2025).

AlignedGen and SynPS adapt attention sharing to DiT-based generation and editing. AlignedGen first introduces Shifted Position Embedding,

$$A_\ell=\mathrm{Softmax}\!\bigl(\tfrac{Q_\ell K_\ell^T}{\sqrt{d_h}}\bigr),\qquad O_\ell=A_\ell V_\ell.$$7

or equivalently $$A_\ell=\mathrm{Softmax}\!\bigl(\tfrac{Q_\ell K_\ell^T}{\sqrt{d_h}}\bigr),\qquad O_\ell=A_\ell V_\ell.$$8, to allocate a non-overlapping positional index set to each image. On this basis it applies Selective Attention Sharing, AdaIN-based alignment of target image queries and keys to the reference style, key scaling by $$A_\ell=\mathrm{Softmax}\!\bigl(\tfrac{Q_\ell K_\ell^T}{\sqrt{d_h}}\bigr),\qquad O_\ell=A_\ell V_\ell.$$9, and layer-selective replacement of MM-Attention in a contiguous set of layers, empirically layers 19–57 in Flux.1-dev. SynPS, by contrast, computes an editing measurement

$s_\ell\in\{0,1\}$0

uses it to derive $s_\ell\in\{0,1\}$1 by a piecewise linear rule with thresholds $s_\ell\in\{0,1\}$2, and then modulates RoPE to balance positional preservation against semantic change during denoising (Zhang et al., 21 Sep 2025, Chen et al., 16 Dec 2025).

HeedVision realizes AAS as a distributed WebXR system built on Three.js, React-Three-Fiber, Spatialstrates/Dashspace, Webstrates and Varv. Each headset runs an Attention Model Service maintaining voxel grids, a Capture Module using ray casting and a GPU-accelerated BVH, an Accumulation & Decay Engine, a WebRTC-based Network Synchronizer, and a rendering pipeline that reveals attention overlays when explicitly triggered. Here sharing is infrastructural and collaborative rather than layerwise (Srinivasan et al., 11 May 2025).

4. Empirical behavior across domains

Reported results span accuracy, throughput, parameter efficiency, fidelity, and collaboration quality. In image recognition, PSViT reports a DeiT-Tiny baseline at $s_\ell\in\{0,1\}$3 GFlops and $s_\ell\in\{0,1\}$4 Top-1, $s_\ell\in\{0,1\}$5 for attention sharing alone with FLOPS kept at $s_\ell\in\{0,1\}$6 G, $s_\ell\in\{0,1\}$7 or $s_\ell\in\{0,1\}$8 for token pooling alone, and $s_\ell\in\{0,1\}$9 or $$A_\ell=(1-s_\ell)\,\mathrm{Softmax}\!\bigl(\tfrac{Q_\ell K_\ell^T}{\sqrt{d_h}}\bigr)+s_\ell A_{\ell-1},$$0 when pooling and sharing are chosen by AutoML; the paper summary also states that the proposed scheme can achieve up to $$A_\ell=(1-s_\ell)\,\mathrm{Softmax}\!\bigl(\tfrac{Q_\ell K_\ell^T}{\sqrt{d_h}}\bigr)+s_\ell A_{\ell-1},$$1 accuracy improvement in ImageNet classification compared with the DeiT. In machine translation, AAS yields an average of $$A_\ell=(1-s_\ell)\,\mathrm{Softmax}\!\bigl(\tfrac{Q_\ell K_\ell^T}{\sqrt{d_h}}\bigr)+s_\ell A_{\ell-1},$$2 speed-up with almost no decrease in BLEU, and $$A_\ell=(1-s_\ell)\,\mathrm{Softmax}\!\bigl(\tfrac{Q_\ell K_\ell^T}{\sqrt{d_h}}\bigr)+s_\ell A_{\ell-1},$$3 speed-up when combined with AAN. In LLMs, LiSA reduces redundant attention calculations within $$A_\ell=(1-s_\ell)\,\mathrm{Softmax}\!\bigl(\tfrac{Q_\ell K_\ell^T}{\sqrt{d_h}}\bigr)+s_\ell A_{\ell-1},$$4 of total layers while achieving a $$A_\ell=(1-s_\ell)\,\mathrm{Softmax}\!\bigl(\tfrac{Q_\ell K_\ell^T}{\sqrt{d_h}}\bigr)+s_\ell A_{\ell-1},$$5 compression of $$A_\ell=(1-s_\ell)\,\mathrm{Softmax}\!\bigl(\tfrac{Q_\ell K_\ell^T}{\sqrt{d_h}}\bigr)+s_\ell A_{\ell-1},$$6 and $$A_\ell=(1-s_\ell)\,\mathrm{Softmax}\!\bigl(\tfrac{Q_\ell K_\ell^T}{\sqrt{d_h}}\bigr)+s_\ell A_{\ell-1},$$7, with maximum throughput improvements of $$A_\ell=(1-s_\ell)\,\mathrm{Softmax}\!\bigl(\tfrac{Q_\ell K_\ell^T}{\sqrt{d_h}}\bigr)+s_\ell A_{\ell-1},$$8 for LLaMA3-8B and $$A_\ell=(1-s_\ell)\,\mathrm{Softmax}\!\bigl(\tfrac{Q_\ell K_\ell^T}{\sqrt{d_h}}\bigr)+s_\ell A_{\ell-1},$$9 for LLaMA2-7B. In diffusion-based generation and editing, the reported gains are framed in terms of style consistency, text alignment, and MLLM-based faithfulness scores. In collaborative immersive analytics, the reported gains are framed in terms of targets found, coverage, coverage efficiency, redundancy, and subjective coordination benefits (Chen et al., 2021, Xiao et al., 2019, Mu et al., 2024, Zhang et al., 21 Sep 2025, Chen et al., 16 Dec 2025, Srinivasan et al., 11 May 2025, Park et al., 27 Jan 2025, Zhao et al., 2023).

System	Setting	Reported result
PSViT	ImageNet, Tiny, $A_\ell=A_{\ell-1}$0 GFlops	$A_\ell=A_{\ell-1}$1 with sharing only; up to $A_\ell=A_{\ell-1}$2 with 2D pooling + sharing
Fast Transformer AAS	10 MT tasks	BLEU $A_\ell=A_{\ell-1}$3; speed $A_\ell=A_{\ell-1}$4 K tok/s
LiSA	LLaMA3-8B / LLaMA2-7B	$A_\ell=A_{\ell-1}$5 and $A_\ell=A_{\ell-1}$6 throughput; $A_\ell=A_{\ell-1}$7 layers shared
KS-DETR	MS COCO val2017	$A_\ell=A_{\ell-1}$8 to $A_\ell=A_{\ell-1}$9 AP over baselines
ASID	$s_\ell=1$0 SR, Set14, 720p	$s_\ell=1$1 K / $s_\ell=1$2 G vs $s_\ell=1$3 K / $s_\ell=1$4 G
AlignedGen	100 style-prompt pairs	$s_\ell=1$5
SynPS	Gemini-2.5-Pro score	PIE-Bench $s_\ell=1$6; Curated $s_\ell=1$7
HeedVision	Collaborative search	Scatter $s_\ell=1$8 vs $s_\ell=1$9; Terrain $m$0 vs $m$1

Qualitative analyses in the supplied works are similarly domain-specific. PSViT reports sharper class-discriminative Grad-CAM heatmaps and reduced redundancy between successive layers. The saliency model reports that classical salience fails when actors’ gaze or body pose point to low-salience objects, whereas the augmented model shifts emphasis to the shared target location. AlignedGen reports that vanilla Flux yields inconsistent styles, StyleAligned on Flux collapses into near-copies of the reference image, and AAS plus ShiftPE preserves brush-stroke, lighting, and color palette while obeying distinct textual concepts. HeedVision reports emergent coordination through voxel footprints and a reduction in duplicate observations as measured by normalized entropy (Chen et al., 2021, Gorji et al., 2016, Zhang et al., 21 Sep 2025, Srinivasan et al., 11 May 2025).

5. Failure modes, constraints, and recurrent misconceptions

A recurrent misconception is that more sharing is necessarily better. The supplied ablations contradict this. In PSViT, no sharing gives $m$2, sharing every two layers gives $m$3, and sharing every three layers gives $m$4, indicating diminishing returns and an accuracy drop under excessive reuse. The same source states that if features change dramatically, such as right after a token-pooling layer, sharing may harm expressiveness, and PSViT therefore never shares across a token-pool boundary (Chen et al., 2021).

Another misconception is that attention weights can be copied across models or images without positional treatment. AlignedGen explicitly reports that naive attention sharing fails in DiT because RoPE causes positional collision: tokens at the same $m$5 across reference and target carry identical RoPE, which spuriously boosts cross-image dot products, causing content leakage and collapse of text control. SynPS formulates a related failure as attention collapse in non-rigid image editing, where positional embeddings or semantic features dominate visual content retrieval, leading to over-editing or under-editing. Its fixed-weight ablations describe the two extremes: $m$6 improves structure but under-follows the prompt, whereas $m$7 follows text but loses layout (Zhang et al., 21 Sep 2025, Chen et al., 16 Dec 2025).

In LLMs, direct sharing without head rearrangement is also ineffective. LiSA states that direct sharing of the weight matrix without carefully rearranging the attention heads proves to be ineffective and that shallow layers are vulnerable to small deviations in attention weights. In saliency prediction, conflicting gaze directions or multiple actors can make push maps compete and degrade $m$8 in aggregate. In immersive analytics, visual clutter remains a concern, head direction is only an approximate proxy for gaze, and high-resolution voxel grids strain CPU and GPU resources (Mu et al., 2024, Gorji et al., 2016, Srinivasan et al., 11 May 2025).

These limitations indicate that successful AAS usually depends on an explicit control mechanism: stage boundaries in PSViT, block thresholds in MT, head-alignment and low-rank residuals in LiSA, positional-index separation in AlignedGen, adaptive $m$9 in SynPS, and explicit overlay triggering or adjustable thresholds in HeedVision.

6. Relation to adjacent methods and prospective directions

AAS is repeatedly presented as complementary to other efficiency or supervision mechanisms rather than a replacement for them. In machine translation it builds on top of an implementation that already adopted an attention cache for fast inference and combines effectively with AAN. LiSA explicitly contrasts cross-layer sharing with intra-layer methods such as KV-cache compression, head grouping, pruning, and quantization, describing AAS as orthogonal to those strategies. KS-DETR presents knowledge sharing as an extension of knowledge distillation in which improved weights and values of teacher attentions are directly shared instead of mimicked (Xiao et al., 2019, Mu et al., 2024, Zhao et al., 2023).

The future directions listed in the supplied sources are correspondingly heterogeneous. PSViT proposes soft sharing with a mixing coefficient $\pi$0, cross-stage or non-adjacent sharing, and dynamic sharing policies driven by a small gating network that inspects $\pi$1. The saliency framework proposes learning $\pi$2 and $\pi$3 from eye-tracking data, incorporating verbal or gestural cues, and extending to 3D and virtual-reality environments. HeedVision points to adaptive or multi-resolution grids, sparse octrees, full GPU compute shaders, larger teams, asynchronous workflows, and true eye-tracking hardware. LiSA proposes automated, data-dependent layer selection, extension beyond adjacent layers, and combination with quantization or head sparsification. These directions suggest that AAS has become a reusable design pattern for exploiting structured redundancy or shared context, with the principal open question being not whether to share attention, but what object should be shared, where, and under what control law (Chen et al., 2021, Gorji et al., 2016, Srinivasan et al., 11 May 2025, Mu et al., 2024).