Shared Cross-Attention in Neural Architectures

• Shared cross-attention is a mechanism that reuses a common attention interface across branches, modalities, or tasks to enforce consistent representation and computational efficiency.
• This architecture reduces parameter redundancy while enabling multi-modal fusion and semantic guidance, as seen in applications like translation, vision, and speech.
• Empirical studies demonstrate performance gains, such as improved BLEU scores, mIoU increases, and significant speedup factors in diverse neural network systems.

Shared cross-attention denotes a family of architectures in which cross-attention, or cross-attention-like conditioning, is reused, coupled, or amortized across branches, modalities, layers, views, tasks, or devices rather than treated as a single isolated encoder-decoder operation. The literature does not use the term uniformly. In some papers it refers to shared information flow without weight tying; in others it denotes a single module reused for all samples, a shared query over heterogeneous controls, a shared per-input encoder plus cross-attentive fusion block, or a shared routing index over a common memory. The common thread is that one representation conditions another through a common attention interface while preserving consistency, common structure, or computational efficiency (Li et al., 2019, Xiao et al., 2022, Tardy et al., 4 Feb 2026, Chen et al., 21 Nov 2025, Sun et al., 4 Jun 2026).

1. Terminological scope

A first sense of shared cross-attention is shared information flow. In "Crossed Co-Attention Networks," the two encoder branches are structurally symmetric and mutually condition one another, but the paper explicitly distinguishes this from shared parameters: the branches are separate but crossed, and only the BPE embeddings are globally shared (Li et al., 2019).

A second sense is module reuse. In "Semantic Cross Attention for Few-shot Learning," the semantic cross-attention module $Cross_\phi$ is a single module with fixed parameters reused for all classes, images, and episodes, while the shared backbone $\mathcal{E}_\theta$ serves both the main classification task and the semantic auxiliary task (Xiao et al., 2022).

A third sense is shared control or shared state. PostCam introduces "query-shared cross-attention," in which the same query tensor derived from the noisy latent attends jointly to camera-pose and rendered-video features (Chen et al., 21 Nov 2025). In joint multi-receiver neural decoding, a shared per-receiver encoder and a shared token-wise cross-attention block are reused across all time-frequency resource elements and all access points (Tardy et al., 4 Feb 2026). In long-context LLM inference, CLSA shares not only the KV cache across cross-decoder layers but also the routing index, so token-level top-$k$ selection is computed once and reused across layers (Sun et al., 4 Jun 2026).

2. Architectural forms

One major pattern is the symmetric multi-branch architecture. The THM/CCN paradigm uses two symmetric encoder modules and one decoder module connected through crossed co-attention. In the MT experiments, both encoder branches receive the same source sentence but with branch-specific perturbations, and the decoder contains two encoder-decoder attention branches whose outputs are concatenated, linearly projected, and sent to an FFN (Li et al., 2019).

A second pattern is auxiliary semantic guidance. In few-shot learning, the main visual feature map $e_{\text{main}}$ and the semantic-space feature $e_{\text{aux}}$ are fused by a semantic cross-attention module. Queries are generated by the semantic auxiliary task, keys and values come from the visual backbone, and the resulting embedding is used by metric-based classifiers such as ProtoNet and ProxyNet. The backbone is shared between the main and auxiliary tasks, and semantic word vectors are not required at meta-test time (Xiao et al., 2022).

A third pattern is shared-query multimodal control. PostCam places a query-shared cross-attention module inside each diffusion transformer block. The noisy latent $z_t$ supplies the query, while camera poses and rendered proxy video provide distinct keys and values; these are concatenated and attended in a single operation. The model is trained in two stages: first pose-only conditioning for coarse control, then joint pose-plus-render conditioning for motion refinement and visual fidelity (Chen et al., 21 Nov 2025).

A fourth pattern is shared-object or shared-goal queries across views. XVTP3D uses a set of 3D queries shared across BEV and FPV to predict multi-goals that are cross-view consistent. Each 3D query is projected into each view, attends to view-specific state features, and is fused back into a unified 3D representation; a random mask and coarse-to-fine cross-attention further regularize the multi-view fusion (Song et al., 2023).

A fifth pattern is distributed exact cross-attention. LV-XAttn keeps large visual key-value blocks local on each GPU and exchanges smaller query blocks across GPUs, exploiting the common MLLM asymmetry in which the query sequence is much shorter than the visual key-value sequence (Chang et al., 4 Feb 2025).

3. Formal mechanisms

The canonical transformer form remains

$$\mathrm{Attn}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V.$$

Shared cross-attention differs primarily in how $Q$, $K$, and $V$ are sourced, coupled, or reused.

In crossed co-attention for MT, each encoder branch keeps its own $\mathcal{E}_\theta$0 but receives $\mathcal{E}_\theta$1 from the opposite branch:

$\mathcal{E}_\theta$2

$\mathcal{E}_\theta$3

The decoder then uses a common decoder-side query and attends separately to left and right encoder memories before concatenation and projection (Li et al., 2019).

In few-shot semantic CAM, the same dot-product template is retained but with semantic queries and visual keys/values:

$\mathcal{E}_\theta$4

Here $\mathcal{E}_\theta$5 is derived from the semantic auxiliary branch, whereas $\mathcal{E}_\theta$6 and $\mathcal{E}_\theta$7 are projected from visual features (Xiao et al., 2022).

In multi-receiver neural decoding, cross-attention is token-wise over aligned receivers. For each resource element, AP 1 provides the query, all APs provide keys and values, and fusion is performed by residual addition and layer normalization:

$\mathcal{E}_\theta$8

$\mathcal{E}_\theta$9

The parameters of this fusion are shared across all time-frequency positions and all receiver sets (Tardy et al., 4 Feb 2026).

In PostCam, the same query $k$0 is shared across two control modalities, while keys and values are concatenated along the token dimension:

$k$1

$k$2

The single attention map over concatenated keys is the defining feature of the query-shared design (Chen et al., 21 Nov 2025).

Not all work using the term is transformer-style. In cross-attention-guided image fusion, attention maps $k$3 and $k$4 are produced from pairs of images through a conv-based module rather than QKV dot products (Shen et al., 2021). In CANet for semantic segmentation, cross-attention is factorized into a spatial map from the shallow branch and a channel map from the deep branch; only the FC layer applied to avg- and max-pooled context vectors is explicitly shared (Liu et al., 2019).

4. Applications and empirical behavior

In machine translation, shared cross-attention has been used to improve sequence transduction quality. On WMT 2014 EN-DE, THM/CCN-Base improves Transformer-Base from 27.21 to 27.95 BLEU and THM/CCN-Big improves Transformer-Big from 28.13 to 28.64; on WMT 2016 EN-FI, THM/CCN-Base improves 16.12 to 16.59 and THM/CCN-Big improves 16.21 to 16.38 (Li et al., 2019). In speech, a multi-task KWS system with a cross-attention decoder over a shared phonetic encoder yields a 12% relative reduction in false reject ratios compared to conventional split branches and a BLSTM decoder (Higuchi et al., 2021).

In recognition and dense prediction, shared cross-attention is often used as a plug-in fusion or summarization mechanism. Semantic CAM improves metric-based few-shot learning on CUB, mini-ImageNet, and tiered-ImageNet, and its ablations show that CAM outperforms naive concatenation and squeeze-and-excitation (Xiao et al., 2022). CANet’s branch-conditioned FCA raises Cityscapes validation mIoU from 67.9% for the two-branch baseline to 73.4% with the final SA$k$5CA design (Liu et al., 2019). In fine-grained visual categorization and Re-ID, DCAL surpasses DeiT-Tiny and ViT-Base by 2.8% and 2.4% mAP on MSMT17 (Zhu et al., 2022).

In fusion and generation, shared cross-attention is used to enforce consistency under heterogeneous views or controls. CADNIF employs cross spatial attention and cross self-attention to improve unsupervised image fusion across multi-modal, multi-exposure, and multi-focus settings (Shen et al., 2021). PostCam reports that its query-shared cross-attention improves camera control precision and view consistency by over 20% while producing the highest video generation quality (Chen et al., 21 Nov 2025). XVTP3D uses shared 3D queries to obtain cross-view-consistent trajectory hypotheses; its ablations report about 10% FDE improvement in BEV and 12–29% in FPV from the shared-query design (Song et al., 2023).

In systems-oriented settings, sharing is often introduced for efficiency. The cross-attention transformer for joint multi-receiver uplink decoding uses 0.15M parameters and 0.24 GFLOPs, and for $k$6 with two pilot columns it is about 1 dB better than the “Perfect-CSI (per-AP) + SNR fusion” baseline at medium/high $k$7 (Tardy et al., 4 Feb 2026). LV-XAttn reports up to $k$8 end-to-end speedup for long-visual-input MLLMs by keeping key-value blocks local and exchanging smaller query blocks (Chang et al., 4 Feb 2025). CLSA reports up to $k$9 decoding speedup and $e_{\text{main}}$0 overall throughput improvement at 128K context by sharing routing indices across layers (Sun et al., 4 Jun 2026).

5. Parameter sharing, information sharing, and common misconceptions

Shared cross-attention is not synonymous with weight tying. CCN is a clear counterexample: the model shares BPE embeddings globally, but the left and right branches are separate, parameter counts roughly double relative to a single Transformer, and the shared aspect is the crossed information flow rather than shared $e_{\text{main}}$1 (Li et al., 2019).

Conversely, some designs do implement direct parameter sharing. The semantic CAM in few-shot learning is a single module reused for all classes, images, and episodes (Xiao et al., 2022). The joint multi-receiver decoder shares both the per-AP encoder and the token-wise cross-attention parameters across all time-frequency positions and receiver links (Tardy et al., 4 Feb 2026). CLSA shares a routing index across all cross-decoder layers on top of a shared memory (Sun et al., 4 Jun 2026). PostCam shares the query stream across camera and render controls, even though the control-side keys and values remain distinct (Chen et al., 21 Nov 2025).

Another recurrent confusion is between cross-attention and cross-conditioned gating. CADNIF’s cross-attention module is conv-based and produces attention maps for each source image conditioned on both images (Shen et al., 2021). CANet’s FCA is also cross-branch but factorized into branch-conditioned spatial and channel attention, not a joint QKV attention over two token sets (Liu et al., 2019). CrossATNet for sketch-based retrieval uses sketch-driven channel-wise gating of image features rather than token-token cross-attention in the transformer sense (Chaudhuri et al., 2021).

A nearby but distinct notion is shared attention without cross-attention. RWSA-MambaUNet shares time- and frequency-attention modules across corresponding U-Net resolutions, but this is cross-layer parameter sharing of self-attention rather than canonical cross-sequence cross-attention (Kühne et al., 2 Oct 2025). The distinction matters because “shared attention” can refer either to shared parameters or to cross-modal conditioning, and the two are only partially overlapping.

6. Interpretability, theory, and outlook

Work on speech-to-text interpretability shows that cross-attention is informative but incomplete as an explanatory signal. When aggregated across heads and layers, cross-attention scores align moderately to strongly with saliency-based explanations, yet they capture only about 50% of input relevance and, at best, only 52–75% of encoder-output saliency (Papi et al., 22 Sep 2025). A plausible implication is that shared cross-attention becomes more faithful when aggregated, but even then it should not be equated with a full causal explanation.

In diffusion models, head-level analyses suggest that shared cross-attention patterns can be concept-stable. Head Relevance Vectors over Stable Diffusion cross-attention heads align with human visual concepts, remain stable across timesteps, and support interventions such as concept strengthening and concept adjusting; in the polysemy experiments, the misinterpretation rate drops from 63.0% to 15.9% (Park et al., 2024). This suggests that some shared cross-attention structure is not merely architectural but also mechanistically reusable.

A theoretical line of work strengthens the case for depth. In a latent multi-modal factor model, single-layer linear self-attention fails to recover the Bayes-optimal predictor uniformly over the task distribution, whereas a multi-layer linearized cross-attention mechanism is provably Bayes optimal under gradient flow in the large-depth, large-context regime (Barnfield et al., 4 Feb 2026). The result does not establish that practical transformer cross-attention always achieves such optimality, but it does isolate a regime in which shared cross-attention is not just expressive but statistically optimal.

Another emerging direction is to turn shared cross-attention into explicit structure. CM-GLasso uses shared prototype queries to distill image and rendered-text patches into semantic nodes and then learns a common precision matrix together with class-specific deviations, yielding a shared graph over cross-modal semantic units (Wang et al., 5 Apr 2026). This suggests a broader trajectory in which shared cross-attention is coupled to explicit topological priors, shared memories, or shared routing policies rather than being left as an unconstrained compatibility operator.

Shared cross-attention is therefore best understood as an architectural principle rather than a single layer type: reuse the conditioning mechanism, the query objects, the memory, the routing, or the latent node basis so that multiple views of the same underlying structure are forced to interact through a common attention interface. Across machine translation, few-shot learning, trajectory prediction, speech, generative video, multimodal diffusion, and long-context systems, that principle has been used to trade parameter duplication for consistency, to trade local fusion for shared semantics, or to trade repeated routing for amortized computation.