Asymmetric Cross-Modal Attention Mechanisms

• The paper demonstrates that asymmetric cross-modal attention updates only the designated modality (e.g., text) by attending over another (e.g., video), preserving crucial modality-specific information.
• It employs distinct projection matrices, normalization, and temporal alignment strategies to achieve directional specialization, leading to improved performance and reduced model complexity.
• The mechanism is versatile, having been successfully integrated into domains such as video moment retrieval, two-stream classification, and medical imaging registration with notable efficiency gains.

Searching arXiv for the cited papers to ground the article in current metadata. An asymmetric cross-modal cross-attention mechanism is a cross-modal fusion operator in which the query stream and the key–value stream are drawn from different modalities, but the interaction is not treated as bidirectionally identical. Across the literature, asymmetry appears in several concrete forms: only one modality may be updated while the other bypasses the block; the two directions may use distinct projection matrices, temperatures, or temporal alignment rules; or each direction may be implemented by a different operator altogether. In its most common form, the mechanism instantiates cross-attention as $Z=\mathrm{softmax}(QK^T/\sqrt{d_k})V$, with $Q$ from one modality and $K,V$ from another, followed by a residual update on only the designated target branch (Panta et al., 2023, Chi et al., 2019). The resulting design has been used for video moment retrieval, two-stream video classification, medical multimodal diagnosis, cross-modal volume registration, cross-modal crowd counting, text-based person search, audio–video generation, and state-based text-to-image generation (Panta et al., 2023, Ming et al., 9 Jul 2025, Song et al., 2021, Zhang et al., 2022, Ji et al., 2022, Zhang et al., 5 Nov 2025, Xiao et al., 19 Apr 2025).

1. Mathematical form and core operator

In the standard formulation, asymmetric cross-modal cross-attention projects one modality into queries and another into keys and values, then computes a weighted aggregation over the source modality. In the video–text asymmetric co-attention block, if $V\in\mathbb{R}^{N\times d_v}$ denotes video features and $T\in\mathbb{R}^{M\times d_t}$ denotes text features, then

$Q=T W_Q,\qquad K=V W_K,\qquad V'=V W_V,$

and the attended update is

$$A=\mathrm{softmax}\!\Bigl(\tfrac{QK^T}{\sqrt{d_k}}\Bigr),\qquad \widetilde T = A V'.$$

The block finishes with residual, LayerNorm, and feed-forward updates on the text side, while the video side is not symmetrically updated inside that block (Panta et al., 2023).

A closely related formulation appears in Cross-Modality Attention for two-stream video classification. With flattened features $X\in\mathbb{R}^{n_1\times d_x}$ and $Y\in\mathbb{R}^{n_2\times d_y}$, one computes

$$Q=XW_q,\qquad K=YW_k,\qquad V=YW_v,\qquad Z=\mathrm{softmax}(QK^T/\sqrt{d_k})V,$$

and wraps the result in a residual block

$Q$0

Here, a CMA block always takes $Q$1 from one modality and $Q$2 from the other modality, so the attention in each block is unidirectional even if the network contains blocks in both directions (Chi et al., 2019).

Not all asymmetric cross-modal mechanisms use the exact Transformer-style $Q$3–$Q$4–$Q$5 implementation. In text-based person search, the asymmetric cross-attention module uses cosine similarity and softmax-like normalization to align image entities with noun phrases and, separately, phrases with image entities. For image $Q$6 text alignment, the raw affinity is

$Q$7

followed by clipping, normalization, phrase-wise softmax weighting, and aggregation into an attended phrase vector $Q$8, after which a cosine similarity $Q$9 is computed. A symmetric-looking reciprocal path also exists, but asymmetry is retained through different temperatures, different pooling order, and potentially different projection weights (Ji et al., 2022).

A further departure appears in CrossWKV for RWKV-7, where cross-modal interaction is implemented through a recurrent Weighted Key-Value mechanism rather than explicit dense attention. The transition matrix

$K,V$0

is input-dependent and non-diagonal, and text features provide receptance gates while image features determine decay and reinforcement terms. This yields an asymmetric cross-modal mapping with linear-time recurrence and constant memory in sequence length (Xiao et al., 19 Apr 2025).

2. What makes the mechanism “asymmetric”

The most direct form of asymmetry is unilateral update. In the asymmetric co-attention network for video moment retrieval, “the asymmetric co-attention block (ACB) only updates the text features by attending over video; video features bypass it,” and “at no point do we symmetrically update $K,V$1 by attending to $K,V$2” (Panta et al., 2023). In the Alzheimer’s multi-omic ACMCA network, only two directed attentions are used—clinical$K,V$3MRI and genetic$K,V$4PET—and the imaging modalities serve only as key/value sources while numeric modalities serve only as queries (Ming et al., 9 Jul 2025).

A second form is directional specialization rather than strict one-way exclusivity. In MRI–ultrasound registration, one block uses Primary=US and Cross=MR, while another uses Primary=MR and Cross=US; each block is itself asymmetric because $K,V$5 comes from the cross input and $K,V$6 come from the primary input (Song et al., 2021). The same principle appears in two-stream video classification, where “RGB$K,V$7Flow” and “Flow$K,V$8RGB” blocks are inserted, but each CMA block computes only one direction of attention (Chi et al., 2019).

A third form is asymmetric alignment strategy. UniAVGen does not use the same temporal rule in both directions. Its audio$K,V$9video aligner allows each video frame to attend to a short temporal window of nearby audio tokens, whereas its video$V\in\mathbb{R}^{N\times d_v}$0audio aligner uses a locally interpolated visual context for each audio token. The design is explicitly contrasted with both symmetric cross-attention and global cross-attention (Zhang et al., 5 Nov 2025). This shows that asymmetry may concern the neighborhood structure and context construction, not only the update target.

A fourth form is asymmetric normalization or pooling. In Asymmetric Cross-Scale Alignment for text-based person search, image$V\in\mathbb{R}^{N\times d_v}$1text and text$V\in\mathbb{R}^{N\times d_v}$2image are both computed, but the two directions differ in softmax temperature, aggregation order, and score construction (Ji et al., 2022). This suggests that “asymmetry” is not reducible to the absence of a reverse path; it can also denote non-equivalent directional operators.

A common misconception is that asymmetry implies a single overall direction for the entire model. The cited work does not support that simplification. Several systems contain two opposite one-way blocks, or two non-identical directed modules, while maintaining per-block asymmetry (Chi et al., 2019, Song et al., 2021, Zhang et al., 2022, Zhang et al., 5 Nov 2025). Another misconception is that asymmetry merely means different parameter tensors. The literature instead associates it with a broader structural decision about which modality probes, which modality is probed, and how information flow is temporally or spatially constrained (Panta et al., 2023, Ming et al., 9 Jul 2025).

3. Architectural insertion patterns

A recurrent pattern is early directed fusion followed by later joint processing. In the video moment retrieval architecture, the transformer backbone is a stack of $V\in\mathbb{R}^{N\times d_v}$3 cross-model skip-connection blocks, with $V\in\mathbb{R}^{N\times d_v}$4 in practice. Within each block, $V\in\mathbb{R}^{N\times d_v}$5 repeated asymmetric co-attention layers first update text using video, after which the video and updated text sequences are concatenated and passed through a Connected Attention Block, a standard multi-head self-attention plus feed-forward layer using $V\in\mathbb{R}^{N\times d_v}$6 heads, head dimension $V\in\mathbb{R}^{N\times d_v}$7, and model dimension $V\in\mathbb{R}^{N\times d_v}$8 (Panta et al., 2023). The asymmetry is therefore front-loaded, while subsequent CAB processing is symmetric over the concatenated stream.

In two-stream video classification, the CMA block is designed as a wrapper that can be inserted into existing architectures. The base network is ResNet-50, viewed as stages conv1/pool, res$V\in\mathbb{R}^{N\times d_v}$9, res$T\in\mathbb{R}^{M\times d_t}$0, res$T\in\mathbb{R}^{M\times d_t}$1, and res$T\in\mathbb{R}^{M\times d_t}$2. A total of five CMA blocks are placed in res$T\in\mathbb{R}^{M\times d_t}$3/res$T\in\mathbb{R}^{M\times d_t}$4 of each branch, inserted every other residual unit. Within a block, the current branch produces queries and the other branch produces keys and values, with spatial max-pooling applied to the other branch before attention. The output is returned to the current branch as a residual feature (Chi et al., 2019).

In ACMCA for Alzheimer’s disease prognosis, the asymmetric cross-modal cross-attention stage follows shallow feature extraction from PET, MRI, genetic, and clinical inputs. PET and MRI pass through a ResNet-50 pretrained backbone to yield 100-dimensional vectors, while genetic and clinical features are transformed by small DNNs into 100-dimensional vectors. Two parallel cross-attention branches—clinical$T\in\mathbb{R}^{M\times d_t}$5MRI and genetic$T\in\mathbb{R}^{M\times d_t}$6PET—produce $T\in\mathbb{R}^{M\times d_t}$7 and $T\in\mathbb{R}^{M\times d_t}$8, after which self-attention and an Fnet module operate on the fused tokens before a four-layer MLP performs three-way classification (Ming et al., 9 Jul 2025).

In cross-modal MRI–ultrasound registration, the attention block is inserted immediately after shallow per-modality encoders and before the deep registration head. The attended outputs from both directions are concatenated channel-wise and fed through convolutional fusion layers and fully connected layers to predict a rigid-transform matrix with 6 DOF (Song et al., 2021). In crowd counting, CSCA is inserted between modality-specific CNN backbones after major convolutional blocks, combining spatial-wise cross-modal attention with channel-wise feature aggregation (Zhang et al., 2022).

In text-based person search, asymmetry is embedded within a multi-scale retrieval model: Swin-Transformer yields a global image vector and four overlapping region vectors, BERT yields a global sentence vector and phrase embeddings, and the asymmetric cross-attention module aligns region/image entities with noun phrases while a separate global image–sentence alignment uses CMPM and CMPC losses (Ji et al., 2022). In UniAVGen, asymmetric cross-modal interaction is injected into each DiT layer, and in CrossWKV it is fused into the recurrent state update of RWKV-7 (Zhang et al., 5 Nov 2025, Xiao et al., 19 Apr 2025).

4. Representative instantiations across domains

The mechanism has been instantiated with markedly different modalities and backbones, but the directional cross-modal principle remains stable.

Setting	Asymmetric design	Reported effect
Video moment retrieval	Text queries attend over video; only text updated in ACB	Better TACoS performance, comparable ActivityNet Captions, 30% fewer parameters (Panta et al., 2023)
Two-stream video classification	Per-block RGB$T\in\mathbb{R}^{M\times d_t}$9Flow or Flow$Q=T W_Q,\qquad K=V W_K,\qquad V'=V W_V,$0RGB attention	72.6% on Kinetics-400 for CMA two-stream (Chi et al., 2019)
Alzheimer’s multi-omics	Clinical$Q=T W_Q,\qquad K=V W_K,\qquad V'=V W_V,$1MRI and genetic$Q=T W_Q,\qquad K=V W_K,\qquad V'=V W_V,$2PET	Test accuracy 94.88% (Ming et al., 9 Jul 2025)
MRI–US registration	One directional block per modality pair direction	Two-stage Attention-Reg reaches SRE$Q=T W_Q,\qquad K=V W_K,\qquad V'=V W_V,$3=3.71 mm (Song et al., 2021)
Cross-modal crowd counting	Spatial cross-attention both ways plus channel gating	State-of-the-art results on RGB-T and RGB-D (Zhang et al., 2022)
Audio–video generation	A2V local window, V2A interpolated visual context	Advantages in LS, TC, and EC (Zhang et al., 5 Nov 2025)

In video moment retrieval, the asymmetric design is paired with momentum contrastive learning. The paper states that video streams are typically an order of magnitude longer than text, and that symmetric cross-attention can dilute language signals across many video time-steps. The proposed design therefore produces a compact, visually grounded text embedding before later multimodal layers, while also reducing parameter count by removing one full branch of attention (Panta et al., 2023). A concrete empirical claim is that this yields “+0.98 pts on TACoS R@1 @0.3,” while matching or exceeding symmetric baselines and being “30 % lighter” (Panta et al., 2023).

In two-stream video classification, the mechanism improves over late fusion and non-local alternatives. On Kinetics-400, the standard two-stream baseline reaches 71.2% top-1, while CMA_iter$Q=T W_Q,\qquad K=V W_K,\qquad V'=V W_V,$4 (two-stream fusion) reaches 72.6%; on P3D ResNet-152 with 16 input frames, two-stream P3D reaches 74.6% and P3D + 3D-CMA reaches 76.0%; and on UCF-101, two-stream ResNet50 reaches 95.5% while CMA two-stream reaches 96.5% (Chi et al., 2019).

In ACMCA, the baseline comparison reports Simple Concatenation at 0.780 accuracy, Symmetric Cross-Attention (MADDi) at 0.859, MADDi + Asymmetric ACM at 0.885, and ACMCA at 0.948. The ablation further reports ACMCA-WCM at 0.785 accuracy and 0.910 AUC, versus ACMCA at 0.946 accuracy and 0.948 AUC, with the text stating that removing the asymmetric CMA block drops accuracy by over 16 points (Ming et al., 9 Jul 2025).

In medical registration, the cross-modal attention block is reported to reduce surface registration error relative to iterative and CNN baselines. The detailed comparison states that against DVNet and MSReg, two-stage Attention-Reg with 1.25 M parameters and 3 ms runtime achieves SRE$Q=T W_Q,\qquad K=V W_K,\qquad V'=V W_V,$5=4.50 mm and SRE$Q=T W_Q,\qquad K=V W_K,\qquad V'=V W_V,$6=3.71 mm, while the heavier baselines remain around 4.75–4.77 mm (Song et al., 2021). In cross-modal crowd counting, CSCA improves several backbones, including BL on RGBT-CC from MAE 18.70 to 14.32 and RMSE 32.64 to 26.01, and BL on ShanghaiTechRGBD from MAE 8.94 to 5.68 and RMSE 12.49 to 8.66 (Zhang et al., 2022).

5. Objectives, optimization, and efficiency

Asymmetric cross-modal cross-attention is frequently paired with auxiliary losses that shape alignment before or alongside fine-grained fusion. In the video moment retrieval model, a MoCo-style video–text contrastive loss is computed from pooled video and word representations using online and momentum encoders, queues of negative examples, and bidirectional softmax distributions $Q=T W_Q,\qquad K=V W_K,\qquad V'=V W_V,$7 and $Q=T W_Q,\qquad K=V W_K,\qquad V'=V W_V,$8. The total loss is

$Q=T W_Q,\qquad K=V W_K,\qquad V'=V W_V,$9

combining representation alignment with proposal boundary regression and ranking (Panta et al., 2023).

In ACMCA, training is simpler: the model is optimized with standard cross-entropy over three classes, with Adam, learning rate $$A=\mathrm{softmax}\!\Bigl(\tfrac{QK^T}{\sqrt{d_k}}\Bigr),\qquad \widetilde T = A V'.$$0, batch size $$A=\mathrm{softmax}\!\Bigl(\tfrac{QK^T}{\sqrt{d_k}}\Bigr),\qquad \widetilde T = A V'.$$1, and $$A=\mathrm{softmax}\!\Bigl(\tfrac{QK^T}{\sqrt{d_k}}\Bigr),\qquad \widetilde T = A V'.$$2 epochs, and “no additional ‘attention’ regularizers” are applied (Ming et al., 9 Jul 2025). In text-based person search, by contrast, the asymmetric cross-attention similarities are inserted into a KL-based matching loss, while global image–sentence alignment is supervised by CMPM and CMPC; the total loss is

$$A=\mathrm{softmax}\!\Bigl(\tfrac{QK^T}{\sqrt{d_k}}\Bigr),\qquad \widetilde T = A V'.$$3

with $$A=\mathrm{softmax}\!\Bigl(\tfrac{QK^T}{\sqrt{d_k}}\Bigr),\qquad \widetilde T = A V'.$$4 and $$A=\mathrm{softmax}\!\Bigl(\tfrac{QK^T}{\sqrt{d_k}}\Bigr),\qquad \widetilde T = A V'.$$5 in the paper (Ji et al., 2022).

Efficiency considerations vary by operator. The MRI–ultrasound registration block has quadratic complexity in the number of spatial positions because computing $$A=\mathrm{softmax}\!\Bigl(\tfrac{QK^T}{\sqrt{d_k}}\Bigr),\qquad \widetilde T = A V'.$$6 costs $$A=\mathrm{softmax}\!\Bigl(\tfrac{QK^T}{\sqrt{d_k}}\Bigr),\qquad \widetilde T = A V'.$$7, multiplying $$A=\mathrm{softmax}\!\Bigl(\tfrac{QK^T}{\sqrt{d_k}}\Bigr),\qquad \widetilde T = A V'.$$8 costs $$A=\mathrm{softmax}\!\Bigl(\tfrac{QK^T}{\sqrt{d_k}}\Bigr),\qquad \widetilde T = A V'.$$9, and storing the affinity matrix requires $X\in\mathbb{R}^{n_1\times d_x}$0 memory (Song et al., 2021). CSCA explicitly reduces spatial complexity by grouping spatial tokens into $X\in\mathbb{R}^{n_1\times d_x}$1 blocks, bringing the cross-modal attention cost down by a factor of $X\in\mathbb{R}^{n_1\times d_x}$2 and yielding SCA blocks that run approximately $X\in\mathbb{R}^{n_1\times d_x}$3–$X\in\mathbb{R}^{n_1\times d_x}$4 faster than a full non-local block, with total parameter increase across four insertions of approximately $X\in\mathbb{R}^{n_1\times d_x}$5–$X\in\mathbb{R}^{n_1\times d_x}$6 (Zhang et al., 2022).

CrossWKV represents a different efficiency regime. Its cross-modal layer runs in linear time with respect to image sequence length and uses constant memory in $X\in\mathbb{R}^{n_1\times d_x}$7, since the persistent state is the recurrent matrix $X\in\mathbb{R}^{n_1\times d_x}$8. The paper reports that on ImageNet 256$X\in\mathbb{R}^{n_1\times d_x}$9256 with 50-token prompts, DIR-7-H runs in 0.52 s using 4.5 GB, versus 0.85 s and 6.5 GB for DiT-XL/2; doubling prompt length increases DIR-7-H time linearly while memory remains approximately 4.7 GB (Xiao et al., 19 Apr 2025). This suggests that asymmetric cross-modal interaction can be embedded either in quadratic attention kernels or in recurrent state-space operators, depending on the surrounding architecture.

6. Interpretation, advantages, and limitations

Several papers attribute the usefulness of asymmetry to modality imbalance. The video moment retrieval work explicitly frames the problem as “information asymmetry due to the difference in length between visual and textual sequences,” and states that video streams are typically much longer than text. Its solution updates only text early, preserving a “pure” video stream until later concatenated self-attention (Panta et al., 2023). In ACMCA, the authors’ hypothesis is that clinical and genetic data must probe into imaging features rather than the reverse, so the imaging modalities remain key/value sources while the numeric modalities act as queries (Ming et al., 9 Jul 2025). A plausible implication is that asymmetry is often introduced when one modality is treated as semantically sparse but decision-critical, and the other as information-rich but structurally diffuse.

Interpretability results also recur. CMA visualizations show that RGB features attend to motion or object cues in the flow branch, that long-range dependencies can be captured, and that discriminative motion is highlighted rather than all moving pixels (Chi et al., 2019). Attention-Reg uses Grad-CAM on attended feature maps and reports that both MR and US branches focus on the true prostate region, indicating spatially consistent correspondences (Song et al., 2021). CSCA is described as correcting alignment errors and channel mis-calibration by allowing each modality to attend to the other where it is most reliable and then gating out noise (Zhang et al., 2022).

The literature also identifies clear failure modes or design trade-offs. CMA reports that when the query falls on pure background, attention can focus on other background regions (Chi et al., 2019). UniAVGen argues that fully global cross-attention converges slowly and includes many irrelevant interactions, whereas symmetric time-aligned cross-attention is too narrow because it omits neighboring phonemes and intermediate visual dynamics; its asymmetric temporal design is presented as a compromise between those two extremes (Zhang et al., 5 Nov 2025). In MRI–US registration, the attention matrix has quadratic memory, so down-sampling and restricted feature sizes are required for tractability (Song et al., 2021).

A final point of clarification is that asymmetry should not be conflated with inferiority to symmetric fusion. The reported evidence includes settings where asymmetric variants outperform symmetric cross-attention baselines, such as MADDi versus MADDi + Asymmetric ACM and full ACMCA, or match and exceed symmetric baselines while using fewer parameters in video grounding (Ming et al., 9 Jul 2025, Panta et al., 2023). At the same time, some architectures preserve two directed paths, later symmetric self-attention, or channel-wise fusion, indicating that asymmetry is typically a local design principle within a larger multimodal system rather than a claim that all stages of fusion must remain directional (Panta et al., 2023, Zhang et al., 2022, Zhang et al., 5 Nov 2025).