Convolution-Attention Fusion Module

• CAFM is a neural network component that fuses convolutional local feature extraction with attention mechanisms to capture both fine-grained details and long-range dependencies.
• It employs design patterns such as dual-path fusion, residual weighted summation, and dynamic gating to effectively balance local and contextual information.
• Empirical results across tasks like detection, segmentation, and multimodal classification demonstrate CAFM's ability to improve accuracy and sensitivity with minimal overhead.

Convolution-and-Attention Fusion Module (CAFM) denotes a class of neural network components that combine convolutional operators with attention mechanisms so that local feature extraction, multi-scale aggregation, long-range dependency modeling, or cross-modal interaction are performed within a single fusion stage. Across the cited literature, the term is used explicitly in some works and, in others, the reported module is described as a concrete instantiation or reusable interpretation of what can be viewed as a Convolution-and-Attention Fusion Module. This suggests that CAFM is best treated as a family of design patterns rather than a single canonical block (Jin et al., 16 Aug 2025).

1. Terminology, scope, and representative formulations

The explicit label appears in the lightweight detection framework YOLO11-CR, where the CAFM is defined as a two-branch unit that takes an input feature map $Y$, extracts local, fine-grained features via a convolutional branch, extracts global, long-range contextual features via a lightweight self-attention branch, and fuses both streams by summation,

$$F_{\text{out}} = F_{\text{conv}} + F_{\text{attn}}.$$

In that work, CAFM replaces the transformer attention layer inside YOLO11’s C2PSA block and forms a C2PSA_CAFM structure in the backbone (Jin et al., 16 Aug 2025).

Several earlier or parallel lines of work instantiate closely related ideas under different names. In multimodal skin cancer classification, the Joint-Individual Fusion with Multi-Modal Fusion Attention module is presented as a reusable convolution-and-attention fusion pattern operating on image and metadata vectors (Tang et al., 2023). In retinal lesion segmentation, the Multi-scale Attention Fusion module is described as “essentially a concrete instantiation” of a CAFM, because it concatenates task-specific convolutional features, applies multi-scale convolution through a split spatial convolutional layer, and generates attention masks for residual reweighting (Zhang et al., 2022). In contour generation for Dunhuang murals, the Ramix block is a residual self-attention and convolution mixed module that fuses local and global features with learnable branch weights (Liu et al., 2022). In vision backbones, Locally Enhanced Self-Attention strengthens the local term with convolution and dynamically couples it with an attention-derived context term (Yang et al., 2021). More abstract feature-fusion formulations, such as Attentional Feature Fusion and its iterative form, define fusion as context-aware selection between convolutional feature maps rather than direct addition or concatenation (Dai et al., 2020).

Representative module	Task domain	Defining fusion pattern
YOLO11-CR CAFM	Fatigue-driving detection	Local conv branch + channel-wise attention branch + summation
JIF-MMFA / MMFA	Multimodal skin cancer classification	Multi-head attention over concatenated image and metadata vectors + residual concat
MAF	Hard exudate segmentation with super-resolution	SSC multi-scale convolution + task-specific attention masks
Ramix	Edge and contour generation	Parallel convolution and self-attention with residual weighted fusion
LESA	Recognition, detection, segmentation	Convolutional unary term + attention binary term + dynamic gate
AFF / iAFF	Generic feature fusion	Attention-based soft selection between two feature maps

2. Core architectural motifs

A recurrent CAFM motif is the explicit division between a convolutional path and an attention path. In YOLO11-CR, the convolutional path is

$$F_{\text{conv}} = W_{3\times3\times3}\bigl(cs(W_{1\times1}(Y))\bigr),$$

while the attention path computes query, key, and value tensors by $1\times1$ and $3\times3$ depthwise convolutions, applies channel-wise scaled dot-product attention,

$$\text{Attention}(Q, K, V) = V \cdot \text{Softmax}\left(\frac{K^T Q}{\alpha}\right),$$

and then uses a residual projection,

$$F_{\text{attn}} = W_{1\times1}\bigl(\text{Attention}(Q, K, V) + Y\bigr).$$

The final representation is the sum of the two branches (Jin et al., 16 Aug 2025).

A second motif is residual weighted fusion of parallel local and contextual operators. Ramix computes convolutional and self-attentional outputs in parallel and fuses them as

$$F_{\text{out}} = \sigma\big(\alpha\,F_{\text{att}}(x) + \beta\,F_{\text{conv}}(x) + x\big),$$

where $\alpha$ and $\beta$ are learnable scalar weights and $$F_{\text{out}} = F_{\text{conv}} + F_{\text{attn}}.$$0 is the residual input. The stated purpose is to exploit local, spatially precise features from convolution and contextual, globally aware features from self-attention simultaneously (Liu et al., 2022).

A third motif is dynamic gating between local and context terms. LESA decomposes self-attention into a local or unary component and a context or binary component, strengthens the unary term with grouped convolution,

$$F_{\text{out}} = F_{\text{conv}} + F_{\text{attn}}.$$1

defines the context term $$F_{\text{out}} = F_{\text{conv}} + F_{\text{attn}}.$$2 by self-attention, and fuses them as

$$F_{\text{out}} = F_{\text{conv}} + F_{\text{attn}}.$$3

The gate is input-adaptive and depends jointly on local and contextual features (Yang et al., 2021).

A fourth motif is context-aware selection between feature sources rather than simple branch addition. Attentional Feature Fusion defines

$$F_{\text{out}} = F_{\text{conv}} + F_{\text{attn}}.$$4

where $$F_{\text{out}} = F_{\text{conv}} + F_{\text{attn}}.$$5 is produced by a multi-scale channel attention module combining local point-wise convolutional context and global average-pooled context. Iterative AFF replaces the initial integration $$F_{\text{out}} = F_{\text{conv}} + F_{\text{attn}}.$$6 with an inner attentional fusion step to alleviate the initial-integration bottleneck (Dai et al., 2020).

A distinct variant moves attention from activation space to weight space. The AW-convolution module computes an attention tensor

$$F_{\text{out}} = F_{\text{conv}} + F_{\text{attn}}.$$7

that matches the convolution kernel shape, forms attentional weights

$$F_{\text{out}} = F_{\text{conv}} + F_{\text{attn}}.$$8

and then performs convolution with $$F_{\text{out}} = F_{\text{conv}} + F_{\text{attn}}.$$9. This is presented as a way to address both the approximation problem and the insufficient capacity problem of activation-based attention maps (Baozhou et al., 2021).

3. Fusion loci: feature maps, vectors, tokens, and weights

CAFM designs differ substantially in the level at which fusion occurs. In multimodal skin cancer classification, fusion happens after backbone feature extraction. A CNN produces an image vector $$F_{\text{conv}} = W_{3\times3\times3}\bigl(cs(W_{1\times1}(Y))\bigr),$$0, an MLP produces a metadata vector $$F_{\text{conv}} = W_{3\times3\times3}\bigl(cs(W_{1\times1}(Y))\bigr),$$1, and the Multi-Modal Fusion Attention module projects each modality into per-modality query, key, and value vectors, concatenates them,

$$F_{\text{conv}} = W_{3\times3\times3}\bigl(cs(W_{1\times1}(Y))\bigr),$$2

applies multi-head self-attention, and adds a residual concatenation,

$$F_{\text{conv}} = W_{3\times3\times3}\bigl(cs(W_{1\times1}(Y))\bigr),$$3

Formally, the module is self-attention over concatenated features; conceptually, it is described as mutual attention because metadata can highlight image channels and image features can highlight metadata dimensions (Tang et al., 2023).

In segmentation, CAFM-like modules often operate on feature maps rather than vectors. In SS-MAF, segmentation and super-resolution decoder outputs $$F_{\text{conv}} = W_{3\times3\times3}\bigl(cs(W_{1\times1}(Y))\bigr),$$4 and $$F_{\text{conv}} = W_{3\times3\times3}\bigl(cs(W_{1\times1}(Y))\bigr),$$5 are concatenated, projected by a $$F_{\text{conv}} = W_{3\times3\times3}\bigl(cs(W_{1\times1}(Y))\bigr),$$6 convolution, processed by a split spatial convolutional layer that mixes a $$F_{\text{conv}} = W_{3\times3\times3}\bigl(cs(W_{1\times1}(Y))\bigr),$$7 group with several dilated $$F_{\text{conv}} = W_{3\times3\times3}\bigl(cs(W_{1\times1}(Y))\bigr),$$8 groups, and then converted into task-specific attention masks,

$$F_{\text{conv}} = W_{3\times3\times3}\bigl(cs(W_{1\times1}(Y))\bigr),$$9

The masks reweight original task features with residual preservation,

$1\times1$0

This design performs cross-task fusion by using jointly computed features to modulate each stream separately (Zhang et al., 2022).

In hybrid CNN-transformer systems, fusion can take place over tokenized multi-level representations. CTRL-F extracts multi-level CNN features, tokenizes them into “large patch tokens” and “small patch tokens,” processes them in two transformer branches, and lets the two branches communicate through cross-attention. The branch outputs are then fused with the CNN prediction through Adaptive Knowledge Fusion or Collaborative Knowledge Fusion (EL-Assiouti et al., 2024). In LV-CadeNet, the encoder uses self-convolution-attention modules and the decoder uses cross-convolution-attention modules over MEG tensors $1\times1$1, combining temporal 1D convolution and multi-head attention across channels to learn waveform morphology and spatial dipole structure jointly (Xiao et al., 2024).

A further generalization appears in multimodal object detection for RGB–IR data. FMCAF treats its Multimodal Cross Attention Fusion block as a convolution-plus-attention fusion design: convolutional Inception-style encoders extract per-modality features, cross-attention exchanges information between RGB and infrared windows, local attention emphasizes informative regions, global attention applies sigmoid-gated residual modulation, and a convolutional fusion block projects the result into a fused 3-channel image for YOLOv11 (Berjawi et al., 20 Oct 2025).

4. Attention mechanisms, collaboration strategies, and supervision

The attention component in CAFM is not uniform across the literature. Transformer-style self-attention and cross-attention are common in token or vector settings, but several influential variants are lighter. MAF uses $1\times1$2 convolution plus sigmoid to produce feature-wise and spatial-wise weighting from fused task features rather than transformer self-attention (Zhang et al., 2022). CAT models collaboration between channel and spatial attention using trainable colla-factors and a three-way pooling scheme built from global average pooling, global maximum pooling, and global entropy pooling. Its output combines channel and spatial attention as

$1\times1$3

with separate interior collaboration factors for pooling choices and exterior collaboration factors for branch balance (Wu et al., 2022). SimAM² instead derives attention from signal theory and uncertainty theory: it treats multimodal fusion as

$1\times1$4

forms an energy-based score $1\times1$5, and rescales the fused representation by

$1\times1$6

It also defines a decoupling-free discrepancy ratio

$1\times1$7

for multimodal gradient modulation (Sun et al., 2023).

Supervision strategies are equally diverse. The multimodal skin cancer model couples CAFM with a Joint-Individual Fusion structure that maintains image-only, metadata-only, and fusion classifiers, combines them by probability averaging at test time, and optimizes the total loss

$1\times1$8

The stated effect is to preserve modality-specific information while learning shared multimodal representations (Tang et al., 2023). SS-MAF explicitly supervises both task streams and the fusion outputs: $1\times1$9 so that region mutual information loss can improve tiny lesion and boundary detection (Zhang et al., 2022). LV-CadeNet embeds its convolution-attention blocks in an encoder-decoder segmentation framework and later fine-tunes the model with teacher–student semi-supervised learning using cross-entropy and KL divergence (Xiao et al., 2024).

These differences make one common misconception untenable: attention in CAFM is not restricted to full transformer attention. The literature includes channel-wise scaled dot-product attention, local-window self-attention, sigmoid-based spatial-channel gating, entropy-based attention, energy-based attention, and weight-shaped attention (Jin et al., 16 Aug 2025).

5. Empirical behavior across application domains

Empirical gains reported for CAFM-style designs are domain-specific but consistently tied to improved fusion of local and contextual information. In multimodal skin cancer classification, JIF-MMFA improves Balanced Accuracy on all three public datasets. On PAD-UFES-20, image-only average BAC is $3\times3$0, the best competing fusion achieves $3\times3$1, and JIF-MMFA (All) reaches $3\times3$2. On SPC, image-only average BAC is $3\times3$3, the best competing fusion is approximately $3\times3$4, and JIF-MMFA (All) reaches $3\times3$5. On ISIC-2019, image-only average BAC is $3\times3$6, the best competing fusion is $3\times3$7, and JIF-MMFA reaches $3\times3$8 for OFB and $3\times3$9 for All. Friedman and Wilcoxon tests indicate significant improvement for most pairs at $$\text{Attention}(Q, K, V) = V \cdot \text{Softmax}\left(\frac{K^T Q}{\alpha}\right),$$0 (Tang et al., 2023).

In hard exudate segmentation, the ablation study isolates the effect of structured convolution-and-attention fusion. On IDRiD, Dice rises from 56.22 for U-Net, to 57.44 for U-Net+Interp+SR, and to 72.55 for U-Net+Interp+SR+MAF; IoU reaches 56.93 and Recall 69.39. On E-Ophtha, Dice rises from 46.59 for U-Net to 52.96 for U-Net+Interp+SR+MAF, with IoU 36.03. The paper also states that on E-Ophtha the method can achieve $$\text{Attention}(Q, K, V) = V \cdot \text{Softmax}\left(\frac{K^T Q}{\alpha}\right),$$1 higher dice and recall compared with the state-of-the-art methods (Zhang et al., 2022).

In fatigue-driving detection, YOLO11-CR reports precision 87.17%, recall 83.86%, mAP@50 88.09%, and mAP@50-95 55.93%. Its ablation shows that replacing YOLO11n with YOLO11n+CAFM changes precision from 0.8648 to 0.8466, recall from 0.7622 to 0.7942, mAP@50 from 0.8228 to 0.8416, and mAP@50–95 from 0.5337 to 0.5363. The reported interpretation is that CAFM substantially increases detection sensitivity while adding minimal overhead (Jin et al., 16 Aug 2025).

In MEG spike detection, LV-CadeNet reports that its long-view convolution-attention encoder-decoder raises F1 on the Sanbo-Clinic clinical test set from 42.31% for the best baseline Crossformer to 54.88% for SSL-LV-CadeNet, with precision 0.5526 and recall 0.6174. The full system combines the convolution-attention blocks with long-view morphology features and semi-supervised learning (Xiao et al., 2024).

In image classification, CTRL-F reports top-1 accuracy of 82.24% on Oxford-102 Flowers and 99.91% on PlantVillage when trained from scratch. Its ablations further indicate that fusion is the critical step: on Oxford-102, the CNN-only variant reaches 76.57%, the MFCA-only variant 72.04%, and the fused AKF version 80.79% (EL-Assiouti et al., 2024). Related general-purpose conv-attention designs also show quantitative gains: AW-convolution improves Top-1 accuracy on ImageNet by 1.00% over a ResNet101 baseline and COCO-style Average Precision by 0.63 on Faster R-CNN with ResNet101-FPN (Baozhou et al., 2021), while Multi-QuartzNet reaches CER 6.77% on AISHELL-1 (Luo et al., 2020).

6. Limitations, misconceptions, and extension paths

The literature also shows that CAFM is not uniformly beneficial under all conditions. Metadata richness matters in multimodal classification: JIF-MMFA is most effective when metadata has sufficient diversity, and benefits are smaller on ISIC-2019, where metadata is sparse (Tang et al., 2023). Frequency-domain preprocessing is not automatically helpful: in FMCAF ablations on VEDAI, concatenation achieves 62.6 mAP@50, Freq-Filter only drops to 52.5, MEFA only reaches 70.4, MCAF only reaches 71.8, and the combined FMCAF reaches 74.6 in ablation and 76.5 in the full setup. This indicates that filtering alone can over-suppress fine detail, whereas attention-based multimodal exchange is the main driver of improvement (Berjawi et al., 20 Oct 2025).

Optimization can also become harder as fusion grows more elaborate. The AFF study reports that iterative attentional feature fusion usually improves over single-stage AFF, but increasing depth together with iAFF can make optimization harder (Dai et al., 2020). In the skin cancer model, MMFA alone performs poorly in a JF setting on ISIC-2019, but becomes among the best modules once combined with JIF supervision, which indicates that mutual-attention mechanisms benefit strongly from branch-specific supervision (Tang et al., 2023). In SS-MAF, simple interpolation does not help and can even hurt, while the structured combination of super-resolution and MAF is what yields the large improvement (Zhang et al., 2022).

Several practical limitations recur. Some vector-based CAFM designs do not explicitly preserve spatial structure in attention, as in MMFA over flattened image features (Tang et al., 2023). YOLO11-CR operates on single images and does not model temporal dependencies such as blink frequency or gradual head drop (Jin et al., 16 Aug 2025). FMCAF assumes paired, aligned RGB–IR images and adds latency through FFT/IFFT and attention (Berjawi et al., 20 Oct 2025). LV-CadeNet notes that sensor grouping and spatial modeling are heuristic and that more principled graph-based attention or 3D head models might improve dipole encoding (Xiao et al., 2024).

A second misconception is that CAFM necessarily implies a heavy transformer inserted into a CNN. The surveyed designs contradict that view. Some use full multi-head attention, some use local-window attention, some rely on sigmoid gates derived from $$\text{Attention}(Q, K, V) = V \cdot \text{Softmax}\left(\frac{K^T Q}{\alpha}\right),$$2 convolutions, some use energy-based closed-form attention, and some modulate convolution weights directly rather than activations (Baozhou et al., 2021). A plausible implication is that CAFM is less a single architecture than a reusable principle: convolution supplies local structure or modality-specific bias, attention supplies selective weighting or long-range interaction, and the fusion rule determines how aggressively the two are coupled.

Current extension paths in the literature follow this principle closely. Proposed directions include extending vector-only fusion to spatial attention over feature maps, adding masking or modality dropout for missing data, integrating temporal modeling for video or biosignals, generalizing from two modalities to richer multimodal settings, and combining frequency-domain filtering with cross-attention in a dataset-agnostic front end (Tang et al., 2023). The cumulative record suggests that future CAFM designs will continue to vary in attention type, fusion locus, and supervision, while preserving the central objective of coupling convolutional locality with attention-driven selectivity.