Fused Attention Blocks (FABlocks)

• Fused Attention Blocks (FABlocks) are modules that adaptively fuse multiple feature streams using attention to preserve distinct stream information and contextual cues.
• They enable iterative, multi-scale feature exchange, as demonstrated in PelFANet where dual-stream data (raw X-rays and bone segmentations) are fused for improved fracture diagnosis.
• FABlocks are implemented in various forms—ranging from CBAM-based dual-stream fusion to relative-location and sparse attention models—optimizing performance across different applications.

Fused Attention Blocks (FABlocks) are attention-guided fusion modules designed to combine multiple feature sources while preserving stream-specific information. In the literature provided here, the term is used most explicitly in PelFANet, where FABlocks iteratively exchange and refine features from raw pelvic X-rays and segmented bone images for fracture diagnosis. Closely related work uses cognate formulations such as attention-fused blocks, attentional feature fusion, location-fused self-attention, and block-level fused sparse attention; despite substantial architectural differences, these methods share a common objective of replacing static feature aggregation with adaptive, learned fusion that is sensitive to context, scale, spatial structure, or sequence regime (Bhuiyan et al., 17 Sep 2025).

1. Scope and terminological usage

The term “Fused Attention Blocks” is not used as a single standardized primitive across the cited papers. Rather, several works instantiate the broader idea of attention-mediated fusion in task-specific ways. PelFANet uses FABlocks in a dual-stream classifier for pelvic fracture diagnosis (Bhuiyan et al., 17 Sep 2025). AFNet introduces a Multipath Attention-Fused Block (MAFB) for multi-source feature fusion and a Refinement Attention-Fused Block (RAFB) for fusing high-level abstract features with low-level spatial features in remote sensing segmentation (Yang et al., 2021). “Attentional Feature Fusion” and its iterative extension formalize feature fusion as a learned soft selection between inputs, with a multi-scale channel attention module to address semantic and scale inconsistency (Dai et al., 2020).

Other papers fuse attention in different senses. DSFNet introduces Location-Fused Self-Attention (LFSA), which augments attention keys with learnable relative-location embeddings for polyp segmentation (Fan et al., 2023). The VQA model based on a Cascade of Self- and Co-Attention blocks fuses intra-modality and cross-modality attention through repeated stacking (Mishra et al., 2023). BlossomRec uses a block-level fused sparse attention mechanism that combines long-term and short-term user-interest pathways through a learnable gate (Ma et al., 15 Dec 2025). Blockbuster addresses fusion at the operator and memory-hierarchy level, automatically rediscovering Flash Attention and producing fused kernels for broader blockwise computation patterns (Dekel, 29 Apr 2025).

A common misconception is that all “fused attention” modules are interchangeable. The cited literature indicates otherwise: some blocks fuse modalities, some fuse feature scales, some fuse content and location inside attention logits, some fuse sparse attention pathways, and some fuse low-level operators into a single kernel. The shared principle is adaptive integration, but the computational object being fused differs substantially.

2. FABlocks in PelFANet

In PelFANet, FABlocks are the central modules of a dual-stream attention network for pelvic fracture diagnosis. One stream processes the raw pelvic X-ray, and the other processes the bone segmentation image. After initial stream-specific feature extraction, the two streams are fused repeatedly through eight stacked FABlocks, and after the final FABlock the features are globally aggregated for classification (Bhuiyan et al., 17 Sep 2025).

The architectural flow is specified as follows. Both streams begin with $3\times 3$ conv $\rightarrow$ BatchNorm $\rightarrow$ ReLU $\rightarrow$ MaxPool. Let $F_1 \in \mathbb{R}^{C \times H \times W}$ denote the raw X-ray feature map and

$F_2 \in \mathbb{R}^{C \times H \times W}$ denote the bone-segmentation feature map. A FABlock first applies stream-specific convolution: $$F_1' = f_{3\times3}(F_1), \quad F_2' = f_{3\times3}(F_2).$$

The refined features are concatenated channel-wise: $$F_{\text{cat}} = \text{Concat}(F_1', F_2') \in \mathbb{R}^{2C \times H \times W}.$$ This concatenated map is then passed through CBAM: $F_{\text{CBAM}} = \text{CBAM}(F_{\text{cat}}).$ A $1\times 1$ convolution projects the result back to the original channel dimension: $\rightarrow$0 Finally, the fused representation is redistributed to both streams by residual update: $\rightarrow$1

$\rightarrow$2

This design has two notable properties. First, the fusion is joint but not destructive: each stream is updated using the same attention-refined shared representation while retaining its own residual state. Second, the process is iterative rather than one-off: each FABlock receives already updated paired features, so the streams repeatedly consult each other across depth. The paper states that this enables “iterative, multi-scale feature exchange,” allowing cues from one modality to influence the other without overwriting the original information (Bhuiyan et al., 17 Sep 2025).

3. Attention mechanics and iterative exchange

The attention mechanism inside PelFANet’s FABlock is CBAM, which applies channel attention and spatial attention sequentially. Channel attention learns “what” to focus on by modeling inter-channel relationships; spatial attention learns “where” to focus within the feature map. The paper describes CBAM through average and max pooling, shared MLPs for channel attention, and a $\rightarrow$3 convolution with sigmoid for spatial attention. Within FABlocks, this sequence reweights the concatenated dual-stream representation before it is projected and redistributed (Bhuiyan et al., 17 Sep 2025).

The same general logic appears in AFNet, but with a different decomposition of the fusion problem. MAFB fuses features from a main branch and an auxiliary branch after concatenation, applies both channel and spatial attention to the combined tensor, and then recombines the weighted features through a $\rightarrow$4 convolution. RAFB fuses low-level spatial and high-level abstract features by using channel attention to weight low-level features and spatial attention to weight high-level features, followed by summation: $\rightarrow$5 This differs from PelFANet’s dual residual redistribution, but both systems compute attention from jointly processed inputs and then use that attention to mediate selective cross-stream or cross-level fusion (Yang et al., 2021).

Attentional Feature Fusion (AFF) makes this idea explicit in a general two-input formulation: $\rightarrow$6 Its iterative extension, iAFF, applies another level of attention because the initial integration itself can become a bottleneck. This suggests a useful way to interpret FABlocks: not merely as attention modules attached to fusion points, but as modules in which the fusion operation itself is parameterized by attention and may be refined across stages (Dai et al., 2020).

4. Related fused-attention formalisms

DSFNet’s Location-Fused Self-Attention (LFSA) provides a different notion of fusion. Instead of fusing two feature streams with CBAM, it fuses content keys with learnable relative-location embeddings inside the attention computation: $\rightarrow$7 Here, $\rightarrow$8 is a learnable embedding of relative location, and $\rightarrow$9 are trainable scalars that normalize the balance between content and location. The paper states that LFSA is used in the decoder, where outputs from several decoding stages are sequentially processed and fused. In this formulation, “fused attention” means adaptive blending of feature similarity and spatial bias within the key space rather than fusion of external branches (Fan et al., 2023).

The VQA model based on a Cascade of Self- and Co-Attention blocks uses yet another fusion pattern. Within each block, self-attention is first applied independently to image and question representations; co-attention is then used to mediate cross-modal interaction, and the output of block $\rightarrow$0 is fed into block $\rightarrow$1. After the final block, pooled image and question vectors are combined by element-wise multiplication: $\rightarrow$2 The paper’s ablation results show that self-attention alone, co-attention alone, and their joint use yield distinct behaviors, with the fused self- and co-attention block outperforming either individual component on VQA2.0 validation (Mishra et al., 2023).

These examples clarify that “fusion” can occur at several levels: between streams, between scales, between modalities, or between content and positional terms inside attention. A plausible implication is that FABlocks are best understood as a design family organized around adaptive relational coupling rather than as one canonical micro-architecture.

5. Block-level fusion for efficiency and kernel generation

BlossomRec extends the fusion concept from spatial feature maps to sequential recommendation. Its block-level fused sparse attention mechanism explicitly separates long-term interest selection (LTIS) from short-term interest selection (STIS), then combines them with a learnable gated output: $\rightarrow$3

$\rightarrow$4

LTIS uses block partitioning, block compression via a learnable MLP, and Top-K block selection; STIS uses a power-law sparse local attention mask. The paper gives overall complexity as

$\rightarrow$5

contrasting it with full attention $\rightarrow$6, and reports interaction reductions from $\rightarrow$7 at sequence length $\rightarrow$8 to $\rightarrow$9 at length $\rightarrow$0 (Ma et al., 15 Dec 2025). In this setting, a fused block is less about multimodal residual redistribution and more about gating between complementary sparse attention pathways.

Blockbuster shifts the meaning of fusion again, from architectural attention design to operator fusion in inference programs. Its block program explicitly models how tensor blocks move between memory tiers, distinguishes buffered from unbuffered edges, and applies substitution rules to eliminate buffered intermediates. The paper states that the framework automatically rediscovers Flash Attention and can fuse LayerNorm with matrix multiplication and RMSNorm with FNN-SwiGLU into single kernels. The same rule system uses companion transformations such as swapping scale and dot or shift and dot to expose fusion opportunities, then recursively fuses maps, reductions, and elementwise operators (Dekel, 29 Apr 2025).

This distinction matters. PelFANet FABlocks are trainable architectural modules used during representation learning. Blockbuster’s block-level fusion is a compiler- or framework-level transformation over lowered AI programs. The shared term “fusion” reflects a common concern with reducing ineffective separation between coupled computations, but the objects and mechanisms of fusion differ.

6. Empirical evidence, limitations, and interpretation

The strongest direct evidence for FABlocks as a named module comes from PelFANet. On the AMERI dataset, PelFANet achieves $\rightarrow$1 accuracy and $\rightarrow$2 AUC on visible fractures, and $\rightarrow$3 accuracy and $\rightarrow$4 AUC on invisible fractures despite not being trained on them (Bhuiyan et al., 17 Sep 2025). The same source reports, in a baseline comparison, AUC $\rightarrow$5 versus $\rightarrow$6 on visible fracture data and AUC $\rightarrow$7, F1 $\rightarrow$8 versus AUC $\rightarrow$9, F1 $F_1 \in \mathbb{R}^{C \times H \times W}$0 on invisible fracture data. However, the paper also notes that direct ablation studies on components like CBAM and FABlocks are planned. This is an important qualification: the performance advantage is consistent with the FABlock design, but the isolated contribution of FABlocks is not yet fully disentangled within that work.

Related evidence supports the broader effectiveness of attention-guided fusion. AFNet reports overall accuracy $F_1 \in \mathbb{R}^{C \times H \times W}$1 and mean F1 $F_1 \in \mathbb{R}^{C \times H \times W}$2 on the ISPRS Vaihingen 2D dataset, and overall accuracy $F_1 \in \mathbb{R}^{C \times H \times W}$3 and mean F1 $F_1 \in \mathbb{R}^{C \times H \times W}$4 on the ISPRS Potsdam 2D dataset (Yang et al., 2021). DSFNet reports that replacing LFSA with standard attention blocks such as CCNet, CSNet, and TransAttUnet reduces performance, especially on Dice and boundary F scores, and states that the proposed model surpasses other state-of-the-art models in Dice, MAE, and IoU (Fan et al., 2023). The VQA model with cascaded self- and co-attention improves VQA2.0 validation overall accuracy from $F_1 \in \mathbb{R}^{C \times H \times W}$5 without attention to $F_1 \in \mathbb{R}^{C \times H \times W}$6 with the full SCA block, and reports TDIUC overall accuracy $F_1 \in \mathbb{R}^{C \times H \times W}$7 (Mishra et al., 2023). BlossomRec reports that at length $F_1 \in \mathbb{R}^{C \times H \times W}$8 it is $F_1 \in \mathbb{R}^{C \times H \times W}$9 faster per epoch in training, $F_2 \in \mathbb{R}^{C \times H \times W}$0 faster in inference for long sequences, and uses $F_2 \in \mathbb{R}^{C \times H \times W}$1th to $F_2 \in \mathbb{R}^{C \times H \times W}$2th of SASRec’s memory, while matching or outperforming full-attention Transformers across four datasets (Ma et al., 15 Dec 2025).

Two interpretive cautions follow from this evidence. First, “fused attention” does not identify a single empirical regime: gains may come from multimodal exchange, hierarchical feature refinement, relative-position bias, pathway sparsification, or kernel-level elimination of intermediates. Second, not all reported improvements isolate the fusion block alone. Where ablations are incomplete, the most rigorous conclusion is that FABlocks or FABlock-like modules are effective components within larger systems, not yet universally characterized independent operators.

A plausible synthesis across the cited work is that FABlocks are best viewed as a recurrent architectural pattern: compute attention from jointly accessible evidence, use that attention to mediate selective information transfer, and repeat or compose the process in a way that preserves useful inductive structure. The specific instantiation—CBAM-based dual-stream redistribution, channel/spatial attention over concatenated branches, content-location key fusion, gated sparse pathway fusion, or compiler-discovered fused kernels—depends on the task, representation, and computational constraints.