MFAVBs: Enhancing Vision Transformer Fusion

• MFAVBs are modular blocks that fuse features from dual streams using shared ViT encoders to promote inter-view complementarity and enhanced unsupervised learning.
• They integrate local window self-attention, global down-sampling, and multi-scale attention fusion to capture both fine-grained details and global contextual information.
• Empirical results indicate improved classification accuracy and accelerated convergence, boosted further by explicit cross-branch interactions and optional CLIP token augmentation.

Multiple Fusing-Augmenting ViT Blocks (MFAVBs) are a family of architectural modules designed to enhance feature representation and fusion in Vision Transformers (ViTs). MFAVBs operate by systematically fusing features from multiple input augmentations, propagating shared and complementary representations across stacked blocks, and, when applied to contrastive clustering, introducing explicit cross-branch interaction coupled with semantic anchors—such as CLIP tokens—enabling improved unsupervised visual understanding and clustering.

1. Block Structure and Design Principles

MFAVBs are founded on a dual-stream processing paradigm wherein two input paths, typically produced via separate data augmentations, are encoded in parallel by shared-weight ViT blocks. The outputs from these branches are concatenated along the token dimension to form an expanded sequence, which is then processed by a larger ViT block ("augmenting") before being split again into two branches for the next MFAVB layer. This iterative fusion and augmentation mechanism enables inter-view complementarity, explicit token-level context sharing, and deeper feature mixing.

In the context of MAFormer (Wang et al., 2022), the fundamental block architecture consists of:

• A Local Window-Attention branch performing multi-head self-attention within non-overlapping windows to aggregate fine-grained representations.
• A Global Learning with Down-sampling (GLD) branch applying a compressive fully-connected transformation to flatten tokens, reducing sequence length and capturing long-range global context.
• A Multi-scale Attention Fusion (MAF) module injecting cross-attention between streams, allowing local features to attend to compressed global tokens, followed by output projections and standard transformer MLPs.
• Stacked block organization, where these operations repeat across layers, increasing representational power.

When applied to contrastive clustering (Wang et al., 12 Nov 2025), MFAVBs consist of repeated pairs of shared-weight ViT encoders, explicit concatenation and augmenting, and explicit re-split operations, handling positive sample pairs through multiple fusion-augment cycles.

2. Mathematical Formulation and Data Flow

The MFAVB composite block within MAFormer can be described in terms of token-wise shape and operation:

1. Input preprocessing: Reshape $X^{\ell-1} \in \mathbb{R}^{H\times W\times C}$ to $X_{\text{flat}} \in \mathbb{R}^{L\times C}$, $L = H \cdot W$.
2. LayerNorm: $Z = \text{LN}(X_{\text{flat}})$.
3. Local Window Self-Attention: Partition $Z$ into windows $(s=M^2)$,

$$A_{\text{local}}(X) = \text{softmax}(QK^T / \sqrt{d})V$$

where $Q = XW_Q$, $K = XW_K$, $V = XW_V$, $d = C$.

1. Global Learning with Down-sampling:
   • Flatten and compress: $$X_G^{\text{mid}} = W_{\text{ds}} \cdot X_G^{\text{in}}$$, $W_{\text{ds}} \in \mathbb{R}^{L \times N\cdot L}$.
   • Add position embeddings: $$X_G = (X_G^{\text{mid}})^T + \text{PosEmb}(N\cdot L, H, W)$$.
2. Multi-scale Attention Fusion:

$$A_{\text{fuse}} = \text{softmax}(Q_L K_G^T / \sqrt{d}) V_G$$

where $Q_L = X_L W_Q^L$, $K_G = X_G W_K^G$, $V_G = X_G W_V^G$. - Residual addition and projection culminates in the output token sequence and is followed by MLP.

In MFAVBs for contrastive learning, the main steps are:

• For $N$ stacked blocks, separately encode positive pairs ($y^a, y^b$), explicitly concatenate $(y^{f}_i = \text{Cat}(y^a_i, y^b_i))$, augment with a larger ViT $(y^{f}_{i,\text{out}})$, then split back $(y^a_{i+1}, y^b_{i+1})$.
• CLS tokens $(h^a, h^b)$ are extracted for downstream contrastive objectives.

A table summarizing the main sub-blocks:

Sub-block	Operation / Formula	Output Shape
Window-SA	$A_{\text{local}}$	$\mathbb{R}^{L\times C}$
GLD Down-sampling	$X_G^{\text{mid}} = W_{\text{ds}} X_G^{\text{in}}$	$\mathbb{R}^{C\times N\cdot L}$
Fusion MAF	$$A_{\text{fuse}} = \text{softmax}(Q_L K_G^T/\sqrt{d}) V_G$$	$\mathbb{R}^{L\times d}$

3. Training Objectives and Contrastive Projections

MFAVBs applied to contrastive clustering employ joint instance-level and clustering-level objectives. After feature fusion and augmenting, the branch CLS tokens are projected via dedicated MLP "projection heads" for contrastive learning:

• Instance-level projection: $P_I(\cdot)$ MLP; InfoNCE loss is defined for each view $a$:

$$L_I^a = -\frac{1}{B} \sum_{i=1}^B \log \left( \frac{\exp(\text{Sim}(I^a_i, I^b_i)/\tau)}{\sum_{j\neq i}\exp(\text{Sim}(I^a_i,I_j)/\tau)} \right)$$

• Clustering-level projection: $P_C(\cdot)$ MLP; InfoNCE is performed on the softmax cluster probabilities.
• Total objective: $L = L_I^a + L_I^b + L_C^a + L_C^b$, backpropagated across all blocks.

Notably, contrastive losses are optimized jointly for instance discrimination and cluster assignment, leveraging the fused and augmented features.

4. CLIP-Pretrained Token Augmentation

MFAVBs optionally incorporate frozen CLIP [CLS] tokens at the input sequence, serving as "multimodal anchors." This augmentation:

• Presents a semantic vector $c^0 \in \mathbb{R}^E$ prepended to the token sequence $(P+2)$, enhancing global representation from early layers.
• Participates naturally in multi-head attention, residual update, and LayerNorm, thereby implicitly conditioning the learned representation.
• Empirically, the presence of CLIP tokens yields higher unsupervised clustering performance across multiple benchmarks.

A plausible implication is that inclusion of semantic anchors modulates attention statistics, improving the robustness and semantic alignment of learned features.

5. Implementation Hyperparameters and Complexity

Canonical MFAVBs instantiations specify:

• Encoder: ViT-Small (embedding dim $E=512$, heads $h=8$).
• Depth: 8 base ViT layers, organized in 4 MFAVBs (each spans 2 layers).
• Sequence length: $T=P+2=51$ tokens.
• Optimization: AdamW (lr=$3\cdot10^{-4}$, weight decay=$1\cdot10^{-4}$, cosine annealing).
• Batch size: $128$; epochs: $500$.
• Data augmentations: multiple geometric and color transformations including Solarize in some branches.
• Block parameter count: for MAFormer, per block $\approx$ $16d^2$; local SA $4d^2$, GLD $C\cdot L\cdot N\cdot L$, fusion $4d^2$, MLP $8d^2$.
• FLOPs per block: Local $O(L\cdot M^2 \cdot d)$, GLD $O(NCL^2)$, Fusion $O(LL_{\text{global}}d)$, MLP $O(Ld4d)$.

Efficiency analysis for MFAVBs-CC (Wang et al., 12 Nov 2025) indicates only $\sim$25% per-epoch runtime increase despite doubling tokens within blocks, but convergence occurs within 60% of normal epochs, leading to net 20–30% training speedup. GPU memory overhead remains tractable ($\sim$1.8 GB at batch size 128 on 32 GB V100).

6. Empirical Performance and Ablation Findings

MFAVBs achieve state-of-the-art results in both supervised and unsupervised settings.

• ImageNet-1K classification (Wang et al., 2022):
  • Top-1 accuracy: MAFormer-L 85.9% (vs. LV-ViT-L 85.3%, CSWin-B 84.2%)
  • Ablations confirm local-global fusion improves accuracy (e.g., GLD + CSWin local yields +1.0%).
• Object detection / segmentation (MSCOCO, Mask R-CNN):
  • MAFormer-L box AP 50.7, mask AP 45.4 (both $>$ CSWin-B).
• Contrastive clustering (Wang et al., 12 Nov 2025):
  • 7 datasets: ACC gain +0.087, NMI +0.072, ARI +0.098 over prior VTCC backbone.
  • CLIP token and MFAVBs contribute additively (ACC +0.067 from MFAVBs only, +0.097 from CLIP only, best when combined).
  • Robust under 30–50% token masking (ACC drop only $\sim$1%).

Ablation studies demonstrate that fusion by explicit co-attention and token splitting is superior to alternatives (e.g., one-way fusion or DS-Net style) by up to +0.2% accuracy, and that both local and global branches contribute distinct, complementary gains.

7. Applications, Limitations, and Outlook

MFAVBs are applicable as backbone modules for tasks demanding nuanced local and global visual representation—classification, detection, segmentation, and unsupervised clustering. Explicit fusion-augment cycles are particularly beneficial for contrastive learning, where intermediate feature mixing enhances cluster separation.

A subtle limitation arises from increased token sequence length and memory overhead, although training runtime is mitigated by accelerated convergence. MFAVBs generalize across domains (e.g., remote sensing, standard vision datasets), suggesting broad transferability.

Future directions plausibly include exploring adaptive fusion and augment strategies, integrating richer multimodal anchors, and extending MFAVBs to multimodal or temporal ViT architectures, particularly where fine-grained view complementarity or cross-modal fusion is required.