Hierarchical Convolutional Fusion Transformers

• HCFTs are hybrid deep neural architectures that combine convolutional feature extractors with transformer blocks using hierarchical fusion and cross-attention.
• They leverage dual-branch designs to capture both local details and global dependencies, improving tasks like EEG decoding, image classification, and semantic segmentation.
• HCFT architectures demonstrate performance gains and efficiency, advancing model generalization in resource-constrained and low-data scenarios.

Hierarchical Convolutional Fusion Transformers (HCFT) are a class of hybrid deep neural architectures integrating convolutional and transformer-based modules using hierarchical, multi-stage fusion and cross-attention mechanisms. These models are designed to simultaneously capture local and global, multi-scale representations across diverse domains such as EEG decoding, image classification, fine-grained recognition, and semantic segmentation. The hallmark of HCFT models is a staged architecture in which convolutional feature extractors are intertwined with transformer blocks and specialized fusion operators, allowing efficient learning of hierarchical representations while leveraging both spatial inductive bias and long-range dependency modeling.

1. Architectural Framework

All reported HCFT instantiations share a multi-stage (typically four-stage) encoder design, with parallel convolution and transformer branches whose outputs are recursively aligned and fused at multiple network depths.

• Dual-Branch Convolutional Encoders: For example, in EEG decoding, the input $X \in \mathbb{R}^{B \times C \times T}$ is split into two streams: a temporal branch with 1D separable convolutions and a spatiotemporal branch employing 2D separable convolutions. Each branch independently extracts locally-specialized features, which are subsequently aligned via cross-attention at each stage (Zhang et al., 18 Jan 2026).
• Hierarchical Transformer Fusion: After each convolutional stage, transformer blocks—with intra- and inter-branch (cross) attention—fuse information across branches and scales. Outputs from all stages are concatenated, and a final multi-head self-attention (MHSA) module encodes global relations before the classification or decoding head.
• Fusion Operators: In computer vision HCFTs, fusion transformer blocks (FT-Blocks) or modules such as correlation-weighted fusion (cwF) and adaptive/collaborative knowledge fusion (AKF/CKF) reweight and merge feature maps from convolutional and transformer paths, often using cross-attention to maintain spatial alignment (EL-Assiouti et al., 2024, Xu et al., 2024, Sahoo et al., 2023).

Summarized HCFT Architectural Elements

Branch/Module	Mechanism	Purpose
Dual conv branches	1D/2D sep convs, depthwise + pointwise	Local, multi-axis feature extraction
Transformer fusion	MHSA, cross-attention blocks	Global, long-range modeling
Fusion ops	FT-Block, cwF, AKF/CKF	Hierarchical multi-modal alignment
Normalization	LayerNorm, Dynamic Tanh (DyT)	Training stability, feature decorrelation

2. Attention and Fusion Strategies

HCFTs rely on multi-level attention to enable feature aggregation within and across stages and modalities.

• Self-Attention: Standard MHSA blocks operate within the transformer or spatiotemporal branch, enhancing feature expressivity by capturing intra-stage dependencies.
• Cross-Attention: Each stage features explicit cross-attention from one branch to the other (e.g., temporal $\to$ spatiotemporal in EEG, large-patch $\leftrightarrow$ small-patch in image MFCA), facilitating local-to-global knowledge transfer (Zhang et al., 18 Jan 2026, EL-Assiouti et al., 2024).
• Hierarchical and Coarse-to-Fine Query Fusion: Fine-level queries are adaptively constructed by fusing base fine queries with projected coarse stage representations, using learned scale factors to ensure sub-linear scaling in settings with many subclasses (Sahoo et al., 2023).
• Fusion Modules:
  • FT-Block: Merges low-resolution, high-semantic feature maps with high-resolution, low-semantic maps using multi-head cross-attention and residual convolutions (Sahoo et al., 2023).
  • cwF: Jointly recalibrates convolutional and transformer outputs using global average pooling and sigmoid weighting before summation (Xu et al., 2024).
  • AKF/CKF: Blend predictions from CNN and transformer heads via adaptive, epoch-dependent linear weighting or through a concatenation-projection pathway with dropout and a final classifier, allowing soft or collaborative fusion (EL-Assiouti et al., 2024).

3. Instantiations and Applications

EEG Decoding (Zhang et al., 18 Jan 2026)

• Architecture: Four-stage encoder with CFT blocks, dual-branch convolutional encoding (temporal + spatiotemporal), hierarchical transformer fusion, and MHSA.
• EEG-specific modules: Dynamic Tanh (DyT) normalization supersedes LayerNorm for gradient flow and redundancy reduction.
• Performance: MI decoding—80.83% accuracy (Cohen’s $\kappa$ 0.6165); Seizure prediction—99.10% sensitivity, 0.0236/h FPR.
• Datasets: BCI Competition IV-2b, CHB-MIT.

Fine-Grained Image Classification (Sahoo et al., 2023)

• Pipeline: Two-level hierarchy with a pre-trained DenseNet-169 CNN backbone; FT-Blocks fuse multi-scale feature maps at each hierarchy level.
• Eigen-query initialization: Class queries initialized via principal component decomposition of class means; queries act as cluster centers in feature space.
• Cluster Focal Loss (CFL): Enhances intra-class cohesion and inter-class separation by modulating query-feature similarities.
• CAMP Block: Cross-attention on coarse+fine queries with the backbone’s global prior, reducing error propagation between hierarchy stages.
• Performance: On GroceryStore, 88.43% (coarse), 81.33% (fine), a +10 pp improvement over SOTA baselines.

Semantic Segmentation (Xu et al., 2024)

• Design: Four-stage CNN with Hierarchy-Aware Pixel-Excitation (HAPE) modules, efficient transformer (ET) encoder reducing O($N^2$) cost, and cwF fusion.
• HAPE: Parallel factorized/dilated convolutions with multi-scale receptive fields and spatial pixel-excitation.
• Results: 74.2% mIoU (Cityscapes), 71.1% mIoU (CamVid), with high FPS and compact model size.

Image Classification (EL-Assiouti et al., 2024)

• CTRL-F (HCFT Editor's term): Two-branch design—MBConv-based convolutional backbone in parallel with MFCA (multi-level feature cross-attention transformer).
• MFCA: Processes feature maps from two distinct convolution stages as large- and small-patch token streams, exchanging information via L rounds of bidirectional cross-attention.
• Fusion: Adaptive knowledge fusion (AKF) and collaborative knowledge fusion (CKF) layer-wise combine CNN and transformer outputs.
• Results: Trained from scratch, CTRLF-B + AKF: 82.24% (Oxford-102 Flowers), 99.89% (PlantVillage); both AKF (Oxford) and CKF (PlantVillage) outperform prior hybrids and both pure CNN or pure transformer baselines.

4. Mathematical Formulations

All HCFT variants utilize standard convolutional and attention formulations. Key equations:

Attention (scaled dot-product):

$$\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\left(\frac{Q K^{T}}{\sqrt{d_k}}\right) V$$

Dynamic Tanh Normalization (DyT):

$\mathrm{DyT}(x) = \tanh(\alpha x)$

$\alpha \in \mathbb{R}^{1 \times D}$ is learnable.

Cluster Focal Loss (fine-grained classification):

$$\mathcal{L}_{\mathrm{CFL}}(x) = -\alpha_t (1 - s_t(x))^{\gamma} \log s_t(x)$$

where $s_t(x)$ is the softened similarity between sample $\to$0 and class-$\to$1 prototype query.

Fusion Module (cwF in segmentation):

$\to$2

5. Training Regimes and Practical Considerations

• Optimization: Predominantly AdamW with staged learning rates and cosine scheduling; batch size and epochs tailored per application.
• Normalization strategies: Choice of DyT vs. LayerNorm (LN) must be dataset/task-matched; DyT particularly effective for MI EEG decoding, LN preferable for seizure prediction.
• Parameter and compute efficiency: HCFT models are typically lightweight (e.g., HCFT-S: 10M parameters, 1.43G FLOPs (EL-Assiouti et al., 2024)) to support deployment in resource-constrained settings, such as embedded BCI systems or real-time segmentation.
• Data augmentation: Standard image classification/segmentation augmentations; EEG tasks rely on preprocessing including channel-wise z-score normalization, frequency domain filtering, and epoch selection.

Table: Exemplary HCFT Results Across Domains

Domain	Model Variant	Metric	Result	Reference
EEG Decoding	HCFT	MI accuracy / $\to$3	80.83% / 0.6165	(Zhang et al., 18 Jan 2026)
		Seizure sens. / FPR / spec.	99.10% / 0.0236/h / 98.82%	(Zhang et al., 18 Jan 2026)
Fine-grained Class	HCFT	Fine accuracy	81.33%	(Sahoo et al., 2023)
Image Class.	CTRLF-B + AKF	Top-1 (Oxford-102 Flowers)	82.24%	(EL-Assiouti et al., 2024)
Semantic Seg.	HAFormer (HCFT)	mIoU (Cityscapes/CamVid)	74.2% / 71.1%	(Xu et al., 2024)

6. Ablation, Limitations, and Extensions

Ablation findings:

• Self- and cross-attention removal degrades accuracy by ≥1–2 pp in EEG decoding; both local and global fusion steps are essential for optimal performance.
• Stage depth and embed size tuning impacts FLOPs and accuracy; trade-off curves inform practical model selection.

Limitations:

• Task-specific hyperparameter sensitivity: The optimal normalization scheme, stage depth, and fusion mechanism can be dataset- and domain-dependent, which limits zero-shot transferability (Zhang et al., 18 Jan 2026).
• Data scale and diversity: Small or homogeneous corpora (especially in EEG) impede robust generalization.

Extension pathways:

• Meta-learning and dynamic adaptation: Automated adaptation of stage depth, normalization, or cross-attention structure per subject or task.
• Federated and foundation-model scaling: Large-scale, multi-center corpora and pretraining with cross-task, masked, or contrastive losses.
• Model compression: Quantization, pruning, and distillation for edge deployment.
• Multi-modal and promptable interfaces: Fusion of EEG, EMG, and eye-tracking for richer BCI paradigms.

7. Comparative Impact and Significance

HCFT-based architectures consistently outperform both single-path CNNs and pure transformers of similar capacity, especially in data-limited and resource-constrained conditions. Their ability to hierarchically aggregate multi-scale local and global information makes them especially suited for applications requiring both fine-grained discrimination (as in medical time-series or product classification) and global context (as in scene parsing). Emerging patterns across studies indicate that explicit cross-attention, dynamic branching, and query-based representation learning are essential for reaching strong generalization and efficiency.

Recent HCFT methodologies are positioned as flexible backbones for future work in both classical supervised tasks and large-scale, foundation-model EEG or multimodal biomedical analysis (Zhang et al., 18 Jan 2026, EL-Assiouti et al., 2024, Sahoo et al., 2023, Xu et al., 2024).