Hierarchical Convolutional Fusion Transformer

• HCFT is an architectural paradigm that fuses local convolutional features with hierarchical Transformer attention to model complex, structured data across modalities.
• It leverages multi-scale fusion and label-query decoding to achieve superior classification accuracy, evidenced by up to 88.43% accuracy in vision tasks.
• Its hierarchical structure and adaptive normalization support scalable processing of large label spaces while enhancing interpretability through attention heatmaps.

A Hierarchical Convolutional Fusion Transformer (HCFT) is an architectural paradigm that systematically combines multi-scale convolutional representations with hierarchical Transformer-based attention for superior modeling of structured data, particularly in fine-grained classification and multimodal decoding tasks. Integrating local feature priors from convolutional backbones with global context extracted via hierarchically arranged attention mechanisms, HCFTs achieve more expressive, context-aware representations than standard CNN or vanilla Transformer models. This entry synthesizes the design principles, algorithmic innovations, and empirical results of the HCFT family, referencing canonical vision and EEG applications, and highlighting the distinct strategies that separate HCFTs from prior fusion architectures (Sahoo et al., 2023, Huo et al., 2022, Zhang et al., 18 Jan 2026).

1. Architectural Blueprint and Theoretical Underpinnings

HCFTs are built upon a hybrid backbone where convolutional feature extractors operate in parallel with Transformer-based global modeling, and dedicated fusion modules unify these branches at multiple semantic hierarchies. The canonical HCFT pipeline comprises:

• Convolutional backbone: Typically a pre-trained, off-the-shelf network such as DenseNet-169 extracts multi-scale feature maps at decreasing spatial resolutions, e.g., $$\{56{\times}56, 28{\times}28, 14{\times}14, 7{\times}7\}$$, with all outputs projected to a shared embedding dimension via $1{\times}1$ convolution.
• Fusion Transformer (FT) blocks: At each hierarchy level, feature maps are fused in a coarse-to-fine sequence. A lower-resolution, semantically-rich feature is treated as the query, while the higher-resolution map serves as key/value, forming cross-scale attention. Formally,

$$\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d}} + P_E\right)V,$$

where $P_E$ denotes positional encoding.

• Hierarchical label-query decoding: After multi-scale fusion, a Transformer-style decoder attends to the fused representations using per-class queries. These queries are either learned or constructed via class-wise covariance eigen-decomposition ("Eigen-queries"), supporting scalable classification across large and hierarchical taxonomies.
• Stage-wise hierarchical query scaling: Fine-class queries inherit structure from their coarse-level parents, yielding

$$Q_\mathrm{fine} = \alpha Q_\mathrm{coarse} + (1-\alpha)Q_\mathrm{init},$$

with a trainable mixing factor $\alpha$.

This architecture is extended to different domains with task-specific adaptations: e.g., for EEG decoding, dual-branch convolutional encoders separately extract temporal and spatiotemporal features, followed by attention-driven cross-branch fusion (Zhang et al., 18 Jan 2026); in medical image classification, parallel local (CNN) and global (Transformer) feature streams are fused via adaptive attention and shortcut mechanisms (Huo et al., 2022).

2. Algorithmic Components and Mathematical Formalism

Principal HCFT modules are formalized as follows:

• Fusion Transformer block: For each pair of source features $X$ (low-res) and $Y$ (high-res),

$$Q_h = XW_q^h, \quad K_h = YW_k^h, \quad V_h = YW_v^h,$$

$$A_h = \mathrm{softmax}\left(\frac{Q_h K_h^T}{\sqrt{d/H}} + B_h\right), \quad \mathrm{head}_h = A_h V_h,$$

with multiple heads concatenated and projected.

• Label-query initialization via PCA: For each class $c$,

$$\mu_c = \frac1{N_c}\sum_{j=1}^{N_c} x_j, \qquad \Sigma_c = \frac1{N_c}\sum_{j=1}^{N_c} (x_j-\mu_c)(x_j-\mu_c)^T,$$

and the query is assembled as a weighted sum of top eigenvectors of $\Sigma_c$.

• Cluster-based (Fisher-style) loss: For an input $x$ and class $i$, cosine similarity $S_i(x)$ and Cluster Focal Loss (CFL) are:

$$S_i(x) = \frac{f(x) \cdot Q(i)}{\|f(x)\|\|Q(i)\|}, \qquad p_i(x) = \frac{e^{\tau S_i(x)}}{\sum_j e^{\tau S_j(x)}},$$

$$\mathcal{L}_\mathrm{CFL} = -\sum_{i=1}^C \alpha_i (1-p_i(x))^\gamma y_i \log p_i(x),$$

with $\gamma$ the focusing parameter.

• CAMP block: Concatenated hierarchical queries and a global-prior feature interact via dot-product cross-attention, and results are regularized with binary cross-entropy.

Analogous cross-attention and multi-stage feature aggregation mechanisms are found in EEG (Zhang et al., 18 Jan 2026) and medical imaging (Huo et al., 2022) HCFTs, often with branch-specific normalization and spatiotemporal reweighting modules.

3. Hierarchical Query Embeddings and Scalability

The hierarchical query mechanism is foundational for handling large label spaces and structured taxonomies. Queries for fine classes are constructed as a trainable combination of learned fine-level embeddings and inherited, projected coarse parent embeddings, controlled by a learned scalar and logical "mask" to accommodate class tree structure. This enables simultaneous coarse and fine prediction and mitigates error propagation between hierarchy levels. Eigen-Query initialization facilitates well-separated embeddings in the learned space, whereas the cluster-based loss further encourages discriminative query structure.

4. Adaptive Fusion and Attention Mechanisms

Multi-stage adaptive fusion—across both spatial scales and semantic hierarchies—is a distinguishing feature of HCFTs:

• Local/global fusion: Local branch CNNs capture short-range patterns; global Transformer blocks model long-range dependencies. Typically, windowed and shifted-window self-attention (W-MSA, SW-MSA) alternate within global blocks to restrict computational complexity while maintaining contextual range.
• Adaptive Hierarchical Feature Fusion (HFF): These modules sequentially apply channel attention to global features, spatial attention to local features, and then concatenate and transform them (plus previous-stage fused outputs) via inverted residual MLPs. This composite output serves as the seed for the next fusion stage (Huo et al., 2022).
• Cross-branch and self-attention: In multimodal tasks (e.g., EEG), cross-attention modules enable information flow between disparate feature branches (temporal and spatiotemporal), augmented by specialized normalization such as Dynamic Tanh (DyT) to maintain gradient stability and highlight salient activity (Zhang et al., 18 Jan 2026).

5. Training, Hyperparameters, and Optimization

HCFTs are trained end-to-end using a combination of standard and bespoke losses tailored to the semantic granularity of the task. In vision applications, coarse-level predictions are supervised with cross-entropy, while fine-level queries employ CFL to enhance inter-class separation. For EEG, binary or multiclass cross-entropy is adopted, with adaptation to the specific task such as motor imagery or seizure detection. Key optimization settings include Adam or AdamW, batch sizes of 32–64, learning rates $10^{-4}$ to $10^{-3}$, and dropout in fusion modules. Hierarchical structures are favored for their modularity and impose only moderate computational overhead (e.g., $\sim8$ GFLOP for HiFuse-Tiny variant) (Huo et al., 2022).

6. Empirical Performance and Ablation Insights

HCFTs have demonstrated state-of-the-art accuracy in multiple domains.

• Vision/fine-grained classification: On the GroceryStore dataset, HCFT achieves coarse accuracy of 88.43% and fine accuracy of 81.33%, surpassing prior art by $\sim10\%$ absolute margin (Sahoo et al., 2023).
• Medical imaging: On ISIC2018, HiFuse-Base reaches 84.12% accuracy, outperforming baseline Transformer and hybrid models (Huo et al., 2022).
• EEG decoding: On BCI IV-2b, HCFT attains 80.83% ± 7.61% accuracy and Cohen's $\kappa=0.6165$; on CHB-MIT, it reaches 99.10% sensitivity with a false positive rate of 0.0236/h (Zhang et al., 18 Jan 2026).

Ablation studies consistently show that hierarchical query fusion, attention-driven fusion blocks, and adaptive normalization provide nontrivial additive gains. Removal of cross-attention, self-attention, or hierarchical fusion blocks reduces performance by 1–3% (absolute), with attention to the cross-branch modules highlighted as most impactful.

7. Broader Applications, Limitations, and Prospects

HCFTs generalize to diverse data modalities, including vision, EEG, and multimodal fusion:

• Potential applications: Robust fine-grained object recognition in e-commerce, motor imagery-based prosthetics and BCI, seizure prediction in ambulatory EEG, and medical classification with complex spatial context.
• Interpretability: Attention heatmaps and UMAPs of learned queries reveal progressive focusing and well-separated class clusters.
• Limitations: Task-specific calibration (stage depths, normalization selection) may hinder plug-and-play deployment, and multi-stage attention incurs moderate compute costs for real-time or edge applications.
• Future trajectory: Scaling to larger, pre-trained HCFT "foundation models" for signal understanding, leveraging meta-learning for cohort adaptation, or model compression for deployment, appears plausible as a research direction (Zhang et al., 18 Jan 2026).

HCFTs thus represent a rigorous, extensible design paradigm for leveraging hierarchical structure in both representation and label space, with strong empirical validation across modalities and fine-grained tasks.