H3Former: Hypergraph FGVC Framework

• H3Former is a framework for fine-grained visual classification that integrates hypergraph semantic-aware aggregation with hyperbolic hierarchical embedding.
• It employs the Semantic-Aware Aggregation Module (SAAM) to dynamically construct weighted hypergraphs, capturing high-order token dependencies.
• The Hyperbolic Hierarchical Contrastive Loss (HHCL) utilizes non-Euclidean geometry to preserve semantic hierarchies, significantly improving classification accuracy.

H3Former is a framework for fine-grained visual classification (FGVC) that addresses the challenges of subtle inter-class differences and substantial intra-class variations by integrating hypergraph-based semantic-aware aggregation and hyperbolic hierarchical embedding. It introduces the Semantic-Aware Aggregation Module (SAAM) to dynamically construct weighted hypergraphs from transformer token features, capturing high-order semantic dependencies, and the Hyperbolic Hierarchical Contrastive Loss (HHCL) to enforce semantic relationships in a non-Euclidean space. This architecture aims to generate region-level representations that are both highly discriminative and structured according to semantic hierarchy (Zhang et al., 13 Nov 2025).

1. Architectural Overview

The H3Former pipeline begins with a Swin-B Transformer backbone that processes an input image, resized to $448\times448$, through four hierarchical stages, producing sets of patch-tokens $X_s$ for $s=1..4$. These representations serve as the basis for a token-to-region aggregation process.

The subsequent SAAM performs multi-scale context generation: at each stage, representations are extracted using average pooling, max pooling, and an attention-weighted mechanism, each projected to a common dimension and concatenated. The composite feature $F\in\mathbb{R}^{3S\times C}$ is then subdivided into $M$ groups and embedded to yield $M$ semantic prototypes $K_m$. Final-stage token features are linearly projected and soft-assigned to these prototypes, creating a weighted hypergraph where nodes are tokens and hyperedges are associated with semantic regions.

Hypergraph convolution is implemented as a two-step message passing:

• Node-to-hyperedge: $H_e = A^\top X W_e$
• Hyperedge-to-node: $X' = A H_e W_v$

where $A$ is the soft-assignment, $W_e$ and $W_v$ are learned matrices, and gating and residual fusion yield refined token features $\hat{X}$. The final pooled, region-aggregated representation is classified using a fully-connected layer, and a combined objective $L_{total} = L_{CE} + \alpha L_{HHCL}$ is applied.

2. Semantic-Aware Aggregation Module (SAAM)

SAAM centrally addresses region-level feature extraction via dynamically constructed hypergraphs:

• For every Swin stage, three contextual vectors—$f^{\text{avg}}_s$, $f^{\text{max}}_s$, $f^{\text{attn}}_s$—are computed via linear projections of average pooled, max pooled, and attention-weighted features. These vectors are concatenated across stages for comprehensive context.
• $F$ is split among $M$ groups, embedded and combined with learnable prototypes $P_m$ to form semantic keys $K_m$.
• Tokens $X\in\mathbb{R}^{N\times C}$ are projected to queries $Q$ and soft-assigned to prototypes via normalized dot product, forming the incidence matrix $A\in\mathbb{R}^{N\times M}$. Each token may belong to several regions, representing high-order semantic dependencies among parts.
• Hypergraph convolution propagates information via "V→E→V" message passing, with learned gates facilitating residual refinement of token features.
• The output hyperedge features $H_e$ act as compact, region-level descriptors, subsequently feeding the hierarchical contrastive structure.

This module enables end-to-end learning of discriminative, context-rich region groupings without explicit region proposals.

3. Hyperbolic Hierarchical Contrastive Loss (HHCL)

HHCL introduces a contrastive learning framework grounded in hyperbolic geometry to reflect semantic hierarchies:

• Motivated by the exponential expansion property of the Lorentz model, which naturally embeds hierarchical structures like visual taxonomies, region-level features (the $M$ hyperedge descriptors) are treated as leaves in a hierarchy.
• Each feature $z\in\mathbb{R}^d$ is mapped into Lorentzian space via $exp_0(z) = (\sqrt{1+\|z\|^2}, z)$ in $\mathbb{R}^{d+1}$.
• Pairwise distances combine Euclidean and hyperbolic metrics: $$D_{ij} = \|z_i-z_j\|_2 + \lambda d_L(exp_0(z_i), exp_0(z_j))$$, with $d_L$ the Lorentzian distance.
• A supervised contrastive loss is computed at every hierarchy level $\ell$ by aggregating over positive and negative pairs, encouraging compact intra-class and separable inter-class clusters.
• A partial-order preservation loss, $L_{hpop}$, enforces that each parent node in the hierarchy is close, in hyperbolic space, to its children: $$L_{hpop} = (1/(L-1)) \sum_{\ell=1}^{L-1} \text{ReLU}(d_L(exp_0(H^{\ell+1}_i), exp_0(H^\ell_i)))$$.
• The total loss is the sum of contrastive and partial-order terms: $L_{HHCL} = L_{con} + \beta L_{hpop}$, ensuring that both taxonomic and discriminative constraints are jointly met.

4. Implementation and Training Procedures

H3Former is implemented in PyTorch and trained on a single NVIDIA A100 GPU. The Swin-B backbone is pre-trained on ImageNet-22K or ImageNet-1K, depending on the downstream dataset. Key configuration parameters include:

• Input size $448 \times 448$; token dimensions $[128, 256, 512, 1024]$; 12 transformer layers with heads $[4, 8, 16, 32]$.
• Number of SAAM hyperedges $M=16$.
• HHCL hyperbolic curvature $\mathcal{K}=0.1$, temperature $\tau=0.1$, geometry weight $\lambda=1.0$, hierarchy weight $\beta=0.1$, $L=4$ hierarchy levels, and loss balance $\alpha=1.0$.
• Optimization proceeds with AdamW (learning rate $\sim$1e-4, weight decay 0.05), 100 epochs, and batch size 32.

Best performance is observed with loss-component weights set to $(0.1, 0.1, 0.1)$ for $$(\mathcal{L}_{hcon}, \mathcal{L}_{econ}, \mathcal{L}_{hpop})$$, $M=16$, and $\mathcal{K}=0.1$.

5. Experimental Results and Comparative Analysis

H3Former is evaluated on CUB-200-2011, NA-Birds, Stanford Dogs, and Oxford Flowers-101 under standard splits, achieving superior or state-of-the-art Top-1 accuracy:

Dataset	H3Former	Best Prior (Method)
CUB	92.7%	91.9% (SR-GNN), 91.8%(IELT)
NA-Birds	91.6%	91.4% (ACC-ViT)
Stanford Dogs	95.8%	93.6% (ViT-Net)
Oxford Flowers	99.7%	99.0% (I2-HOFI)

Ablation experiments show:

• Removal of both SAAM and HHCL degrades performance (CUB: 90.9%, Dogs: 91.1%).
• Implementing only HHCL yields CUB 91.2%, Dogs 92.6%; only SAAM gives 92.5%/95.2%.
• The complete H3Former (with both modules) attains the highest accuracy.

Deviation from optimal loss-component weights or hyperparameters results in reduced performance, indicating the necessity of precise calibration for maximum benefit.

6. Qualitative Evaluation and Interpretability

Qualitative analysis involves visualization of the $M=16$ hyperedges as spatial activation maps. Hyperedges are consistently associated with semantically meaningful image regions (e.g., bird beak, wing, tail, eye) across diverse samples and are robust to pose and background variation. Models trained with alternative hyperbolic (non-HHCL) losses exhibit less coherent or overlapping region groupings, whereas HHCL ensures tight alignment between learned groups and annotated parts.

t-SNE visualizations demonstrate that HHCL increases feature cluster compactness and inter-class separation, while SAAM sharpens decision boundaries. Their combination produces well-separated, low-variance feature clusters.

7. Significance, Limitations, and Future Directions

H3Former introduces a principled integration of hypergraph-based region modeling and hyperbolic contrastive embedding for FGVC, producing structured, interpretable, and highly discriminative representations. The synergy of SAAM and HHCL leads to improvements in both accuracy and semantic region discovery, with demonstrated transfer across multiple benchmark datasets.

A plausible implication is that this framework could extend to other tasks requiring structured semantic grouping (e.g., part discovery, hierarchical retrieval, and taxonomic visual understanding). The design relies on precise hyperparameter tuning and batched supervision at hierarchical levels. Addressing scalability to larger taxonomies or improving computational efficiency under extremely fine-grained regimes represents possible avenues for future research (Zhang et al., 13 Nov 2025).