Semantic-Aware Aggregation Module

• SAAM is a hypergraph-based aggregation method that groups local transformer tokens into semantically coherent regions for effective fine-grained classification.
• It dynamically constructs weighted hypergraphs using learnable semantic prototypes and applies a two-step hypergraph convolution to refine token embeddings.
• Empirical evaluations show SAAM improves top-1 accuracy on datasets like CUB-200-2011 and Stanford-Dogs by up to 4.1%, outperforming traditional region proposal methods.

The Semantic-Aware Aggregation Module (SAAM) is a hypergraph-based aggregation mechanism introduced within the H³Former architecture for fine-grained visual classification. SAAM addresses key limitations in previous feature localization and region proposal methods, specifically their inability to comprehensively capture discriminative cues and their propensity for introducing category-agnostic redundancy. By dynamically constructing a weighted hypergraph structure atop transformer tokens, SAAM enables the aggregation of local features into a small number of semantically coherent region-level descriptors, facilitating the modeling of high-order semantic dependencies among tokens (Zhang et al., 13 Nov 2025).

1. Conceptual Role within H³Former

SAAM operates on the visual tokens extracted by a multi-stage Swin Transformer backbone, specifically targeting the patch-level embeddings from its final stage. Its primary objective is to adaptively group (aggregate) these local tokens into a limited set of semantically meaningful regions. Each region, interpreted as a hyperedge within a token–region hypergraph, serves to (a) model high-order correlations among multiple tokens, (b) facilitate message passing through a hypergraph convolution procedure, and (c) output refined token embeddings enriched with discriminative, part-aware cues relevant for fine-grained categorization. SAAM thus forms the core token-to-region aggregation mechanism, converting transformer outputs into representations suitable for structured region-level modeling (Zhang et al., 13 Nov 2025).

2. Dynamic Weighted Hypergraph Construction

SAAM’s aggregation is formalized as a dynamic, learnable hypergraph over transformer tokens. The construction process comprises token and prototype definition, context extraction, and assignment computation:

• Nodes ($V$): Set of $N$ visual tokens $X \in \mathbb{R}^{N \times C}$ obtained from the final transformer stage.
• Hyperedges ($E$): $M$ learnable semantic prototypes $K_m \in \mathbb{R}^{d_k}$, each encoding a distinct contextual pattern (e.g., "wing texture," "beak shape").
• Prototype Initialization: For each backbone stage $s=1...S$, SAAM extracts three context vectors via (i) average pooling + linear, (ii) max pooling + linear, and (iii) attention-weighted pooling. The resulting $3S$ context vectors are projected and concatenated ($F \in \mathbb{R}^{3S \times C}$), then split into $M$ groups and passed through an MLP, each sum with a learnable bias, yielding $K_m = \phi(F_{(m)}) + P_m$.
• Participation Weights (Incidence Matrix): Token features are projected to the prototype space: $Q = XW_q$. Soft assignment $A_{i, m}$ (incidence of token $i$ in hyperedge $m$) uses a scaled dot product with softmax:

$$A_{i, m} = \frac{\exp(Q_i^\top K_m / \sqrt{d_k})}{\sum_{m'=1}^M \exp(Q_i^\top K_{m'} / \sqrt{d_k})}$$

Tokens can participate in multiple hyperedges with different strengths, defining a soft, adaptive hypergraph structure (Zhang et al., 13 Nov 2025).

3. Two-Step Hypergraph Convolution

SAAM applies a two-phase (node-to-hyperedge and hyperedge-to-node) message passing scheme—hypergraph convolution—as follows:

• Node$\to$Hyperedge Aggregation: Region features are computed by weighted sum:

$$H_e = A^\top X W_e, \quad H_e \in \mathbb{R}^{M \times C}, \quad W_e \in \mathbb{R}^{C \times C}$$

• Hyperedge$\to$Node Update: Token features are updated using refined region-level cues:

$$X' = A H_e W_v, \quad W_v \in \mathbb{R}^{C \times C}$$

• Residual Gating: Each token receives a learnable gate $g \in \mathbb{R}^N$ and the final output is given by:

$\widehat{X} = X + (g \odot X')$

where $\odot$ denotes channel-wise broadcasting multiplication.

This mechanism enables SAAM to model many-to-many (high-order) relationships among tokens and aggregate contextualized semantic information far beyond pairwise graph structures (Zhang et al., 13 Nov 2025).

4. Multi-Scale Contextual Prototype Extraction

Prototypes $K_m$ are not limited to local or high-level cues; they are initialized with information from every Swin Transformer stage. For $S=4$ backbone stages, each with average, max, and attention-pooling, $3S$ context vectors are computed, projected, concatenated, and divided into $M$ channel-wise groups. This architecture ensures that region prototypes integrate fine details (texture), intermediate patterns (part shape), and global compositions (layout), supporting feature aggregation that is sensitive to both coarse and fine hierarchical structures prevalent in fine-grained visual categories (Zhang et al., 13 Nov 2025).

5. Region-Level Representation and Read-out

The aggregated region features $H_e$ (of shape $M \times C$) are utilized as compact, semantically meaningful descriptors for downstream processing. These $M$ region-level vectors can either serve as direct input to next-stage modules—such as the Hyperbolic Hierarchical Contrastive Loss (HHCL)—or be further pooled into a global feature for classification. This hierarchical read-out preserves semantically crucial information discovered by the soft, prototype-driven aggregation mechanism (Zhang et al., 13 Nov 2025).

6. Architectural Specifications

Key architectural and hyperparameter settings in SAAM are as follows:

Parameter	Description	Value in H³Former
Number of hyperedges ($M$)	Number of semantic prototypes / regions	16
Prototype dim ($d_k$)	Dimension of token/prototype embeddings	Equal to $C$ (1024)
Backbone stages ($S$)	Multi-scale context sources	4 (Swin Transformer)
Gate ($g$)	Per-token, learned gating scalar	$\mathbb{R}^N$
Embedding dims per stage	Transformer channel width per stage	{128, 256, 512, 1024}
MLP hidden dims	Hidden layer sizes for MLP $\phi$	{512, 1024, 2048, 4096}
Transformer heads	Per-stage multi-head self-attn. configuration	{4, 8, 16, 32}
Weight matrices ($W_q, W_e, W_v$)	Learnable projections	$W_q \in \mathbb{R}^{C \times d_k}$, $W_e, W_v \in \mathbb{R}^{C \times C}$

These architectural details determine the SAAM’s representational capacity, its ability to encode multi-scale region prototypes, and the flexibility of the aggregation process (Zhang et al., 13 Nov 2025).

7. Empirical Impact and Comparative Analysis

Ablation studies on CUB-200-2011 and Stanford-Dogs datasets quantify the empirical contribution of SAAM:

Variant	CUB-200-2011 (%)	Stanford-Dogs (%)
Baseline (no SAAM, no HHCL)	90.9	91.1
+HHCL only	91.2	92.6
+SAAM only	92.5	95.2
+SAAM + HHCL (full)	92.7	95.8

Replacing SAAM with a standard graph neural network (GNN) or a fixed hypergraph (HGNN) results in a 0.4–0.5% decrease in top-1 accuracy, confirming that the soft, prototype-guided hypergraph construction in SAAM is crucial for capturing the high-order, fine-grained semantics required for discriminative visual categorization. The introduction of SAAM alone yields a 1.6% (CUB) to 4.1% (Dogs) increase in accuracy over the baseline. This suggests that semantic-aware, learnable aggregation via soft hyperedges substantially improves local-region discrimination compared to both prior token selection and region proposal strategies (Zhang et al., 13 Nov 2025).