HG-TNet (Hybrid) for Histopathology

• The paper presents a hybrid deep learning architecture (HG-TNet) that integrates transformer, CNN, capsule, and graph attention modules for enhanced histopathology classification.
• The model achieves 96% accuracy on the LC25000 dataset and significantly improves class-level metrics in diagnosing colorectal and lung cancers.
• HG-TNet employs dual computational streams to capture global context and fine-grained details, fusing features with cross-attention and graph attention for robust spatial reasoning.

HG-TNet is a hybrid multi-scale deep learning architecture developed to improve classification performance on histopathological images of colon and lung tissues. Its design integrates transformer modules, convolutional neural networks (CNNs), capsule networks, and graph attention mechanisms within a unified framework, explicitly targeting the extraction and fusion of both global contextual and fine-grained spatial features. The architecture is evaluated on the LC25000 dataset, achieving substantial gains in diagnostic accuracy and class-level metrics for colorectal and lung cancer histopathology (Saremi et al., 2 Sep 2025).

1. Architectural Overview

HG-TNet processes each $224 \times 224$ RGB histopathology image through two nearly parallel computational streams. The transformer branch employs a convolution-based patch embedding that divides the image into non-overlapping patches, each mapped to a $d$-dimensional feature vector without explicit positional encoding. These embeddings are then processed by a stack of $L = 4$ transformer encoder blocks, each composed of multi-head self-attention (MHSA), a feed-forward network (FFN), residual connections, and layer normalization. This stream is designed for global context aggregation.

Concurrently, the CNN branch implements a deep convolutional backbone in which every odd-numbered convolutional layer is followed by max-pooling and dropout to capture fine-grained and progressively more abstract local features. Details such as the number of filters, kernel sizes, or exact feature dimensionalities are not specified in the published record.

Feature maps from both branches are concatenated channel-wise in a cross-attention fusion layer and projected with a linear transformation. Capsule networks are integrated post-fusion to preserve spatial relationships and compositionality, although explicit formation or routing details are not included. The fused features are treated as nodes in a graph, passed through a graph attention module (GAT) to re-weight pairwise relations, and finally reduced to a single vector for classification.

2. Transformer Branch and Patch Embedding

The transformer stream begins with a PatchEmbed layer that divides the input image into a grid of non-overlapping patches. Each patch $x_i \in \mathbb{R}^{P \times P \times 3}$ is vectorized and projected into a $d$-dimensional embedding through a learned linear transformation:

$$e_i = W_e \cdot \mathrm{vec}(x_i), \quad e_i \in \mathbb{R}^d$$

Neither the patch size $P$ nor projection matrix shape $W_e$ is published. The absence of explicit positional encoding is noted.

Transformer encoder blocks ($L = 4$) then operate as follows: MHSA computes queries, keys, and values with learnable projection matrices, and self-attention coefficients are defined as:

$$A = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_h}}\right)$$

where $d_h$ is the token dimension per head (not specified). Attention outputs are concatenated, linearly projected, passed through an FFN using the GELU activation, and wrapped with residual and layer-norm structures: $$\begin{align*} \text{FFN}(Z) &= W_2 \cdot \text{GELU}(W_1 Z + b_1) + b_2\ X' &= \text{LayerNorm}(X + Z)\ Y &= \text{LayerNorm}(X' + \text{FFN}(X')) \end{align*}$$

3. CNN Branch and Local Feature Extraction

The CNN branch architecture is described as a deep convolutional hierarchy, with odd-numbered layers followed by max-pooling and dropout regularization to enhance generalization and progressively expand receptive fields. While the sequence of convolutions, activations, and pooling is highlighted, the report does not enumerate kernel sizes, strides, or filter counts. Emphasis is placed on the branch’s role as a “fine-grained detail extractor,” complementing the global context aggregation by the transformer.

4. Feature Fusion, Graph Attention, and Capsule Modules

After their parallel computations, the transformer and CNN branches’ outputs are concatenated and passed through a linear “cross-attention fusion” (Editor’s term) layer:

$$f_\mathrm{fused} = \text{Linear}([f_\mathrm{trans}; f_\mathrm{cnn}])$$

No additional gating, weighting, or attention beyond this projection is present.

The fused features serve as nodes in a graph $G = (V, E)$, constructed so that each node $v_i$ represents a feature vector and edges encode spatial/semantic proximity (not numerically described). Pairwise attention coefficients for nodes $i$ and $j$ are computed by:

$$\alpha_{ij} = \mathrm{softmax}_j(\mathrm{LeakyReLU}(a^T [W h_i \parallel W h_j]))$$

with updated node representations aggregated as:

$$h_i' = \sigma\left(\sum_{j \in \mathcal{N}(i)} \alpha_{ij} W h_j\right)$$

Exact weight shapes, node feature dimensions, and topological specification of $E$ are unreported.

Capsule networks are introduced for spatial structure preservation, though the paper does not specify how capsules are formed, routed, or parameterized. No details on squash functions or routing-by-agreement are given.

5. Auxiliary Objectives and Classification Layer

To encourage robust feature learning, HG-TNet incorporates a self-supervised rotation prediction task. A dedicated rotation head predicts which discrete rotation $$R_k \in \{0^\circ, 90^\circ, 180^\circ, 270^\circ\}$$ was applied to each input. The auxiliary loss is:

$$\mathcal{L}_\mathrm{rot} = -\sum_{k=1}^K y^k \log p^k$$

where $K = 4$. The overall objective combines classification and rotation prediction:

$$\mathcal{L}_\mathrm{total} = \mathcal{L}_\mathrm{class} + \lambda \mathcal{L}_\mathrm{rot}$$

The weighting parameter $\lambda$ is not specified.

After graph attention and pooling, the resulting feature vector is passed through layer normalization, dropout, a fully connected layer, and softmax to yield final class probabilities.

6. Benchmark Results and Evaluation

HG-TNet is evaluated on the LC25000 dataset, containing five classes spanning colon and lung subtypes. The reported performance is summarized as follows:

Metric	Value
Overall Accuracy	96.0%
Macro Precision/Recall/F1	0.96 (each)
Class-wise AUC	0.98–0.99 (see below)

Class-wise confusion matrix recalls:

• colon_aca: 94.8%
• colon_n: 98.0%
• lung_aca: 94.4%
• lung_n: 100.0%
• lung_scc: 90.8%

Class-wise AUC values: [0.98, 0.99, 0.97, 0.99, 0.99].

7. Design Implications and Practical Considerations

HG-TNet’s architecture is characterized by its synergy of global and local feature modeling, graph-structured relational reasoning, and spatial hierarchy preservation. The absence of explicit hyperparameterization details (e.g., patch size, capsule routing) precludes direct replication without informed selection of standard values from related architectures. The modular design allows adaptation and extension to other histopathological analysis tasks contingent upon empirical validation. The inclusion of a rotation prediction auxiliary task indicates an empirical strategy to enforce equivariant, augmentation-robust representations.

The reported gains over standard methods reinforce the efficacy of hybrid graph-transformer pipelines in computational pathology. A plausible implication is the potential for transfer to other domains requiring multi-scale, structured, and context-aware feature aggregation (Saremi et al., 2 Sep 2025).