SemaMIL for WSI Classification

• SemaMIL is a multiple instance learning framework that semantically reorders whole slide image patches to optimize contextual modeling.
• It integrates a Semantic-guided Retrieval State Space Module that conditions state transitions on representative patch queries.
• The framework overcomes attention and Transformer limitations, achieving state-of-the-art accuracy with reduced computational cost.

Searching arXiv for SemaMIL and closely related WSI MIL/SSM papers. SemaMIL is a multiple instance learning framework for whole slide image classification that combines a semantically structured sequence construction step with retrieval-guided state space modeling. It was introduced as “Semantic Reordering with Retrieval-Guided State Space Modeling for Whole Slide Image Classification” (Gan et al., 30 Aug 2025). In the reported formulation, a whole slide image is partitioned into patches, encoded as patch embeddings, reordered so that semantically similar patches become adjacent in sequence space, and then processed by a Semantic-guided Retrieval State Space Module that uses representative patches to modulate state-space dynamics. The method is positioned against attention-based MIL, Transformer-based MIL, and vanilla state space model pipelines, with the central claim that arbitrary patch ordering weakens histological semantics and interpretability, whereas semantic reordering and retrieval-conditioned scanning improve long-range contextual modeling while retaining the linear-complexity advantage of state space models (Gan et al., 30 Aug 2025).

1. Problem setting and motivation

SemaMIL addresses whole slide image classification under the standard MIL formulation used in computational pathology. Whole slide images are described as gigapixel pathology images for which direct end-to-end training on full-resolution slides is impractical because slides are extremely large, only slide-level labels are typically available, and pixel or patch annotations are scarce (Gan et al., 30 Aug 2025). The resulting pipeline partitions a slide into many tissue patches and maps them with a pretrained feature extractor into a bag or sequence of patch embeddings, turning WSI classification into a long-sequence modeling problem.

The motivation of SemaMIL is framed through a comparison among three methodological families. Attention-based MIL methods such as ABMIL and CLAM can highlight important patches, but they “typically treat each patch independently and neglect the contextual dependencies inherent to tissue architecture” (Gan et al., 30 Aug 2025). Transformer models can model patch interactions explicitly, but self-attention has quadratic complexity in sequence length and is described as computationally expensive for WSIs with thousands to tens of thousands of patches, with a tendency to overfit when supervision is weak (Gan et al., 30 Aug 2025). State space models offer linear complexity and a global receptive field, but SemaMIL identifies a pathology-specific failure mode: arbitrary or random patch ordering discards histological priors and separates semantically related regions in the sequence, which undermines contextual modeling and interpretability (Gan et al., 30 Aug 2025).

Within this framing, SemaMIL’s central premise is twofold. First, the sequential order fed to an SSM should be semantically meaningful rather than arbitrary. Second, representative slide content should influence sequence dynamics globally rather than only through local recurrence. This yields the two principal modules of the framework: Semantic Reordering (SR) and the Semantic-guided Retrieval State Space Module (SRSM) (Gan et al., 30 Aug 2025).

2. Semantic Reordering

Semantic Reordering is the order-sensitive front end of SemaMIL. Given projected patch embeddings $\{x_i\}_{i=1}^L$, the method applies a lightweight router composed of two linear layers with GELU:

$h_i = GELU(W_1 x_i), \qquad z_i = W_2 h_i.$

After a softmax operation on $z_i$, the model obtains a probability vector $p_i$ and assigns a hard semantic label

$c_i = \arg\max(p_i).$

The permutation is then defined by sorting the semantic labels,

$\pi = \arg sort(c_1,\cdots,c_L),$

which produces the reordered sequence

$X' = [x_{\pi(1)},x_{\pi(2)},\dots,x_{\pi(L)}].$

After state-space processing, the original index order is restored through the inverse permutation,

$y_i = y'_{\pi^{-1}(i)}.$

These equations and the accompanying interpretation are given explicitly in the method description (Gan et al., 30 Aug 2025).

This mechanism does not use an explicit clustering algorithm such as k-means. Instead, grouping is induced by the learned router: each patch receives a semantic score vector, softmax yields assignment probabilities, hard assignment places the patch into a semantic group, and sorting by the group index makes semantically similar patches adjacent (Gan et al., 30 Aug 2025). The permutation is reversible by construction, which allows the model to exploit semantically favorable ordering internally while preserving output alignment to the original patch positions if needed.

The paper states that hard assignment via $\arg\max$ is used, but also notes that “when using soft assignments including Gumbel variants the process is fully differentiable and reversible” (Gan et al., 30 Aug 2025). No full training algorithm for such variants is specified in the provided description. This suggests that differentiable routing is contemplated as an extension, but the exact experimental role of Gumbel-based routing is not established in the reported details.

The intended pathological significance of SR is that dispersed yet histologically related regions are brought close together in sequence space. The paper argues that this reduces effective interaction distance for the causal scan, lets the SSM propagate discriminative pathology signals more effectively, preserves semantic coherence better than random or arbitrary shuffling, and improves interpretability because neighboring sequential interactions correspond more directly to related tissue patterns (Gan et al., 30 Aug 2025).

3. Retrieval-guided state space modeling

The second defining component of SemaMIL is the Semantic-guided Retrieval State Space Module (SRSM). Its purpose is to augment the reordered sequence with a form of retrieval-conditioned global context. After semantic alignment, the model scores patches with a lightweight linear projection and selects the top-$K$ most important ones as a query set,

$h_i = GELU(W_1 x_i), \qquad z_i = W_2 h_i.$0

The remaining $h_i = GELU(W_1 x_i), \qquad z_i = W_2 h_i.$1 patches form the context sequence

$h_i = GELU(W_1 x_i), \qquad z_i = W_2 h_i.$2

The exact scoring function is not specified in the reported details, but top-$h_i = GELU(W_1 x_i), \qquad z_i = W_2 h_i.$3 query selection is explicitly part of the method (Gan et al., 30 Aug 2025).

SRSM starts from the continuous-time linear time-invariant state-space system

$h_i = GELU(W_1 x_i), \qquad z_i = W_2 h_i.$4

and its zero-order-hold discretization

$h_i = GELU(W_1 x_i), \qquad z_i = W_2 h_i.$5

Instead of using fixed or purely input-conditioned SSM parameters, SemaMIL generates dynamic parameters from the retrieved query set:

$h_i = GELU(W_1 x_i), \qquad z_i = W_2 h_i.$6

Using a fixed base transition matrix $h_i = GELU(W_1 x_i), \qquad z_i = W_2 h_i.$7, the method defines

$h_i = GELU(W_1 x_i), \qquad z_i = W_2 h_i.$8

and

$h_i = GELU(W_1 x_i), \qquad z_i = W_2 h_i.$9

The context sequence is then processed causally as

$z_i$0

All of these expressions are stated directly in the mathematical formulation (Gan et al., 30 Aug 2025).

The paper explains the intended interpretation of these parameters: $z_i$1 captures local tissue continuity, whereas $z_i$2 gates global semantic querying and suppresses noise under guidance from $z_i$3 (Gan et al., 30 Aug 2025). The distinction from standard SSM or Mamba-style layers is therefore explicit. In SemaMIL, state-space dynamics are conditioned on retrieved representative patches rather than only on the current sequential input. This gives the method a hybrid structure that combines linear-time scanning, representative-patch retrieval, and dynamic transition adaptation.

To better capture the two-dimensional structure of histopathology slides, SRSM is executed in parallel along four scan directions, producing outputs $z_i$4, which are fused before global pooling and classification (Gan et al., 30 Aug 2025). The exact fusion equation is not provided in the reported details.

4. End-to-end pipeline and implementation profile

The end-to-end SemaMIL pipeline begins by partitioning a WSI into $z_i$5 patches,

$z_i$6

and extracting patch features

$z_i$7

with a pretrained encoder (Gan et al., 30 Aug 2025). The features are then linearly projected to a lower-dimensional space of dimension $z_i$8, yielding the sequence supplied to SR.

The reported architecture can be summarized as follows. First come pretrained patch embeddings and a linear projection to the lower-dimensional space. Second, Semantic Reordering reorders the patch sequence through the learned router and reversible permutation. Third, SRSM performs retrieval-guided causal scanning in four directions. Fourth, fused outputs are globally pooled to obtain a slide-level representation, which is passed to the subtype classification head (Gan et al., 30 Aug 2025).

Several implementation details are explicit. The experiments use patch features extracted by the TITAN model, and the data preprocessing pipeline is stated to follow CLAM for fair comparison (Gan et al., 30 Aug 2025). The reported optimization setting specifies a learning rate of

$z_i$9

The paper further states that training is slide-level supervised MIL training using pre-extracted TITAN patch features (Gan et al., 30 Aug 2025).

At the same time, a substantial set of details is not specified in the reported description. The paper does not provide the embedding dimension $p_i$0, the number of SRSM layers, hidden-state size, normalization layers, residual connections, positional encodings, exact classifier structure, top-$p_i$1 value, number of semantic categories, batch size, number of epochs, weight decay, scheduler, or optimizer type (Gan et al., 30 Aug 2025). It also does not report the exact training treatment of the hard routing step beyond noting the existence of possible soft or Gumbel variants. For an encyclopedia treatment, this is significant because the conceptual architecture is well defined, while some implementation-critical hyperparameters remain under-specified in the available description.

5. Experimental results and efficiency

SemaMIL is evaluated on four public pathology datasets for subtype classification: EBRAINS, BRACS, IPD-Brain (reported as IPD in tables), and TCGA (Gan et al., 30 Aug 2025). The evaluation protocol uses ten-fold Monte Carlo cross-validation with a train/validation/test split of

$p_i$2

For BRACS, an official split is also evaluated (Gan et al., 30 Aug 2025). The reported metrics are AUC and ACC.

The compared baselines include Max-Pooling, Mean-Pooling, ABMIL, CLAM-MB, DSMIL, DTFDMIL, TransMIL, S4MIL, and MambaMIL (Gan et al., 30 Aug 2025). The paper reports that SemaMIL achieves the best performance across all listed datasets and splits. The principal results are summarized below.

Dataset / split	SemaMIL result	Best baseline reported
EBRAINS	AUC $p_i$3, ACC $p_i$4	MambaMIL: AUC $p_i$5, ACC $p_i$6
BRACS official split	AUC $p_i$7, ACC $p_i$8	MambaMIL: AUC $p_i$9, ACC $c_i = \arg\max(p_i).$0
BRACS cross-validation	AUC $c_i = \arg\max(p_i).$1, ACC $c_i = \arg\max(p_i).$2	DSMIL: AUC $c_i = \arg\max(p_i).$3; MambaMIL: ACC $c_i = \arg\max(p_i).$4
IPD	AUC $c_i = \arg\max(p_i).$5, ACC $c_i = \arg\max(p_i).$6	Best baseline: AUC $c_i = \arg\max(p_i).$7, ACC $c_i = \arg\max(p_i).$8
TCGA	AUC $c_i = \arg\max(p_i).$9, ACC $\pi = \arg sort(c_1,\cdots,c_L),$0	S4MIL: AUC $\pi = \arg sort(c_1,\cdots,c_L),$1; DTFDMIL: ACC $\pi = \arg sort(c_1,\cdots,c_L),$2

The efficiency comparison in Table 3 reports FLOPs, parameter counts, and ACC for several methods. In that table, SemaMIL has 0.248 G FLOPs, 0.464 M parameters, and 0.751 ACC, while the competing methods are listed as ABMIL (0.352 G, 0.467 M, 0.740), CLAM (0.354 G, 0.662 M, 0.722), DSMIL (0.624 G, 0.476 M, 0.722), DTFDMIL (0.318 G, 0.586 M, 0.741), TransMIL (0.502 G, 0.542 M, 0.711), S4MIL (0.706 G, 0.924 M, 0.715), and MambaMIL (0.255 G, 0.470 M, 0.744) (Gan et al., 30 Aug 2025). In the reported comparison, SemaMIL therefore has the lowest FLOPs, the fewest parameters, and the best ACC.

The paper attributes this efficiency to the linear-time advantage of state space models relative to Transformer self-attention (Gan et al., 30 Aug 2025). No explicit asymptotic formula for SemaMIL itself is given, but the reported interpretation is that SemaMIL preserves the linear-complexity character of SSM scanning while improving contextual modeling through SR and SRSM.

6. Ablation, interpretation, and limitations

The principal ablation isolates the contributions of SR and SRSM. On EBRAINS, the baseline without SR and SRSM reaches AUC $\pi = \arg sort(c_1,\cdots,c_L),$3 and ACC $\pi = \arg sort(c_1,\cdots,c_L),$4; SRSM only yields AUC $\pi = \arg sort(c_1,\cdots,c_L),$5 and ACC $\pi = \arg sort(c_1,\cdots,c_L),$6; SR only yields AUC $\pi = \arg sort(c_1,\cdots,c_L),$7 and ACC $\pi = \arg sort(c_1,\cdots,c_L),$8; and the combination SR + SRSM yields AUC $\pi = \arg sort(c_1,\cdots,c_L),$9 and ACC $X' = [x_{\pi(1)},x_{\pi(2)},\dots,x_{\pi(L)}].$0 (Gan et al., 30 Aug 2025). On BRACS, the baseline gives AUC $X' = [x_{\pi(1)},x_{\pi(2)},\dots,x_{\pi(L)}].$1 and ACC $X' = [x_{\pi(1)},x_{\pi(2)},\dots,x_{\pi(L)}].$2; SRSM only gives AUC $X' = [x_{\pi(1)},x_{\pi(2)},\dots,x_{\pi(L)}].$3 and ACC $X' = [x_{\pi(1)},x_{\pi(2)},\dots,x_{\pi(L)}].$4; SR only gives AUC $X' = [x_{\pi(1)},x_{\pi(2)},\dots,x_{\pi(L)}].$5 and ACC $X' = [x_{\pi(1)},x_{\pi(2)},\dots,x_{\pi(L)}].$6; and SR + SRSM yields AUC $X' = [x_{\pi(1)},x_{\pi(2)},\dots,x_{\pi(L)}].$7 and ACC $X' = [x_{\pi(1)},x_{\pi(2)},\dots,x_{\pi(L)}].$8 (Gan et al., 30 Aug 2025).

These ablations support three specific conclusions stated or directly implied by the reported numbers. Both modules improve over the baseline. SR alone is slightly stronger than SRSM alone in the listed ablations. The combination performs best, which indicates complementarity between semantically meaningful ordering and retrieval-guided state-space conditioning (Gan et al., 30 Aug 2025).

The interpretability argument is primarily conceptual. The paper emphasizes that semantic reordering makes the sequence more coherent because histologically similar patches become adjacent, and that the retrieved top-$X' = [x_{\pi(1)},x_{\pi(2)},\dots,x_{\pi(L)}].$9 queries can be understood as representative evidence guiding global contextual modulation (Gan et al., 30 Aug 2025). Figure-level descriptions reportedly compare ordering strategies for Mamba-based modeling and present hard cases correctly classified by SemaMIL but misclassified by competing methods. However, the paper does not provide detailed visualizations of selected query patches, semantic group assignments, saliency maps, attention maps, or retrieval heatmaps in the reported description (Gan et al., 30 Aug 2025).

Several limitations are explicit or directly implied. SemaMIL depends on the quality of pretrained TITAN features. The discrete routing step uses hard $y_i = y'_{\pi^{-1}(i)}.$0 assignment, but the exact training treatment is not fully specified. Important hyperparameters such as the number of semantic groups, top-$y_i = y'_{\pi^{-1}(i)}.$1, hidden dimensions, and number of layers are not reported in the available description. The paper also does not provide ablations for different $y_i = y'_{\pi^{-1}(i)}.$2 values, numbers of semantic groups, sorting strategies, random-order sensitivity, or the effect of four-direction scanning relative to a single direction (Gan et al., 30 Aug 2025). This suggests that while the method’s conceptual contributions are clear, several design choices remain empirically under-characterized in the reported material.

A recurrent misconception is to conflate SemaMIL with SEMA, the distinct vision model introduced in “SEMA: a Scalable and Efficient Mamba like Attention via Token Localization and Averaging” (Tran et al., 10 Jun 2025). The two methods are unrelated in task and formulation: SEMA is a vision attention mechanism motivated by attention dispersion, whereas SemaMIL is a WSI MIL framework based on semantic reordering and retrieval-guided state-space modeling. The shared prefix is nominal rather than methodological.

7. Position within WSI MIL and state space research

SemaMIL occupies a specific position at the intersection of multiple instance learning and long-sequence state-space modeling for pathology. Relative to classical attention MIL, it is designed to incorporate contextual structure beyond patch-level importance weighting. Relative to Transformer MIL, it aims to avoid quadratic sequence complexity. Relative to vanilla SSM pipelines, it introduces semantic sequence construction and retrieval-guided parameter adaptation so that state transitions reflect representative slide content rather than only arbitrary scan order (Gan et al., 30 Aug 2025).

The method’s core technical contribution is therefore not merely the substitution of an SSM for attention, but the assertion that sequence geometry in pathology should be semantically engineered. Semantic Reordering addresses the order sensitivity of causal scans by making semantically related regions adjacent. SRSM then uses a representative query subset to generate or modulate discrete SSM parameters, introducing a form of global conditioning into an otherwise linear sequential mechanism (Gan et al., 30 Aug 2025).

A plausible implication is that SemaMIL should be most relevant in WSI regimes where patch sequences are long enough for Transformer costs to be burdensome, while tissue architecture is structured enough that random or rasterized ordering is semantically suboptimal. Another plausible implication is that its main advantages may depend on the quality of the learned routing function: if semantic grouping is poor, the benefits of sequence reordering would likely diminish. These are interpretive extensions of the reported design logic rather than separately demonstrated claims.

Taken as a whole, SemaMIL is best understood as a semantic-guided WSI MIL framework in which the slide is first reordered into a histology-aware sequence and then processed by a retrieval-conditioned state-space module operating in four scan directions. In the reported experiments, this combination yields state-of-the-art AUC and ACC on EBRAINS, BRACS, IPD, and TCGA, while also using fewer FLOPs and parameters than the compared baselines (Gan et al., 30 Aug 2025).