COSFormer: Continual WSI Analysis

• The paper introduces COSFormer, a unified Transformer-based framework leveraging expert consultation and autoregressive decoding for continual WSI analysis.
• It reformulates classification as sequence generation, enabling a dynamically expanding diagnostic vocabulary without retraining on previous data.
• By mitigating inter-task interference with text-based replay and task-adaptive modules, COSFormer outperforms baselines in both task- and class-incremental settings.

COSFormer is a unified Transformer-based framework for continual whole-slide-image (WSI) analysis that is designed to learn sequentially from new tasks without revisiting the full historical data. Introduced for multi-task WSI analysis, it integrates three components—Expert Consultation (EC), a Transformer encoder–decoder backbone, and autoregressive inference—within a single model that can grow dynamically as new tasks arrive. The framework is evaluated on a sequence of seven WSI datasets spanning seven organs and six WSI-related tasks under both task-incremental and class-incremental settings, where it is reported to show superior generalizability and effectiveness relative to the listed continual-learning baselines (Bui et al., 4 Aug 2025).

1. Problem formulation and domain setting

COSFormer is situated in continual learning for gigapixel pathology images. Whole Slide Images reveal detailed tissue structures in magnified views and are central to cancer diagnosis and prognosis, but their giga-sized nature imposes substantial storage and computational costs. The motivating requirement is a continual learning system that can adapt an existing model to new tasks without retraining or fine-tuning on previous tasks while maintaining high performance and resource efficiency (Bui et al., 4 Aug 2025).

The framework assumes a sequential task stream. At step $t$, the model observes dataset $\mathcal D_t$, learns to classify its $c_t$ classes, and must retain performance on $\{\mathcal D_1,\dots,\mathcal D_{t-1}\}$. Two standard continual-learning regimes are used. In TASK-IL, the active task is known and prediction is restricted to its $c_t$ classes. In CLASS-IL, the task is unknown and the model must discriminate jointly among all classes accumulated up to step $t$, namely $\sum_{i=1}^{t} c_i$. This distinction is operationally important because COSFormer’s inference procedure uses task-conditioned masking only in TASK-IL, whereas CLASS-IL requires generation over the full label vocabulary (Bui et al., 4 Aug 2025).

A notable design choice is to cast classification as sequence generation rather than fixed-head discriminative prediction. This suggests that COSFormer is intended not merely as a rehearsal-based classifier, but as a label-generation system whose output space can expand as new diagnostic terms appear. The paper explicitly frames this as support for a dynamically growing vocabulary of diagnostic terms, thereby avoiding architecture redesign when classes change (Bui et al., 4 Aug 2025).

2. Core architecture

The model begins with WSI tiling and frozen visual feature extraction. Each WSI is partitioned into non-overlapping patches, and a frozen visual encoder $F_{\mathrm{visual}}$, with UNI given as an example, maps patch $x_{i,j}$ to feature vector $z^v_{i,j}\in\mathbb R^{d_f}$. For slide $\mathcal D_t$0, the patch embeddings are collected as $\mathcal D_t$1 (Bui et al., 4 Aug 2025).

These visual features are projected into the Transformer embedding space through the EC module, producing $\mathcal D_t$2. The projected sequence is then processed by a Transformer encoder $\mathcal D_t$3 with $\mathcal D_t$4 layers using Nystrom self-attention to model long-range patch interactions: $\mathcal D_t$5 The decoder $\mathcal D_t$6, with $\mathcal D_t$7 layers, performs standard Transformer decoding using multi-head self-attention over prior word embeddings, cross-attention over the encoder output, and a feed-forward network after each cross-attention block. If $\mathcal D_t$8 denotes the decoder state at time step $\mathcal D_t$9, the next-word logits are

$c_t$0

This encoder–decoder factorization links patch-level slide representation to token-level label generation (Bui et al., 4 Aug 2025).

The architecture therefore differs from fixed-output continual classifiers in two ways. First, representation learning is task-adaptive at the input projection stage through EC. Second, prediction is autoregressive rather than single-shot. A plausible implication is that COSFormer’s extensibility is distributed across both the input-conditioning mechanism and the output vocabulary, rather than localized in a task-specific classifier head.

3. Expert Consultation module

The Expert Consultation module is the defining mechanism of COSFormer. It maintains one generalist weight matrix $c_t$1 and one expert matrix $c_t$2 for each of the $c_t$3 tasks observed so far. A router network $c_t$4 computes unnormalized task scores for a patch embedding sequence: $c_t$5 The current task $c_t$6 is emphasized by scaling and averaging: $c_t$7

$c_t$8

The final projection is

$c_t$9

The paper explicitly interprets EC as a Mixture-of-Experts layer in which each $\{\mathcal D_1,\dots,\mathcal D_{t-1}\}$0 specializes in one task, while $\{\mathcal D_1,\dots,\mathcal D_{t-1}\}$1 provides a shared component (Bui et al., 4 Aug 2025).

This mechanism is described as “doctor-like” expert consultation, with task-adaptive embeddings intended to mitigate inter-task interference. The claim is architectural rather than metaphorical: the router computes patch-wise task weights, emphasizes the current task through the scaling factor $\{\mathcal D_1,\dots,\mathcal D_{t-1}\}$2, averages across patches, and fuses the experts into a single projection matrix. In effect, task adaptation occurs before the slide enters the Transformer proper. This suggests that COSFormer treats continual learning primarily as conditional representation routing, supplemented by replay and autoregressive decoding, rather than as parameter isolation alone (Bui et al., 4 Aug 2025).

The parameter-growth behavior is also explicit. The parameter budget increases by $\{\mathcal D_1,\dots,\mathcal D_{t-1}\}$3 for each new task, since a new expert matrix is added per task. The paper identifies this as a limitation and points to more parameter-efficient expert modules as a future direction (Bui et al., 4 Aug 2025).

4. Autoregressive inference and continual-learning objective

COSFormer formulates diagnosis as sequence generation. Starting from a beginning-of-sequence token $\{\mathcal D_1,\dots,\mathcal D_{t-1}\}$4, the decoder iteratively emits

$\{\mathcal D_1,\dots,\mathcal D_{t-1}\}$5

The conditional factorization over a diagnostic label sequence $\{\mathcal D_1,\dots,\mathcal D_{t-1}\}$6 is

$\{\mathcal D_1,\dots,\mathcal D_{t-1}\}$7

Equivalently, the decoder produces next-token logits conditioned on both the encoder output and previously generated tokens: $\{\mathcal D_1,\dots,\mathcal D_{t-1}\}$8 Under TASK-IL, irrelevant words outside a task-specific “Words of Interest” set are masked at each step: $\{\mathcal D_1,\dots,\mathcal D_{t-1}\}$9 Under CLASS-IL, no mask is used, so the model must select from the full vocabulary (Bui et al., 4 Aug 2025).

The training objective combines current-task learning, replay, and dark replay. For slide $c_t$0 and decoding step $c_t$1,

$c_t$2

Here $c_t$3 denotes a small replay buffer, and the MSE term aligns current logits with those from the previous model on replayed samples in order to mitigate forgetting (Bui et al., 4 Aug 2025).

This design ties the continual-learning objective to the autoregressive formulation. Because the model replays token-level outputs rather than only class logits from a fixed head, the dark-replay term regularizes the evolving vocabulary-conditioned decoder. A plausible implication is that forgetting can arise not only in class boundaries but also in the sequential structure of label generation, which the formulation attempts to preserve.

5. Training protocol, buffer construction, and evaluation

The reported task order is CAMELYON16 $c_t$4 TCGA-NSCLC $c_t$5 TCGA-BRCA $c_t$6 TCGA-RCC $c_t$7 TCGA-ESCA $c_t$8 TCGA-TGCT $c_t$9 TCGA-CESC, together with the reverse order. The model hyperparameters are specified as embedding dimension $t$0, $t$1, router scaling $t$2, shift $t$3, replay buffer size $t$4 WSIs, Lookahead-RAdam with learning rate $t$5, and early stopping after 5 epochs. Accuracy is the primary metric and F1 score is also reported. The listed baselines are GDumb, A-GEM, ER-ACE, and DER++, all using the same Transformer-encoder backbone (Bui et al., 4 Aug 2025).

Replay-buffer construction uses a text-based retrieval strategy. The most representative slide per class cluster is selected by computing the similarity

$t$6

and then applying $t$7-means to enforce diversity. The paper characterizes this strategy as ensuring both representativeness and diversity in stored exemplars, improving replay quality (Bui et al., 4 Aug 2025).

The empirical results are reported for both continual-learning regimes.

Setting	COSFormer	DER++
TASK-IL, forward order	93.1%	92.2%
TASK-IL, reverse order	91.14%	90.9%
CLASS-IL, forward order	81.09%	71.20%
CLASS-IL, reverse order	79.36%	77.75%

In the ablation study, removing task-ID $t$8 or Words-of-Interest masking degrades performance by up to 12% in CLASS-IL. Omitting EC, the autoregressive decoder, or the text-based buffer each lowers accuracy by up to 11% in CLASS-IL. The t-SNE visualization in Figure 1 is reported to show that COSFormer embeddings have the highest silhouette scores, which the paper interprets as evidence of well-separated task clusters (Bui et al., 4 Aug 2025).

The evaluation scope is substantial within the stated setup: seven WSI datasets, seven organs, six WSI-related tasks, and two task orders. At the same time, the paper notes that only two task orders were tested, and indicates that more randomized sequences would probe robustness further (Bui et al., 4 Aug 2025).

6. Interpretation, limitations, and nomenclature

The paper attributes COSFormer’s performance to the interaction of three mechanisms: expert consultation for task-adaptive embeddings, autoregressive decoding for dynamically growing diagnostic vocabularies, and text-based buffer sampling for representative and diverse rehearsal data. These are presented as complementary rather than interchangeable components, which is consistent with the ablation results showing performance drops when any of them is removed (Bui et al., 4 Aug 2025).

Several limitations are stated directly. Per-task performance can vary, and some tasks, such as TCGA-BRCA in CLASS-IL, showed lower accuracy. Only two task orders were examined. The parameter budget grows by $t$9 with each new task because each task introduces a new expert matrix. Autoregressive decoding adds inference overhead, motivating future work on faster decoding or distillation. The paper also identifies extension to other pathology tasks, including survival prediction and treatment-response, as a natural next step (Bui et al., 4 Aug 2025).

A recurrent source of confusion is nomenclature. COSFormer in WSI continual learning should be distinguished from “cosFormer: Rethinking Softmax in Attention”, a linear Transformer that replaces the softmax kernel with a ReLU-based linear kernel and cosine distance re-weighting for efficient attention (Qin et al., 2022), and from “CoSformer: Detecting Co-Salient Object with Transformers”, which addresses co-salient object detection through Transformer-based intra-image and inter-image modeling with contrastive learning (Tang et al., 2021). The names are similar, but the problem settings, architectures, and objectives are different. This suggests that the capitalization variants map to separate research lines rather than successive versions of a single method.