--- title: 'COSFormer: Continual WSI Analysis' url: https://www.emergentmind.com/topics/cosformer type: topic --- # COSFormer: Continual WSI Analysis COSFormer is a unified Transformer-based framework for continual whole-slide-image (WSI) analysis that is designed to learn sequentially from new tasks without revisiting the full historical data. Introduced for multi-task WSI analysis, it integrates three components—Expert Consultation (EC), a Transformer encoder–decoder backbone, and autoregressive inference—within a single model that can grow dynamically as new tasks arrive. The framework is evaluated on a sequence of seven WSI datasets spanning seven organs and six WSI-related tasks under both task-incremental and class-incremental settings, where it is reported to show superior generalizability and effectiveness relative to the listed continual-learning baselines [2508.02220]. ## 1. Problem formulation and domain setting COSFormer is situated in continual learning for gigapixel pathology images. Whole Slide Images reveal detailed tissue structures in magnified views and are central to cancer diagnosis and prognosis, but their giga-sized nature imposes substantial storage and computational costs. The motivating requirement is a continual learning system that can adapt an existing model to new tasks without retraining or fine-tuning on previous tasks while maintaining high performance and resource efficiency [2508.02220]. The framework assumes a sequential task stream. At step \(t\), the model observes dataset \(\mathcal D_t\), learns to classify its \(c_t\) classes, and must retain performance on \(\{\mathcal D_1,\dots,\mathcal D_{t-1}\}\). Two standard continual-learning regimes are used. In **TASK-IL**, the active task is known and prediction is restricted to its \(c_t\) classes. In **CLASS-IL**, the task is unknown and the model must discriminate jointly among all classes accumulated up to step \(t\), namely \(\sum_{i=1}^{t} c_i\). This distinction is operationally important because COSFormer’s inference procedure uses task-conditioned masking only in TASK-IL, whereas CLASS-IL requires generation over the full label vocabulary [2508.02220]. A notable design choice is to cast classification as sequence generation rather than fixed-head discriminative prediction. This suggests that COSFormer is intended not merely as a rehearsal-based classifier, but as a label-generation system whose output space can expand as new diagnostic terms appear. The paper explicitly frames this as support for a dynamically growing vocabulary of diagnostic terms, thereby avoiding architecture redesign when classes change [2508.02220]. ## 2. Core architecture The model begins with WSI tiling and frozen visual feature extraction. Each WSI is partitioned into non-overlapping patches, and a frozen visual encoder \(F_{\mathrm{visual}}\), with UNI given as an example, maps patch \(x_{i,j}\) to feature vector \(z^v_{i,j}\in\mathbb R^{d_f}\). For slide \(i\), the patch embeddings are collected as \(\mathbf z_i=\{z^v_{i,j}\}_{j=1}^{N_i}\) [2508.02220]. These visual features are projected into the Transformer embedding space through the EC module, producing \(\mathbf z' \in \mathbb R^{N\times d_{\mathrm{model}}}\). The projected sequence is then processed by a Transformer encoder \(\mathcal E\) with \(N_e\) layers using Nystrom self-attention to model long-range patch interactions: \[ \mathbf z^{(\ell+1)} = \mathbf z^{(\ell)}+\mathrm{Norm}\bigl(\mathrm{NA}(\mathbf z^{(\ell)})\bigr),\quad \ell=0,\dots,N_e-1. \] The decoder \(\mathcal D\), with \(N_d\) layers, performs standard Transformer decoding using multi-head self-attention over prior word embeddings, cross-attention over the encoder output, and a feed-forward network after each cross-attention block. If \(\mathbf h^{(k)}\) denotes the decoder state at time step \(k\), the next-word logits are \[ p^{(k+1)} = \mathrm{Linear}\bigl(\mathbf h_{N_d}^{(k)}\bigr)\in\mathbb R^{|\mathcal V|}. \] This encoder–decoder factorization links patch-level slide representation to token-level label generation [2508.02220]. The architecture therefore differs from fixed-output continual classifiers in two ways. First, representation learning is task-adaptive at the input projection stage through EC. Second, prediction is autoregressive rather than single-shot. A plausible implication is that COSFormer’s extensibility is distributed across both the input-conditioning mechanism and the output vocabulary, rather than localized in a task-specific classifier head. ## 3. Expert Consultation module The Expert Consultation module is the defining mechanism of COSFormer. It maintains one generalist weight matrix \(\theta_{\mathrm{gen}}\in\mathbb R^{d_f\times d_{\mathrm{model}}}\) and one expert matrix \(\theta_k\in\mathbb R^{d_f\times d_{\mathrm{model}}}\) for each of the \(N_T\) tasks observed so far. A router network \(\mathcal R\) computes unnormalized task scores for a patch embedding sequence: \[ \mathbf W = \{\mathbf w_k\}_{k=1}^{N_T} = \mathrm{FC}_2\bigl(\mathrm{ReLU}(\mathrm{FC}_1(\mathbf z))\bigr),\quad \mathbf w_k\in\mathbb R^N. \] The current task \(\mathcal T\) is emphasized by scaling and averaging: \[ \tilde w_{k,j} = \frac{\exp\bigl(\mathbbm1_{k=\mathcal T}(\mathbf w_k\cdot\mathbf1)\gamma+\mathbbm1_{k\neq\mathcal T}\mathbf w_k(j)\bigr)} {\sum_{k'\neq\mathcal T}\exp(\mathbf w_{k'}(j))+\exp(\mathbf w_{\mathcal T}\cdot\mathbf1\,\gamma)}, \] \[ \bar w_k = \frac1N\sum_{j=1}^N\tilde w_{k,j} + \mathbbm1_{k=\mathcal T}\,\beta. \] The final projection is \[ \theta_{EC} = \theta_{\mathrm{gen}} + \sum_{k=1}^{N_T}\bar w_k\,\theta_k, \qquad \mathbf z' = \mathbf z\,\theta_{EC}. \] The paper explicitly interprets EC as a Mixture-of-Experts layer in which each \(\theta_k\) specializes in one task, while \(\theta_{\mathrm{gen}}\) provides a shared component [2508.02220]. This mechanism is described as “doctor-like” expert consultation, with task-adaptive embeddings intended to mitigate inter-task interference. The claim is architectural rather than metaphorical: the router computes patch-wise task weights, emphasizes the current task through the scaling factor \(\gamma\), averages across patches, and fuses the experts into a single projection matrix. In effect, task adaptation occurs before the slide enters the Transformer proper. This suggests that COSFormer treats continual learning primarily as conditional representation routing, supplemented by replay and autoregressive decoding, rather than as parameter isolation alone [2508.02220]. The parameter-growth behavior is also explicit. The parameter budget increases by \(d_f\times d_{\mathrm{model}}\) for each new task, since a new expert matrix is added per task. The paper identifies this as a limitation and points to more parameter-efficient expert modules as a future direction [2508.02220]. ## 4. Autoregressive inference and continual-learning objective COSFormer formulates diagnosis as sequence generation. Starting from a beginning-of-sequence token \(\langle\mathrm{BOS}\rangle\), the decoder iteratively emits \[ \hat y^{(k)}=\arg\max_u\,p^{(k)}_u. \] The conditional factorization over a diagnostic label sequence \((y_1,\dots,y_K)\) is \[ P(y_1,\dots,y_K\mid x) = \prod_{k=1}^K P(y_k\mid y_{