Aggregation Schemes for Single-Vector WSI Representation Learning in Digital Pathology

Abstract: A crucial step to efficiently integrate Whole Slide Images (WSIs) in computational pathology is assigning a single high-quality feature vector, i.e., one embedding, to each WSI. With the existence of many pre-trained deep neural networks and the emergence of foundation models, extracting embeddings for sub-images (i.e., tiles or patches) is straightforward. However, for WSIs, given their high resolution and gigapixel nature, inputting them into existing GPUs as a single image is not feasible. As a result, WSIs are usually split into many patches. Feeding each patch to a pre-trained model, each WSI can then be represented by a set of patches, hence, a set of embeddings. Hence, in such a setup, WSI representation learning reduces to set representation learning where for each WSI we have access to a set of patch embeddings. To obtain a single embedding from a set of patch embeddings for each WSI, multiple set-based learning schemes have been proposed in the literature. In this paper, we evaluate the WSI search performance of multiple recently developed aggregation techniques (mainly set representation learning techniques) including simple average or max pooling operations, Deep Sets, Memory networks, Focal attention, Gaussian Mixture Model (GMM) Fisher Vector, and deep sparse and binary Fisher Vector on four different primary sites including bladder, breast, kidney, and Colon from TCGA. Further, we benchmark the search performance of these methods against the median of minimum distances of patch embeddings, a non-aggregating approach used for WSI retrieval.

• The paper proposes and assesses innovative aggregation methods that merge patch embeddings into a unified WSI representation.
• It demonstrates that deep Fisher Vector variants, especially sparse and binary forms, outperform traditional pooling techniques in retrieval speed and accuracy.
• Experimental results on TCGA datasets show that optimal regularization enhances both the quality and efficiency of WSI representations.

Evaluation of Aggregation Schemes for WSI Representation Learning

The paper by Hemati et al. addresses a significant challenge in computational pathology: the efficient representation of Whole Slide Images (WSIs) through a singular feature vector. As WSIs are too large and complex to be processed directly by GPUs, the conventional approach involves breaking them down into smaller image patches, which are subsequently analyzed using pre-trained deep learning models. This leads to the necessity of developing techniques to aggregate these multiple patch embeddings into a single vector representation of the WSI, thus facilitating efficient storage, retrieval, and comparison for downstream tasks.

Methodological Overview

The authors investigate several aggregation techniques that have been proposed for synthesizing a single embedding from sets of patch embeddings, which are:

• Traditional Pooling Operations: Simple average and max pooling.
• Deep Sets: A permutation-invariant approach that learns representations through deep neural networks.
• Memory Networks: Employ memory units and attention mechanisms for complex set functions.
• Focal Attention: Integrates attention mechanisms with focal loss for enhanced embedding aggregation.
• Gaussian Mixture Model (GMM) Fisher Vector: Extends the Bag of Visual Words with higher-order statistics.
• Deep Fisher Vector: Uses Variational Autoencoders (VAEs) to derive Fisher Vectors adapted for deep learning, with focus on its sparse and binary variations.

Experimental Validation

The study conducts exhaustive evaluations across datasets from TCGA (The Cancer Genome Atlas) covering histopathological sites of bladder, breast, kidney, and colon. The benchmarking process focuses on the $k$-nearest neighbors ($k$-NN) retrieval task, which emulates practical applications in pathology for WSI search and retrieval.

Quantitative metrics utilized in their evaluation include classification accuracy, macro F1-score, and weighted F1-score. Deep Fisher Vector, along with its sparse and binary variations, generally surpassed other methods, demonstrating particularly strong performance in retrieval tasks across all datasets.

Noteworthy Findings

1. Deep Sparse and Binary Fisher Vectors: These emerged as superior methodologies, delivering the best balance of accuracy and computational efficiency, especially leveraging the sparse and binary nature of embeddings for faster retrieval, which is crucial for practical deployment in clinical settings.
2. Search Performance: The binary Fisher Vector, using Hamming distance, significantly reduced search times compared to the Euclidean distance-based methods, underscoring its utility for rapid indexing and querying.
3. Impact of Regularization: An ablation study on the regularization parameter $\alpha$ in the calculation of Fisher Vectors highlighted its effect on embedding quality and efficiency, with optimum values enhancing the capability for data representation while maintaining reduced dimensionality.

Implications and Future Directions

The authors' work suggests not only the efficiency of Fisher Vector-based approaches but also sets a precedent for further exploration in permutation-invariant representations within digital pathology. The adaptability of these methods in encoding complex patterns with reduced computational footprint presents considerable promise for enhancing diagnostic workflows.

Looking forward, integration with more varied pathology data types and coupling with other AI-driven systems for histopathology could advance these aggregation strategies into holistic diagnostic tools. Furthermore, future explorations might involve hybrid models that leverage deep Fisher Vectors with other state-of-the-art techniques in deep learning, potentially unveiling new avenues for accurate and scalable pathology image analysis.