Transcriptomics-guided Slide Representation Learning in Computational Pathology (2405.11618v1)

Abstract: Self-supervised learning (SSL) has been successful in building patch embeddings of small histology images (e.g., 224x224 pixels), but scaling these models to learn slide embeddings from the entirety of giga-pixel whole-slide images (WSIs) remains challenging. Here, we leverage complementary information from gene expression profiles to guide slide representation learning using multimodal pre-training. Expression profiles constitute highly detailed molecular descriptions of a tissue that we hypothesize offer a strong task-agnostic training signal for learning slide embeddings. Our slide and expression (S+E) pre-training strategy, called Tangle, employs modality-specific encoders, the outputs of which are aligned via contrastive learning. Tangle was pre-trained on samples from three different organs: liver (n=6,597 S+E pairs), breast (n=1,020), and lung (n=1,012) from two different species (Homo sapiens and Rattus norvegicus). Across three independent test datasets consisting of 1,265 breast WSIs, 1,946 lung WSIs, and 4,584 liver WSIs, Tangle shows significantly better few-shot performance compared to supervised and SSL baselines. When assessed using prototype-based classification and slide retrieval, Tangle also shows a substantial performance improvement over all baselines. Code available at https://github.com/mahmoodlab/TANGLE.

• The paper introduces Tangle, a novel self-supervised learning approach that integrates gene expression profiles with histology slide images to improve slide representation learning in computational pathology.
• Tangle demonstrated significant performance improvements over existing methods in comprehensive evaluations on multiple tissue types, particularly excelling in few-shot learning tasks and prototype-based classifications.
• This multimodal approach reduces the need for large labeled datasets, potentially leading to more efficient diagnostic tools and advancing precision medicine by capturing underlying biological states more accurately.

An Exploration of Transcriptomics-guided Slide Representation Learning in Computational Pathology

The paper, "Transcriptomics-guided Slide Representation Learning in Computational Pathology," presents a novel approach to self-supervised learning (SSL) in digital pathology that reinforces the utility of integrating multimodal data. This method, termed $Tangle$, leverages gene expression profiles to enhance slide representation learning, offering improved performance across major tasks in computational pathology, particularly in few-shot learning settings.

The primary challenge addressed is the development of scalable models capable of processing whole-slide images (WSIs), which can exceed the dimensions of 150,000$\times$150,000 pixels. Conventional methods split these WSIs into smaller, manageable patches, each represented by a lower-dimensional embedding obtained via pre-trained networks, historically reliant on datasets like ImageNet. The integration of SSL, specifically through histopathology-specific encoders, such as Vision Transformers (ViTs) and Convolutional Neural Networks (CNNs), has optimized this methodology significantly.

$Tangle$ innovatively employs multimodal pre-training to merge slide images with their corresponding gene expression data, hypothesizing that this creates a potent, task-agnostic training signal for slide embeddings. The multimodal aspect is facilitated by modality-specific encoders whose outputs are aligned through contrastive learning, effectively capturing the textual and visual narratives present within the data. Gene expression data provide a molecular lens, capturing the biological processes within a tissue, while histology slides offer spatial context with detailed morphological descriptions.

The method's pre-training was executed across substantial datasets comprising liver, breast, and lung samples from humans and rats, illustrating $Tangle$'s robustness and adaptability across diverse biological data. In comprehensive evaluations, $Tangle$ demonstrated a marked improvement over existing benchmarks in both supervised and SSL paradigms, reflected by its performance in few-shot learning tasks and prototype-based classifications.

Three independent test datasets—encompassing breast, lung, and liver WSIs—served as the evaluation grounds. Here, the algorithm notably surpassed supervised and intra-modality SSL baselines, showcasing its capability in extracting meaningful and predictive embeddings that could inform lesion classification and cancer subtype differentiation with limited labeled data. This is especially significant in contexts like rare disease evaluations or early-phase clinical studies where labeled data is scarce.

From a theoretical perspective, $Tangle$ underscores the potential and importance of multimodal learning in pathology. By learning from both transcriptomics and visual data, models like $Tangle$ can better approximate the underlying biological state than unimodal approaches. Practically, by reducing dependencies on huge labeled datasets, $Tangle$ empowers more efficient and effective diagnostic tools, potentially accelerating clinical workflows and enhancing precision medicine.

Looking forward, $Tangle$ lays the groundwork for enriching computational pathology with further multimodal approaches. Future research could extend this framework to include other modalities, such as imaging genomics or proteomics, offering even more comprehensive insights into the pathophysiological processes. Moreover, exploring novel SSL objectives that can function complementary to the contrastive approaches employed here might reveal more nuanced and scalable solutions to slide representation challenges.

In conclusion, the work outlined in this paper significantly contributes to the burgeoning field of multimodal learning within computational pathology. By demonstrating the efficacy of integrating transcriptomics with histopathology, the authors offer a compelling vision for the future of disease diagnosis and characterization, one that potentially integrates seamlessly into clinical practice, ensuring advancements not just in AI, but in patient care itself.