GeMM-GAN: Multimodal Gene Expression GAN

• The paper introduces GeMM-GAN, a multimodal conditional generative model that produces high-dimensional gene expression profiles (18,868 genes) from whole-slide images and clinical descriptions.
• It employs a fusion of a pretrained Vision Transformer, Clinical ModernBERT, FiLM modulation, and bidirectional cross-attention to integrate visual and textual modalities.
• Empirical evaluations on TCGA data demonstrate improved recall, gene-gene correlation preservation, and enhanced disease-type prediction relative to baseline generative models.

Searching arXiv for the named method and closely related mixture-GAN references.

GeMM-GAN is a multimodal conditional generative model for synthesizing gene expression profiles from histopathology whole-slide images and clinical descriptions. It is presented as a Generative Adversarial Network conditioned jointly on image patches and clinical text, with a multimodal conditioning stack that combines a pretrained pathology Vision Transformer, a pretrained clinical text encoder, FiLM modulation, image-patch self-attention, and bidirectional cross-attention, and with a WGAN-GP used for gene-expression generation (Panaccione et al., 21 Jan 2026). In the reported formulation, the model operates on TCGA-derived triplets of whole-slide image, metadata-derived text, and matched RNA-seq profile, and targets a high-dimensional output space of 18,868 genes after preprocessing (Panaccione et al., 21 Jan 2026).

1. Problem formulation and modeling objective

GeMM-GAN is proposed to address a biomedical setting in which histopathology slides and patient metadata are routinely available, whereas matched transcriptomic profiles are often difficult to obtain because RNA-seq is costly, infrastructure-intensive, and privacy-sensitive (Panaccione et al., 21 Jan 2026). Within that framing, the model aims to generate realistic gene expression profiles from two more accessible modalities—whole-slide histopathology images and clinical descriptions—so that transcriptomic information can be synthesized when sequencing is unavailable.

The paper formalizes the task with a gene expression profile $x_g \in \mathbb{R}^g$, a whole-slide image $I \in \mathbb{R}^{W \times H \times 3}$, and a textual description $x_t \in T$. Because the whole-slide image is too large to process directly, the slide is represented as a set of patches $\{x_p^{(i)}\}$, where each patch satisfies $x_p \in \mathbb{R}^{P \times P \times 3}$. The learning objective is stated as

$$\text{learn a conditional generative model } G(x_g \mid \{x_p^{(i)}\}, x_t)$$

that captures the joint distribution of gene expression patterns given the visual and textual context (Panaccione et al., 21 Jan 2026).

A central conceptual point is that the model learns a conditional distribution rather than a deterministic predictor from pathology to transcriptomics. This matters because, as the paper notes, expression is inherently variable and multimodal even within similar histology. The authors further position GeMM-GAN as the first model to generate gene expression conditioned jointly on images and text rather than using only omics-domain covariates or only pathology images (Panaccione et al., 21 Jan 2026).

2. Data modalities and preprocessing pipeline

The image modality begins with whole-slide histopathology images. Tissue regions are segmented using Otsu thresholding, and high-resolution tiles of size $P=256$ are extracted; only tiles containing more than 20% tissue are retained (Panaccione et al., 21 Jan 2026). Training does not consume all valid tiles from a slide. Instead, $N=256$ patches are randomly sampled per training step, and each selected patch is processed independently by the image encoder (Panaccione et al., 21 Jan 2026).

The clinical modality is derived from structured TCGA JSON metadata, including demographics, cancer subtype, treatment history, and related disease-context fields. The retained fields are converted into short natural-language case summaries using a quantized instruction-tuned biomedical LLM, specifically a medical-data-finetuned Llama3-8B, yielding descriptions of approximately 200 words (Panaccione et al., 21 Jan 2026). The stated motivation is partly deployment realism: a user could provide a whole-slide image and a text description to obtain a plausible gene expression profile.

The output modality is a near-full-transcriptome tabular vector. The starting expression matrix contains RNA-seq FPKM values. Genes with more than 90% missing values are removed, and the remaining gene measurements are z-score normalized (Panaccione et al., 21 Jan 2026). After preprocessing, each sample is represented by 18,868 genes (Panaccione et al., 21 Jan 2026).

The dataset is drawn from TCGA and spans nineteen tumor types. After filtering to retain only samples with whole-slide image file size below 100 MB, the cohort contains 1,944 clinical cases (Panaccione et al., 21 Jan 2026). The reported experimental protocol uses a single 80/20 train-test split with consistent partitioning across image, text, and expression modalities so that all data from a given patient remain in the same partition (Panaccione et al., 21 Jan 2026). The paper does not describe a separate validation set, cross-validation, or stratification procedure.

3. Architecture and multimodal fusion

GeMM-GAN is organized into three stages: single-modality encoding, multimodal fusion, and conditional gene-expression generation with a WGAN-GP (Panaccione et al., 21 Jan 2026). The image encoder is UNI, a pretrained Vision Transformer foundation model for computational pathology. For each tile, UNI produces a 1024-dimensional embedding, which is linearly projected to a shared multimodal width $d=256$; the UNI backbone is frozen and only the projection layer is trained (Panaccione et al., 21 Jan 2026). The text encoder is Clinical ModernBERT, which outputs 768-dimensional token embeddings that are likewise projected to $d=256$, with the pretrained text encoder frozen (Panaccione et al., 21 Jan 2026).

The text encoder output is denoted

$I \in \mathbb{R}^{W \times H \times 3}$0

where $I \in \mathbb{R}^{W \times H \times 3}$1 is the maximum number of text tokens. Its first row is the text CLS token,

$I \in \mathbb{R}^{W \times H \times 3}$2

The image branch outputs

$I \in \mathbb{R}^{W \times H \times 3}$3

and because the image encoder processes patches independently, the model introduces a learnable surrogate image CLS vector

$I \in \mathbb{R}^{W \times H \times 3}$4

to represent global slide information (Panaccione et al., 21 Jan 2026).

The multimodal conditioning vector is defined as

$I \in \mathbb{R}^{W \times H \times 3}$5

Fusion proceeds in three stages. First, GeMM-GAN uses Feature-wise Linear Modulation (FiLM) to inject global textual information into each patch embedding: $I \in \mathbb{R}^{W \times H \times 3}$6 The paper notes that the notation is typographically corrupted, but the intended operation is standard FiLM: each patch-embedding channel is scaled and shifted using parameters derived from the text CLS embedding (Panaccione et al., 21 Jan 2026).

Second, the model prepends the learnable image CLS token to the FiLM-modulated image sequence and applies a Transformer encoder, producing

$I \in \mathbb{R}^{W \times H \times 3}$7

Only the existence and role of this Transformer encoder are specified. The paper does not report the number of layers, number of heads, positional encoding type, normalization placement, or activation choice (Panaccione et al., 21 Jan 2026).

Third, the model applies bidirectional cross-attention between text and image. The text CLS token queries the image token set: $I \in \mathbb{R}^{W \times H \times 3}$8 and the updated image CLS token queries the text token set: $I \in \mathbb{R}^{W \times H \times 3}$9 The final conditioning vector is

$x_t \in T$0

The paper does not provide the scaled dot-product attention equations or state whether residual connections, layer normalization, or feed-forward blocks are used around the cross-attention modules (Panaccione et al., 21 Jan 2026).

On the generative side, the model uses two instances of the multimodal fusion network with separate learnable parameters,

$x_t \in T$1

$x_t \in T$2

rather than sharing the same conditioning vector between generator and discriminator (Panaccione et al., 21 Jan 2026). The generator is an MLP taking a Gaussian noise vector $x_t \in T$3 together with $x_t \in T$4, and the discriminator is an MLP taking either a real or generated expression profile together with $x_t \in T$5. Conditioning is injected by concatenation: the effective inputs are $x_t \in T$6, $x_t \in T$7, and $x_t \in T$8 (Panaccione et al., 21 Jan 2026). Both generator and discriminator have two hidden layers of size 256 (Panaccione et al., 21 Jan 2026).

4. Adversarial learning protocol and evaluation methodology

GeMM-GAN uses a Wasserstein GAN with Gradient Penalty. The paper states that WGAN-GP is chosen because the authors regard it as still state-of-the-art for transcriptomic generation due to stable training on high-dimensional data, while diffusion models for transcriptomics are described as emerging but still exploratory (Panaccione et al., 21 Jan 2026). However, the manuscript does not write out the WGAN-GP loss explicitly, and it does not report the gradient penalty coefficient, critic update schedule, optimizer type, learning rate, weight decay, or other standard low-level hyperparameters (Panaccione et al., 21 Jan 2026).

Reported implementation details are limited but concrete. The system uses PyTorch 2.6.0 and CUDA 12.4 on a workstation with an AMD Ryzen 1950X CPU, 128 GB RAM, and two NVIDIA A6000 GPUs with 48 GB VRAM each (Panaccione et al., 21 Jan 2026). The shared latent dimension is $x_t \in T$9, which is used both for aligned modality embeddings and for the noise vector dimensionality. Batch size is 64, and $\{x_p^{(i)}\}$0 random patches are sampled from each whole-slide image at every training step (Panaccione et al., 21 Jan 2026).

The baselines are an unconditional WGAN-GP trained only on gene expression, a conditional WGAN-GP conditioned on disease type and primary site, and a CVAE conditioned on the same two categorical variables (Panaccione et al., 21 Jan 2026). The utility evaluation is omitted for the unconditional WGAN-GP because generated samples from that model cannot be assigned disease-type or primary-site labels in a principled manner (Panaccione et al., 21 Jan 2026).

Evaluation is divided into unsupervised metrics, detectability, and utility. The unsupervised metrics include precision and recall for generative models following Kynkäänniemi et al., using manifolds defined by the Euclidean distance to the 10th nearest neighbor: $\{x_p^{(i)}\}$1

$\{x_p^{(i)}\}$2

The paper also uses “C. MSE,” the mean squared error between real and synthetic gene-gene correlation matrices, as a measure of co-expression preservation (Panaccione et al., 21 Jan 2026). Detectability is assessed by logistic regression and MLP classifiers that distinguish real from synthetic data, with accuracy and F1 as metrics. Utility is assessed by training Random Forest and MLP classifiers on synthetic gene expression and testing on real data for disease-type and primary-site prediction (Panaccione et al., 21 Jan 2026).

5. Empirical findings and ablation structure

The principal empirical claim is that GeMM-GAN produces more realistic and functionally meaningful gene expression profiles than the reported baselines (Panaccione et al., 21 Jan 2026). On unsupervised evaluation, GeMM-GAN attains precision 0.7103 and recall 0.8190, whereas conditional WGAN-GP attains precision 0.8436 and recall 0.7372 (Panaccione et al., 21 Jan 2026). The result reflects lower precision but notably higher recall, which the authors interpret as better support coverage and reduced mode collapse. On the biologically motivated correlation-structure metric, GeMM-GAN achieves C. MSE $\{x_p^{(i)}\}$3, compared with $\{x_p^{(i)}\}$4 for conditional WGAN-GP and $\{x_p^{(i)}\}$5 for CVAE; the paper explicitly describes this as a 98.2% improvement over the conditional WGAN-GP (Panaccione et al., 21 Jan 2026).

Detectability results are mixed. GeMM-GAN is substantially harder to distinguish from real data using logistic regression, with accuracy 0.5236 and F1 0.5444, compared with 0.8801/0.8925 for conditional WGAN-GP and 0.8931/0.9003 for CVAE (Panaccione et al., 21 Jan 2026). By contrast, MLP detectability remains high for all methods, including GeMM-GAN at accuracy 0.9797 and F1 0.9794, which indicates that stronger classifiers still detect residual distribution mismatch (Panaccione et al., 21 Jan 2026).

Utility results favor GeMM-GAN most strongly in Random Forest evaluation. For disease type prediction, Random Forest accuracy/F1 are 0.7449/0.8249 for GeMM-GAN versus 0.6367/0.7422 for conditional WGAN-GP (Panaccione et al., 21 Jan 2026). For primary site prediction, Random Forest accuracy/F1 are 0.6936/0.7478 versus 0.6118/0.6800 (Panaccione et al., 21 Jan 2026). MLP utility also improves, though more modestly: disease type accuracy/F1 are 0.9354/0.9322 versus 0.9262/0.9247, and primary site accuracy/F1 are 0.8436/0.8264 versus 0.8254/0.8094 (Panaccione et al., 21 Jan 2026). The abstract states that the model improves “by more than 11% the accuracy on downstream disease type prediction compared to current state-of-the-art generative models,” while the detailed table supports a clearest $\{x_p^{(i)}\}$6 relative gain in disease-type Random Forest F1 and a larger relative gain in disease-type Random Forest accuracy (Panaccione et al., 21 Jan 2026).

The ablation study compares six conditioning variants: Mean Image, Text CLS Token, Patch Transformer, FiLM, Cross Attention, and the full model (Panaccione et al., 21 Jan 2026). The text-only variant shows very high precision, 0.9513, but poor recall, 0.5205, and poor utility, with Random Forest accuracy 0.5000 and F1 0.6330 (Panaccione et al., 21 Jan 2026). The Patch Transformer improves on mean pooling, supporting the importance of modeling inter-patch relationships. FiLM alone gives the highest recall, 0.8821, but lower downstream utility; Cross Attention alone gives better C. MSE, 0.0011, and good utility; the full model gives the best overall balance, including C. MSE $\{x_p^{(i)}\}$7, lowest logistic-regression detectability accuracy 0.5236, and highest Random Forest disease-type utility 0.7449 accuracy and 0.8249 F1 (Panaccione et al., 21 Jan 2026). This suggests that FiLM and cross-attention play complementary roles, with FiLM injecting coarse textual context early and cross-attention refining bidirectional alignment.

The evidence for biological validity is indirect but nontrivial. The paper’s strongest support is preservation of gene-gene correlation structure and the ability of synthetic profiles to train classifiers that generalize to real disease-type and primary-site prediction (Panaccione et al., 21 Jan 2026). At the same time, the manuscript does not provide pathway enrichment, gene-set consistency, subtype-specific molecular findings, or external biological validation.

6. Terminology, neighboring methods, and recognized limitations

GeMM-GAN should not be conflated with GMGAN from Graphical-GAN. GMGAN denotes “Gaussian Mixture GAN,” a 2018 instance of the Graphical-GAN framework for discrete latent structure, with latent variables $\{x_p^{(i)}\}$8 and generative process

$\{x_p^{(i)}\}$9

where the mixture prior is imposed in latent space for image generation (Li et al., 2018). GeMM-GAN, by contrast, is a multimodal conditional transcriptomic generator from pathology and clinical text, and the two models address different modalities, outputs, and problem formulations.

It is also distinct from MGAN, or “Multi-Generator Generative Adversarial Nets,” which models the generator as a probabilistic mixture of multiple generators to mitigate mode collapse and includes a classifier that predicts generator identity (Hoang et al., 2017). GeMM-GAN does not use a mixture-of-generators design in the reported description. It is likewise different from GMM-Based Generative Adversarial Encoder Learning, which augments a GAN with a discriminator-shared encoder and fits a Gaussian Mixture Model to encoder outputs after training for clustering and improved sampling (Feigin et al., 2020). These methods are conceptually adjacent because all involve GANs and multimodality or mixture structure, but they are not alternative names for the same model.

Within multimodal biomedical generative modeling, the paper places GeMM-GAN between three existing lines of work: discriminative models that predict gene expression from images, generative transcriptomic models that do not use imaging or clinical modalities, and image-generation models conditioned on transcriptomics (Panaccione et al., 21 Jan 2026). A plausible implication is that GeMM-GAN’s novelty lies less in GAN mechanics alone than in the joint conditioning interface across pathology and clinical language.

The reported limitations are substantial. The study uses only TCGA, with 1,944 cases after filtering, and external generalization is not tested (Panaccione et al., 21 Jan 2026). The whole-slide image file-size criterion below 100 MB may bias the retained cohort. The clinical descriptions are generated by an LLM from structured metadata, introducing an additional preprocessing dependency whose errors or hallucinations are not analyzed. Reproducibility is constrained by missing optimization and architecture details. Detectability by MLP remains very high, indicating that synthetic and real profiles remain separable by stronger classifiers. Privacy is a motivating theme, but no formal privacy-risk analysis is reported. Biological validation is limited to correlation structure and downstream predictive utility rather than pathway-level, survival-level, subtype-level, or wet-lab validation (Panaccione et al., 21 Jan 2026).

For future work, the authors explicitly propose extending the framework in the reverse direction by generating histopathology images from gene expression and clinical data in order to study how genetic changes manifest morphologically in tissue (Panaccione et al., 21 Jan 2026). That proposal reinforces the broader view of GeMM-GAN as a cross-modal simulator for computational pathology and multimodal precision medicine.