A multimodal and temporal foundation model for virtual patient representations at healthcare system scale

Abstract: Modern medicine generates vast multimodal data across siloed systems, yet no existing model integrates the full breadth and temporal depth of the clinical record into a unified patient representation. We introduce Apollo, a multimodal temporal foundation model trained and evaluated on over three decades of longitudinal hospital records from a major US hospital system, composed of 25 billion records from 7.2 million patients, representing 28 distinct medical modalities and 12 major medical specialties. Apollo learns a unified representation space integrating over 100 thousand unique medical events in our clinical vocabulary as well as images and clinical text. This "atlas of medical concepts" forms a computational substrate for modeling entire patient care journeys comprised of sequences of structured and unstructured events, which are compressed by Apollo into virtual patient representations. To assess the potential of these whole-patient representations, we created 322 prognosis and retrieval tasks from a held-out test set of 1.4 million patients. We demonstrate the generalized clinical forecasting potential of Apollo embeddings, including predicting new disease onset risk up to five years in advance (95 tasks), disease progression (78 tasks), treatment response (59 tasks), risk of treatment-related adverse events (17 tasks), and hospital operations endpoints (12 tasks). Using feature attribution techniques, we show that model predictions align with clinically-interpretable multimodal biomarkers. We evaluate semantic similarity search on 61 retrieval tasks, and moreover demonstrate the potential of Apollo as a multimodal medical search engine using text and image queries. Together, these modeling capabilities establish the foundation for computable medicine, where the full context of patient care becomes accessible to computational reasoning.

• The paper introduces Apollo, a novel large-scale multimodal temporal foundation model for representing virtual patient trajectories using diverse EHR data.
• It employs modality-specific encoders and a transformer architecture to integrate structured, text, and image data spanning decades of care.
• Apollo achieves significant AUROC improvements across 261 prognostic tasks, underscoring its potential for precision medicine and operational optimization.

Multimodal Temporal Foundation Models for Virtual Patient Representation: An Analysis of Apollo

Introduction

The proliferation of multimodal data within longitudinal electronic health records (EHRs), spanning diverse clinical specialities, laboratory results, diagnostic imaging, free-text notes, and structured codes, poses a significant challenge for unified computational modeling of patient trajectories. The paper "A multimodal and temporal foundation model for virtual patient representations at healthcare system scale" (2604.18570) introduces Apollo, a large-scale, multimodal temporal foundation model. Apollo is trained on over 25 billion medical events from more than 7 million patients, integrating structured, textual, and imaging data across three decades of care at a major academic US health system. This essay analyzes Apollo's design, benchmarking methodology, performance on downstream clinical tasks, interpretability, and implications for precision medicine.

Figure 1: Comprehensive depiction of the multi-institutional MGB-7M dataset, Apollo's multimodal temporal transformer architecture, pretraining and inference workflow, and core downstream evaluation strategies.

Architecture and Data Integration

Apollo extends the transformer paradigm to model full patient trajectories by tokenizing multimodal EHR data—structured events (diagnoses, medications, labs, vitals), free-text clinical notes, and images—via modality-specific encoders. Structured modalities leverage learnable embeddings; text and image modalities are encoded using clinical domain-specific language and vision models. Embeddings are projected to a shared latent space and temporally contextualized via a transformer backbone, with patient demographics and age information embedded explicitly.

The training corpus (MGB-7M) encompasses 28 distinct modalities over 33 years, sourced from 17 hospitals, capturing both inpatient and outpatient care and the full diagnostic spectrum. Pretraining employs a multimodal masking strategy, reconstructing masked tokens within each modality. The final patient representation is derived by appending a diagnostic modality prompt and extracting the contextual embedding at the last time point, enabling unsupervised, temporally-grounded patient state representation.

Emergence of a Unified Medical Concept Space

A salient property of Apollo is the emergence of semantically coherent, cross-modal embeddings. Uniform Manifold Approximation and Projection (UMAP) visualizations illustrate the clustering of discrete medical events—diagnosis codes, labs, medications—by semantic similarity, recapitulating clinical ontologies without explicit supervision.

Figure 2: UMAP projections demonstrate Apollo's learned atlas of medical concepts, capturing intra- and inter-modality relationships, with coherent clustering of diseases and pharmacologic agents, and cross-modal alignment of structured/unstructured representations.

These embeddings unify otherwise siloed modalities. Clinical notes and image representations for specific encounters are embedded near relevant structured events (e.g., bone marrow biopsy notes, blood smear images, and hematology labs in a leukemia workup).

Patient Embedding Phenotyping and Temporal Trajectories

Apollo’s temporal transformer composes event embeddings to yield dynamic patient representations. These aggregate the entire clinical trajectory into single vectors, facilitating phenotypic clustering. UMAP of patient embeddings reveals that local neighborhoods encode disease phenotypes, age, and trajectory. Disease-specific trajectories show, for instance, convergence in patient embedding space prior to schizophrenia diagnosis and divergence in glaucoma patients who progress to blindness.

Figure 3: UMAP visualizations and trajectory analyses illustrate temporal evolution of patient states and clustering by phenotype; AUROC summaries for major prognostic categories and ablation results highlight architectural contributions.

Performance on Prognosis and Operational Benchmarks

Apollo's embeddings support linear Cox time-to-event modeling for 261 prognostic tasks and 61 retrieval tasks, covering risk of new disease onset, disease progression, treatment response, adverse events, and hospital operations.

Key numerical results:

• New disease onset: AUROC improvements over age-sex baseline are substantial (e.g., heart failure 0.88 vs. 0.77; type 2 diabetes 0.85 vs. 0.61; schizophrenia 0.92 vs. 0.65).
• Disease progression: Gains are marked in chronic and acute scenarios (e.g., hypertension to heart failure 0.86 vs. 0.75; stroke survival 0.84 vs. 0.72).
• Treatment response: Breast cancer (trastuzumab) 0.93 AUROC; prostate cancer hormone therapy 0.86; COPD therapy with tiotropium 0.75.
• Adverse events: NSAID-induced acute kidney injury 0.91 AUROC; glucocorticoid-induced diabetes 0.87.
• Hospital operations: 30-day readmission prediction (0.62 vs. 0.54, balanced accuracy); dialysis dependence (0.97 vs. 0.65).

Apollo consistently outperforms single-modality, text-snapshot, and even fully supervised transformer baselines, demonstrated particularly in cancer progression benchmarks.

Semantic Patient Retrieval and Multimodal Medical Search

Patient embeddings serve as indices for large-scale semantic retrieval. On 61 diagnosis-treatment cohort queries, Apollo achieves higher Acc@5 than progress-note baselines, particularly in complex disease states (malignancy, severe cardiovascular disease). Qualitative examples confirm that embedding-based retrieval identifies clinically similar patients despite incomplete structured Coding, and enables multimodal queries (arbitrary text or images).

Figure 4: Quantitative and qualitative assessment of Apollo's patient retrieval capabilities using both structured and unstructured (text/image) queries.

Interpretability: Local and Global Attribution

Apollo's risk predictions are amenable to both local and global interpretation:

• Local analysis: Leave-one-token-out (LOTO) sensitivity highlights individual events—labs, diagnoses, notes—that drive acute shifts in predicted risk on the patient's timeline.
• Global analysis: Integrated Gradients across patient cohorts identify top contributors to risk for specific outcomes, recapitulating known biomarkers and risk factors, e.g., iron deficiency in COPD, hypertension in diabetes complications, comorbidities impacting cancer survival.

  Figure 5: Both example patient risk trajectories and population-level feature attributions support interpretability; canonical drivers and novel, hypothesis-generating associations are surfaced for further epidemiological study.

Calibration and Stratification

Apollo's risk scores yield clear stratification into high/low-risk groups and demonstrate strong calibration across tasks. Kaplan-Meier and calibration plots consistently show significant survival differences and low calibration error across endpoints ranging from disease onset to operational management.

Ablation and Model Components

Ablation studies isolate the contribution of multimodality, temporal context, and pretraining. The removal of unstructured modalities, temporal integration, or pretraining leads to significant drops in AUROC, confirming the necessity of each architectural component. The multimodal transformer yields consistent gains in neoplastic disease progression compared to strong structured-only or progress-note baselines.

Implications and Future Perspectives

Apollo sets a precedent for system-scale, multimodal longitudinal foundation modeling in healthcare, with broad implications:

• Precision Medicine: The unified representation enables continuous risk monitoring, earlier disease detection, and operational optimization.
• Cohort discovery and trial matching: Embeddings facilitate high-fidelity patient retrieval for research and clinical trials, even amid noisy or incomplete coding.
• Interpretability: The ability to surface both canonical and novel risk drivers opens avenues for biomarker discovery and validation.
• Generalization: Real-world deployment necessitates validation across health systems and geographic populations beyond a single EHR platform; future efforts should address cross-system generalization and adaptability.
• Causal Inference: Current association-focused modeling must be extended toward counterfactual and individualized treatment effect estimation, possibly via integration of interventional and multi-arm clinical data.
• Architectural Evolution: Further end-to-end pretraining, advanced aggregation, and direct incorporation of wearable/lifestyle data are promising directions.

Conclusion

Apollo establishes that at the scale of entire healthcare systems, integration of all available EHR modalities within a temporal transformer framework yields representations that are phenotypically meaningful, predictive across hundreds of clinical tasks, highly interpretable, and operationally useful. The demonstrated performance on diverse clinical outcomes and operational benchmarks positions unified multimodal patient representations as foundational elements in the computational infrastructure for AI-enabled healthcare.