Foundation Models Overview

• Foundation models are large pre-trained neural networks that use broad, self-supervised data to create general-purpose representations.
• They exhibit emergent capabilities such as zero-shot and few-shot learning, enabling effective adaptation to a variety of downstream tasks.
• Empirical studies highlight the balance between model scale and domain-specific performance, emphasizing the need for high-quality, diverse training data.

Foundation models are large neural networks pre-trained on broad and diverse data, typically with self-supervised, semi-supervised, or weakly supervised objectives, in order to produce general-purpose representations that can be adapted, fine-tuned, or prompted for many downstream tasks. Across the literature, their characteristic properties are scale, breadth of pre-training data, multimodality, and emergent behavior such as zero-shot, few-shot, or in-context learning; at the same time, several domain studies argue that “foundationality” is not guaranteed by architecture or parameter count alone, and must be established empirically in the target domain (Schneider, 2022, Alfasly et al., 2023, Paschali et al., 2024).

1. Historical trajectory

The concept emerged from a longer sequence of developments in machine learning. A concise lineage places McCulloch and Pitts’ 1943 mathematical neuron model at the beginning, followed by expert systems and early neural-network experiments in the 1970s and 1980s, representation learning and large curated datasets in the 1990s and 2000s, deep convolutional networks and word embeddings in 2012–2015, the Transformer architecture in 2017, pre-trained transformers such as BERT and early GPT systems in 2018–2020, and multimodal and vision foundation models such as CLIP, DALL·E, and Florence from 2021 onward (Schneider, 2022).

Within this trajectory, foundation models are distinguished from earlier task-specific systems by a combination of orders-of-magnitude larger parameter counts, pre-training on generic web-scale or multimodal corpora, and the appearance of capabilities that were not present in smaller, narrowly trained networks. The historical argument is not merely architectural. It is also organizational: the transition is from many models specialized per task toward fewer general backbones that are subsequently adapted to diverse downstream settings (Schneider, 2022).

This historical framing also explains why the term has remained contested. Some papers use it primarily for very LLMs, while others extend it to vision, vision-language, Earth-system, radiology, materials-science, and behavior-modeling systems. A plausible implication is that “foundation model” has become a cross-domain category whose core semantics are defined by pre-training regime and transfer behavior rather than by a single modality or architecture family.

2. Definitional core and criteria of foundationality

A common informal definition describes a foundation model as a neural network $f_\theta$ trained on large, broad datasets with the goal of producing a general-purpose representation that can be adapted or prompted for many downstream tasks (Schneider, 2022). More abstractly, environmental-science work writes a foundation model as a parameterized map

$$f_\theta:\mathcal{X}\to\mathcal{Z},\qquad \theta\in\mathbb{R}^P,$$

optionally paired with a decoder or prediction head $g_\phi:\mathcal{Z}\to\mathcal{Y}$, and trained under a generic pre-training objective

$$L(\theta,\phi)=\mathbb{E}_{x\sim D}\bigl[\ell\bigl(g_\phi(f_\theta(x)),\tau(x)\bigr)\bigr],$$

where $\tau(x)$ is a target derived from $x$ itself, as in masked modeling or reconstruction (Yu et al., 5 Apr 2025).

A more restrictive criterion is proposed in the digital-pathology study of Alfasly et al. Let $f_\theta$ denote a neural network, $P=\dim(\theta)$ its parameter count, $D_{\rm pre}$ the pre-training corpus, and $\{T_i\}$ a family of downstream tasks. On that account, foundationality rests on three pillars: model size, training dataset scale and diversity, and task generality. Their summarized criterion states that a model qualifies if $$f_\theta:\mathcal{X}\to\mathcal{Z},\qquad \theta\in\mathbb{R}^P,$$0 exceeds a threshold on the order of $$f_\theta:\mathcal{X}\to\mathcal{Z},\qquad \theta\in\mathbb{R}^P,$$1, if $$f_\theta:\mathcal{X}\to\mathcal{Z},\qquad \theta\in\mathbb{R}^P,$$2 with at least two modalities or a very large single modality, and if for a large set of unseen tasks $$f_\theta:\mathcal{X}\to\mathcal{Z},\qquad \theta\in\mathbb{R}^P,$$3, zero-shot or minimally fine-tuned performance exceeds a baseline threshold $$f_\theta:\mathcal{X}\to\mathcal{Z},\qquad \theta\in\mathbb{R}^P,$$4 (Alfasly et al., 2023).

Radiology reviews retain the same structure but emphasize standardized terminology: scale, self-supervision, multimodality, and emergence. They also separate uni-modal from multi-modal systems, including vision-only, language-only, vision-language, and extended multimodal models that incorporate structured clinical signals such as EHR fields, lab values, omics, and biosensor data (Paschali et al., 2024).

These formulations converge on an important distinction. Large pre-training is necessary in the concept, but not sufficient. Foundationality is not merely a property of parameter count or of pre-training corpus size; it is also a claim about transfer, reuse, and robustness of the learned representation across a sufficiently wide downstream task family (Alfasly et al., 2023).

3. Pre-training paradigms, architectures, and adaptation

Foundation models are usually trained with objectives that derive supervision from the input itself or from weak cross-modal alignment. In auto-regressive language modeling, the standard objective is next-token prediction,

$$f_\theta:\mathcal{X}\to\mathcal{Z},\qquad \theta\in\mathbb{R}^P,$$5

while in masked modeling or autoencoding the target is reconstruction of missing or masked content. Environmental and radiology surveys describe masked modeling, contrastive learning, and physics-guided self-supervision as recurring regimes; radiology work also gives the masked-image-modeling objective

$$f_\theta:\mathcal{X}\to\mathcal{Z},\qquad \theta\in\mathbb{R}^P,$$6

and a CLIP-style contrastive loss for image-report alignment (Paschali et al., 2024, Yu et al., 5 Apr 2025).

Architecturally, the dominant building block is the Transformer. A standard attention layer is written as

$$f_\theta:\mathcal{X}\to\mathcal{Z},\qquad \theta\in\mathbb{R}^P,$$7

with multi-head composition and feed-forward sublayers stacked deeply (Yu et al., 5 Apr 2025). Representative implementations include decoder-only LLMs, Vision Transformers, hybrid CNN-Transformer systems, and multimodal systems with separate backbones—such as ViT and BERT—followed by fusion through cross-attention or lightweight adapters (Paschali et al., 2024). The historical shift to the Transformer is central because it enabled massive parallelism and large-scale pre-training in NLP and then beyond it (Schneider, 2022).

Adaptation methods fall into several regimes. Fine-tuning updates all or part of the model parameters on downstream data. Prompting and in-context learning instead condition inference on a prompt or support set without gradient updates; in language modeling terms, the model estimates

$$f_\theta:\mathcal{X}\to\mathcal{Z},\qquad \theta\in\mathbb{R}^P,$$8

for a prompt $$f_\theta:\mathcal{X}\to\mathcal{Z},\qquad \theta\in\mathbb{R}^P,$$9 (Schneider, 2022). Domain reviews also distinguish zero-shot evaluation, linear probing with a frozen encoder, and full fine-tuning, sometimes with parameter-efficient methods such as LoRA, adapters, or prompt tuning (Paschali et al., 2024, Yu et al., 5 Apr 2025).

Evaluation is correspondingly heterogeneous. Reported metrics include accuracy, F1, AUC-ROC, Dice coefficient, IoU, BLEU, ROUGE, retrieval F1, weighted Kendall’s $g_\phi:\mathcal{Z}\to\mathcal{Y}$0, RMSE, $g_\phi:\mathcal{Z}\to\mathcal{Y}$1, CRPS, and task-specific clinical or physical validation measures (Paschali et al., 2024, Yu et al., 5 Apr 2025). This diversity reflects the fact that foundation models are not tied to a single task ontology; they are judged by transfer performance under a specified adaptation protocol.

4. Domain-specific realizations

The foundation-model paradigm has been instantiated across many research domains, often with domain-specific refinements of data, objectives, and evaluation.

In medicine, radiology reviews describe uni-modal vision models, LLMs, vision-LLMs, and extended multimodal systems. They emphasize large unlabeled corpora, self-supervision, and clinical evaluation by reader studies, subgroup analyses, and workflow integration, while also noting risks such as hallucinations, automation bias, dataset bias, and computational cost (Paschali et al., 2024).

In Earth and climate science, the concept is adapted to spatiotemporal, multi-variable physical data. One survey defines an Earth and climate foundation model as a model $g_\phi:\mathcal{Z}\to\mathcal{Y}$2 trained by self-supervised learning on a very large and heterogeneous Earth/climate dataset and efficiently adapted to a wide variety of downstream tasks; it further proposes eleven desirable features, including geolocation embedding, scale awareness, wavelength embedding, the time variable, multisensory inputs, uncertainty quantification, physical consistency, and carbon minimization (Zhu et al., 2024). ESFM then presents a fully open Earth System Foundation Model based on a 3D Swin-UNet backbone, extended to dense gridded data, sparse satellite swaths, and irregular station data with native handling of missing values and adaptive layer-norm-based ensembles for probabilistic forecasting (Ozdemir et al., 20 Apr 2026).

Environmental-science surveys generalize this perspective further, positioning foundation models as backbones for forward prediction, data generation, data assimilation, downscaling, inverse modeling, model ensembling, and decision-making across heterogeneous environmental data sources (Yu et al., 5 Apr 2025).

Outside the natural sciences, behavior modeling provides a different realization. Be.FM is built on LLaMA-3.1 Instruct backbones and supervised fine-tuned with LoRA on literature, experimental, and survey data. It is evaluated on behavior prediction, latent-trait inference, contextual-factor inference, research-workflow reasoning, and complex problem solving, thereby treating human behavior itself as a target domain for foundation-model transfer (Xie et al., 29 May 2025).

Foundation models have also been reframed as systems infrastructure. CHORUS applies LLMs to table-class detection, column-type annotation, and join-column prediction via natural-language prompting rather than task-specific model architectures (Kayali et al., 2023). GR-T proposes a mobile foundation model as firmware: a single immutable multimodal model residing in the NPU, with per-application adapters supplying downstream specialization (Yuan et al., 2023). The Danish Foundation Models project emphasizes nationally controlled, open, and documented language and speech models for Danish, motivated by low-resource-language risk and governance requirements (Enevoldsen et al., 2023). In materials informatics, a 3D polycrystal foundation model learns voxel-based microstructure representations through large-scale self-supervised masking and transfers them to stiffness prediction and nonlinear response modeling (Wei et al., 7 Dec 2025). In anomaly detection, surveys classify foundation-model usage into three roles—encoder, detector, and interpreter—rather than into a single canonical architecture (Ren et al., 10 Feb 2025).

5. Empirical limits of transfer and the problem of model selection

The strongest challenge to naïve accounts of foundationality comes from domain-transfer failures. In digital pathology, Alfasly et al. compared three CLIP variants—CLIP, BiomedCLIP, and PLIP—against two smaller pathology-specific networks, KimiaNet and DinoSSLPath, using patch embeddings, Yottixel-based whole-slide matching, and average F1 over leave-one-out runs. Their reported aggregate retrieval results were as follows (Alfasly et al., 2023):

Model	Total F1 (internal)	Total F1 (public)
CLIP	55 % ± 10 %	77.2 % ± 25 %
BiomedCLIP	59 % ± 8 %	78.7 % ± 22 %
PLIP	60 % ± 6 %	79.1 % ± 23 %
DinoSSLPath	65 % ± 5 %	82.3 % ± 22 %
KimiaNet	65 % ± 6 %	79.5 % ± 24 %

These results are central because the larger, more broadly pre-trained models did not surpass the smaller pathology-trained networks on internal whole-slide-image retrieval, even after domain-specific fine-tuning. The paper’s interpretation is explicit: data quality and domain relevance matter as much as, or more than, sheer scale; a purported foundation model can “fail” to be foundational in a given domain if it cannot surpass, or at least match, smaller domain-specific networks on core downstream tasks (Alfasly et al., 2023).

A related issue is selecting among pre-trained models before expensive adaptation. EMMS addresses this by using foundation encoders such as CLIP, BERT, and GPT-2 to convert heterogeneous downstream labels into unified noisy label embeddings (“F-Labels”), and then fitting a weighted linear regression whose optimum yields a transferability score. Across 5 downstream tasks and 24 datasets, EMMS reports average gains over LogME+F-Label of $g_\phi:\mathcal{Z}\to\mathcal{Y}$3 on image classification, $g_\phi:\mathcal{Z}\to\mathcal{Y}$4 on referring comprehension, $g_\phi:\mathcal{Z}\to\mathcal{Y}$5 on image captioning, $g_\phi:\mathcal{Z}\to\mathcal{Y}$6 on visual QA, and $g_\phi:\mathcal{Z}\to\mathcal{Y}$7 on text QA, together with wall-clock speedups of $g_\phi:\mathcal{Z}\to\mathcal{Y}$8, $g_\phi:\mathcal{Z}\to\mathcal{Y}$9, $$L(\theta,\phi)=\mathbb{E}_{x\sim D}\bigl[\ell\bigl(g_\phi(f_\theta(x)),\tau(x)\bigr)\bigr],$$0, $$L(\theta,\phi)=\mathbb{E}_{x\sim D}\bigl[\ell\bigl(g_\phi(f_\theta(x)),\tau(x)\bigr)\bigr],$$1, and $$L(\theta,\phi)=\mathbb{E}_{x\sim D}\bigl[\ell\bigl(g_\phi(f_\theta(x)),\tau(x)\bigr)\bigr],$$2, respectively (Meng et al., 2023). This suggests that foundation models can function not only as universal backbones, but also as meta-level infrastructure for model assessment and selection.

6. Socio-technical, engineering, and theoretical perspectives

Foundation models alter not only technical pipelines but also institutional structure. Socio-technical analyses emphasize homogenization—the replacement of many specialist models by a small number of very large models—and concentration of control in organizations able to finance large-scale training. They also identify a new prompt-based mode of end-user interaction, together with fairness, bias, accountability, and governance concerns (Schneider, 2022).

This institutional shift has motivated an explicit engineering response. Foundation Model Engineering proposes treating data and model architecture as “source code,” training as the “compiler,” and the trained foundation model as the “executable binary.” On that analogy, it advocates declarative, automated, and unified interfaces for data cleaning, labeling, access control, model selection, fine-tuning, merging, and deployment, including a “Git for models” perspective on parameter deltas and merge workflows (Ran et al., 2024).

At the same time, theoretical work argues that post-hoc explainability is insufficient at foundation-model scale and instead organizes interpretable analysis around three classical axes: generalization-analysis methods, expressive-power analysis, and dynamic-behavior analysis. The surveyed tools include VC-dimension bounds, Rademacher complexity, stability-based generalization, PAC-Bayesian bounds, universal approximation, computational-power characterizations of attention, Neural Tangent Kernel analysis, gradient-flow and SGD convergence, RLHF dynamics, and formal treatments of privacy, fairness, and hallucination (Fu et al., 2024).

Across application surveys, several recurring open problems appear. Radiology work highlights hallucinations, automation bias, dataset bias, computational cost, and environmental footprint (Paschali et al., 2024). Earth and climate work emphasizes uncertainty quantification, physical consistency, adversarial defenses, interpretability, and carbon minimization as desirable properties of an “ideal” Earth foundation model (Zhu et al., 2024). Anomaly-detection surveys stress efficiency, bias, explainability of the foundation model itself, multimodal fusion beyond simple alignment, and the cost of prompt engineering (Ren et al., 10 Feb 2025). The common pattern is that scale broadens capability, but also amplifies questions of robustness, calibration, governance, and reproducibility.

Foundation models therefore denote not a single architecture, but a research program: very large pre-trained models that aspire to serve as reusable substrates across tasks and domains. The literature supports that ambition, but it also narrows it. A model is “foundational” only to the extent that its representations transfer reliably, its adaptation pathways are efficient, and its behavior remains technically and institutionally tractable under real downstream conditions (Alfasly et al., 2023).