3D Foundation Model Overview

• 3D foundation models are pre-trained neural architectures that extract transferable representations directly from 3D data such as medical images, point clouds, and meshes.
• They leverage self-supervised techniques like contrastive learning and masked autoencoding to capture global and local spatial structures, overcoming data annotation scarcity.
• These models enable robust downstream tasks including classification, segmentation, and open-vocabulary recognition across diverse domains like medical imaging and urban scene analysis.

A 3D foundation model is a pre-trained, typically large-scale, neural architecture designed to process and extract general-purpose, transferable representations directly from three-dimensional (3D) data—such as volumetric medical images, point clouds, or 3D mesh data—using self-supervised learning (SSL) or multitask objectives on vast, heterogeneous datasets. These models supply downstream tasks (e.g., classification, segmentation, scene understanding) with powerful, domain-agnostic features in data-scarce or label-efficient regimes, often vastly improving generalization and sample efficiency across modalities, populations, and data sources.

1. Conceptual Foundations and Motivation

The rise of foundation models in NLP—notably pre-trained LLMs—and in 2D computer vision (e.g., SimCLR, CLIP, and DINO-v2) has catalyzed a parallel thrust for foundational architectures focused on 3D data. The motivation for 3D foundation models (“3DFMs” as an Editor’s term) derives from several challenges:

• Data Annotation Scarcity: Manual labeling for 3D data (MRI, CT scans, LiDAR, point clouds) is resource-intensive.
• Cross-Domain Generalization: Conventional supervised architectures typically overfit to specific datasets, protocols, or populations, limiting transferability.
• Complexity of 3D Structure: 3D data encode richer spatial relationships (e.g., anatomical structure, physical geometry) than their 2D projections, demanding architectural modifications (3D convolutions, point set/matrix encoding, 3D transformers).

Foundation models learn from large-scale, highly varied, and often unlabeled 3D datasets using self-supervision (contrastive or masked modeling) or cross-modal objectives, leading to universal, domain-robust representations capable of broad transfer, few-shot adaptation, and open-vocabulary generalization (Kaczmarek et al., 12 Sep 2025, Lee et al., 4 Feb 2025, Zhu et al., 4 Feb 2025, Pai et al., 15 Jan 2025, Lai et al., 18 Oct 2024, Mazher et al., 27 Oct 2025, Wang et al., 19 Feb 2025).

2. Core Architectural Patterns

3D foundation models fall into several principal categories, tailored to the input domain:

• 3D CNN/ResNet Backbones: Direct extension of 2D ConvNets using 3D operators. Example: 3D ResNet-18 with 3×3×3 convolutions, batch normalization, and MLP projection heads for contrastive learning on volumetric MRI data (Kaczmarek et al., 12 Sep 2025).
• 3D Vision Transformers (ViT/MAE/DINO Variants): Patching and embedding 3D volumetric input (voxels or local point clusters) into sequence tokens for transformer-style attention. Absolutepositional encoding is adapted to 3D grid indices (Lai et al., 18 Oct 2024, Zhu et al., 4 Feb 2025, Pai et al., 15 Jan 2025, Mazher et al., 27 Oct 2025).
• Point Cloud Models: Sparse 3D U-Nets with submanifold convolutions (SparseConvUNet), or hierarchical set encoders (e.g., Point-JEPA) mapping unstructured points (Lee et al., 4 Feb 2025, Letellier et al., 25 Nov 2025).
• Hybrid Encoder-Decoders: Models such as Dens3R leverage coupled pointmap representations predicting dense 3D positions, depth, and normals through lightweight shared transformer backbones (Fang et al., 22 Jul 2025).
• Cross-Modal and Vision-Language Architectures: For scene understanding and open-vocabulary 3D recognition, 3DFMs entangle point cloud/mesh/voxel features with vector representations derived from foundation vision-LLMs (CLIP, DINO), leveraging cross-attention, InfoNCE contrastive heads, and masked reconstruction losses (Wang et al., 17 Jun 2024, Zhang et al., 2023, Zuo et al., 3 Jan 2024).

A key property across these architectures is emphasis on preserving the volumetric and spatial context inherent in 3D data—and, where applicable, fusing it with 2D or language-driven context for broad semantic understanding.

3. Pre-training Objectives and Data Regimes

Self-supervised pre-training is foundational, enabling the extraction of representations untethered from task-specific, annotated datasets:

• Contrastive Objectives: SimCLR- and InfoNCE-based frameworks perturbed each input volume/point cloud into multiple “views” via heavy 3D augmentation (crops, rotations, flips, intensity shifts), optimizing temperature-scaled cosine similarity in embedding space (Kaczmarek et al., 12 Sep 2025, Pai et al., 15 Jan 2025, Lee et al., 4 Feb 2025).
• Masked Autoencoding (MAE): Models randomly mask a high fraction of input volume patches or points, reconstructing the missing data from visible regions. This forces capture of global and local spatial structure, promoting features that generalize across tasks (Lai et al., 18 Oct 2024, Wang et al., 19 Feb 2025, Wei et al., 7 Dec 2025).
• Cross-modal Distillation: Embeddings from 2D foundation models (e.g., CLIP, DINOv2) are distilled into 3D volumetric fields, e.g., by regressing 3D feature fields to match 2D projections, or through pixel/photoalignment losses as in FMGS and DistillNeRF (Zuo et al., 3 Jan 2024, Wang et al., 17 Jun 2024).
• Autoregressive Generative Objectives: In large language–vision models, foundation models may optimize to predict interleaved text/image tokens from multimodal input, using cross-attention Perceiver modules and LLMs (e.g., LLaMA) (Wu et al., 2023).

Pre-training data typically cover extremely heterogeneous, multi-institutional, multi-contrast, and multi-condition sources (medicine: ADNI, NACC, OASIS, BraTS, etc.; urban scenes: BuildingWorld; robotics: DROID). Dataset sizes range from tens of thousands (MRI) (Kaczmarek et al., 12 Sep 2025), to hundreds of thousands (head CT) (Zhu et al., 4 Feb 2025), to millions (urban buildings, point clouds).

4. Downstream Adaptation and Transfer

3D foundation models deliver features optimized for transferability and label efficiency:

• Linear Probing and Full Fine-tuning: After pre-training, downstream tasks (e.g., disease classification, age regression, 3D segmentation, registration, open-world recognition) are addressed by fine-tuning a minimal number of additional task-specific layers (e.g., linear heads, MLPs, decoder branches), or via LoRA-style adaptation for policy learning in robotics (Kaczmarek et al., 12 Sep 2025, Yang et al., 11 Mar 2025, He et al., 7 Jun 2024).
• Few-shot Learning: Performance remains robust even with only 10–20% of labeled samples (Alzheimer’s AUC ~0.89 with 20% label fraction, outperforming fully supervised baselines (Kaczmarek et al., 12 Sep 2025)). Similar findings appear in segmentation/registration (Triad, VISTA3D), open-vocabulary 3D detection (Zhang et al., 2023), and robotics (Yang et al., 11 Mar 2025).
• Zero-shot and Open-vocabulary Capabilities: By coupling 3D feature spaces with language representations (e.g., CLIP text embeddings), models like Mosaic3D, FM-OV3D, and FMGS achieve open-vocabulary segmentation, free-form referring detection, and compositional 3D scene generation (Lee et al., 4 Feb 2025, Zhang et al., 2023, Tang et al., 29 Nov 2025, Zuo et al., 3 Jan 2024). Cross-modal distillation further strengthens model’s ability to handle out-of-distribution concepts.
• Generalization Across Institutions/Protocols: In brain MRI, 3D SimCLR and BrainFound models yield state-of-the-art AUROC for AD detection across multiple external datasets. FM-CT for head CT achieves 12–21% macro-AUC gain over training from scratch on unseen test sets (Mazher et al., 27 Oct 2025, Zhu et al., 4 Feb 2025).

5. Representative Results and Empirical Highlights

Key quantitative outcomes across domains demonstrate the impact of 3D foundation models:

Model/Study	Task	Metric/Result	Baseline/Comparison
3D SimCLR (Kaczmarek et al., 12 Sep 2025)	Alzheimer’s (AIBL) classification	AUC = 0.929 (FT, 100% data)	MAE-FT: 0.798; ResNet-18: 0.869
3D SimCLR (Kaczmarek et al., 12 Sep 2025)	Stroke regression (SOOP)	MAE = 5.37	ResNet-18: 5.47; MAE-FT: 6.15
Triad-3D MRI (Wang et al., 19 Feb 2025)	Segmentation (17 datasets)	Dice: 79.09% (Triad)	72.21% (scratch)
FM-CT (Zhu et al., 4 Feb 2025)	Head CT disease detection (NYU)	Macro-AUC: 0.852	0.734 (scratch); 0.748 (external)
VISTA3D (He et al., 7 Jun 2024)	3D segmentation (127 classes)	Dice: 0.792 (auto+point)	Auto3DSeg: 0.706; nnUNet: 0.718
Mosaic3D (Lee et al., 4 Feb 2025)	ScanNet20 zero-shot segmentation	f-mIoU: 65.0	RegionPLC: 57.8; OpenScene-3D:41.2
BuildingWorld (Huang et al., 9 Nov 2025)	3D building recon/data diversity	5M buildings, 44 cities	Enabling diverse urban 3DFMs

Interpretation: Self-supervised, volumetric pre-training with even relatively modest ResNet- or autoencoder-scale 3D backbones, when coupled to aggressive dataset scaling and task-agnostic objectives, delivers improvements over strong supervised and alternative self-supervised baselines.

6. Limitations, Insights, and Future Directions

Key insights have emerged from recent 3D foundation model research:

• Anatomical and Spatial Inductive Bias: 3D convolutions and transformers force the model to capture spatial correlations critical for domains like neuroimaging and materials science (sulci, grain texture, local curvature)—improving downstream transfer (Kaczmarek et al., 12 Sep 2025, Wei et al., 7 Dec 2025).
• Global Contrastive and Masked Objectives: Encouraging models to learn invariants across data sources, morphologies, and disease types boosts robustness to scanner and site variability (Kaczmarek et al., 12 Sep 2025, Mazher et al., 27 Oct 2025, Pai et al., 15 Jan 2025).
• Few-shot and Data-scarce Regimes: Large-scale unsupervised pre-training enables high fidelity with minimal labels, supporting realistic clinical or industrial deployment.
• Computational Bottlenecks: Full 3D ViT-style models demand significant memory and integrate hundreds of millions to billions of parameters, motivating efficient distillation (Foundry SuperTokens (Letellier et al., 25 Nov 2025)) and lightweight encoding for the edge.
• Data Quality and Diversity Constraints: Foundation models’ performance remains sensitive to the diversity and representativeness of pre-training datasets. Imbalances in organ/protocol coverage, domain shifts, or poor curation may limit generalization (Triad, RadFM).
• Inter-modality and Multi-modal Generalization: Vision-language coupling—either through instruction-tuned LLMs, or explicit cross-modal encoders—is an emerging frontier, as is the expansion from pure vision to multi-modal (image, text, time-series) 3D FMs (Wu et al., 2023, Lai et al., 18 Oct 2024).

Future directions involve: (i) scaling pre-training sets to millions of 3D volumes (medicine, urban, scientific), (ii) advancing open-vocabulary and open-set 3D understanding, (iii) integrating video and temporally-resolved 3D data, and (iv) unifying 2D, 2.5D, and 3D architectures in common frameworks.

7. Impact Across Applications and Domains

3D foundation models are catalyzing transformative capabilities across sectors:

• Medical Imaging: State-of-the-art in MRI/CT disease classification, segmentation (organ/tumor), diagnosis, report generation, and few-shot/fine-tuned interactive annotation (Kaczmarek et al., 12 Sep 2025, Zhu et al., 4 Feb 2025, Mazher et al., 27 Oct 2025, Lai et al., 18 Oct 2024, He et al., 7 Jun 2024, Wang et al., 19 Feb 2025).
• Urban and Materials Informatics: Robust modeling of polycrystalline microstructures (stiffness and nonlinear property prediction) (Wei et al., 7 Dec 2025), procedural city-scale reconstruction and segmentation (Huang et al., 9 Nov 2025), architectural-style transfer, and zero/few-shot adaptation to new cities.
• Holistic 3D Scene Understanding: Joint depth, normal, occupancy, and open-vocabulary segmentation and detection by integrating geometry, language, and semantics across input modalities (Lee et al., 4 Feb 2025, Zhang et al., 2023, Zuo et al., 3 Jan 2024, Wang et al., 17 Jun 2024, Tang et al., 29 Nov 2025).
• Robotics and Embodied Policy: 3D policy foundation models enable robust generalization to unseen objects/environments with minimal demonstration data, leveraging point cloud and language-conditioned diffusion transformers (Yang et al., 11 Mar 2025).

Collectively, 3D foundation models establish a scalable paradigm that decouples representation learning from task supervision, underpinning generalist AI for complex physical and clinical domains. They instantiate a universal “backbone” for transfer, few/zero-shot adaptation, and cross-domain reasoning in 3D-structured environments.