3D Foundation Models: An Overview

• 3D foundation models are large-scale, pre-trained deep learning architectures designed to process, represent, and generate complex 3D data such as point clouds, meshes, and volumetric scenes.
• They employ unified, multi-task prediction strategies that integrate geometric reconstruction, semantic reasoning, and cross-modal feature fusion using state-of-the-art techniques like transformers and convolutional networks.
• These models are widely applied in robotics, medical imaging, and autonomous systems, yet they face challenges in efficient real-time inference, out-of-distribution generalization, and explicit geometric reasoning.

3D foundation models are large-scale, pre-trained deep learning frameworks that ingest 3D data and support a diverse array of spatial tasks spanning geometric reconstruction, perception, reasoning, and even generative modeling. Distinguished from their 2D or multimodal predecessors by their explicit spatial reasoning capabilities and unified end-to-end architectures, these models are designed to process, represent, or generate complex 3D structures—including point clouds, meshes, volumetric scenes, or direct 3D outputs from imagery. Recent advances have catalyzed a surge in research, resulting in a heterogeneous but rapidly maturing ecosystem of methods tailored to domains such as robotics, medical imaging, autonomous systems, and open-vocabulary scene understanding.

1. Key Taxonomies and Model Design

3D foundation models (FMs) encompass several broad paradigms distinguished by their architectural objectives and task specializations:

Model Class	Primary Output/Use	Representative Approaches
3D Geometric Foundation Models	Dense geometry (depth, normals, pts)	DUSt3R, Dens3R, MASt3R, MonST3R, VGGT (Cong et al., 2 Jun 2025, Fang et al., 22 Jul 2025)
Generative 3D Models	Mesh/Scene generation	MeshXL (autoregressive mesh LLM) (Chen et al., 31 May 2024)
Cross-modal/Multimodal Models	Vision-language-3D fusion	FM-OV3D, FMGS (Zhang et al., 2023, Zuo et al., 3 Jan 2024)
Policy/Agent Foundation Models	Action policy from 3D	FP3 (robotics) (Yang et al., 11 Mar 2025)
Medical Imaging/Volume Models	3D segmentation (auto/interactive)	VISTA3D, CT-FM (He et al., 7 Jun 2024, Pai et al., 15 Jan 2025)
Labeling and Self-Supervision Tools	3D labeling/specialization	LeAP, LoRA3D (Gebraad et al., 6 Feb 2025, Lu et al., 10 Dec 2024)

Architecturally, the backbone networks span transformer variants (ViT, decoder-only, diffusion-tranformer combos), convolutional/residual 3D CNNs (for volumes), UNets (medical, voxel-based), and hybrid representations (Gaussian Splatting + feature fields in FMGS (Zuo et al., 3 Jan 2024)). Most models implement modular heads to support multiple geometric or semantic tasks, and many leverage advanced positional encoding schemes—such as position-interpolated rotary encoding in Dens3R (Fang et al., 22 Jul 2025)—for robust high-resolution performance.

2. Unified and Multi-Task Prediction Strategies

A defining feature of recent 3D FMs is joint prediction of interdependent geometric quantities rather than isolated outputs. For example, Dens3R (Fang et al., 22 Jul 2025) unifies depth, surface normal, and 3D pointmap regression using a two-stage pipeline: initial scale-invariant pointmap training (including local/global 3D losses and image-pair InfoNCE matching) followed by intrinsic-invariant refinement with explicit normal supervision. The final architecture concatenates surface normal outputs with pointmap features, enforcing geometric consistency and resolving ambiguities that single-task pipelines often cannot.

Similarly, FMGS (Zuo et al., 3 Jan 2024) melds pre-computed 2D vision-language embeddings (from CLIP, DINO) into an efficient 3D representation by distilling those features via a multi-resolution hash encoder and training with view-consistent rendering losses, including a specially designed pixel alignment loss for robust spatial semantics.

Joint learning enables robust multi-view 3D reasoning, augments downstream tasks (e.g. object detection, scene segmentation, pose estimation), and ensures intrinsic coherence between predicted modalities.

3. Cross-Modal Extensions and Fusion

Cross-modal 3D FMs capitalize on synergies between 2D vision-LLMs, generative text/image models, and 3D geometric representations:

• FM-OV3D (Zhang et al., 2023) fuses open-vocabulary localization (via Grounded-SAM 2D pseudo-labels mapped to 3D) with cross-modal feature alignment from GPT-3, Stable Diffusion, and CLIP, using contrastive learning to unify point cloud, visual, and textual spaces.
• Bridge3D (Chen et al., 2023) and OpenSU3D (Mohiuddin et al., 19 Jul 2024) exploit semantic masks, captions, and object features from 2D FMs (SAM, CLIP, GPT-4V) to guide 3D representation learning, enabling scene- and object-level knowledge distillation, open-vocabulary annotation, and instance-level query.
• Approaches such as PointSeg (He et al., 11 Mar 2024), LeAP (Gebraad et al., 6 Feb 2025), and others map and fuse 2D segmentation outputs into 3D point clouds or voxel grids, enforcing spatial, temporal, and Bayesian consistency to overcome projection noise and semantic ambiguity.

These models drive open-set classification, zero-shot generalization, and multi-modal retrieval tasks in 3D, extending applicability to dynamic or otherwise insufficiently annotated domains.

4. Benchmarks, Evaluation Protocols, and Challenges

The proliferation of 3D FMs has motivated the assembly of standardized benchmarks:

• E3D-Bench (Cong et al., 2 Jun 2025) systematically evaluates 16 leading end-to-end geometric FMs across sparse-view depth estimation, video depth estimation, 3D reconstruction, multi-view pose estimation, and novel view synthesis, all in both familiar and challenging out-of-distribution settings. Core metrics include Absolute Relative Error (AbsRel), inlier ratio ($\delta < \tau$), and standard photometric and reconstruction baselines.
• EFM3D (Straub et al., 14 Jun 2024) establishes egocentric 3D tasks—object detection and surface regression—on wearable sensor data using Project Aria, with baseline models such as Egocentric Voxel Lifting (EVL) that integrate 2D foundation features with local volumetric grids.
• GIQ (Michalkiewicz et al., 9 Jun 2025) probes geometric reasoning by evaluating depth reconstruction, symmetry detection (via linear probes on feature embeddings), mental rotation (synthetic/wild image splits), and zero-shot classification on a polyhedra corpus with known ground truth.

Benchmarks reveal that while current 3D FMs excel on sub-tasks (e.g. pairwise matching or depth maps), several challenges persist:

• Joint dense 3D reconstruction is more difficult than individual element estimation.
• Generalization degrades with extreme out-of-distribution shifts (e.g., air-ground, high-altitude, or “wild” imagery).
• Explicit geometric reasoning (e.g., mental rotation, convexity, symmetry beyond trivial cases) remains a fundamental gap (Michalkiewicz et al., 9 Jun 2025).
• Real-time inference and memory efficiency—critical for robotics and wearable deployments—are not yet achieved by most methods (Cong et al., 2 Jun 2025, Straub et al., 14 Jun 2024).

5. Resource Utilization, Training, and Specialization

Scaling 3D FMs for giant models or datasets is non-trivial given their memory and computation profiles:

• Merak (Lai et al., 2022) demonstrates automated 3D parallelism integrating data, tensor, and pipeline model parallelism. Notable runtime contributions include a shifted critical path pipeline schedule, stage-aware activation recomputation, and sub-pipelined tensor parallelism for overlapping communication/computation. This system achieves up to 1.61× speedup over prior state-of-the-art frameworks (on 20B parameter models with 64 GPUs).
• Specialized adaptation methods like LoRA3D (Lu et al., 10 Dec 2024) and Endo3DAC (Cui et al., 20 Mar 2025) refine large pre-trained FMs to target scenes—using only a handful of unlabeled images—with parameter-efficient low-rank adaptation, automatic confidence calibration, or dynamic task-specific gating, without external priors or manual labels.
• Downstream fine-tuning approaches (e.g., FP3’s LoRA policy adaptation (Yang et al., 11 Mar 2025), VISTA3D’s interactive branch for unexpected clinical anatomy (He et al., 7 Jun 2024)) require only modest additional data/compute and enable rapid domain transfer.

6. Applications and Domain-Specific Extension

3D FMs support an expanding array of domains and practical tasks:

• Robotics and Manipulation: FP3 (Yang et al., 11 Mar 2025), by directly consuming point cloud streams and language, generalizes with high sample efficiency (>90% success with 80 demos in unseen manipulation tasks) and robust zero-shot generalization. Robotic calibration and scene mapping, as in JCR (Zhi et al., 17 Apr 2024), are enabled with few images and commodity hardware, bypassing specialized calibration targets and unreliable photometric systems.
• Medical Imaging: VISTA3D (He et al., 7 Jun 2024) and CT-FM (Pai et al., 15 Jan 2025) provide state-of-the-art segmentation and triage for large-scale volumetric data, boasting improvements in Dice coefficients, zero-shot generalization to rare/novel structures, and robust anatomical clustering. Efficient adaptation (Endo3DAC (Cui et al., 20 Mar 2025)) for surgical 3D reconstruction matches or exceeds prior methods even in low-annotation regimes.
• Open-World 3D Understanding: Models such as OpenSU3D (Mohiuddin et al., 19 Jul 2024) and Bridge3D (Chen et al., 2023) incrementally build open-vocabulary 3D representations, supporting object-centric queries, spatial reasoning via LLMs, and annotation across dynamic indoor/outdoor environments without requiring full-scene preconstruction.
• Content Creation and Simulation: MeshXL (Chen et al., 31 May 2024) demonstrates that LLM–style architectures, trained autoregressively on sequentially ordered mesh tokenizations (via neural coordinate fields), yield output with coverage and MMD improvements over prior generative models and support conditional (image/text-guided) mesh synthesis at scale.

7. Future Directions and Unaddressed Challenges

Despite rapid progress, key open directions remain:

• Robust real-time inference will require further innovations in model compression, runtime scheduling, and memory partitioning (Lai et al., 2022, Cong et al., 2 Jun 2025).
• Explicit geometric priors or hybrid “learning + geometry” modules are needed to close performance gaps in challenging out-of-distribution tasks and geometric reasoning—specifically symmetry detection, mental rotation, and convexity classification (Michalkiewicz et al., 9 Jun 2025).
• Multimodal integration with audio, tactile, or language for universal physical world modeling is yet to be fully explored.
• Techniques for unsupervised adaptation, compositional scene editing, and large-scale synthetic data utilization (e.g., for outdoor, aerial, or underrepresented domains) are poised for expansion.

Continued open-source release of both code and curated evaluation data (as exemplified by CT-FM (Pai et al., 15 Jan 2025), VISTA3D (He et al., 7 Jun 2024), and E3D-Bench (Cong et al., 2 Jun 2025)) is expected to accelerate progress in building, refining, and deploying truly generalist 3D foundation models.

---

In conclusion, 3D foundation models unify geometric perception, semantic understanding, and generative capabilities across a growing spectrum of applications. Recent research demonstrates substantial progress in joint modeling, cross-modal fusion, resource orchestration, and incremental adaptation, but highlights persistent challenges in generalization, efficient inference, and geometry-aware reasoning. Evaluative benchmarks and cross-domain demonstration evidence provide a robust framework for ongoing development in this quickly evolving field.