Robotic Foundation Models in Robotics

• Robotic Foundation Models (RFMs) are large-scale, multimodal controllers that map visual, linguistic, and proprioceptive inputs to low-level actions for diverse tasks.
• They employ unified architectures like transformers, diffusion-based policies, and modular components to integrate perception, planning, and control.
• RFMs aim to simplify robotic system design with reusable cores while addressing challenges in adaptation, visual robustness, and safety in dynamic settings.

Robotic Foundation Models (RFMs) are large, generally pre-trained models designed as general-purpose controllers for robots: they take multimodal observations and task specifications—typically visual observations, natural language instructions, and sometimes proprioception or history—and output low-level actions, action chunks, or higher-level plans for diverse tasks, scenes, and embodiments (Quarantiello et al., 21 Oct 2025, Peng et al., 13 May 2026). Across the recent literature, the term covers Vision-Language-Action (VLA) models such as RT-1, RT-2, and OpenVLA, diffusion and flow-based policies such as Octo and the $\pi$-series, world-action models, and broader modular systems that couple foundation perception, reasoning, and control components (Xu et al., 2024, Khan et al., 14 Jul 2025). RFMs replace narrowly engineered pipelines with reusable generalist cores, but the same literature identifies persistent limitations in adaptation, visual robustness, safety, compute efficiency, evaluation, and deployment readiness (Dey et al., 2024, Kube et al., 6 Mar 2026).

1. Definition and conceptual scope

The literature does not fix a single universal boundary for the term “robotic foundation model,” but several common elements recur. One line of work frames RFMs as high-capacity, Transformer-based models used directly as control policies and trained on large-scale robot datasets across many robots and tasks, often in VLA form, with policies of the form $\pi_\theta(a_t \mid o_t, g)$, where $o_t$ denotes multimodal observation and $g$ a language or symbolic goal (Quarantiello et al., 21 Oct 2025). Another line describes RFMs as large, generally pre-trained models designed as general-purpose controllers that map image sequences and instructions to low-level actions or trajectories in physical space, distinguishing discriminative VLAs from World-Action Models (WAMs) that jointly model world dynamics and action generation (Peng et al., 13 May 2026). Survey work for industrial control sharpens the definition further by reserving “RFM” for models with a generalist core: systems that can efficiently adapt across varied tasks, settings, embodiments, and hardware configurations with low dedicated engineering or retraining effort, rather than pure perception encoders or single-task controllers (Kube et al., 6 Mar 2026).

This scope places RFMs between generic foundation models and classical robot stacks. Traditional systems decompose perception, planning, and control into task-specific modules. By contrast, the modern RFM approach uses a single neural network or a tightly integrated multimodal architecture that can be adapted across robots, tasks, and environments (Sartor et al., 2024). At the same time, RFMs remain distinct from generic LLMs and VLMs because they are embedded in closed-loop embodied decision-making: actions alter subsequent observations, safety constraints matter, and input-output interfaces must match robot kinematics, action spaces, and latency constraints (Kube et al., 6 Mar 2026, Kawaharazuka et al., 2024).

The literature also reflects a definitional tension between narrow and broad uses of the term. Narrow definitions emphasize robot-centric policies and world models trained directly on embodied data, such as RT-1, RT-2, RT-X, Octo, RoboCat, or OpenVLA (Xu et al., 2024, Kawaharazuka et al., 2024). Broader reviews include modular systems in which LLMs, VLMs, open-vocabulary detectors, and task-specific policies jointly constitute the effective robotic foundation layer, especially in unstructured environments where foundation models dominate the cognitive stack before full end-to-end embodiment is achieved (Naderi et al., 2024, Mirjalili, 30 Oct 2025). This suggests that “RFM” is best understood as a family of architectures organized around reusable, large-scale priors for robotic behavior rather than a single canonical model class.

2. Architectural patterns and multimodal interfaces

A common formalization treats an RFM as a conditional policy over multimodal observations and task context. In discriminative VLAs, image sequence $\mathcal{I} = \{i_1,\dots,i_T\}$ and instruction $L = [l_1,\dots,l_N]$ are mapped to an action sequence $\mathcal{A} = \{a_1,\dots,a_T\}$ by a model $f_\theta$, with training often based on an unweighted per-timestep loss over action dimensions (Peng et al., 13 May 2026). Autoregressive variants, exemplified by OpenVLA, tokenize actions and predict them sequentially through a language-model-style decoder; diffusion and flow-based variants such as Octo and $\pi_0$ instead generate action chunks through iterative denoising or vector-field matching (Wu et al., 19 May 2025, Peng et al., 13 May 2026).

Within this common interface, the architectural space is heterogeneous. Survey work separates high-level planners, perception backbones, world models, and low-level control policies, but recent systems increasingly blur those boundaries (Khan et al., 14 Jul 2025, Xu et al., 2024). RT-1 and RT-2 exemplify monolithic VLA architectures that fuse vision, language, and action into a unified sequence model. OpenVLA adapts a large language backbone for robotic control. WAMs such as Motus, LingBot-VA, and Fast-WAM explicitly couple latent world modeling and action generation (Peng et al., 13 May 2026). Code-as-policy systems and LLM-based planners remain more modular, emitting plans, code, or skill sequences to be executed by downstream controllers (Kawaharazuka et al., 2024, Khan et al., 14 Jul 2025).

A second axis of variation concerns perceptual representation. Most early RFMs are 2D-image-based, but FP3 introduces a “3D foundation policy” centered on point clouds and a diffusion transformer. FP3 has 1.3B parameters, is pre-trained on 60k DROID trajectories spanning 86 tasks and 564 scenes, and models language-conditioned action chunks from point-cloud observations, language, and proprioception (Yang et al., 11 Mar 2025). In real-robot experiments, FP3 fine-tuned with only 80 demonstrations per downstream task achieved over 90% success rates in novel environments with unseen objects, while image-only variants degraded much more strongly in the wild (Yang et al., 11 Mar 2025). This establishes 3D geometry as a first-class design choice rather than a peripheral modality.

A third axis is compositionality. The compositional paradigm proposed in “A Compositional Paradigm for Foundation Models: Towards Smarter Robotic Agents” frames RFMs not as monolithic parameter blocks but as reusable assemblies of modular components, such as LoRA adapters, small heads, and modality-specific encoders (Quarantiello et al., 21 Oct 2025). In the robotics-specific WSA architecture, pre-trained vision and language encoders are mapped into a common latent space through small adapters and combined through attention or learned weighting before an RL policy head acts on the fused state. The same paper argues that dynamic composition at inference time and hierarchical module reuse are central to more adaptable RFMs (Quarantiello et al., 21 Oct 2025). A plausible implication is that future RFMs will increasingly resemble systems of interoperable foundation components rather than single end-to-end transformers.

3. Training regimes, data, and scaling behavior

The training literature emphasizes large, heterogeneous embodied datasets. Open X-Embodiment and related corpora are repeatedly cited as the substrate for multi-task, multi-robot training (Quarantiello et al., 21 Oct 2025, Dey et al., 2024). OpenVLA is presented as a VLA model based on a LLaMA 6.7B reasoning core paired with DINO-v2 and SigLIP vision backbones and action-token outputs for 7-DoF end-effector deltas (Dey et al., 2024). RT-X and related efforts extend this logic to cross-embodiment training; RT-X is described as involving 22 robots, 527 skills, and 160k tasks in the survey literature (Xu et al., 2024). FP3 illustrates the complementary 3D route, using point-cloud observations and a diffusion transformer rather than purely 2D visual tokens (Yang et al., 11 Mar 2025).

A meta-analysis of 198 research papers on embodied AI and robotics reports that RFM performance follows power-law scaling of the form $y = \alpha x^\beta + \gamma$, where $\pi_\theta(a_t \mid o_t, g)$0 is failure rate and $\pi_\theta(a_t \mid o_t, g)$1 is compute, data size, or model size (Sartor et al., 2024). For high-quality fits with $\pi_\theta(a_t \mid o_t, g)$2, the reported median exponents are $\pi_\theta(a_t \mid o_t, g)$3 for compute, $\pi_\theta(a_t \mid o_t, g)$4 for data size, and $\pi_\theta(a_t \mid o_t, g)$5 for model size; for data scaling, seen tasks scale more favorably than unseen tasks, with median exponents $\pi_\theta(a_t \mid o_t, g)$6 and $\pi_\theta(a_t \mid o_t, g)$7 respectively (Sartor et al., 2024). The same study argues that robotic performance improves faster with added resources than classical language modeling, though still with diminishing returns. This provides a quantitative basis for the widespread empirical strategy of expanding model size and dataset diversity.

At the same time, several papers argue that scaling alone is insufficient. The compositional-paradigm paper explicitly states that scaling parameters and data has hit diminishing returns and that current RFMs are brittle in dynamic environments, hard to adapt without large-scale retraining, and inefficient in terms of compute and data (Quarantiello et al., 21 Oct 2025). AttenA+ makes a related argument from a different angle: current RFMs inherit a “flat” training paradigm from language modeling, weighting every timestep equally even though low-velocity manipulation phases are often physically critical (Peng et al., 13 May 2026). By reweighting loss contributions according to inverse action velocity without changing architecture or parameters, AttenA+ improves OpenVLA-OFT to $\pi_\theta(a_t \mid o_t, g)$8 average success on Libero and improves Fast-WAM to $\pi_\theta(a_t \mid o_t, g)$9 average success on RoboTwin 2.0 (Peng et al., 13 May 2026). This suggests that training objective design, not only model scale, remains a major lever in RFMs.

Training-free inference modifications can also matter. “Policy Contrastive Decoding for Robotic Foundation Models” identifies spurious visual correlations in OpenVLA, Octo, and $o_t$0, and proposes a decoding rule that contrasts action distributions under original and object-masked observations to emphasize object-relevant cues (Wu et al., 19 May 2025). Reported gains include an 8% improvement for $o_t$1 in simulation and a 108% improvement in real-world evaluation (Wu et al., 19 May 2025). The result is notable because it improves pre-trained policies without weight updates, implying that part of the generalization deficit lies in decoding and attention allocation rather than only in representation quality.

4. Adaptation, continual learning, and robustness

Adaptation is a central weakness of current RFMs. The compositional-paradigm paper argues that RFMs are usually trained in an offline, one-shot way on a fixed corpus, while real robots must adapt continuously to new objects, environments, and users (Quarantiello et al., 21 Oct 2025). Its proposed solution combines continual learning (CL) with compositionality. In the ViT-LoRA setting, each task receives its own adapter, and Hierarchical Adapter Merging (HAM) later consolidates adapters through similarity-aware merging rather than overwriting a single parameter set. On CUB-200 with 50 tasks, HAM reaches $o_t$2 accuracy in $o_t$3 s, compared with $o_t$4 and $o_t$5 s for SD-LoRA, and $o_t$6 and $o_t$7 s for InfLoRA (Quarantiello et al., 21 Oct 2025). In robotic manipulation, the same paper reports that WSA achieves reward/step $o_t$8, success rate $o_t$9, and training time $g$0 h, compared with OpenVLA at reward/step $g$1, success rate $g$2, and training time $g$3 h under similar budgets (Quarantiello et al., 21 Oct 2025).

A different robustness problem is visual catastrophic forgetting. “ReVLA: Reverting Visual Domain Limitation of Robotic Foundation Models” studies RT-1, Octo, and OpenVLA in a SIMPLER-based evaluation with novel YCB objects and distractors and shows that all three suffer large out-of-domain degradation (Dey et al., 2024). The paper attributes part of OpenVLA’s weakness to forgetting in its DINO-v2 backbone: after end-to-end robot fine-tuning, the same DINO-v2 encoder that originally supports high-quality depth regression produces nearly constant or low-detail depth maps under DPT and linear-probe evaluations (Dey et al., 2024). ReVLA addresses this by gradually reversing DINO-v2 and SigLIP weights toward their original pre-trained states during Fractal fine-tuning. In out-of-domain evaluation, OpenVLA attains $g$4 overall success, whereas ReVLA (DS flip) reaches $g$5 and ReVLA (DS gradual) reaches $g$6; grasp success improves from $g$7 for OpenVLA to $g$8 for ReVLA (DS gradual) (Dey et al., 2024). The paper’s reported “77%” and “66%” relative improvements for lifting and grasping directly target the visual generalization bottleneck of VLAs.

Robustness issues also appear in the language channel. “Embodied Red Teaming for Auditing Robotic Foundation Models” treats language-conditioned policies as RFMs and uses a VLM-driven red-teaming procedure to generate contextually grounded, failure-inducing instructions (Karnik et al., 2024). On CALVIN, 3D-Diffuser Actor drops from about $g$9 success on training instructions to about $\mathcal{I} = \{i_1,\dots,i_T\}$0 on ERT($\mathcal{I} = \{i_1,\dots,i_T\}$1), while GR-1 drops from about $\mathcal{I} = \{i_1,\dots,i_T\}$2 on training instructions to about $\mathcal{I} = \{i_1,\dots,i_T\}$3 on ERT($\mathcal{I} = \{i_1,\dots,i_T\}$4); on RLBench, 3D-Diffuser falls from about $\mathcal{I} = \{i_1,\dots,i_T\}$5 on benchmark instructions to about $\mathcal{I} = \{i_1,\dots,i_T\}$6 on ERT-generated instructions (Karnik et al., 2024). OpenVLA’s success on a SimplerEnv Coke-can task falls from $\mathcal{I} = \{i_1,\dots,i_T\}$7 on the original instruction to $\mathcal{I} = \{i_1,\dots,i_T\}$8 on ERT instructions (Karnik et al., 2024). This indicates that high benchmark success can coexist with brittle instruction generalization, and it suggests that future RFMs must learn “embodied similarity” between semantically equivalent instructions rather than relying on narrow training phrasings.

5. Safety, auditing, and human-facing evaluation

Safety emerges in the literature as a structurally separate problem from generalization. “Towards Safe Robot Foundation Models Using Inductive Biases” explicitly treats RFMs as unsafe-by-default generalist policies and wraps them in ATACOM, a geometric safety layer that modifies policy outputs without retraining (Tölle et al., 15 May 2025). Assuming a control-affine system $\mathcal{I} = \{i_1,\dots,i_T\}$9 and analytic safety constraints $L = [l_1,\dots,l_N]$0, the safe set is $L = [l_1,\dots,l_N]$1, and ATACOM enforces forward invariance of $L = [l_1,\dots,l_N]$2 through a tangent-space action transformation (Tölle et al., 15 May 2025). The appeal is that safety comes from geometry and control structure rather than from hoping safe behavior emerges from demonstration data. In Franka manipulation, the paper reports that trajectories with ATACOM respect workspace, obstacle, and joint-limit constraints while maintaining task performance comparable to the base $L = [l_1,\dots,l_N]$3 policy; in dynamic AirHockey with Octo, no safety violations are observed across training checkpoints once the safety layer is used (Tölle et al., 15 May 2025).

Auditing work underscores why such explicit safety mechanisms are needed. ERT shows that state-of-the-art RFMs not only fail under adversarially varied instructions but can also behave unsafely, including causing objects to fall off the table under both explicitly unsafe and apparently neutral commands (Karnik et al., 2024). The industrial-readiness survey reinforces this diagnosis at scale. It evaluates 324 manipulation-capable RFMs using 149 criteria and 48,276 criterion-level decisions, and finds that even the highest-rated models satisfy only about 11–12% of all criteria (Kube et al., 6 Mar 2026). Safety and compliance are particularly underdeveloped: across the full corpus, 1+-Coverage for safety is $L = [l_1,\dots,l_N]$4 and the average criteria coverage for that implication is $L = [l_1,\dots,l_N]$5, while real-time performance and cost-effective integration are similarly low (Kube et al., 6 Mar 2026). The survey’s conclusion is that industrial maturity is limited and uneven and that benchmark success has not translated into auditable deployment stacks (Kube et al., 6 Mar 2026).

Evaluation is not only a technical matter but also a communicative one. “How Users Understand Robot Foundation Model Performance through Task Success Rates and Beyond” studies how non-experts interpret RFM evaluation data and finds that task success rate (TSR) is used largely as experts intend, namely as an indication of how likely the robot is to succeed on a task (Sheidlower et al., 3 Feb 2026). At the same time, participants strongly value other information types, especially natural-language failure descriptions and evidence from related tasks. Estimated TSR improves binary success prediction accuracy, but users want access to both previous real evaluation data and robot-provided estimates for novel tasks (Sheidlower et al., 3 Feb 2026). For encyclopedia purposes, this reframes RFM evaluation as a dual requirement: metrics must support research comparison, and they must also support calibrated trust, supervision, and risk management in actual use.

6. Deployment settings and future research directions

As RFMs move from laboratory policies to deployed systems, serving and systems design become part of the definition of the field. “ROSA: A Robotics Foundation Model Serving System for Robot Factories” treats RFMs not merely as individual models but as a shared AI substrate for fleets of factory robots, including System 1 action models, System 2 planners, safety models, and task-progress monitors (Jiang et al., 1 Jul 2026). ROSA replaces the single-robot, single-model serving assumption with shared GPU-pool serving and a robotics-aware scheduler that optimizes SLO-qualified factory productivity rather than individual-request latency. Implemented on top of Ray Serve with vLLM, PyTorch, and JAX backends, ROSA improves factory productivity by up to 12.06x over conventional dedicated serving systems (Jiang et al., 1 Jul 2026). This is a systems-level reminder that deployment bottlenecks are increasingly organizational and infrastructural, not only algorithmic.

Broader deployment surveys place current RFMs short of full autonomy in unstructured or industrial settings. A review of foundation models for autonomous robots in unstructured environments places current systems near Level 3, “Conditional Automation,” on a five-level autonomy scale, citing strong cognitive gains in perception, planning, and human-robot interaction but limited embodied robustness in construction, mining, urban outdoor, housing, and post-disaster domains (Naderi et al., 2024). The industrial-readiness survey reaches a compatible conclusion for collaborative robot platforms: progress toward industry-grade RFMs depends less on isolated benchmark peaks than on systematic incorporation of safety, real-time feasibility, robust perception, interaction, and cost-effective integration (Kube et al., 6 Mar 2026).

Future directions in the literature are correspondingly plural. One direction is richer structure: compositional CL, hierarchical adapter reuse, skill libraries, and modular world models (Quarantiello et al., 21 Oct 2025). A second is richer physics: velocity-aware training such as AttenA+, geometric safety layers such as ATACOM, and 3D-centric policies such as FP3 (Peng et al., 13 May 2026, Tölle et al., 15 May 2025, Yang et al., 11 Mar 2025). A third is richer evaluation: embodied red teaming, user-facing performance communication, and standardized industrial criteria (Karnik et al., 2024, Sheidlower et al., 3 Feb 2026, Kube et al., 6 Mar 2026). A fourth is richer systems support: shared serving fabrics, admission control, failover, and multi-model orchestration (Jiang et al., 1 Jul 2026). Taken together, these lines suggest that the next phase of RFMs is unlikely to be defined by model scale alone. A plausible implication is that the field is moving toward architectures in which large generalist policies are only one layer inside broader compositional, safety-aware, and deployment-oriented robotic systems.