---
title: Robotic Foundation Models in Robotics
url: https://www.emergentmind.com/topics/robotic-foundation-models-rfms
type: topic
---

# Robotic Foundation Models in Robotics

Robotic Foundation Models (RFMs) are large, generally pre-trained models designed as general-purpose controllers for robots: they take multimodal observations and task specifications—typically visual observations, natural language instructions, and sometimes proprioception or history—and output low-level actions, action chunks, or higher-level plans for diverse tasks, scenes, and embodiments [2510.18608][2605.13548]. Across the recent literature, the term covers Vision-Language-Action (VLA) models such as RT-1, RT-2, and OpenVLA, diffusion and flow-based policies such as Octo and the $\pi$-series, world-action models, and broader modular systems that couple foundation perception, reasoning, and control components [2402.02385][2507.10087]. RFMs replace narrowly engineered pipelines with reusable generalist cores, but the same literature identifies persistent limitations in adaptation, visual robustness, safety, compute efficiency, evaluation, and deployment readiness [2409.15250][2603.06749].

## 1. Definition and conceptual scope

The literature does not fix a single universal boundary for the term “robotic foundation model,” but several common elements recur. One line of work frames RFMs as high-capacity, Transformer-based models used directly as control policies and trained on large-scale robot datasets across many robots and tasks, often in VLA form, with policies of the form $\pi_\theta(a_t \mid o_t, g)$, where $o_t$ denotes multimodal observation and $g$ a language or symbolic goal [2510.18608]. Another line describes RFMs as large, generally pre-trained models designed as general-purpose controllers that map image sequences and instructions to low-level actions or trajectories in physical space, distinguishing discriminative VLAs from World-Action Models (WAMs) that jointly model world dynamics and action generation [2605.13548]. Survey work for industrial control sharpens the definition further by reserving “RFM” for models with a generalist core: systems that can efficiently adapt across varied tasks, settings, embodiments, and hardware configurations with low dedicated engineering or retraining effort, rather than pure perception encoders or single-task controllers [2603.06749].

This scope places RFMs between generic foundation models and classical robot stacks. Traditional systems decompose perception, planning, and control into task-specific modules. By contrast, the modern RFM approach uses a single neural network or a tightly integrated multimodal architecture that can be adapted across robots, tasks, and environments [2405.14005]. At the same time, RFMs remain distinct from generic LLMs and VLMs because they are embedded in closed-loop embodied decision-making: actions alter subsequent observations, safety constraints matter, and input-output interfaces must match robot kinematics, action spaces, and latency constraints [2603.06749][2402.05741].

The literature also reflects a definitional tension between narrow and broad uses of the term. Narrow definitions emphasize robot-centric policies and world models trained directly on embodied data, such as RT-1, RT-2, RT-X, Octo, RoboCat, or OpenVLA [2402.02385][2402.05741]. Broader reviews include modular systems in which LLMs, VLMs, open-vocabulary detectors, and task-specific policies jointly constitute the effective robotic foundation layer, especially in unstructured environments where foundation models dominate the cognitive stack before full end-to-end embodiment is achieved [2407.14296][2510.26855]. This suggests that “RFM” is best understood as a family of architectures organized around reusable, large-scale priors for robotic behavior rather than a single canonical model class.

## 2. Architectural patterns and multimodal interfaces

A common formalization treats an RFM as a conditional policy over multimodal observations and task context. In discriminative VLAs, image sequence $\mathcal{I} = \{i_1,\dots,i_T\}$ and instruction $L = [l_1,\dots,l_N]$ are mapped to an action sequence $\mathcal{A} = \{a_1,\dots,a_T\}$ by a model $f_\theta$, with training often based on an unweighted per-timestep loss over action dimensions [2605.13548]. Autoregressive variants, exemplified by OpenVLA, tokenize actions and predict them sequentially through a language-model-style decoder; diffusion and flow-based variants such as Octo and $\pi_0$ instead generate action chunks through iterative denoising or vector-field matching [2505.13255][2605.13548].

Within this common interface, the architectural space is heterogeneous. Survey work separates high-level planners, perception backbones, world models, and low-level control policies, but recent systems increasingly blur those boundaries [2507.10087][2402.02385]. RT-1 and RT-2 exemplify monolithic VLA architectures that fuse vision, language, and action into a unified sequence model. OpenVLA adapts a large language backbone for robotic control. WAMs such as Motus, LingBot-VA, and Fast-WAM explicitly couple latent world modeling and action generation [2605.13548]. Code-as-policy systems and LLM-based planners remain more modular, emitting plans, code, or skill sequences to be executed by downstream controllers [2402.05741][2507.10087].

A second axis of variation concerns perceptual representation. Most early RFMs are 2D-image-based, but FP3 introduces a “3D foundation policy” centered on point clouds and a diffusion transformer. FP3 has 1.3B parameters, is pre-trained on 60k DROID trajectories spanning 86 tasks and 564 scenes, and models language-conditioned action chunks from point-cloud observations, language, and proprioception [2503.08950]. In real-robot experiments, FP3 fine-tuned with only 80 demonstrations per downstream task achieved over 90% success rates in novel environments with unseen objects, while image-only variants degraded much more strongly in the wild [2503.08950]. This establishes 3D geometry as a first-class design choice rather than a peripheral modality.

A third axis is compositionality. The compositional paradigm proposed in “A Compositional Paradigm for Foundation Models: Towards Smarter Robotic Agents” frames RFMs not as monolithic parameter blocks but as reusable assemblies of modular components, such as LoRA adapters, small heads, and modality-specific encoders [2510.18608]. In the robotics-specific WSA architecture, pre-trained vision and language encoders are mapped into a common latent space through small adapters and combined through attention or learned weighting before an RL policy head acts on the fused state. The same paper argues that dynamic composition at inference time and hierarchical module reuse are central to more adaptable RFMs [2510.18608]. A plausible implication is that future RFMs will increasingly resemble systems of interoperable foundation components rather than single end-to-end transformers.

## 3. Training regimes, data, and scaling behavior

The training literature emphasizes large, heterogeneous embodied datasets. Open X-Embodiment and related corpora are repeatedly cited as the substrate for multi-task, multi-robot training [2510.18608][2409.15250]. OpenVLA is presented as a VLA model based on a LLaMA 6.7B reasoning core paired with DINO-v2 and SigLIP vision backbones and action-token outputs for 7-DoF end-effector deltas [2409.15250]. RT-X and related efforts extend this logic to cross-embodiment training; RT-X is described as involving 22 robots, 527 skills, and 160k tasks in the survey literature [2402.02385]. FP3 illustrates the complementary 3D route, using point-cloud observations and a diffusion transformer rather than purely 2D visual tokens [2503.08950].

A meta-analysis of 198 research papers on embodied AI and robotics reports that RFM performance follows power-law scaling of the form $y = \alpha x^\beta + \gamma$, where $y$ is failure rate and $x$ is compute, data size, or model size [2405.14005]. For high-quality fits with $R^2 > 0.7$, the reported median exponents are $\beta=-0.1503$ for compute, $\beta=-0.2663$ for data size, and $\beta=-0.3757$ for model size; for data scaling, seen tasks scale more favorably than unseen tasks, with median exponents $-0.3766$ and $-0.2313$ respectively [2405.14005]. The same study argues that robotic performance improves faster with added resources than classical language modeling, though still with diminishing returns. This provides a quantitative basis for the widespread empirical strategy of expanding model size and dataset diversity.

At the same time, several papers argue that scaling alone is insufficient. The compositional-paradigm paper explicitly states that scaling parameters and data has hit diminishing returns and that current RFMs are brittle in dynamic environments, hard to adapt without large-scale retraining, and inefficient in terms of compute and data [2510.18608]. AttenA+ makes a related argument from a different angle: current RFMs inherit a “flat” training paradigm from language modeling, weighting every timestep equally even though low-velocity manipulation phases are often physically critical [2605.13548]. By reweighting loss contributions according to inverse action velocity without changing architecture or parameters, AttenA+ improves OpenVLA-OFT to $98.60\%$ average success on Libero and improves Fast-WAM to $92.46\%$ average success on RoboTwin 2.0 [2605.13548]. This suggests that training objective design, not only model scale, remains a major lever in RFMs.

Training-free inference modifications can also matter. “Policy Contrastive Decoding for Robotic Foundation Models” identifies spurious visual correlations in OpenVLA, Octo, and $\pi_0$, and proposes a decoding rule that contrasts action distributions under original and object-masked observations to emphasize object-relevant cues [2505.13255]. Reported gains include an 8% improvement for $\pi_0$ in simulation and a 108% improvement in real-world evaluation [2505.13255]. The result is notable because it improves pre-trained policies without weight updates, implying that part of the generalization deficit lies in decoding and attention allocation rather than only in representation quality.

## 4. Adaptation, continual learning, and robustness

Adaptation is a central weakness of current RFMs. The compositional-paradigm paper argues that RFMs are usually trained in an offline, one-shot way on a fixed corpus, while real robots must adapt continuously to new objects, environments, and users [2510.18608]. Its proposed solution combines continual learning (CL) with compositionality. In the ViT-LoRA setting, each task receives its own adapter, and Hierarchical Adapter Merging (HAM) later consolidates adapters through similarity-aware merging rather than overwriting a single parameter set. On CUB-200 with 50 tasks, HAM reaches $55.17\% \pm 1.04$ accuracy in $170.61$ s, compared with $47.56\% \pm 3.77$ and $318.22$ s for SD-LoRA, and $36.02\% \pm 3.26$ and $196.89$ s for InfLoRA [2510.18608]. In robotic manipulation, the same paper reports that WSA achieves reward/step $0.60$, success rate $0.91$, and training time $14$ h, compared with OpenVLA at reward/step $0.10$, success rate $0.0$, and training time $92$ h under similar budgets [2510.18608].

A different robustness problem is visual catastrophic forgetting. “ReVLA: Reverting Visual Domain Limitation of Robotic Foundation Models” studies RT-1, Octo, and OpenVLA in a SIMPLER-based evaluation with novel YCB objects and distractors and shows that all three suffer large out-of-domain degradation [2409.15250]. The paper attributes part of OpenVLA’s weakness to forgetting in its DINO-v2 backbone: after end-to-end robot fine-tuning, the same DINO-v2 encoder that originally supports high-quality depth regression produces nearly constant or low-detail depth maps under DPT and linear-probe evaluations [2409.15250]. ReVLA addresses this by gradually reversing DINO-v2 and SigLIP weights toward their original pre-trained states during Fractal fine-tuning. In out-of-domain evaluation, OpenVLA attains $0.162$ overall success, whereas ReVLA (DS flip) reaches $0.287$ and ReVLA (DS gradual) reaches $0.278$; grasp success improves from $0.348$ for OpenVLA to $0.579$ for ReVLA (DS gradual) [2409.15250]. The paper’s reported “77%” and “66%” relative improvements for lifting and grasping directly target the visual generalization bottleneck of VLAs.

Robustness issues also appear in the language channel. “Embodied Red Teaming for Auditing Robotic Foundation Models” treats language-conditioned policies as RFMs and uses a VLM-driven red-teaming procedure to generate contextually grounded, failure-inducing instructions [2411.18676]. On CALVIN, 3D-Diffuser Actor drops from about $92.9\%$ success on training instructions to about $66.9\%$ on ERT($k=0$), while GR-1 drops from about $94.1\%$ on training instructions to about $54.7\%$ on ERT($k=1$); on RLBench, 3D-Diffuser falls from about $60.4\%$ on benchmark instructions to about $3.1\%$ on ERT-generated instructions [2411.18676]. OpenVLA’s success on a SimplerEnv Coke-can task falls from $76.0\%$ on the original instruction to $30.8\% \pm 3.8\%$ on ERT instructions [2411.18676]. This indicates that high benchmark success can coexist with brittle instruction generalization, and it suggests that future RFMs must learn “embodied similarity” between semantically equivalent instructions rather than relying on narrow training phrasings.

## 5. Safety, auditing, and human-facing evaluation

Safety emerges in the literature as a structurally separate problem from generalization. “Towards Safe Robot Foundation Models Using Inductive Biases” explicitly treats RFMs as unsafe-by-default generalist policies and wraps them in ATACOM, a geometric safety layer that modifies policy outputs without retraining [2505.10219]. Assuming a control-affine system $\dot{x}=f(x)+G(x)u$ and analytic safety constraints $g(x)\le 0$, the safe set is $\mathcal{S}=\{x\in\mathcal{X}\mid g(x)\le 0\}$, and ATACOM enforces forward invariance of $\mathcal{S}$ through a tangent-space action transformation [2505.10219]. The appeal is that safety comes from geometry and control structure rather than from hoping safe behavior emerges from demonstration data. In Franka manipulation, the paper reports that trajectories with ATACOM respect workspace, obstacle, and joint-limit constraints while maintaining task performance comparable to the base $\pi_0$ policy; in dynamic AirHockey with Octo, no safety violations are observed across training checkpoints once the safety layer is used [2505.10219].

Auditing work underscores why such explicit safety mechanisms are needed. ERT shows that state-of-the-art RFMs not only fail under adversarially varied instructions but can also behave unsafely, including causing objects to fall off the table under both explicitly unsafe and apparently neutral commands [2411.18676]. The industrial-readiness survey reinforces this diagnosis at scale. It evaluates 324 manipulation-capable RFMs using 149 criteria and 48,276 criterion-level decisions, and finds that even the highest-rated models satisfy only about 11–12% of all criteria [2603.06749]. Safety and compliance are particularly underdeveloped: across the full corpus, 1+-Coverage for safety is $0.034$ and the average criteria coverage for that implication is $0.003$, while real-time performance and cost-effective integration are similarly low [2603.06749]. The survey’s conclusion is that industrial maturity is limited and uneven and that benchmark success has not translated into auditable deployment stacks [2603.06749].

Evaluation is not only a technical matter but also a communicative one. “How Users Understand Robot Foundation Model Performance through Task Success Rates and Beyond” studies how non-experts interpret RFM evaluation data and finds that task success rate (TSR) is used largely as experts intend, namely as an indication of how likely the robot is to succeed on a task [2602.03920]. At the same time, participants strongly value other information types, especially natural-language failure descriptions and evidence from related tasks. Estimated TSR improves binary success prediction accuracy, but users want access to both previous real evaluation data and robot-provided estimates for novel tasks [2602.03920]. For encyclopedia purposes, this reframes RFM evaluation as a dual requirement: metrics must support research comparison, and they must also support calibrated trust, supervision, and risk management in actual use.

## 6. Deployment settings and future research directions

As RFMs move from laboratory policies to deployed systems, serving and systems design become part of the definition of the field. “ROSA: A Robotics Foundation Model Serving System for Robot Factories” treats RFMs not merely as individual models but as a shared AI substrate for fleets of factory robots, including System 1 action models, System 2 planners, safety models, and task-progress monitors [2607.01088]. ROSA replaces the single-robot, single-model serving assumption with shared GPU-pool serving and a robotics-aware scheduler that optimizes SLO-qualified factory productivity rather than individual-request latency. Implemented on top of Ray Serve with vLLM, PyTorch, and JAX backends, ROSA improves factory productivity by up to 12.06x over conventional dedicated serving systems [2607.01088]. This is a systems-level reminder that deployment bottlenecks are increasingly organizational and infrastructural, not only algorithmic.

Broader deployment surveys place current RFMs short of full autonomy in unstructured or industrial settings. A review of foundation models for autonomous robots in unstructured environments places current systems near Level 3, “Conditional Automation,” on a five-level autonomy scale, citing strong cognitive gains in perception, planning, and human-robot interaction but limited embodied robustness in construction, mining, urban outdoor, housing, and post-disaster domains [2407.14296]. The industrial-readiness survey reaches a compatible conclusion for collaborative robot platforms: progress toward industry-grade RFMs depends less on isolated benchmark peaks than on systematic incorporation of safety, real-time feasibility, robust perception, interaction, and cost-effective integration [2603.06749].

Future directions in the literature are correspondingly plural. One direction is richer structure: compositional CL, hierarchical adapter reuse, skill libraries, and modular world models [2510.18608]. A second is richer physics: velocity-aware training such as AttenA+, geometric safety layers such as ATACOM, and 3D-centric policies such as FP3 [2605.13548][2505.10219][2503.08950]. A third is richer evaluation: embodied red teaming, user-facing performance communication, and standardized industrial criteria [2411.18676][2602.03920][2603.06749]. A fourth is richer systems support: shared serving fabrics, admission control, failover, and multi-model orchestration [2607.01088]. Taken together, these lines suggest that the next phase of RFMs is unlikely to be defined by model scale alone. A plausible implication is that the field is moving toward architectures in which large generalist policies are only one layer inside broader compositional, safety-aware, and deployment-oriented robotic systems.

Source: https://www.emergentmind.com/topics/robotic-foundation-models-rfms