Foundation Model Embodied Reasoning (FoMER)

• FoMER is a framework that integrates foundation models with robotic systems, grounding high-level reasoning in physical sensorimotor actions.
• It employs hierarchical sense–plan–act architectures, affordance scoring, and world-model grounding to generate feasible, context-aware actions.
• FoMER is applied in robotic task planning, navigation, and interaction, with benchmarks evaluating spatial perception, safety, and temporal coherence.

Searching arXiv for recent FoMER and embodied foundation model papers to ground the article. Foundation Model Embodied Reasoning (FoMER) denotes the use of foundation models inside embodied systems so that semantic, symbolic, and commonsense reasoning is grounded in robot bodies, sensors, actuators, and dynamic environments, and directly drives perception, planning, control, and interaction (Khan et al., 14 Jul 2025). In the literature, the term names both a broader research program spanning multimodal robotics, embodied task planning, and world-model-based control, and a benchmarked capability of large multimodal models to produce step-by-step, physically grounded next actions in embodied settings (Dissanayake et al., 18 Sep 2025). Across these usages, FoMER concerns the conversion of foundation-model representations—typically learned from large-scale text, image, video, and multimodal corpora—into situated decisions whose adequacy depends on spatial relations, affordances, temporal structure, physical feasibility, safety, and feedback from physical or simulated worlds (Smeaton, 2024).

1. Definition and conceptual scope

FoMER extends the notion of reasoning with foundation models from disembodied prediction over text or images to action-conditioned interaction with environments. In the robotics review literature, this is formulated as placing LLMs, vision-LLMs (VLMs), or multimodal foundation models inside robotic stacks so that high-level reasoning is constrained, informed, and validated by physics, sensors, and environment feedback (Khan et al., 14 Jul 2025). In the benchmarking literature, FoMER is operationalized as the ability of large multimodal models to interpret multimodal observations, reason about physical constraints and safety, and generate valid next actions together with coherent reasoning trails (Dissanayake et al., 18 Sep 2025).

A recurrent conceptual distinction is that between reasoning-like behavior and mechanistic understanding of that behavior. One position paper argues that foundation models compress “logic, reasoning and inference into their structure and weights in ways we do not understand,” while warning against anthropomorphizing this as human understanding (Smeaton, 2024). This distinction matters in FoMER because embodied success can arise from sub-symbolic pattern structure in embedding spaces, affordance scoring, or policy regularities, even when the internal mechanism remains opaque.

FoMER also subsumes several adjacent notions that appear under different names. Embodied task planning benchmarks frame the problem as producing feasible ordered symbolic action sequences from visual observations and task descriptions. Humanoid loco-manipulation work frames it as embodied chain of action reasoning over an action library. Navigation work frames it as coherent reasoning in embodied scenes with explicit chain-of-thought and reactive control. Despite the variation, the common structure is a foundation model that must connect observation, latent reasoning, and action under embodiment-specific constraints (zhang et al., 2024).

2. Representations, architectures, and grounding mechanisms

A central representational assumption in FoMER is inherited from foundation models more broadly: knowledge is stored as distributed “parametric memory” in weights and embedding spaces rather than as explicit symbolic rules (Smeaton, 2024). Tokens, images, and multimodal inputs are mapped to high-dimensional vectors, and embodied use requires those representations to be connected to sensorimotor streams. The robotics review makes this explicit through a shared latent representation,

$$z = f_{\text{FM}}(x_{\text{vision}}, x_{\text{text}}, x_{\text{tactile}}, x_{\text{audio}}, x_{\text{state}}),$$

which functions as the interface between semantic interpretation and downstream planning or control (Khan et al., 14 Jul 2025).

Architecturally, the literature converges on several recurrent patterns. One is the hierarchical sense–plan–act stack, where VLMs provide semantic scene representations, LLMs convert goals and observations into action sequences, and either end-to-end vision-language-action policies or classical controllers execute those plans (Khan et al., 14 Jul 2025). A second is affordance-constrained action selection, formalized in the review through the SayCan-style composition

$$a^* = \arg\max_a \big(\pi_{\text{LLM}}(a \mid g)\cdot Q_{\text{aff}}(s,a)\big),$$

which multiplies linguistic appropriateness by physical feasibility (Khan et al., 14 Jul 2025). A third is explicit world-model grounding: FOUNDER learns a mapping from foundation-model embeddings to world-model states, $z_t \sim Q_\psi(\cdot \mid e_t)$, and then defines reward through predicted temporal distance to a mapped goal state, $r_t^{\text{TD}} = -D_\omega(z_t, z_g)$, so that task prompts become actionable goals in latent dynamics space (Wang et al., 15 Jul 2025).

Another important architectural theme is action abstraction. UniAct addresses cross-robot action heterogeneity by learning a discrete universal action space

$$\mathcal{U} = (u_1,\dots,u_N), \quad u_i \in \mathbb{R}^D,$$

with $N=256$ and $D=128$, so that the shared model reasons over embodiment-agnostic atomic behaviors and small embodiment-specific decoders map these back to robot-specific commands (Zheng et al., 17 Jan 2025). This makes the action interface itself part of FoMER’s representational problem, rather than an afterthought.

A common misconception is that FoMER is synonymous with LLM-driven symbolic planning. The manipulation literature shows that robust embodied reasoning can also be instantiated directly in the perception–control stack. SAM-E, for example, uses Segment Anything as a promptable object-centric visual encoder and predicts action sequences with multi-channel heatmaps, explicitly presenting FoMER as a visual foundation model plus structured sequence policy head rather than an LLM-centric planner (Zhang et al., 2024).

3. Data, training dynamics, and adaptation

FoMER systems rely on diverse training regimes that combine internet-scale priors with embodied specialization. A recurring argument is that improved reasoning cannot be explained by scale alone. One position paper emphasizes that novel training techniques and efficient architectures matter, and links qualitative capability transitions to grokking: prolonged training can move a model from memorization and shallow pattern matching toward more structured generalization (Smeaton, 2024). This observation reappears in embodied settings, where curriculum design, self-reflection, parameter-efficient adaptation, and reinforcement learning are repeatedly used to elicit reasoning from limited embodied data.

Several systems make this concrete. SAM-E adapts a SAM ViT-B encoder with LoRA on attention query and value projections, keeping total parameters at approximately 122M with about 35.7M trainable, and learns sequence imitation over RLBench with a horizon $h=5$; it reports $70.6\%\pm0.7$ average success across 18 tasks, few-shot adaptation performance of $63.2\%$ versus RVT’s $$a^* = \arg\max_a \big(\pi_{\text{LLM}}(a \mid g)\cdot Q_{\text{aff}}(s,a)\big),$$0, and about 1130 inference calls versus RVT’s about 6158 (Zhang et al., 2024). Embodied-R couples Qwen2.5-VL-72B-Instruct for perception with Qwen2.5-3B-Instruct for reasoning, trains with Group Relative Policy Optimization and a think–answer logical consistency reward on only 5,415 QA pairs across 1,492 videos, and reaches $$a^* = \arg\max_a \big(\pi_{\text{LLM}}(a \mid g)\cdot Q_{\text{aff}}(s,a)\big),$$1 average accuracy on eight embodied spatial reasoning tasks while reportedly surpassing OpenAI-o1 by $$a^* = \arg\max_a \big(\pi_{\text{LLM}}(a \mid g)\cdot Q_{\text{aff}}(s,a)\big),$$2 and Gemini-2.5-Pro by $$a^* = \arg\max_a \big(\pi_{\text{LLM}}(a \mid g)\cdot Q_{\text{aff}}(s,a)\big),$$3 on that test set (Zhao et al., 17 Apr 2025).

Reinforcement learning over reasoning traces has become a prominent FoMER training pattern. Nav-R1 constructs Nav-CoT-110K for embodied tasks, then applies GRPO with format, understanding, and navigation rewards together with a Fast-in-Slow policy split that separates deliberate semantic reasoning from low-latency control; it reports over $$a^* = \arg\max_a \big(\pi_{\text{LLM}}(a \mid g)\cdot Q_{\text{aff}}(s,a)\big),$$4 average improvement in reasoning and navigation performance and a successful real-world mobile robot deployment under limited onboard resources (Liu et al., 13 Sep 2025). Embodied-Reasoner uses a three-stage pipeline—imitation learning, rejection-sampled self-exploration, and reflection tuning—over 9.3k Observation-Thought-Action trajectories with 64k interactive images and 90k thinking processes, and reports $$a^* = \arg\max_a \big(\pi_{\text{LLM}}(a \mid g)\cdot Q_{\text{aff}}(s,a)\big),$$5 overall success, ahead of OpenAI o1, o3-mini, and Claude-3.7 on its embodied search benchmark (Zhang et al., 27 Mar 2025).

Large-scale embodied data engines amplify these effects. ABot-N0 trains on 16.9M expert trajectories and 5.0M reasoning samples across 7,802 high-fidelity 3D scenes covering $$a^* = \arg\max_a \big(\pi_{\text{LLM}}(a \mid g)\cdot Q_{\text{aff}}(s,a)\big),$$6, while Embodied-R1.5 builds a data system of over 15B tokens from 34 datasets and then applies a multi-task balanced RL recipe tailored to heterogeneous embodied tasks (Chu et al., 12 Feb 2026, Yuan et al., 9 Jun 2026). A plausible implication is that FoMER increasingly depends on data systems that jointly supervise spatial cognition, planning, correction, grounding, and action, rather than on action logs alone.

4. Evaluation, benchmarks, and what they measure

Evaluation in FoMER has bifurcated into capability-disentangled benchmarks and integrated embodied reasoning benchmarks. MFE-ETP defines four capability dimensions—Object Understanding, Spatio-Temporal Perception, Task Understanding, and Embodied Reasoning—and packages them into 1,184 test cases across six instruction formats. On that benchmark, GPT-4V obtains $$a^* = \arg\max_a \big(\pi_{\text{LLM}}(a \mid g)\cdot Q_{\text{aff}}(s,a)\big),$$7 on overall embodied QA aggregate and $$a^* = \arg\max_a \big(\pi_{\text{LLM}}(a \mid g)\cdot Q_{\text{aff}}(s,a)\big),$$8 success on 100 general-description embodied reasoning tasks, while MiniCPM records 0 correct plans out of 270 planning cases despite competitive QA scores. A regression analysis assigns the largest positive coefficient to spatial perception ($$a^* = \arg\max_a \big(\pi_{\text{LLM}}(a \mid g)\cdot Q_{\text{aff}}(s,a)\big),$$9), followed by object type recognition ($z_t \sim Q_\psi(\cdot \mid e_t)$0) and sequence understanding ($z_t \sim Q_\psi(\cdot \mid e_t)$1), indicating that spatial grounding is the strongest predictor of planning success (zhang et al., 2024).

The FoMER benchmark introduced for step-by-step embodied reasoning evaluates 1,112 samples across 10 task categories, 8 embodiments, and 3 robot types, and explicitly separates perceptual grounding from action reasoning through 10 scoring dimensions including faithfulness, spatial reasoning, physical causality, safety, commonsense, hallucination, redundancy, semantic coverage, reasoning alignment, and missing steps (Dissanayake et al., 18 Sep 2025). In that setting, True/False questions are easiest, multiple choice is hardest, and human performance remains substantially ahead of current large multimodal models; on a human-evaluated subset, the human baseline is $z_t \sim Q_\psi(\cdot \mid e_t)$2, whereas Gemini 2.5 Pro is in the $z_t \sim Q_\psi(\cdot \mid e_t)$3 range and OpenAI o4-mini is the strongest overall model at roughly $z_t \sim Q_\psi(\cdot \mid e_t)$4 reasoning accuracy (Dissanayake et al., 18 Sep 2025).

These evaluations also sharpen an important caution already present in the conceptual literature: narrow benchmark scores do not certify embodied competence. The position paper on foundation-model reasoning stresses contamination, preference-vs-correctness mismatch, and the difficulty of judging free-form reasoning responses, especially when current suites focus on narrow aspects of performance (Smeaton, 2024). This suggests that FoMER evaluation must combine textual reasoning quality with task success, physical feasibility, temporal coherence, safety, and robustness to distribution shift.

5. Representative systems and application domains

The current FoMER landscape spans manipulation, navigation, humanoid loco-manipulation, cloud-side annotation, and unified embodied foundation models. Representative systems reported in the literature include SAM-E, Embodied-R, Nav-R1, Embodied-R1.5, and ABot-N0, each emphasizing a different point in the perception–reasoning–action spectrum (Zhang et al., 2024, Zhao et al., 17 Apr 2025, Liu et al., 13 Sep 2025, Yuan et al., 9 Jun 2026, Chu et al., 12 Feb 2026).

System	Embodied emphasis	Reported headline result
SAM-E	Visual FM + sequence imitation for 3D manipulation	$z_t \sim Q_\psi(\cdot \mid e_t)$5 average success on 18 RLBench tasks
Embodied-R	VLM perception + 3B LM reasoning	$z_t \sim Q_\psi(\cdot \mid e_t)$6 average on 8 embodied spatial tasks
Nav-R1	3D VLM + CoT + GRPO + Fast-in-Slow navigation	Over $z_t \sim Q_\psi(\cdot \mid e_t)$7 average improvement on reasoning and navigation
Embodied-R1.5	Unified EFM with planning, grounding, correction, pointing	SOTA on 16 of 24 embodied VLM benchmarks
ABot-N0	Unified VLA for five navigation tasks	New SOTA across 7 benchmarks

Manipulation-oriented FoMER includes both direct policy learning and closed-loop reasoning. SAM-E treats long-horizon motor behavior as sequence prediction over multi-view heatmaps rather than step-wise action selection (Zhang et al., 2024). Embodied-R1.5 unifies embodied cognition, planning, correction, and pointing inside one 8B model, then fine-tunes a VLA head with comparatively small action datasets; it further exposes a Planner-Grounder-Corrector closed loop that supports instruction following, affordance grounding, articulated object manipulation, and long-horizon tasks in real robots (Yuan et al., 9 Jun 2026). UniAct contributes a complementary action-side abstraction by introducing learned universal actions that eliminate cross-embodiment action heterogeneity and reportedly allow a 0.5B model to outperform embodied foundation models 14 times larger in cross-embodiment evaluations (Zheng et al., 17 Jan 2025).

Navigation-focused FoMER increasingly combines explicit reasoning with real-time control. Nav-R1 treats navigation as coherent embodied chain-of-thought in 3D scenes and explicitly decouples slow semantic reasoning from fast reactive control (Liu et al., 13 Sep 2025). ABot-N0 pushes unification further by combining a Qwen3-4B Cognitive Brain with a flow-matching Action Expert and a hierarchical “Brain-Action” architecture across Point-Goal, Object-Goal, Instruction-Following, POI-Goal, and Person-Following, together with an agentic planner and topological memory (Chu et al., 12 Feb 2026). In humanoid robotics, embodied chain of action reasoning has been instantiated as a two-stage pipeline in which GPT-4 first produces scene descriptions and then generates action chains over a humanoid action library; on three real-world tasks, this framework reports $z_t \sim Q_\psi(\cdot \mid e_t)$8 executable action sequences and $z_t \sim Q_\psi(\cdot \mid e_t)$9 success on average, improving over an Action Behavior Planner baseline (Hao et al., 13 Apr 2025).

FoMER also includes systems whose primary role is not direct control but embodied understanding. Thinker targets first-person video reasoning, explicitly addressing confusion between third-person and first-person perspectives and the tendency to overlook information in video endings by jointly incorporating key frames and full video sequences; it reports state-of-the-art results on RoboVQA and Egoplan-Bench2 (Pan et al., 29 Jan 2026). XEmbodied augments a large multimodal model with a structured 3D Adapter and an Efficient Image-Embodied Adapter so that 3D geometry and physical cues become first-class inputs for scenario mining and embodied VQA, improving spatial reasoning, traffic semantics, and embodied affordance across 18 public benchmarks (Qian et al., 20 Apr 2026). FOUNDER, in turn, exemplifies the world-model branch of FoMER by grounding foundation-model prompt embeddings into world-model state space and learning reward-free goal-conditioned policies through imagination (Wang et al., 15 Jul 2025).

6. Limitations, systems constraints, and open problems

The main limitations of FoMER are consistently structural rather than merely incremental. First, current foundation models remain weakly embodied: they are still predominantly trained on internet text and images and therefore lack direct experience with contact physics, friction, compliance, safety-critical corner cases, and continual sensorimotor adaptation (Khan et al., 14 Jul 2025). Second, mechanistic understanding remains limited. The neuroscience analogy advanced in the position paper is explicitly cautionary: probing neurons or attention heads can reveal structured features, including monosemantic combinations, but it does not yet yield a deep theory of how reasoning is implemented internally, nor does it justify treating foundation models as brains (Smeaton, 2024).

Third, deployment is constrained by systems realities. The edge-systems survey frames embodied foundation model deployment as a “Deployment Gauntlet” of eight coupled barriers—sensor fusion tax, heterogeneous compute mismatch, unified memory bottleneck, energy and thermal ceiling, long-horizon execution drift, safety and verification gap, OS and scheduling bottleneck, and I/O and communication bottleneck—and argues that reliable deployment requires system-level co-design across memory, scheduling, communication, and architecture (Grover et al., 16 Mar 2026). In that analysis, autoregressive VLA policies are primarily memory-bandwidth-limited, whereas diffusion-based controllers are more compute- and thermal-limited, implying that FoMER architectures must often separate fast control from slower semantic reasoning.

Fourth, evaluation remains imperfect. Benchmarks such as MFE-ETP and FoMER-Bench show substantial human–model gaps and expose frequent failures in spatial grounding, temporal integration, social navigation, and safety-aware reasoning, but they still simplify the open-world complexity of physical environments (zhang et al., 2024, Dissanayake et al., 18 Sep 2025). This suggests that benchmark progress should be interpreted as evidence of partial competence rather than as proof of general embodied intelligence.

The most consistent future directions in the literature are tightly coupled embodied models, richer multimodal datasets, better grounding, stronger interpretability, and hybrid control architectures. The robotics review explicitly recommends moving from “language/vision first, physics later” toward models that jointly learn semantics and dynamics, and combining foundation models with world models, MPC, and verification layers (Khan et al., 14 Jul 2025). A plausible implication is that FoMER will increasingly be built as a layered system: foundation models for semantics and long-horizon reasoning, world models for counterfactual simulation, explicit grounding modules for geometry and affordance, and certified low-level controllers for timing and safety.