Embodied-LM: Grounded Multimodal Cognition

• Embodied-LM is a framework that couples language models with sensory inputs and motor control to achieve grounded sensorimotor reasoning.
• It employs integrated modules for active perception, planning, and memory to support real-time adaptation and effective task execution.
• Applications span robotics, social VR, autonomous driving, and domestic manipulation, highlighting its versatility and practical impact.

Embodied-LM denotes a family of systems in which LLMs, and more broadly multimodal LLMs, are coupled to perception, memory, planning, and action so that natural-language reasoning is grounded in sensorimotor context. In the dominant embodied-AI formulation, intelligence emerges from the closed-loop coupling of active perception, embodied cognition, and dynamic interaction, with hardware embodiment serving as a critical enabler under real-world latency and power constraints; within that formulation, “Embodied-LM” refers specifically to the use of LLMs or MLLMs to empower embodied cognition and control. The term has also been used for specific system designs, including decoder-only multimodal models that ingest continuous sensor modalities, GPT-4–driven embodied conversational agents in social VR, and prototype neurosymbolic systems that compile language into executable schematic representations (Feng et al., 24 Sep 2025, Driess et al., 2023, Pan et al., 2024, Olivier et al., 3 Sep 2025).

1. Conceptual scope and terminological variants

In contemporary embodied AI, LLM-only approaches center on text-only semantics, planning, and action sequencing. They are strongest at semantic reasoning and task decomposition, converting natural-language instructions into natural-language action sequences and plans, but they remain constrained by fixed action libraries and environment-specific feasibility checks, and they struggle with physical grounding and real-time adaptation. MLLMs extend this regime to vision, language, audio, and tactile signals, and can act across the entire embodied pipeline, bridging high-level multimodal inputs to low-level motor actions. Within this broader class, VLMs pair visual and language understanding for goal-driven tasks with fixed action spaces, whereas VLAs integrate multimodal inputs with low-level control via differentiable pipelines, tokenization schemes, and action de-tokenization; the survey literature explicitly cites PaLM-E, RT-2, and Octo as representative instances of these distinctions (Feng et al., 24 Sep 2025).

A separate but related usage treats “Embodied-LM” not as a generic category but as the name of a neurosymbolic prototype. In that formulation, natural-language scenarios are interpreted through image schemas such as PATH and CONTAINER, then compiled into Answer Set Programming with a spatial constraint backend. The result is an executable schematic representation whose logical consequences can be inspected through stable models and geometric witnesses. This usage narrows the term to embodied cognitive structures rather than to multimodal control stacks, but it shares the broader objective of replacing ungrounded textual reasoning with interpretable, physically constrained inference (Olivier et al., 3 Sep 2025).

2. Architectural organization

Across implementations, Embodied-LM systems tend to factor into a small set of recurring modules: an instruction interpreter, a planner, a skill library or primitive-action interface, a controller, a memory or map module, and a closed feedback loop. Active perception covers SLAM, 3D scene understanding, and exploration; embodied cognition covers task-driven self-planning, memory-driven self-reflection, and multimodal foundation models; dynamic interaction covers low-level action control, high-level behavioral interaction, and collaborative decision-making. The same literature also treats hardware embodiment as a first-class component, emphasizing quantization, pruning, compiler-level optimization with TVM/LLVM/CUDA, and domain-specific accelerators such as TPU, FPGA, CGRA, and ASIC to satisfy latency and power constraints (Feng et al., 24 Sep 2025).

PaLM-E provides a canonical architectural instance of this pattern. It extends a decoder-only LLM by injecting continuous observations from vision and state estimation directly into the language embedding space, so that the model processes multi-modal sentences in which modality placeholders are replaced by learned modality embeddings. Visual encoders, state encoders, and text tokens are therefore handled by a single self-attentive transformer stack, allowing the model to plan sequential robotic manipulation, answer visual questions, and perform captioning within one multimodal token space (Driess et al., 2023).

OPEx makes the decomposition explicit for embodied instruction following. It separates the policy into Observer, Planner, and Executor, grounding the LLM-centric components in semantic mapping and deterministic low-level control. In ALFRED and ALFWorld, the Planner produces subgoals, the Executor selects skills such as NavigateToObject, Explore, LookAround, or RequireReplan, and the low-level policy translates those skills into navigation and interaction primitives. Its ablations show that better perception and better low-level action heuristics materially alter success rates, indicating that Embodied-LM competence depends not only on language reasoning but also on the quality of the grounding substrate beneath it (Shi et al., 2024).

3. Grounding, memory, and embodied knowledge

Grounding in Embodied-LM is ordinarily achieved through a combination of vision-language alignment, semantic augmentation of SLAM, scene graphs and semantic lexicons, affordance grounding through action libraries, episodic memory buffers, and closed-loop revision from collision, affordance, and sensor feedback. Mapping is not merely geometric: object classes, relations, and goal-conditioned semantics are injected into memory so that later planning can be conditioned on what was observed, what was attempted, and what succeeded or failed (Feng et al., 24 Sep 2025).

The memory problem is central because egocentric perception is partial and dynamic. The abstract of "LLM as A Robotic Brain: Unifying Egocentric Memory and Control" presents LLM-Brain as a generalizable framework that uses a large-scale LLM as a robotic brain to unify egocentric memory and control, integrates multiple multimodal LLMs for robotic tasks, adopts a zero-shot learning approach, and has components that communicate using natural language in closed-loop multi-round dialogues encompassing perception, planning, control, and memory. The two downstream tasks emphasized there—active exploration and embodied question answering—illustrate a recurring Embodied-LM claim: long-horizon action becomes materially easier when memory and control are expressed through a common representational interface (Mai et al., 2023).

Empirical work on physical concepts sharpens this point. The VEC benchmark distinguishes visual concepts from embodied concepts such as mass, temperature, and hardness. There, OPT-175B reaches 50.21% on mass, 59.83% on temperature, and 57.33% on hardness, whereas CLIP-ViT/L-14 reaches 73.15%, 65.88%, and 69.57%, and BLIP-base reaches 83.94%, 74.98%, and 56.93%, respectively. The same study reports that distilling embodied knowledge from a VLM teacher into an LM yields performance gain comparable with that by scaling up the parameters of LMs 134x. In Embodied-LM terms, this indicates that textual scale alone does not reliably produce embodied semantics, while multimodal representations can supply missing physical priors (Li et al., 2023).

A complementary strategy is to expose the LM directly to simulated embodied trajectories. "LLMs Meet World Models: Embodied Experiences Enhance LLMs" deploys an embodied agent in VirtualHome, collects both goal-oriented planning traces and random exploration traces, and fine-tunes LLMs with EWC-LoRA. The paper reports an average improvement of 64.28% over base LMs across 18 tasks, with large gains in counting, object path tracking, activity recognition, and plan generation. A plausible implication is that Embodied-LM need not always emerge from end-to-end multimodal architectures; it can also be induced by translating world-model experience into language-supervised traces that preserve state transitions and affordances (Xiang et al., 2023).

4. Training paradigms, adaptation, and optimization

Embodied-LM training spans prompt engineering, supervised fine-tuning, reinforcement learning, and test-time adaptation. In the Minecraft study "The Embodied World Model Based on LLM with Visual Information and Prediction-Oriented Prompts," no supervised training or RL of the LLM is introduced; instead, the system strengthens world-model behavior through multimodal prompting and explicit prediction-oriented outputs. Indirect visual utilization is markedly stronger than direct image input for the target task of crafting a golden pickaxe: success within 70 iterations occurs only with indirect visual utilization, and the predictive variant reaches gold ingots in 20.67 average iterations versus 32.33 for the conventional variant, approximately 1.56× faster. This result suggests that Embodied-LM performance can sometimes be improved not by larger models, but by forcing the model to externalize anticipated state changes before acting (Haijima et al., 2024).

At the scale of embodied foundation models, EmbodiedBrain formalizes a more aggressive recipe. Built on Qwen2.5-VL, it imposes an agent-aligned schema with <response>, <plans>, and <actions> fields, then trains with large-scale supervised fine-tuning followed by Step-Augmented Group Relative Policy Optimization. Step-GRPO injects Guided Precursors—prefixes of preceding steps—so that long-horizon planning is learned conditionally rather than from scratch at every episode. On VLM-PlanSim-99, the paper reports end-to-end task success of 31.31% for the 7B model and 46.46% for the 32B model, and its analysis attributes part of the gain to the structured output contract that reduces the design-agent mismatch between MLLM generation and executable robotic interfaces (Zou et al., 23 Oct 2025).

EmboMatrix pushes this logic from model training to infrastructure. It defines a training ground consisting of an embodied task set, an interactive physical simulator, and a reward architecture, then trains EmboBrain-7B with GRPO under a multi-level reward decomposition combining format adherence, semantic relevance, and goal-oriented success. The distributed heterogeneous system reduces average per-rollout simulation latency from 3.48 s in the naive setup to 0.07 s in the full system, and the resulting EmboBrain-7B is reported to surpass the 671B DeepSeek-R1 baseline by 9.5% on average over two embodied decision-making benchmarks. Here the distinctive claim is not only model quality but the proposition that Embodied-LM competence can be cultivated by a sufficiently scalable training ground rather than by static corpora alone (Lei et al., 14 Oct 2025).

Two other lines address post-training adaptability and efficiency. TMoW equips a Llama-3.2-1B policy with LoRA world-model adapters and a prototype-based router that is updated at test time, yielding 80.16% Success Rate on unseen VirtualHome domains and 68.83% on unseen ALFWorld domains, while keeping average latency at 700 ms. NeSyPr instead compiles symbolic plans into vector-quantized procedural memory integrated into the LM decoder, achieving 5.2±0.7 s latency with 89.0±2.0% CSR in its proceduralization efficiency comparison, versus 59.5±1.9 s for BoT and 111.0±3.3 s for LRM. Together, these systems demonstrate that Embodied-LM optimization now includes test-time routing, knowledge compilation, and memory shaping, not only larger-scale pretraining (Jang et al., 30 Jan 2026, Choi et al., 22 Oct 2025).

5. Representative systems and application domains

Embodied-LM systems now span domestic manipulation, embodied instruction following, region-centric video cognition, social VR tutoring, affect-aware perception, and cross-domain embodied foundation modeling. The range of application domains explicitly identified in the literature includes navigation, manipulation, household tasks, mobile robotics, trajectory learning and behavior understanding, multi-agent collaboration, human-in-the-loop interaction, service robotics, rescue UAVs, industrial inspection and sorting, education, virtual agents, and space exploration; standard evaluation metrics include Task Success Rate, Real-time Responsiveness, Energy Efficiency, Path Length, Generalization Score, Safety Violations, and, for multi-agent settings, Coordination Efficiency and Communication Overhead (Feng et al., 24 Sep 2025).

System	Embodiment interface	Reported scope
PaLM-E	Interleaved visual, state, and text tokens	Robotic manipulation planning, VQA, captioning
OPEx	Observer-Planner-Executor with semantic maps	Embodied instruction following
RynnEC	Video, region masks, and [SEG] token	Object properties, segmentation, spatial reasoning
EmbodiedBrain	Agent-aligned plans and executable actions	Task planning and end-to-end simulation
MiMo-Embodied	Shared ViT-projector-LLM across robotics and driving	17 embodied AI and 12 driving benchmarks
ELLMA-T	VRChat avatar with voice, gestures, and memory	English-language learning in social VR

RynnEC illustrates a region-centric variant of Embodied-LM. Built on VideoLLaMA3-Image with a region encoder and a SAM2-based mask decoder, it allows either mask-conditioned question answering or language-conditioned mask generation. On RynnEC-Bench, which spans 22 tasks, RynnEC-7B reaches an overall score of 56.2 and RynnEC-2B reaches 54.4, while the authors emphasize strong performance in object property understanding, referring segmentation, and spatial cognition. The system is particularly notable because it treats region masks as first-class interaction objects rather than as downstream annotations (Dang et al., 19 Aug 2025).

A broader unification appears in MiMo-Embodied, which shares one ViT encoder, one projector, and one 7B LLM across embodied AI and autonomous driving. It reports an Embodied Avg. of 62.4% and an Autonomous Driving Avg. of 63.3%, with a four-stage curriculum consisting of embodied SFT, driving SFT, CoT SFT, and GRPO-based RL fine-tuning. The paper frames this as positive transfer between two physical domains that are usually studied separately, implying that Embodied-LM may eventually be better understood as a cross-embodied foundation model rather than as a robotics-only planner (Hao et al., 20 Nov 2025).

Embodied-LM has also expanded into situated social interaction. ELLMA-T instantiates GPT-4 as a conversational avatar in VRChat with Whisper speech recognition, OpenAI TTS, VRChat OSC, Unity animation clips, keyword-based emotion mirroring, and short- and long-term memory. In a qualitative study with $N=12$, participants reported realistic, believable, and context-specific role plays, and the paper positions the agent as an embodied conversational agent for contextualized English-language learning in shared VR spaces (Pan et al., 2024).

Affective and body-grounded perception form another adjacent domain. ELENA formalizes Embodied LVLM Emotion Narratives as a mapping from an image to an emotion label, explicit embodied evidence, implicit bodily sensations, a narrative, and a body-part set. In zero-shot evaluation it improves macro-F1 over naive prompts across 12 of 15 configurations, with one of the largest gains occurring on HECO_Masked for Gemini, from 16.9 to 31.5, a +14.6 increase. This does not turn emotion narration into control, but it broadens Embodied-LM toward body-grounded multimodal inference beyond action planning (Saim et al., 23 Sep 2025).

6. Limitations, controversies, and future directions

The current literature converges on a common set of limitations. MLLMs remain stronger at contextual task reasoning, language understanding, and subtask planning than at physics grounding and real-time responsiveness, whereas world models are stronger at internal representations and future prediction than at open-ended semantic reasoning and generalizable task decomposition. Accordingly, recent surveys argue for a joint MLLM–WM architecture in which semantic planning, physical simulation, memory updating, and hardware embodiment are tightly coupled. The anticipated benefits are semantic–physical alignment, safer and more executable plans, long-horizon foresight, real-time adaptation, and integrated lifelong learning; the corresponding challenges are synchronization between high-latency semantic processing and real-time physics modeling, semantic–physical misalignment, scalable memory management, multimodal data requirements, sim-to-real transfer, and robustness to noise, adversarial conditions, and partial observability (Feng et al., 24 Sep 2025).

Embodiment itself is not uniformly beneficial. In a mixed-methods within-subjects study of cooperative tasks, the non-embodied text-only agent was rated significantly higher on competence than the embodied MetaHumans-based agent, with means of 4.26 versus 3.54 and $p=0.01$. Qualitatively, six participants complained about lack of pushback from the embodied agent versus two for the non-embodied one, and the paper argues that embodiment can become detrimental when combined with sycophancy. This introduces an important caution: adding an avatar, voice, or nonverbal signals does not automatically increase credibility in LLM-based collaboration, and in some settings may intensify perceptions of inauthenticity (Wang et al., 3 Jun 2025).

Text-only spatial benchmarks expose another persistent bottleneck. GSU shows that models often struggle with frames of reference relative to an embodied agent and with identifying 3D shapes from coordinate lists; it also reports that exposure to a visual modality does not provide a generalizable understanding of 3D space that VLMs are able to utilize for these tasks. Even when frontier models solve the base tasks, harder variants still cause failures, especially in egocentric navigation and allocentric localization. This suggests that reference-frame maintenance and structured 3D composition remain central unsolved problems for Embodied-LM (Sidhu et al., 18 Mar 2026).

A further theoretical extension proposes that embodiment should include not only external perception–action loops but also internal embodiment: interoceptive monitoring, homeostatic control, affective state, and motivational drives. In that dual-embodiment framework, external state and internal state are co-equal variables in the computational graph, with internal drives modulating attention, memory, reward, and policy temperature. A plausible implication is that future Embodied-LM systems may be judged not only by how well they ground language in the world, but by how well they integrate external affordances with internal regulation, social simulation, and safety-sensitive preference formation (Kadambi et al., 11 Oct 2025).