Reinforcement Learning Foundation Models

• Reinforcement Learning Foundation Models are large-scale architectures that integrate pre-trained FM priors with RL to enhance generalization and in-context planning.
• They leverage multimodal models from language, vision, and synthetic MDPs to deliver improved sample efficiency and robust semantic state understanding.
• Practical implementations in robotics and simulation demonstrate accelerated skill acquisition and effective reward shaping, reshaping autonomous agent design.

Reinforcement Learning Foundation Models

Reinforcement Learning Foundation Models (RL-FMs) are large-scale models built to provide generalizable priors, semantic understanding, and in-context planning capabilities for sequential decision-making tasks, especially when sample efficiency, interpretability, or direct supervision are bottlenecks. RL-FMs span architectures adapted from language, vision, or multimodal foundation models, as well as bespoke architectures trained on large distributions of synthetic or real Markov Decision Processes (MDPs). Their integration with reinforcement learning (RL) frameworks is fundamentally reshaping the construction of autonomous agents in robotics, embodied AI, simulation, and structured control domains.

1. Foundations: Definitions and Motivations

Foundation Models (FMs) are defined by their pretraining on vast, often internet-scale, multi-modal or structured datasets and their ability to generalize via in-context learning, zero/few-shot adaptation, and the provision of rich semantic representations (Moroncelli et al., 2024). RL-FMs draw from the strengths of FMs—strong generalization, in-context adaptation, and high-level reasoning—while grounding them in the action/state space and real-time feedback of RL.

Key motivation:

• Standard FMs lack embodiment, sample efficiency, or the ability to perform trial-and-error adaptation in interactive environments.
• RL is powerful in principle for continual, embodied learning, but suffers from poor generalization, high sample complexity, and brittle reward design (Moroncelli et al., 2024).
• RL-FMs thus combine FM priors with RL’s on-policy adaptation, posing a unique paradigm for constructing agents that can reason, generalize, and learn from interaction.

2. Mathematical Formulations and Principles

The integration of foundation models with reinforcement learning is formalized within the MDP framework or its extensions:

For an MDP $(\mathcal{S}, \mathcal{A}, P, r, \gamma)$, the standard RL objective is

$$\max_\theta\ \mathbb{E}_{\tau \sim \pi_\theta}\left[\sum_{t=0}^T r(s_t,a_t) \right].$$

To inject FM priors—$p_\text{FM}(a|s)$—the objective can be regularized as

$$\max_\theta\ \mathbb{E}_{\tau \sim \pi_\theta}\left[\sum_t r(s_t, a_t) + \lambda \log p_\text{FM}(a_t|s_t)\right],$$

where $\lambda$ trades off RL and prior alignment (Moroncelli et al., 2024).

Alternatively, in unsupervised RL settings, foundation models (VLMs, LLMs) supply both imagined tasks and reward signals (Nam et al., 2023): $$\max_\pi\ \sum_{i=1}^N \mathbb{E}_{s_0\sim\rho} \mathbb{E}_{a_{0:T-1}\sim\pi}\left[\sum_{t=0}^{T-1}\hat{r}(o_{t-H:t}, \delta^{(i)})\right],$$ with LLM–proposed instructions $\delta^{(i)}$ and VLM-based reward $\hat{r}$.

In the context of preference-based RL, FMs are used for synthetic trajectory evaluation and reward modeling via generalized Bradley–Terry models with multimodal feedback (Wang et al., 19 Sep 2025).

3. Taxonomies and Architectures for RL-FM Integration

A comprehensive taxonomy groups RL-FM integration approaches into four broad categories (Moroncelli et al., 2024):

FM type	Role in RL-FM Integration	Example Pipelines
LLM Planners + RL Agents	LLMs generate high-level task plans; RL grounds	SayCan, PSL, Grounded Decoding
VLMs Guiding Policy/Reward	VLMs supply reward functions or state encodings	CLIPort, ZeST, FoMoRL, RoboCLIP
Diffusion Policy Models	Diffusion models sample actions/sequences in RL	Diffusion-QL, SGP, RL4Med-DDPO
Transformer-RL Agents	RL as sequence modeling; pretrain multi-task	Decision Transformer, Gato, Q-Transformer

Notable instantiations:

• LiFT—LLM task proposal + VLM reward for unsupervised skill discovery (Nam et al., 2023).
• RLFP/FAC—Distills foundational policy, value, and reward priors for dense shaping and fast learning (Ye et al., 2023).
• FoMoRL—CLIP-based intrinsic reward for curiosity-driven exploration (Andres et al., 2024, Gupta et al., 2022).
• Found-RL—Asynchronous VLM feedback with value-margin and advantage-weighted action regularization for real-time autonomous driving (Qu et al., 11 Feb 2026).
• CRAFT—LLMs as autonomous coaches for MARL curriculum/reward decomposition, refined by VLM feedback (Choi et al., 17 Sep 2025).
• PRIMT—Hierarchical fusion of LLM and VLM for preference-based RL in robotics and counterfactual reasoning (Wang et al., 19 Sep 2025).
• InnateCoder—Foundation model–driven zero-shot option discovery and programmatic policy synthesis (Moraes et al., 18 May 2025).
• DINO-R1—RL-based fine-tuning of vision foundation models for in-context visual reasoning (Pan et al., 29 May 2025).
• S2E—Navigation FMs post-trained with RL (residual adaptation) for safe, interactive urban navigation (He et al., 29 Jul 2025).
• Tabular RL-FM—Direct pretraining of in-context RL models on synthetic MDP priors with graph attention networks, bypassing standard data collection (Zighem et al., 17 Jun 2026).

4. RL with Foundation Model Priors: Mechanisms and Empirical Results

The principal mechanisms for leveraging FMs in RL systems include:

• Task/skill proposal and decomposition: LLMs generate sub-tasks, curricula, or options based on grounded, scene-aware prompts (Nam et al., 2023, Moraes et al., 18 May 2025, Choi et al., 17 Sep 2025).
• Reward shaping or replacement: VLM/CLIP-based intrinsic or external rewards directly measure semantic state novelty, task completion, or alignment with language instructions (Nam et al., 2023, Andres et al., 2024, Gupta et al., 2022, Qu et al., 11 Feb 2026).
• Representation transfer: Perception pipelines are enhanced via frozen or adapted FM encoders, providing agents with robust state embeddings for policy/value networks (Farooq et al., 7 Aug 2025, Ye et al., 2023).
• Imitation and regularization: RL policies may be regularized toward FM output distributions or plans via explicit KL penalties or advantage-weighted action guidance (Moroncelli et al., 2024, Qu et al., 11 Feb 2026, Ye et al., 2023).
• World model simulation: LLMs and VLMs act as generative world simulators (FWMs), allowing for pretraining of RL agents entirely within FM “simulated experience” before finetuning in the real environment (Sasso et al., 19 Sep 2025).
• Preference aggregation and causal credit assignment: Hierarchical fusion of LLM/VLM feedback, foresight/hindsight synthesis, and counterfactuals provide robust, interpretable reward models in preference-based RL (Wang et al., 19 Sep 2025).

Empirical benchmarks demonstrate:

• Marked acceleration in sample efficiency for skill acquisition in both simulation and real-robot settings, often reducing required frames or environment interactions by 5–10× (Ye et al., 2023, Nam et al., 2023, Choi et al., 17 Sep 2025, He et al., 29 Jul 2025).
• Significant improvements in sparse- or zero-shot reward regimes, with FM-based intrinsic or language-grounded rewards driving exploration that outpaces classic count-, prediction error–, or RIDE-style bonuses (Andres et al., 2024, Gupta et al., 2022).
• In fully tabular domains, synthetic MDP-trained Graph Attention Networks solve held-out in-context RL tasks in far fewer episodes than UCB-VI or Q-learning (Zighem et al., 17 Jun 2026).

5. Practical Implementations, Challenges, and Limitations

RL-FM systems are realized through carefully orchestrated architectural modules, training schedules, and FM integration points:

• Batch asynchronous VLM-server architectures are essential for RL in time-constrained settings, amortizing expensive FM inference (Qu et al., 11 Feb 2026).
• FM-based reward signals require stabilization (moving average, peak clipping, positive gating) due to noise and credit assignment drift (Nam et al., 2023).
• Policy/value priors must be regularized to prevent overfitting to misleading FM suggestions or catastrophic forgetting when fine-tuned (Ye et al., 2023, Moroncelli et al., 2024).
• Visual/semantic grounding is not universally robust—off-the-shelf FMs may supply subcomponent-level rewards or hallucinated, unachievable tasks (Nam et al., 2023, Moroncelli et al., 2024).
• Reliance on trained FM object detectors, text models, or synthetic MDP priors restricts direct applicability to complex environments or real-world scale, and may incur API and computational costs (Andres et al., 2024, Wang et al., 19 Sep 2025, Zighem et al., 17 Jun 2026).

Further, limitations include the sensitivity of RL to FM prior misspecification, limits of FM simulation accuracy in stochastic or partially observable regimes, and the potential for RL fine-tuning to destroy valuable pre-trained knowledge (He et al., 29 Jul 2025, Moroncelli et al., 2024).

6. Extensions, Impact, and Future Directions

RL-FMs are defining new research frontiers and application domains:

• Model-based and world modeling: FM-powered world models (LLMs, VLMs) can serve as high-fidelity simulators for offline RL, model-based RL, and safe planning (Sasso et al., 19 Sep 2025, Moroncelli et al., 2024).
• Robotics and multi-agent systems: RL-FMs underpin sample-efficient manipulation, navigation, and multi-agent coordination, with transfer demonstrated to real quadruped platforms and zero-shot cross-platform generalization (Choi et al., 17 Sep 2025, He et al., 29 Jul 2025).
• Task-level reasoning and composition: Hierarchical/hybrid frameworks integrate LLM planning, VLM evaluation, and transformer-RL policies under modular, protocolized interfaces (Moroncelli et al., 2024, Nam et al., 2023).
• Reasoning and interpretability in medical and spatial domains: RL refines FMs for grounded, step-by-step interpretability (e.g., chest X-ray analysis, spatial reasoning with explicit local-region justification) (Lin et al., 4 Sep 2025, Zhao et al., 17 Apr 2025).
• Programmatic and option-based RL: Foundation models generate compositional, temporally extended skill modules (options), drastically increasing sample efficiency and enabling program synthesis–driven behavior (Moraes et al., 18 May 2025).
• Autonomous curriculum and reward synthesis: FMs automate task curriculum design, reward function specification, and reward refinement through learning or human-in-the-loop interaction (Nam et al., 2023, Choi et al., 17 Sep 2025, Wang et al., 19 Sep 2025).

Key open research directions include:

• Efficient and robust fine-tuning of RL-FMs under hardware and data constraints, including adapter architectures and prompt-based specialization (Moroncelli et al., 2024).
• Scalable, safe, and certifiable policy grounding of FM priors in non-trivial robot or agent dynamics (Moroncelli et al., 2024, He et al., 29 Jul 2025).
• Generalization to continuous, high-dimensional, and partially observable domains, including integration of visual, semantic, and geometric cues (He et al., 29 Jul 2025, Zighem et al., 17 Jun 2026).
• Memory and continual learning in RL-FMs to avoid catastrophic forgetting during lifelong adaptation (Moroncelli et al., 2024).
• Systematic evaluation benchmarks (e.g., NavBench-GS, Report Arena) to rigorously quantify generalization, interpretability, and safety (He et al., 29 Jul 2025, Lin et al., 4 Sep 2025).
• Prompt engineering and prior design for synthetic MDPs as foundational resources for in-context RL (Zighem et al., 17 Jun 2026).

In summary, RL-FMs represent a convergence of generative AI and embodied RL, where rich semantic priors and reasoning capability are tightly coupled with reward-driven adaptation. As algorithmic and computational strategies continue to mature, RL-FMs are expected to define the core paradigm for the next generation of autonomous, interactive, and generalist agents across domains (Moroncelli et al., 2024, Nam et al., 2023, Zighem et al., 17 Jun 2026).