Imagine-Bench: LLM Rollouts in RL

• Imagine-Bench is a benchmark that systematically evaluates reinforcement learning using both real trajectories and LLM-generated synthetic (imaginary) rollouts.
• It employs a dual-data protocol with expert-collected and LLM-imagined trajectories to assess success rates across tasks varying in complexity.
• The framework reveals that while synthetic rollouts boost sample efficiency on simpler tasks, they fall short on novel compositional tasks, underscoring the need for improved data filtering.

Imagine-Bench is a benchmark framework introduced to systematically evaluate how reinforcement learning (RL) algorithms exploit "imaginary rollouts" generated by LLMs alongside standard environment-collected trajectories. Its primary goal is to enable rigorous assessment of offline RL algorithms in regimes where both real and synthetic experience can be used to facilitate task learning, especially in settings characterized by language-conditioned instructions and generalization to novel tasks (2505.10010).

1. Motivation and Problem Scope

The central challenge addressed by Imagine-Bench is the sample inefficiency of conventional RL, which typically requires extensive real-world or simulator interaction for policy optimization. Emerging research demonstrates that LLMs, when trained or fine-tuned on environment-goal-trajectory pairs, can "imagine" plausible trajectories for new, previously unseen tasks, producing so-called "imaginary rollouts." These synthetic experiences hold the promise of enhancing learning, but until the introduction of Imagine-Bench, systematic, cross-domain evaluation of RL using imaginary rollouts lacked a standardized protocol. Imagine-Bench is thus designed as a unified evaluation suite for "RL from Imaginary Rollouts" (RLIM), similar in spirit to foundational offline RL benchmarks such as D4RL or RL Unplugged, but explicitly incorporating both real and LLM-generated data (2505.10010).

2. Benchmark Composition and Dataset Structure

Imagine-Bench provides datasets founded on a dual-source protocol. For each covered environment:

• Real rollout datasets consist of trajectories collected via expert agents or teacher policies interacting with the environment for a specific sequence of states, actions, and achieved goals.
• LLM-imaginary rollout datasets are synthesized by prompting a fine-tuned LLM (concretely, Llama-2-7b-chat-hf) to generate plausible action sequences conditioned on initial states and natural language goals, leveraging supervised objectives: dynamics prediction, rollout explanation, and rollout generation.

Each data sample is structured as

$$\{ G^k, (s_0^k, a_0^k, s_1^k, a_1^k, \dots) \}_{k=1}^K,$$

with $G^k$ as the task goal in natural language and the sequence $(s_0^k, a_0^k, \dots)$ as the corresponding trajectory.

The environments span:

• Locomotion (MuJoCo HalfCheetah)
• Robotic Manipulation (Meta-World, CLEVR-Robot, LIBERO)
• Navigation (BabyAI)

Tasks are labeled with four levels of language-conditioned complexity: training (seen), rephrasing (paraphrased goals), easy (unseen but related), and hard (compositional or novel).

3. Evaluation Protocol and Metrics

The primary metric used is success rate, defined by task-specific criteria (e.g., 5 cm positional accuracy for manipulation, 85% semantic consistency for locomotion). Algorithms are trained and evaluated in matched settings on real rollouts, imaginary rollouts, and mixtures thereof. Key offline RL baselines include Behavior Cloning (BC), Conservative Q-Learning (CQL), BCQ, TD3+BC, PRDC, COMBO, and SAC, although not all methods are run in every domain due to domain-method compatibility constraints.

Each offline RL algorithm is trained on mixed datasets (real+imaginary, denoted w/ IR), with sampling balanced across data sources. For robust evaluation, results are reported as the average over the last five checkpoints and three random seeds.

4. Key Empirical Findings

• Usefulness of Imaginary Rollouts: LLM-imagined rollouts provide some utility, particularly for goals close to the training distribution (e.g., paraphrased or related tasks), but their effectiveness decreases sharply on "hard" or compositional tasks.
• Generalization Gaps: There is a significant performance gap between models trained on real rollouts for novel tasks (64.37% success on hard tasks) and those trained with imaginary rollouts (35.44% success). This demonstrates that synthetic experience cannot yet substitute for real experience on challenging instructions, largely due to compounding errors, imperfect physical plausibility, and hinted suboptimalities in the imagined data (2505.10010).
• Algorithm Robustness: Among offline RL baselines, CQL and BCQ tend to outperform others (such as SAC) in stability and sample-efficiency across both real and synthetic datasets, yet none closes the generalization gap.
• Degradation by Task Hardness: As tasks become more compositionally or instructionally demanding (as in Meta-World or BabyAI hard splits), model success rates often fall below 10%. Imaginary trajectories are less consistent, less dynamically correct, and less legal as difficulty increases.
• Imaginary Rollout Quality Analysis: On BabyAI rephrasing tasks, LLM rollouts achieve 88.0% consistency, 96.0% transition correctness, and 98.5% dynamics legality, but for hard tasks, these drop to 25.8%, 72.9%, and 66.8% respectively.

5. Pipeline for LLM-Imaginary Rollout Generation

The LLM-based rollout generator is fine-tuned on real environment-goal-trajectory pairs using:

1. Dynamics prediction—next state given $(s_t, a_t)$;
2. Rollout explanation—natural language description of past rollouts;
3. Rollout generation—complete action sequences for a specified new goal.

At deployment, a "goal-oriented prompt" (GOP) is constructed: $$\text{GOP} = \text{“Generate a rollout for the following goal: [GOAL]. Rollout:”},$$ and the model outputs: $$\{ a_0, s_1, a_1, \dots \} \leftarrow (\text{GOP}, s_0).$$

6. Limitations and Future Directions

While Imagine-Bench demonstrates that LLM-imaginary rollouts can benefit language-conditioned RL, their current utility is hampered by three main issues:

• Suboptimal exploitation of synthetic data: Offline RL algorithms do not adequately filter or weight varied-quality rollouts, struggle with distributional shift, and do not robustly utilize the structure or uncertainty of imagined experiences.
• Quality of imaginary rollouts: LLMs trained on real data may produce trajectories that are less physically plausible or dynamically correct as goal novelty grows, limiting their downstream effectiveness.
• Generalization to multimodal contexts: Present evaluations are limited to vectorized or symbolic state-space representations. Extension to multimodal (e.g., vision-language) tasks would require novel cross-modal rollout generation architectures.

Explicit research directions proposed include: (1) algorithmic designs for better filtering and leveraging of imaginary rollouts, (2) fast online adaptation and continual learning for integrating real and synthetic data during deployment, (3) enhancements to LLMs for improved physics-aware imagination, and (4) vision-language extensions to support more complex multimodal RL environments (2505.10010).

7. Significance and Role in RL Research

Imagine-Bench establishes a foundational standard for evaluating RL with hybrid real-imagined data sources and is unique in explicitly targeting language-driven generalization. It exposes a quantifiable gap between the promise of LLM-augmented experience and current algorithmic capacity for generalizable, language-conditioned RL. The benchmark consequently serves both as an evaluative baseline and as a driver for methodological innovation in synthetic experience utilization, offering a roadmap for future embodied intelligence and instruction-following RL research (2505.10010).