Prompt-Driven Exploration

Abstract: Exploration is essential to RL since a policy cannot improve by repeatedly sampling the behaviors it already prefers. Standard methods inject stochasticity in the action space, but such jitter only yields rollouts close to the original. Escaping a weak policy often requires global perturbations that action noise cannot produce. LLMs and vision-language-action (VLA) models offer a pathway: they condition the policy on a natural language prompt, and since the rollout follows from it, modifying the prompt induces global changes. The challenge is finding prompts that induce useful global changes. With a weak policy that rarely succeeds, reward is too sparse to select on. Our idea is to refine prompts from the rollouts themselves: a vision-LLM (VLM) reasons over the rollout video, diagnoses how the policy responded, and rewrites the prompt to elicit better behavior next time. This procedure realizes posterior sampling, a classical RL exploration framework, at the level of prompts: the VLM maintains an implicit distribution over useful prompts and updates it from observed rollouts. We call this strategy Prompt-Driven Exploration (PDE). Across manipulation and reasoning tasks, PDE enables RL to learn successful policies even from zero-reward starts, and improves sample efficiency more broadly. Our website is available at https://xinyunsunshine.github.io/prompt-rl.

• The paper introduces a novel framework that leverages language-driven prompt manipulation to achieve global policy changes in reinforcement learning.
• It employs posterior sampling over prompts integrated with PPO, enabling significant gains in sample efficiency and success rates compared to traditional action noise.
• Empirical evaluations across robotics, language tasks, and generalization benchmarks validate PDE's ability to overcome reward sparsity and refine behavioral strategies.

Prompt-Driven Exploration: Expanding the Horizons of RL through Language Conditioning

Motivation and Problem Formulation

Exploration is a central challenge in reinforcement learning (RL), especially in high-dimensional control settings such as robotics or complex reasoning in LLMs. Standard exploration methods, which inject randomness into the action space, only yield local perturbations in behavior. As policy rollouts remain near the existing behavioral distribution, escaping from weak or overfit initializations—ubiquitous in large vision-language-action (VLA) models and LLMs—becomes intractable due to reward sparsity and insufficient behavioral diversity. This paper introduces an alternative paradigm: language-driven exploration through prompt manipulation. By leveraging the natural language interface of modern VLAs and LLMs, the proposed Prompt-Driven Exploration (PDE) framework explores in the space of prompts, inducing global policy shifts that are unreachable by action-level noise.

Methodology: Posterior Sampling over Prompts

Prompt-Driven Exploration recasts classical posterior sampling RL (PSRL) in the prompt space. For a pretrained policy $\pi_\theta$ conditioned on an input prompt $p$, the exploration strategy is implemented via two key components:

1. Prompt Posterior Maintenance: A vision-LLM (VLM) prompt sampler, $\rho(\cdot \mid g, \mathcal{H})$, maintains an implicit posterior over prompts for each task $g$. This distribution is iteratively updated based on interaction history $\mathcal{H}$—sequences $(g, p, \tau, R(\tau, g))$—where $\tau$ is the trajectory collected under prompt $p$ and $R$ is the terminal reward.
2. Prompt-Based Exploration: Candidate prompts are sampled from $\rho$, not restricted to canonical instructions. After executing the policy under these prompts, the resulting trajectories are analyzed via the VLM, which diagnoses behavioral modes and proposes new prompts to target failure modes or maximize reward.
3. Integration with Policy Optimization: RL updates are coordinated via Proximal Policy Optimization (PPO) using a mixture of rollouts from canonical and exploratory prompts. To ensure transferability, the PPO objective couples log-probabilities from both the exploratory and canonical prompts, forcing the learned policy to improve under the deployment prompt even as training signal is unlocked by prompt-driven exploration.

   Figure 1: Overview of the PDE methodology—using a VLM-based prompt sampler to iteratively refine candidate prompts and collect diverse successful rollouts, which are then used for PPO-based policy updates.

This structured exploration enables global behavioral changes unachievable by step-wise action noise, as demonstrated in motivating experiments.

Empirical Evaluation

PDE is systematically evaluated on a suite of challenging robotics and LLM benchmarks: LIBERO/LIBERO-PRO for robotic manipulation, ManiSkill for generalization in tabletop manipulation, and LiveCodeBench/AIME 2026 for language-model RL. The evaluation focuses on three central questions:

• Can prompt-space exploration unlock nonzero reward from weak or zero-reward initial policies?
• Does PDE enable sample-efficient RL fine-tuning where action-space perturbations fail?
• Is prompt-driven exploration generalizable to modalities beyond robotics?

Illustrative Case: Prompt Refinement in Robotic Manipulation

Initial rollouts under canonical prompts exhibit systematic failures due to overfitting to spurious correlations in training data. For example, with the prompt "close the microwave," the policy consistently manipulates a mug rather than interacting with the microwave door—demonstrating the limitations of action-space noise for local exploration.

Figure 2: Top—failure under canonical prompt; bottom—prompt optimization discovers an alternative phrasing that elicits successful behavior. Right panel—PDE bootstraps RL from 0% to nearly 98% success, while action-noise exploration remains stagnant.

PDE, by contrast, discovers and curates alternative prompts—e.g., "push on the microwave door until it shuts"—that globally shift the policy distribution toward desirable behaviors. Prompt discovery is efficient and interpretable, consistently surfacing 9–12 unique nonzero-success prompts per task within 85 rollouts, unlocking reward regimes that trigger further RL improvement.

Figure 3: Prompt rephrasing transitions a failure mode into success without training—highlighting that prompts induce global behavioral change.

LIBERO-PRO, ManiSkill, and Cross-Modal Extensions

In large-scale benchmarks, PDE demonstrates substantial gains:

• LIBERO-PRO: On 120 tasks spanning varied difficulty, PDE achieves the highest success rates across all tiers, with a striking 60% relative improvement on hard tasks (initial SR < 10%), where competing methods gain no traction.
• ManiSkill: Across diverse generalization variants (novel objects, backgrounds, robot states), PDE consistently boosts performance, with improvements observed both in sample efficiency and final success rates.

  Figure 4: Learning curves for Pi0.5 on LIBERO-PRO, aggregated by task difficulty. PDE achieves high final success and faster convergence, with the greatest advantage for difficult tasks.

• LLM RL: In LiveCodeBench and AIME 2026, PDE accelerates early learning in code generation and mathematical reasoning tasks, closing the gap faster under the same optimization budget, with gains evident in the first few iterations. This implies that prompt-driven exploration is not inherently tied to perception-based control but can expose useful solution modes in any prompt-conditioned foundation model.

Ablation Analysis and Theoretical Implications

Ablative studies demonstrate that the observed gains are not simply due to prompt diversity. Unoptimized paraphrase pools yield only marginal improvements over PPO, underscoring that optimization—grounded in rollout analysis and reinforcement—is essential. Furthermore, policies trained under a curriculum of optimized prompts exhibit robust generalization to unseen paraphrasings, attaining near-identical success rates on both in-distribution and out-of-distribution prompt sets.

Figure 5: Optimized prompt pools (PDE) substantially outperform random prompt paraphrasing and canonical-only training, highlighting the critical impact of prompt selection and optimization.

Figure 6: Number of unique successful prompts discovered grows linearly with rollout budget, indicating effective coverage of the prompt space.

The transferability of learned behaviors also hinges critically on the policy update mechanism. Mixed backpropagation—where gradients are shared between canonical and exploratory prompts—outperforms updates conditioned only on the current or curriculum prompt, ensuring that reward discovered through alternative linguistic grounding translates to improved deployment under the canonical instruction.

Real-World Robotic Manipulation

PDE's gains translate directly to hardware, where high sample-efficiency is paramount due to the costliness of physical rollouts. PPO+PDE achieves more than 2x the average success rate of action noise PPO after only 128 rollouts. Optimized prompts routinely disambiguate under-specified original instructions and resolve systematic failure cases in multi-object and spatial settings.

Implications and Future Directions

The implications of PDE are both practical and theoretical:

• Sample-Efficient RL: PDE enables successful RL fine-tuning from weak initializations—often the bare minimum in foundation models—and drastically reduces the number of required environment interactions.
• Exploration in Context Space: By leveraging the prompt interface of foundation models, PDE offers a tractable and scalable alternative to Bayesian neural-network posteriors or intrinsic-motivation-based exploration, particularly in domains where environment reward is sparse and state spaces are high-dimensional.
• Prompt Optimization as Policy Structure Discovery: Discovering effective linguistic groundings reveals latent structure in policy space, with potential for better curriculum learning and transfer, modular task specification, and explainable skill discovery.
• Modality-General Exploration: PDE can be applied whenever a policy is prompt-conditioned, regardless of whether the underlying task is robotic manipulation, code synthesis, or mathematical reasoning.

Future work may investigate integrating prompt-driven exploration with auxiliary reward models for dense feedback, meta-learning of prompt posterior update rules, or the use of PDE in multi-agent or human-in-the-loop RL. Additionally, PDE opens new avenues for foundation model alignment: since global changes in behavior can be elicited via contextual prompts, language interfaces may become the principal locus for both exploration and control in RL paradigms for foundation models.

Conclusion

Prompt-Driven Exploration introduces prompt-space posterior sampling as a principled and highly effective approach for exploration in RL with foundation models. PDE demonstrates strong empirical gains in sample efficiency, learning capacity from zero-reward starts, transferability across prompt forms, and broad applicability from robotics to language tasks. Its central insight—that exploration in context space is both feasible and powerful for language-conditioned policies—suggests a paradigm shift for RL in the era of multimodal and language-driven foundation models.