Curiosity-Driven Vision-Language RL

• Curiosity-driven vision-language RL is a framework where agents integrate visual and language inputs with both intrinsic and extrinsic rewards to explore and learn from complex environments.
• Researchers model agent-environment interaction as an MDP with multimodal states, leveraging visual operations and language-guided actions optimized by methods like PPO and SAC.
• Empirical results show that incorporating curiosity signals improves sample efficiency, generalization, and systematic reasoning across benchmarks such as VQA, image captioning, and embodied tasks.

Curiosity-driven vision-language reinforcement learning (VL-RL) refers to a family of frameworks in which agents equipped with both vision and language capabilities are trained via reinforcement learning, guided not only by extrinsic task rewards but also by intrinsic curiosity mechanisms. These agents leverage curiosity signals to efficiently explore high-dimensional spaces—pixel-level visual inputs and structured language sequences—resulting in improved generalization, sample efficiency, and systematic reasoning. Contemporary research formalizes curiosity using prediction error, semantic uncertainty, epistemic novelty, or exploration-balancing terms, and rigorously quantifies their impact on complex vision-language benchmarks and embodied robotic tasks.

1. Formalization of Curiosity-Driven Vision-Language RL

In curiosity-driven VL-RL settings, agent-environment interaction is modeled as a Markov Decision Process (MDP) $(S, A, P, R)$ with multimodal states $s_t = (v_t, l_t)$—typically concatenating visual (images/video) and linguistic (text/commands) representations. The action space $A$ may comprise natural language utterances, discrete visual reasoning operations (e.g., zoom-in, frame select), and, in embodied contexts, motor commands. The reward function $R(s_t,a_t)$ is often decomposed into:

$R(s_t,a_t) = r_e(s_t,a_t) + r_{curio}(s_t,a_t)$

where $r_e$ encodes extrinsic task success (e.g., QA accuracy, coverage ratio), and $r_{curio}$ denotes intrinsic bonuses. Notable variants include:

• Pixel-space reasoning reward: Intrinsic bonus based on under-explored use of visual ops (Su et al., 21 May 2025).
• Prediction error curiosity: SP-Net-based state-prediction error for paragraph generation (Luo et al., 2019).
• Active inference–driven complexity: KL-divergence between posterior and prior in latent state inference (Tinker et al., 6 Oct 2025).
• Semantic reward shaping via LLM feedback: LLM-based alignment incentives supplant explicit intrinsic motivation (Margapuri, 14 Jul 2025).

Agents optimize the expected discounted sum of these combined rewards via policy-gradient methods such as PPO, self-critical REINFORCE, or entropy-regularized actor–critic.

2. Curiosity Mechanisms in Pixel-Space Reasoning

"Pixel Reasoner" exemplifies vision-language RL agents incentivized to perform chain-of-thought reasoning in pixel space rather than text alone. The agent is equipped with visual operations:

• CropImage(bbox₂d, img_id): Selects and inspects image regions.
• SelectFrames(frame_list): Chooses salient video frames.

Curiosity is formalized via the Rate of Pixel-space Reasoning (RaPR), targeting a minimum threshold $H$ for visual-op engagement:

$$r_{curio}(x,y) = \max(H - RaPR(x), 0) \cdot 1_{PR(y)}$$

An efficiency penalty prevents overuse, and combined reward is:

$$r'(x,y) = r_e(x,y) + \alpha r_c(x,y) + \beta r_p(y)$$

Training proceeds via a two-phase pipeline: warm-started instruction tuning on synthetic reasoning traces followed by reinforcement learning incentivized by curiosity (Su et al., 21 May 2025). Empirical results show that inclusion of the curiosity bonus maintains high RaPR and yields state-of-the-art performance on VQA, counting, and infographic benchmarks.

3. Curiosity-Driven Language-Conditional Exploration and Learning

"Visual Curiosity" (Yang et al., 2018) operationalizes curiosity in agents that ask unambiguous questions to an Oracle to accelerate visual recognition. States are graph memories tracking the agent’s attribute knowledge, actions are templated queries, and the reward is the marginal gain in scene-graph recall. The RL policy, parameterized via RNN+GCN, learns to disambiguate and diversify questions, exploring object–attribute space driven by knowledge gaps (entropy). Empirical evaluations demonstrate robust generalization to novel objects and environments. The framework includes modules for vision (Faster R-CNN), language policy (LSTM+GCN), answer digestion, and graph working memory, all learned end-to-end.

4. Joint Intrinsic and Extrinsic RL for Diverse Vision-Language Generation

In curiosity-driven visual paragraph generation (Luo et al., 2019), agents generate diverse, coherent image descriptions via RL with both extrinsic (BLEU, CIDEr) and intrinsic (state-prediction error) rewards. The intrinsic bonus is derived from a state-prediction network (SP-Net):

$$r^i_t = V^\pi(s_t) = \frac{\rho}{2} \lVert \tilde\phi(s_t) - \phi(s_t) \rVert^2$$

Extrinsic rewards are provided only for full paragraphs; TD($\lambda$) learning propagates feedback. Training integrates RL loss, discounted imitation learning (cross-entropy on human paragraphs), and an action-prediction auxiliary task. Empirical results show substantial gains in output diversity and overall metrics when curiosity is present: CIDEr boosts up to 38.4% vs. RL-only baselines.

5. Embodied Co-Development of Vision, Language, and Action via Curiosity

Simulated robotic agents learn action-language mappings (e.g., "push red cube") through self-guided exploration driven by curiosity (Tinker et al., 6 Oct 2025). The architecture integrates variational recurrent neural networks (VRNN) for forward modeling with Soft Actor-Critic (SAC) for control. Intrinsic reward is computed via KL-divergence between posterior and prior latents in the active inference model, plus policy entropy:

$$r_{intrinsic}(t) = \eta \mathrm{D}_{KL}[q(z_t|o_t,h_{t-1}) || p(z_t|h_{t-1})] + \alpha H[\pi(\cdot|h_t)]$$

Empirical studies show pronounced gains in generalization to unseen sentence–action pairs when curiosity covers both sensory and linguistic feedback. The developmental trajectory mirrors human infant learning: prerequisite actions (perception → locomotion → manipulation) emerge sequentially; compositional generalization supersedes rote pairing as vocabulary scale increases.

6. Semantic Reward Shaping via LLMs: An Alternative to Curiosity

Prompt-Informed RL (PIRL) demonstrates that semantic feedback from LLMs (GPT-3.5) can substitute for explicit intrinsic curiosity in complex visual coverage tasks (Margapuri, 14 Jul 2025). The agent’s reward combines environment-based coverage, battery, and collision metrics with LLM-guided alignment penalties and incentives:

$$r_{total}(s_t,a_t) = r_{env}(s_t,a_t) + \alpha_L r_{LLM}(s_t,a_t)$$

Here, $r_{LLM}$ enforces alignment between the agent’s actions and LLM recommendations on camera parameters and movement vectors, parsed at each step from structured zero-shot prompts. PIRL achieves higher visual coverage (up to +14% Gym, +27% Webots), battery efficiency (+25%), and lower redundancy (–18%) than PPO, imitation learning, and LLM-only controllers. This suggests that semantic priors encoded by LLMs can guide exploration in vision-language RL without handcrafted curiosity bonuses, enabling robust sim-to-real transfer and generalization.

7. Future Directions and Challenges

Research across these frameworks highlights multiple open directions:

• Scaling pixel-space reasoning: Robust, curiosity-driven exploration of fine-grained visual input remains computationally intensive; efficient tool-selection policies and scalable reward mechanisms are active areas.
• Multi-agent and social learning: Emergent communication and coordination via curiosity-driven RL in multi-agent vision-language systems have potential for collective cognition (Tinker et al., 6 Oct 2025).
• Semantic grounding of curiosity: Combining structured LLM-driven reward shaping with epistemic uncertainty bonuses could maintain both semantic alignment and explorative diversity (Margapuri, 14 Jul 2025).
• Towards active tutor–learner interaction: Integration of dialogic scaffolding with curiosity-driven exploration may further approach the efficiency of human developmental learning (Tinker et al., 6 Oct 2025).
• Benchmarks and evaluation: New tasks that require high-fidelity vision-language reasoning and measure both sample efficiency and generalization are needed.

Curiosity-driven vision-language reinforcement learning synthesizes advances in exploration theory, multimodal neural architectures, and semantic reasoning, enabling agents to learn faster, generalize better, and reason with greater fidelity over multimodal tasks (Su et al., 21 May 2025, Yang et al., 2018, Luo et al., 2019, Tinker et al., 6 Oct 2025, Margapuri, 14 Jul 2025).