Double Interactive Reinforcement Learning

• DIRL is a reinforcement learning framework that fuses environmental rewards with human feedback to overcome sparse signals.
• It employs dual-signal fusion and a two-phase training approach to improve policy robustness in tasks like AUV control and autonomous driving.
• Empirical results show DIRL achieves faster convergence and higher performance compared to traditional RL methods in safety-critical environments.

Double Interactive Reinforcement Learning (DIRL) refers to a class of methods in reinforcement learning (RL) wherein agents integrate two distinct sources of interactive guidance—often environmental and human interaction—through mathematically principled mechanisms, reward fusion, or two-stage training. Approaches under this term target domains where environmental signals alone are insufficient for efficient learning or generalization, and where human knowledge, judgment, or demonstration can accelerate or supplement autonomous exploration. DIRL has been implemented in a range of tasks, including autonomous underwater vehicle (AUV) control (Zhang et al., 2020), autonomous driving (Sygkounas et al., 28 Apr 2025), and spatial reasoning in embodied vision-language agents (Chen et al., 3 Dec 2025).

1. Core Principles and Definitions

DIRL frameworks are characterized by the simultaneous or sequential incorporation of two forms of interaction:

1. Dual-Signal Fusion: Merging of environmental rewards (derived from the task environment) with human-derived signals (evaluative feedback, interventions, demonstration) to guide RL policy updates.
2. Two-Phase Learning: Sequential structure—first leveraging one teacher or subdomain (e.g., human demonstration or restricted tool use), then expanding to interactive RL over the broader task or toolset.

Key objectives of DIRL include overcoming sparse/imperfect reward structures, accelerating convergence, and enhancing robustness or generalization, particularly in high-stakes or hard-to-specify domains.

2. Algorithmic Instantiations

A. Reward-Level Fusion (AUV Path Following) (Zhang et al., 2020)

• Agent Architecture: Deep Q-Network (DQN) approximates action-value $Q(s, a;\theta)$ given low-dimensional state.
• Human Reward: $r_h(s_t, a_t) = R_h$ ($R_h \in \{\pm0.8, \pm0.5\}$) is provided by the trainer via GUI during each timestep.
• Environmental Reward (for path tracking tasks):

$$r_e(s_t, a_t) = -0.9|c_d - c| + 0.1 \cdot 2^{(2 - d/10)}$$

where $c$ = current heading, $c_d$ = desired heading, $d$ = lateral deviation.

• Combination:

$$r_{\mathrm{combined}}(s_t, a_t) = r_e(s_t, a_t) + r_h(s_t, a_t)$$

• Training: Standard DQN with the combined reward used for all updates; human feedback can be sporadic (missing steps use $r_h=0$).

B. Q-value Blending and Human Interventions (iDDQN) (Sygkounas et al., 28 Apr 2025)

• Architecture: Two dueling Double DQN Q-networks process stacked image and sensor inputs; agent actions and possible human interventions are both recorded.
• Action Selection: If human overrides, action $a_\mathrm{human}$ is executed and stored for later blended updates.
• Blended Q-Update:

$r_h(s_t, a_t) = R_h$0

where $r_h(s_t, a_t) = R_h$1 is a decaying weight prioritizing human input early in training, vanishing over time.

• Targets and Optimization: All losses, targets, and soft updates follow Double DQN conventions, but over this blended Q-value stream.
• Offline Evaluative Prediction: Post-hoc analysis replaces human actions with agent actions via model-based simulation to estimate true value of interventions.

C. Two-Phase Interactive RL (SpaceTools) (Chen et al., 3 Dec 2025)

• Phase 1 (Teaching):
  • Train a VLM policy $r_h(s_t, a_t) = R_h$2 via supervised fine-tuning on traces from (a) an RL-trained single-tool specialist, and (b) a frontier LM using all tools successfully.
• Phase 2 (Exploration):
  • Initialize $r_h(s_t, a_t) = R_h$3 from teaching phase; continue RL with Group Relative Policy Optimization (GRPO) on all tool interactions, using rollouts and reward functions specific to spatial reasoning outcomes.
• Reward Types: Task-specific, e.g., accuracy for VQA, normalized negative distance to centroid (NNDC) for pointing, mean IoU for bounding boxes, or mean angular coordinate error (MACE) for grasp tasks.

3. Experimental Application Domains

Autonomous Underwater Vehicles (AUVs) (Zhang et al., 2020):

• DIRL allows AUVs to efficiently learn path tracking in sparse-reward settings. Empirically, DIRL agents achieve 4–5× faster convergence and lower variance compared to pure environmental RL and match/frequently surpass human-only interactive RL in both straight-line and sinusoidal path-following.

Autonomous Driving (Sygkounas et al., 28 Apr 2025):

• iDDQN augments Double DQN with human actions and targeted Q-value blending. Human intervention is most beneficial in early-stage training, as reflected in dynamic $r_h(s_t, a_t) = R_h$4 scheduling. iDDQN outperforms standard DRL, behavioral cloning, DQfD, and HG-DAgger both in sample efficiency and final performance, especially in generalization to novel driving environments. The offline evaluative framework confirms high alignment (94.2%) between human interventions and true avoidance of catastrophes.

Spatial VLMs and Embodied Agents (Chen et al., 3 Dec 2025):

• DIRL's phased approach yields SpaceTools, a VLM-capable agent that orchestrates multiple vision and robot tools to achieve state-of-the-art metrics on RoboSpatial-Home, BLINK, BOP-ASK, and real robot manipulation (Kinova Jaco arm). With DIRL, SpaceTools achieves 70.0% overall on RoboSpatial-Home (vs. best prior at 59.4%) and shows robust closed-loop task execution in real 7-DOF robot pick-and-place.

4. Comparative Performance and Empirical Benchmarks

Table: DIRL Method Variants and Key Results

Domain	DIRL Variant	Baseline(s) Outperformed	Empirical Speedup
AUV Path Following	DQNHE (DIRL)	DQNE, DQNH	4–5x faster convergence
Autonomous Driving	iDDQN	DQfD, HG-DAgger, BC, vanilla DRL	+97.3% (test env reward vs. vanilla)
VLM Spatial Reasoning	DIRL (SpaceTools)	GPT-5, Claude 4.5, RoboRefer	+12–16% on RoboSpatial-Home

Performance improvements in DIRL are consistently attributed to: (1) leveraging human intuition/guidance to traverse difficult reward landscapes, (2) structured two-phase training for complex tool composition, and (3) flexible blending of agent and human policies/feedback signals through explicit decay scheduling or adaptive control.

5. Limitations and Open Challenges

Several limitations are documented across DIRL implementations:

• Human Feedback Saturation: Sustained human involvement is expensive or infeasible in long deployments (AUVs, driving). DIRL mitigates this via decaying usage or handover to environmental reward exclusively, but optimizing this transition remains an open problem (Zhang et al., 2020, Sygkounas et al., 28 Apr 2025).
• Reward Scale–Sensitivity: Direct additive fusion of human and environmental signals may require normalization/tuning if tasks, signal scales, or domains differ substantially (Zhang et al., 2020).
• Blending Dynamics: Fixed or naive decay of human weights may not optimize generalization for all scenarios; adaptive $r_h(s_t, a_t) = R_h$5 scheduling or confidence-based intervention remains future work (Sygkounas et al., 28 Apr 2025).
• Sim2Real Gaps: Most DIRL research to date is simulation-centric; real robot deployments expose latency, safety, and reward noise challenges (Chen et al., 3 Dec 2025).
• Methodological Bottlenecks: Complex, multi-tool RL may eventually require hierarchical RL, explicit error-recovery, or integration of raw sensory output into the learning loop (Chen et al., 3 Dec 2025).

6. Extensions, Prospective Research, and Application Scope

Future directions for DIRL involve:

• Curricula and Expertise Scaling: Systematic progression from novice to expert intervention or graded guidance scheduling (Sygkounas et al., 28 Apr 2025).
• Adaptive Blending: Dynamic human/agent fusion based on agent uncertainty or novelty detection (Sygkounas et al., 28 Apr 2025).
• Expanded Feedback Modalities: Integration of richer human signals (demonstration, advice, natural language feedback) and better temporal credit-assignment for delayed human input (Zhang et al., 2020).
• Hierarchical and Long-Horizon Tasks: Application of DIRL to longer, more structured mission planning—spanning simulation-to-real transfer and real-world multi-stage tasks (Chen et al., 3 Dec 2025).
• Tool Infrastructure: More scalable, lower-latency tool execution and inference in high-throughput, large-scale embodied contexts (Chen et al., 3 Dec 2025).

7. Conclusion

Double Interactive Reinforcement Learning constitutes a principled, empirically tested paradigm for accelerating and enhancing RL in domains where environmental feedback alone is insufficient. By explicitly integrating human or demonstration signals—either at the reward, Q-value, or trajectory level—and, in some settings, leveraging a staged two-phase learning process, DIRL achieves faster convergence, lower variance, and improved generalization in safety-critical and high-complexity environments. Empirical evidence across AUVs, autonomous driving, and multimodal spatial reasoning consistently highlights DIRL’s sample efficiency and real-world utility, even as scalability, dynamic blending, and infrastructure remain active areas of refinement (Zhang et al., 2020, Sygkounas et al., 28 Apr 2025, Chen et al., 3 Dec 2025).