Dual-Decision Agents Overview

• Dual-decision agents are embodied systems that interleave fast, reactive control with slow, deliberative planning, drawing on Kahneman’s dual-process theory.
• They employ adaptive gating and proficiency-based memory mechanisms to dynamically assign tasks between RL agents and Vision-Language Models.
• Empirical studies demonstrate that these architectures achieve higher success rates, faster convergence, and enhanced adaptability in dynamic and uncertain settings.

Dual-decision agents are embodied decision-making systems that explicitly interleave two complementary kinds of reasoning or control, typically orchestrating fast, intuitive actions with slower, more deliberative planning. These architectures draw on cognitive models such as Kahneman’s dual-process theory, and are formalized algorithmically both for general decision-making (e.g., reinforcement learning) and for agent/controller switching scenarios. Dual-decision frameworks are designed to achieve robust generalization, rapid response, and improved adaptability in dynamic or uncertain environments.

1. Theoretical Foundations of Dual-Decision Architectures

Dual-decision paradigms are grounded in the systematic decomposition of decision workflows into hierarchies or coordinated systems, each optimized for particular capabilities. In the context of the Dual-System Adaptive Decision Framework (DSADF), System 1 operates as a fast, goal-conditioned RL agent, responsible for atomic action selection given short-horizon goals. This subsystem leverages an agent-centered memory space to track and exploit proficiency on familiar subtasks. System 2, in contrast, is a slow, deliberative subsystem powered by a large Vision-LLM (VLM), capable of high-level task decomposition, self-reflective planning, and fallback execution on subtasks beyond the agent’s proficiency (Dou et al., 13 May 2025).

Another prominent dual-layer abstraction models the explicit switching among multiple agents (human and machine) as a two-layer Markov Decision Process (2-layer MDP). Here, the top layer supervises agent selection and switching, while the bottom layer delegates task execution to the chosen agent, whose internal policies may be unknown or stochastic. The aim is to learn switching strategies that trade off environment cost, control cost, and switching cost, enabling efficient exploitation of heterogeneous agents while minimizing regret (Balazadeh et al., 2020).

2. Formal Structure and Algorithmic Realizations

A representative dual-decision architecture is realized as follows:

Dual-System Adaptive Decision Framework (DSADF)

• System 1 (Fast/Reactive):

A goal-conditioned policy $\pi_\theta(a|s,g)$ trained using off-policy RL (DQN) with a progressive hierarchical reward:

$$R = \gamma_1 R_T + \gamma_2 R_{sub} + \gamma_3 R_{proxi}$$

where $R_T$ penalizes or rewards final target completion, $R_{sub}$ provides dense feedback for subgoal achievement, and $R_{proxi}$ aligns transitions with subgoal semantics via high embedding similarity.

• System 2 (Deliberative/Planner & Gating):
  • Task decomposition via Chain-of-Thought and self-reflection (generating $G_{\mathrm{init}}, G$).
  • Gating: assigning each subgoal $g$ based on proficiency $p(g)$ from the memory space $M$; if $p \geq \tau$, $g$ is delegated to System 1, otherwise System 2 executes via instruction or direct control.
• Interaction:

Alternating execution proceeds according to the partitioned subgoals ($G_{RL}$, $G_{VLM}$) with emergency fallback to System 2 for unexpected events (Dou et al., 13 May 2025).

Two-Layer MDP for Agent Switching

• Formalism:

States are extended to $S^1 = S \times D$ (agent-annotated) and $S^2 = S \times A$. Top-layer actions select which agent will act, followed by bottom-layer execution by the agent’s policy, all within an episodic finite-horizon framework.

• Learning Algorithm:

UCRL2-MC extends upper-confidence bounds (UCB) principles to the two-layer setting, with dynamic programming leveraging empirical confidence sets for both agent and environment transitions. The regret relative to an optimal switching policy is provably sublinear, with further improvements under shared environments and multiple teams (Balazadeh et al., 2020).

3. Memory, Gating, and Task Routing

A critical enabler is the memory/gating mechanism for skill-aware assignment. In DSADF, the memory space $M$ maintains $(u_i, p_i)$ pairs, indexing subtasks with quantified agent proficiency as determined by the VLM. The routing logic partitions goals such that:

$$\text{assign}(g) = \begin{cases} \text{System 1} & \text{if } p \geq \tau \ \text{System 2} & \text{otherwise} \end{cases}$$

This method enables the RL agent to exploit known skills, while the VLM handles unfamiliar or low-proficiency cases and emergencies. The proficiency scores $p$ are continuously updated based on episode outcomes and optionally smoothed over recent experience (Dou et al., 13 May 2025).

Tables of the gating mechanism and memory update rules are shown below.

Component	Description
Memory $M$	Stores $(u_i, p_i)$; updated after every training episode
Gating rule	$g \in G_{RL}$ if $p(g) \geq \tau$; otherwise $g \in G_{VLM}$

4. Empirical Validation and Performance

Empirical results validate the dual-decision approach on long-horizon environments demanding both reactive control and generalization:

• Crafter Environment:

DSADF achieves higher Task Success Rate (TSR), faster convergence, and greater subtask completion rates compared to baselines such as sparse-RL, curiosity-driven RL, LINVIT, and small VLM agents. For complex, out-of-distribution tasks (e.g., "mine diamond"), DSADF attains a TSR of 68.3% at 2767s, while strong baseline Qwen-2.5 yields 7.5% at 8495s, evidencing substantial sample and execution efficiency gains (Dou et al., 13 May 2025).

• HouseKeep Environment:

Tasked with implicit-object rearrangement, DSADF delivers Average Object Success Rates (AOSR) of 92–96%, outperforming Sparse-RL (~75%), LINVIT (~85%), GPT-4o (~85%), and Qwen (~88%). This demonstrates the approach’s strength in inferring implicit goals and handling novel situations.

In dual-decision agent switching (2-layer MDP), UCRL2-MC dynamically selects control between human and machine agents. In obstacle avoidance and RiverSwim benchmarks, UCRL2-MC achieves sublinear regret and demonstrates robust exploitation of environmental structure, with up to 40% regret reduction via shared confidence bounds in multi-team regimes (Balazadeh et al., 2020).

5. Key Insights, Limitations, and Open Problems

Dual-decision architectures unify the strengths of fast, reactive control (RL) and slow, analytical planning (VLM or agent-planner switching), yielding improved efficiency and generalization relative to single-system baselines. Progressive hierarchical rewards and adaptive gating mechanisms mitigate sparse reward and credit assignment issues.

Principal limitations include reliance on external VLM APIs (introducing latency, cost, and hallucination risk), a fixed gating threshold $\tau$ that may be suboptimal, and coarse memory statistics that do not capture context-dependent proficiency variance. In the 2-layer agent-switching model, the Markov policy assumption and finite state/action spaces may constrain real-world applicability, and agent policy non-stationarity is not addressed.

Areas for further investigation include:

• End-to-end differentiable integration of RL and VLM with joint fine-tuning.
• Adaptive or learned gating functions (e.g., meta-learning $\tau$).
• Extension to hierarchical/multi-objective, multi-agent, or continuous control domains.
• Richer, context-sensitive memory statistics for improved task routing (Dou et al., 13 May 2025, Balazadeh et al., 2020).

6. Comparative Overview and Research Directions

Dual-decision agents provide a principled mechanism for blending low-latency, skill-grounded control and high-level, compositional reasoning. As summarized below, distinct models occupy different methodological and application spectra but share common motivations.

Framework	Fast System	Slow System	Key Allocation Mechanism
DSADF	RL agent + memory	Vision-LLM (VLM)	Proficiency-based gating
2-layer MDP	Each agent policy	Agent selection (switching)	UCB-based policy switching

This suggests that dual-decision architectures are increasingly central in achieving scalable, robust autonomy, particularly for environments with compositional structure, implicit objectives, or the need for real-time adaptation. A plausible implication is that future models will incorporate more nuanced memory/statistics and flexible gating, as well as more seamless fusion of policy learning and deliberative planning components (Dou et al., 13 May 2025, Balazadeh et al., 2020).