CogDDN: Cognitive Demand-Driven Navigation

• CogDDN is a vision-language framework that integrates fast heuristic actions with analytic reflection to navigate unknown environments based on natural language demands.
• It employs demand matching via fine-tuned visual-language models and chain-of-thought prompting to semantically align detected objects with implicit user instructions.
• The dual-process design, combining rapid exploration/exploitation with GPT-4 driven corrective reflection, significantly boosts navigation success over traditional data-driven methods.

CogDDN, short for “Cognitive Demand-Driven Navigation with Decision Optimization and Dual-Process Thinking,” is a Vision-Language-Model-based framework for demand-driven navigation (DDN) in unknown and unstructured environments. It is designed for settings in which a mobile robot must identify and locate objects from implicit natural-language demands, such as “I am thirsty,” even when object locations are not known in advance. The framework is presented as an alternative to traditional data-driven DDN methods that rely on pre-collected data for model training and decision-making, a dependence that is described as limiting generalization in unseen scenarios. CogDDN emulates human cognitive and learning mechanisms by combining fast and slow thinking systems, semantically aligning detected objects with the given instruction, and selectively identifying key objects essential to fulfilling user demands (Huang et al., 15 Jul 2025).

1. Conceptual scope and system architecture

CogDDN is organized around five core components: 3D Robot Perception, Demand Matching (LLM), Dual-Process Decision-Making (Heuristic Process + Analytic Process), Knowledge Base $K$, and an Environmental Actuator implemented through the AI2Thor interface (Huang et al., 15 Jul 2025). The framework operates at each time step $t$ from three primary inputs: a natural-language demand $M$, a front-facing RGB image $I_t$, and 3D detections $O_t$ supplied by the UniMODE perception module.

The processing pipeline begins with Demand Matching, which uses a fine-tuned LLM to map the detected objects $O_t$ and the instruction $M$ to a possibly empty set of candidate target objects $O_m \subseteq O_t$. This step determines whether the agent has a semantically plausible target in the current observation. If $O_m = \varnothing$, the system enters an exploratory mode; if $O_m \neq \varnothing$, it instead performs target-directed navigation.

The Heuristic Process implements this branching explicitly. In the empty-target case, it invokes Explore to generate a short sequence of exploratory actions through VLM-prompted Chain-of-Thought (CoT). In the nonempty-target case, it invokes Exploit to generate a single goal-directed step toward the selected object, again through a fine-tuned VLM and CoT. When this heuristic action is impeded, such as by collision or lack of progress, the Analytic Process is triggered. A GPT-4-based reflection sub-module then analyzes the last observation, identifies the error, proposes a corrected reasoning and action tuple $t$0, and appends the experience to the knowledge base $t$1 (Huang et al., 15 Jul 2025).

This architecture yields a closed-loop decision cycle in which new corrective experiences are subsequently used offline for supervised fine-tuning of the heuristic VLM. A plausible implication is that CogDDN treats navigation not merely as action selection under partial observability, but as a continually revised reasoning problem in which reflective correction augments future feed-forward competence.

2. Formalization as demand-driven navigation

CogDDN casts DDN as a Partially Observable Markov Decision Process, written as

$t$2

where $t$3 denotes the agent’s hidden state, $t$4 denotes observations $t$5, $t$6 denotes discrete actions such as $t$7, and $t$8 denotes deterministic transition or execution (Huang et al., 15 Jul 2025).

User demand is encoded by processing the natural-language request $t$9 with an LLM to produce an attribute descriptor $M$0. Each detected object $M$1 is associated with a visual embedding $M$2. The framework then defines a semantic-alignment score between the demand and each detected object using the frozen or fine-tuned encoders of the VLM, with the score described as a cosine-based compatibility between the language representation of $M$3 and the visual representation of $M$4 (Huang et al., 15 Jul 2025).

Target selection is expressed as an optimization over these scores. The candidate target set is

$M$5

or, equivalently, the top-$M$6 ranked objects may be selected. If $M$7, the navigation objective is defined as reaching the position $M$8 of the chosen object $M$9 (Huang et al., 15 Jul 2025).

The navigation policy factorizes into heuristic and analytic components. By default, policy execution follows the Heuristic Process, but if the agent is impeded it switches to the Analytic Process. At time $I_t$0, the action is determined by exploitative behavior if a target exists and exploratory behavior otherwise: $I_t$1 Both $I_t$2 and $I_t$3 are implemented through CoT prompting of the fine-tuned VLM (Huang et al., 15 Jul 2025).

This formalization places semantic grounding and navigation under a single decision-theoretic frame. This suggests that CogDDN is intended to bridge instruction interpretation, target discovery, and action generation within one POMDP-style formulation rather than treating them as independent modules.

3. Dual-process decision-making and reflective correction

The central design principle of CogDDN is a two-system structure inspired by dual-process theory. The Heuristic Process corresponds to System-I and is characterized as fast and feed-forward. The Analytic Process corresponds to System-II and is invoked only when the heuristic action fails because a hindrance has been detected (Huang et al., 15 Jul 2025).

Within the Heuristic Process, Explore and Exploit serve complementary roles. Explore is used when $I_t$4, and it generates a sequence $I_t$5 of exploratory actions conditioned on the current image $I_t$6, a past path summary $I_t$7 used to avoid loops, and an obstacle flag $I_t$8. Exploit is used when $I_t$9, and it generates exactly one action $O_t$0 that advances toward the target. Both operations use CoT prompting with a three-stage output structure: Scene Description $O_t$1, Reasoning $O_t$2, and Decision $O_t$3 (Huang et al., 15 Jul 2025).

The heuristic decision at time $O_t$4 is given by

$O_t$5

This decomposition is significant because it makes the intermediate textual rationale an explicit part of the action-generation pipeline rather than a post hoc explanation (Huang et al., 15 Jul 2025).

The Analytic Process uses GPT-4 to reflect on the last frame’s $O_t$6, identify the error, and produce a corrected $O_t$7. The knowledge base is updated according to

$O_t$8

Across episodes, $O_t$9 accumulates approximately $O_t$0 $O_t$1 experiences, which are then used for supervised fine-tuning of the Heuristic VLM (Huang et al., 15 Jul 2025).

At a high level, the decision loop proceeds as follows: the system perceives $O_t$2, computes $O_t$3, chooses Heuristic Explore or Heuristic Exploit depending on whether $O_t$4 is empty, executes action $O_t$5, and, if impeded, applies Analytic Reflection, updates $O_t$6, and sends the corrected action $O_t$7 (Huang et al., 15 Jul 2025). A plausible implication is that CogDDN operationalizes reflection not as online policy optimization in the strict gradient-based sense, but as error-conditioned data generation for subsequent supervised refinement.

4. Learning pipeline and optimization procedures

The perception module is based on UniMODE, which is trained on 55K AI2Thor frames using a 70%/10%/20% split to predict 3D bounding boxes. The detection loss $O_t$8 combines IoU, classification cross-entropy, and box-regression terms as specified in the UniMODE paper (Huang et al., 15 Jul 2025).

Demand Matching is trained from 7.2K/1.4K/2.1K question-answer pairs of the form $O_t$9. A Qwen2-VL-7B model is fine-tuned via LoRA with rank 64 and $M$0, using cross-entropy on the target attributes: $M$1 This training stage provides the semantic mapping from natural-language demand to object-level candidate selection (Huang et al., 15 Jul 2025).

The Heuristic VLM, which implements both Explore and Exploit, is supervised-fine-tuned from the knowledge base of approximately 72K trajectories. The model is Qwen2-VL-7B, trained over 5 epochs with batch size 1, learning rate $M$2, and gradient accumulation 8. Its loss $M$3 is the token-wise cross-entropy between generated CoT plus final action sequence and the ground-truth VQA-format examples containing scene description, reasoning, and decision (Huang et al., 15 Jul 2025).

The Analytic Process itself is not differentiated through. GPT-4 remains frozen, and learning proceeds indirectly through data augmentation into $M$4. Whenever a hindrance occurs, the GPT-4 reflection prompt produces a corrected triplet that is appended to the knowledge base (Huang et al., 15 Jul 2025). This establishes a division of labor between a frozen reflection engine and an updateable heuristic executor.

The overall optimization strategy therefore combines standard supervised learning for perception, LoRA fine-tuning for semantic demand matching, supervised fine-tuning for action reasoning, and reflection-driven augmentation for continual improvement. This suggests a modular training regime in which each stage addresses a distinct source of failure: perception errors, demand-grounding errors, and action-selection errors.

5. Experimental protocol and quantitative results

CogDDN is evaluated in AI2Thor using 400 indoor scenes from ProcThor, partitioned into 200 training scenes and 200 test scenes. The environment contains 109 object categories and 1692 training, 241 validation, and 485 test natural-language demand instructions. Evaluation is conducted in a closed-loop setup in which the agent may see only a single front-facing RGB view at each step (Huang et al., 15 Jul 2025).

Three metrics are reported. Navigation Success Rate (NSR) is defined as

$M$5

Success weighted by Path Length (SPL), following Anderson et al. ’18, is

$M$6

where $M$7 is the agent’s path length and $M$8 is the shortest-path length. Selection Success Rate (SSR) is the fraction of episodes in which the agent correctly identifies an object satisfying $M$9 (Huang et al., 15 Jul 2025).

In baseline comparison, CogDDN achieves, for Seen Scene / Seen Instruction, an NSR of 38.3% versus a best baseline of 21.5%, corresponding to an improvement of 16.8 percentage points. For Unseen Scene / Unseen Instruction, it achieves an NSR of 34.5% versus 16.1%, corresponding to an improvement of 18.4 percentage points. The report summarizes this as an approximately 15% absolute NSR improvement over the strongest single-view, camera-only methods (Huang et al., 15 Jul 2025).

A separate comparison with depth-equipped methods reports that InstructNav using RGB+D achieves NSR 30.0% and SPL 14.2%, whereas CogDDN using RGB only with reflection achieves NSR 34.5% and SPL 17.1% (Huang et al., 15 Jul 2025). Within the scope of the reported experiments, this indicates that the dual-process RGB-only design remains competitive even against an RGB+D baseline.

The reported quantitative gains are especially notable because the evaluation regime is closed loop and single view. A plausible implication is that performance is being attributed not merely to better scene encoding, but to the coupling of semantic target selection, CoT-guided action generation, and reflection-based correction under partial observability.

6. Ablations, case analyses, and stated limitations

The ablation studies isolate three core contributors. First, when the fine-tuned Exploit module is removed and replaced with vanilla GPT-4, NSR drops from 34.5% to 24.2% in the single-step setting or 29.1% in the full-sequence setting. Second, disabling Chain-of-Thought and requiring the VLM to output an action directly reduces NSR from 34.5% to 27.3%. Third, the reflection mechanism is evaluated through four rounds of 500 epochs each, in which GPT-4-generated reflection samples are added to $O_m \subseteq O_t$0 and the heuristic model is re-fine-tuned; SSR and NSR improve marginally after each round, while SPL increases substantially (Huang et al., 15 Jul 2025).

The qualitative analyses provide examples of the intended functional role of each submodule. In Explore, the system avoids redundant loops by tracking prior rotations in the CoT prompt. In Exploit, it correctly angles and navigates around obstacles such as beds and chairs en route to the target. In Reflection, it diagnoses that a prior “MoveAhead” would collide with a chair and instead issues a corrective “RotateLeft” (Huang et al., 15 Jul 2025). These examples align the reported metric gains with specific behavioral phenomena: loop avoidance, obstacle-aware local planning, and error-conditioned policy revision.

The paper states three principal advances: VLM-based demand matching that generalizes to unseen instructions; a dual-process System-I/System-II decision module integrating rapid heuristics with analytic reflection; and a continually updated knowledge base that allows self-supervised bootstrapping of the heuristic VLM (Huang et al., 15 Jul 2025). It also identifies several limitations: the lack of long-term memory, with only short-term memory in Explore; the computational cost of GPT-4 in practice; and the need to re-supervised-fine-tune the Heuristic model whenever $O_m \subseteq O_t$1 grows (Huang et al., 15 Jul 2025).

The future directions are likewise explicit. These include incorporating a long-horizon spatial memory module, developing lighter-weight reflection engines or end-to-end differentiable reflection, and exploring end-to-end navigation training that fuses System-I and System-II within a single differentiable architecture (Huang et al., 15 Jul 2025). These proposals suggest that CogDDN is positioned not as a final unified solution, but as a modular prototype for integrating demand grounding, heuristic control, and reflective correction in embodied navigation.