Duty-Distinct Chain-of-Thought (DDCoT)

• DDCoT is a zero-shot framework that decomposes complex vision-and-language questions into sub-questions, clearly separating reasoning from visual recognition.
• It employs negative-space prompting to label uncertain visual aspects, ensuring the VQA model exclusively handles vision tasks and minimizes hallucinations.
• DDCoT achieves significant performance gains and enhanced explainability, demonstrating superior generalizability across both zero-shot and fine-tuning paradigms.

Duty-Distinct Chain-of-Thought (DDCoT) is a zero-shot prompting framework engineered to elicit accurate, generalizable, and explainable multimodal rationales from LLMs. By explicitly decomposing complex vision-and-language (V&L) questions into a sequence of reasoning sub-questions, DDCoT delegates pure visual recognition tasks to a dedicated visual question answering (VQA) model and enlists the LLM to integrate only the reliable information into a human-like chain-of-thought (CoT). This structured approach addresses core challenges of multimodal reasoning, including annotation inefficiency, inflexibility across modes, limited generalizability, and explainability failures prevalent in prior CoT methods (Zheng et al., 2023).

1. Motivating Challenges in Multimodal CoT Reasoning

Multimodal CoT reasoning confronts four principal obstacles:

• Labor-intensive annotation: Manual creation of multimodal rationales at scale is costly and inefficient.
• Flexibility: Preceding techniques tend to specialize, functioning solely in either zero-shot or fine-tuning regimes, but not both.
• Generalizability: Existing CoT approaches falter on out-of-distribution queries, especially those demanding novel inference trajectories.
• Explainability: Hallucinations—generating incorrect or unsubstantiated visual facts—are frequent in multimodal CoTs, undermining trust.

DDCoT addresses these with a design focused on critical skepticism and meticulous role separation between reasoning and recognition.

2. Core Insights: Critical Thinking and Division of Labor

Two principal insights underpin the DDCoT framework:

1. Keeping Critical Thinking: LLMs, when exposed to multimodal prompts, exhibit a tendency to treat all information as factual, often hallucinating visual aspects. By introducing explicit uncertainty in sub-answers through negative-space prompting—where the LLM marks visually-dependent questions as "Uncertain"—DDCoT enforces skepticism and compels the LLM to defer vision-based inferences to a VQA model.
2. Letting Everyone Do Their Jobs: Attempting joint reasoning over both visual and textual inputs in a single step leads to a proliferation of hallucinations due to untrustworthy integration. DDCoT separates concerns by allocating pure reasoning to the LLM and pure visual recognition to a VQA model. This division of responsibility leverages each model’s inherent strengths and curtails error amplification.

3. Mechanisms: Negative-Space Prompting and Responsibility Allocation

Negative-space prompting is the core architectural innovation in DDCoT. The framework decomposes an input question into sub-questions $q_i$. For each $q_i$:

• The LLM answers assuming no image is provided:
  • If answerable with world knowledge, the LLM responds concretely.
  • Otherwise, it outputs "Uncertain."

All sub-questions marked "Uncertain" create a "negative space"—gaps that a VQA model must fill. The process follows these steps:

1. Decomposition: The LLM produces $\{q_i, a^0_i\}$, with $a^0_i \in \{\text{"Uncertain"}\} \cup \mathcal{A}$.
2. Recognition: For all $i$ where $a^0_i =$ "Uncertain," a VQA model $f_{\mathrm{VQA}}$ processes $(\text{image}, q_i)$ to yield $a^{\text{vis}}_i$.
3. Joint Reasoning: Aggregate all $(q_i, a_i)$ pairs, where

$$a_i = \begin{cases} a^0_i & a^0_i \neq \text{“Uncertain”} \ a^{\text{vis}}_i & a^0_i = \text{“Uncertain”} \end{cases}$$

The LLM is then prompted to construct a global rationale, vigilantly integrating only valid sub-answers ("Note that some $a_i$ may be incorrect—select and integrate only the valid ones to produce a coherent rationale and final answer.").

Fine-tuning incorporates visual-text fusion via Rationale-Compressed Visual Embedding (RCVE) and Deep-Layer Prompting (DLP). Let $T \in \mathbb{R}^{N_t \times C}$ be the text embedding, $V_g \in \mathbb{R}^C$ and $V_l \in \mathbb{R}^{N_v \times C}$ as global and local image features:

• $V_t = \mathrm{CrossAttn}(V_g, T)$
• $$V_r = \mathrm{reshape}(\mathrm{MLP}(V_t)) \in \mathbb{R}^{N_r \times C_r}$$
• $V = \mathrm{CrossAttn}(V_r, V_l)$

$V$ is injected into encoder layers alongside learnable prompts $P_l$.

4. Prompting Workflow and Rationale Generation

The DDCoT procedure unfolds as follows:

• Step A: Decomposition
  • LLM generates sub-questions (e.g., “What foods are shown?”)
• Step B: Negative-Space Answering
  • LLM responds with either knowledge-based answers or “Uncertain.”
• Step C: Visual Filling
  • The VQA model fills all "Uncertain" responses with its own outputs.
• Step D: Chain-of-Thought Integration
  • LLM aggregates and synthesizes all facts (visual and textual) to construct a coherent rationale and final answer.

For example, for the question “Which nutrient is mainly provided by the foods shown?” given an image of fruits: - The LLM identifies food types as "Uncertain" (needing vision), but can specify "Fats" for an orange via world knowledge. - The VQA model identifies “Orange, banana” in the image. - The LLM merges these to justify a final answer such as “Vitamin C,” explaining the relationship through stepwise reasoning.

5. Experimental Setup, Performance, and Evaluation

Dataset: ScienceQA (21,000 multi-choice questions spanning NAT/SOC/LAN domains).

Models:

• Zero-shot: GPT-3, ChatGPT (with BLIP-2 for image captioning).
• Fine-tuning: UnifiedQA (T5-base) + CLIP ViT-L/14 encoder with RCVE and DLP.

Metrics: Accuracy on ScienceQA splits ({IMG, TXT, NO}, Grades 1–6, 7–12).

Performance Outcomes

Method	Setting	IMG Split Accuracy
GPT-3 (CoT)	Zero-shot	67.43%
DDCoT (GPT-3)	Zero-shot	69.96% (+2.53%)
ChatGPT (CoT)	Zero-shot	67.92%
DDCoT (ChatGPT)	Zero-shot	72.53% (+4.61%)
UnifiedQA	Fine-tuning	66.53%
DDCoT	Fine-tuning	83.34% (+16.81%)
MM-CoT†	Fine-tuning	75.11%
DDCoT	Fine-tuning	83.34% (+8.23%)

Fine-tuned DDCoT achieves an accuracy of 83.34% on image splits, substantially outperforming previous methods by a margin of up to 16.81%. Zero-shot gains ranged from +2.53% to +4.61% over baseline.

6. Generalizability, Explainability, and Ablation Analysis

Generalizability: When trained on two domains and evaluated on an unseen third (NAT/SOC/LAN in ScienceQA), DDCoT surpassed MM-CoT by +15.5%, +9.6%, and +12.2% respectively.

Ablation Studies:

• Naïve CoT rationales (without negative space) provided no gain on image splits.
• Duty-Distinct without uncertainty yielded a +2.58% gain.
• Duty-Distinct with explicit uncertainty led to +5.15%.
• Removing RCVE or DLP reduced accuracy by 3.02% and 0.99%, respectively.

Human Evaluation on 200 samples (12 groups, 3 raters each):

Rationale Quality	MM-CoT	DDCoT (Ours)
Relevance	70.8%	92.0%
Correctness	67.9%	86.4%
Completeness	64.8%	85.7%
Coherence	57.9%	84.3%
Explainability	58.7%	83.3%

Qualitative analyses confirm DDCoT’s ability to accurately identify map shapes, object-level attributes, and to incorporate factual world knowledge, whereas baselines frequently hallucinate or omit essential steps.

7. Conclusion and Future Prospects

DDCoT establishes a principled methodology for robust multimodal chain-of-thought reasoning by enforcing critical thinking through negative-space prompting and a duty-distinct division between LLM reasoning and VQA recognition. It achieves state-of-the-art results across both zero-shot and fine-tuning paradigms, and exhibits superior generalizability and human-rated explainability.

Identified future directions include:

• Reducing residual hallucinations via tighter verification or explicit uncertainty quantification.
• Incorporating multimodal pre-training to strengthen vision-language alignment prior to CoT induction.
• Extending DDCoT methodology to tasks such as image captioning, video QA, and exploring bias mitigation strategies in zero-shot prompting (Zheng et al., 2023).