CoQ: Multimodal Curiosity & Evidence Acquisition

• Chain of Questions (CoQ) is a reasoning paradigm that actively acquires evidence by generating targeted, modality-specific questions.
• It employs a prompt-to-question-to-task-to-sensor pipeline that maps queries to perceptual tasks, guiding the selection of vision, audio, spatial, and text modalities.
• The framework reframes multimodal reasoning from passive fusion to active inquiry, enhancing interpretability while highlighting challenges in precise sensor activation.

Chain of Questions (CoQ) is a structured reasoning paradigm in which a LLM generates intermediate questions before producing an answer. In the multimodal formulation introduced in "Chain of Questions: Guiding Multimodal Curiosity in LLMs" (Iji et al., 6 Aug 2025), those questions are not merely explanatory text: they identify missing evidence, specify the perceptual task needed to obtain that evidence, trigger the corresponding sensor or tool, and then support grounded answer generation. CoQ therefore reframes multimodal reasoning as active evidence acquisition across text, vision, audio, and spatial perception, rather than passive consumption of auxiliary inputs that are already present in the prompt (Iji et al., 6 Aug 2025).

1. Conceptual basis and problem setting

Conventional chain-of-thought and other step-by-step prompting methods improve textual reasoning by eliciting intermediate steps, but they remain largely unimodal and passive with respect to non-text inputs. In multimodal settings, however, a model must decide not only how to reason, but also what information to obtain and from which channel. The CoQ framework is proposed for precisely this setting: instead of assuming that images, audio, or 3D scans are already available and relevant, it makes the model ask targeted questions about the environment and use those questions to guide modality activation (Iji et al., 6 Aug 2025).

The motivating claim is that existing multimodal LLMs often treat vision, audio, or spatial data as supplementary context already provided. This is limiting in dynamic environments, where the central problem is often evidence selection rather than evidence fusion. CoQ operationalizes what the paper terms "multimodal curiosity": when information is missing, the model first asks for it; when a modality is relevant, it explicitly engages it. The intended result is a procedure that more closely mirrors inquiry-driven reasoning than standard multimodal prompting (Iji et al., 6 Aug 2025).

Within this formulation, the relevant modalities are vision, audio, spatial perception, and text. Vision is associated with functions such as object identity or scene description, audio with speech content and sound events, and spatial perception with geometry and location. This division does not define a learned sensory policy; rather, it establishes a procedural framework for selecting and invoking perceptual resources when the prompt underdetermines the answer (Iji et al., 6 Aug 2025).

2. Formal structure of the multimodal CoQ framework

The conceptual pipeline is defined as

$$\text{Prompt} \rightarrow \text{Question} \rightarrow \text{Task} \rightarrow \text{Sensor}.$$

Given a textual prompt $P$, the model generates a sequence of modality-specific questions

$Q = \{ q_1, q_2, \dots, q_k \}.$

Each question $q_i$ is mapped to a perceptual task by a task selection function $\mathcal{T}$, and each task is mapped to a sensor by a sensor assignment function $\mathcal{S}$:

$$T_i = \mathcal{T}(q_i), \qquad S_i = \mathcal{S}(T_i).$$

Executing $T_i$ with $S_i$ yields an observation $o_i$, and the resulting observation set $P$0 is aggregated into a multimodal context $P$1. The final answer is then produced by integrating the original prompt with that aggregated context:

$P$2

This formalization is notable for what it excludes. The paper does not introduce a probabilistic or reinforcement-learning treatment of curiosity, does not define a modality-selection policy $P$3, and does not specify information-gain objectives or explicit training losses for question quality or modality gating. The framework is instead procedural, with deterministic functions $P$4 and $P$5 plus an aggregation-and-answer function $P$6, instantiated either by few-shot prompting or by fine-tuning (Iji et al., 6 Aug 2025).

Representative question-to-task mappings are explicit. "What do I see?" maps to object detection; "Who am I looking at?" to captioning; "What are they saying?" to speech-to-text; "What am I hearing?" to sound event detection; "What is the sentiment?" to sentiment analysis; "What is the spatial location?" to spatial detection; "What is the pose?" to pose estimation; "What are they doing?" to action recognition; "Who is talking?" to speaker ID; and "What language?" to language ID (Iji et al., 6 Aug 2025).

3. Inference workflow and system integration

At inference time, CoQ follows an explicit run-time procedure. The model reads the prompt $P$7, generates targeted multimodal questions $P$8, maps each question to a perceptual task with $P$9, assigns the appropriate sensor with $Q = \{ q_1, q_2, \dots, q_k \}.$0, executes each task-sensor pair to obtain observations $Q = \{ q_1, q_2, \dots, q_k \}.$1, aggregates those observations into a multimodal context $Q = \{ q_1, q_2, \dots, q_k \}.$2, and finally produces a grounded answer $Q = \{ q_1, q_2, \dots, q_k \}.$3. The framework also allows the chain of questions, tasks, sensors, and observations to be logged as a reasoning trace for interpretability (Iji et al., 6 Aug 2025).

Two implementation paths are discussed. The first, and the one emphasized experimentally, is few-shot prompting: the model is primed with instructions and examples demonstrating curiosity-driven multimodal question generation and the mapping from questions to tasks and sensors. The second is fine-tuning, in which the model is explicitly trained to produce modality-specific questions during reasoning. The paper presents fine-tuning as a possible extension that may improve consistency and accuracy, but it also notes the increased compute cost. No reinforcement learning or policy-gradient training is introduced (Iji et al., 6 Aug 2025).

Architecturally, CoQ is instantiated on top of general-purpose LLMs rather than through internal redesign of the base model. The reported experiments evaluate FLAN-T5 variants, which are encoder-decoder models, and Llama 2, which is decoder-only, without modifying their internal architectures. External perception is delegated to task-specific modules and sensors selected through $Q = \{ q_1, q_2, \dots, q_k \}.$4 and $Q = \{ q_1, q_2, \dots, q_k \}.$5, including cameras, microphones, LiDAR, speech-to-text engines, object detectors, captioning models, and 3D spatial analyzers (Iji et al., 6 Aug 2025).

The fusion mechanism is left intentionally underspecified. Observations are aggregated into a context $Q = \{ q_1, q_2, \dots, q_k \}.$6 and fed back to the LLM for final reasoning, but the paper does not define a concrete fusion architecture such as cross-attention layers. Aggregation is described conceptually and may be implemented as textual summaries, structured metadata, or embeddings. This suggests that CoQ is intended as a control framework over perception and reasoning components rather than as a specific multimodal backbone design (Iji et al., 6 Aug 2025).

4. Benchmark construction and empirical findings

To evaluate multimodal curiosity, the paper constructs a composite benchmark by integrating four datasets with different modality requirements: WebGPT, ScienceQA, AVSD, and ScanQA. WebGPT contributes 19,578 human-generated prompts and is marked as not requiring additional modalities. ScienceQA is divided into cases that require visual evidence and strictly textual cases. AVSD requires joint audio-visual context for dialog grounded in video sequences. ScanQA contributes 41,363 question-answer pairs derived from 800 ScanNet indoor 3D scans and explicitly requires spatial reasoning. The resulting integrated benchmark contains 180,629 labeled instances spanning text-only, image-supported, audio-visual, and spatial tasks (Iji et al., 6 Aug 2025).

The evaluation is centered on question generation and modality alignment rather than on downstream task accuracy. The primary metric is whether the generated questions align with the modality requirements of each instance, recorded as Match versus Mismatch. A secondary metric measures whether the model attempts curiosity-driven engagement at all, recorded as Asked versus Did Not Ask. Standard downstream QA metrics such as accuracy, BLEU, METEOR, and F1 are explicitly not the focus (Iji et al., 6 Aug 2025).

The evaluated models are FLAN T5 base (250M), FLAN T5 large (780M), FLAN T5 xl (3B), and Llama 2 (7B). Decoding strategies include greedy, sampling, and beam search, with beam search noted to incur higher computational overhead. The FLAN T5 models are run on P100 GPUs, Llama 2 on A100 GPUs, and batched inference is used over the 180,629-instance benchmark (Iji et al., 6 Aug 2025).

Model	Match %	Asked %
FLAN T5 XL	76.2	80.0
LLaMA 7B	43.9	41.0
FLAN T5 Large	26.3	72.5
FLAN T5 Base	17.6	33.6

In absolute terms, FLAN T5 XL records 137,701 matches against 42,928 mismatches, and 144,547 asked cases against 36,082 did-not-ask cases. LLaMA 7B records 79,355 matches against 101,274 mismatches, and 74,036 asked cases against 106,593 did-not-ask cases. FLAN T5 Large shows a high curiosity rate with 130,941 asked cases but much weaker targeting precision, with only 47,511 matches. FLAN T5 Base records 31,861 matches and 60,773 asked cases (Iji et al., 6 Aug 2025).

The reported interpretation is that both architecture and scale matter for precise question generation. FLAN T5 XL, an encoder-decoder model at 3B parameters, substantially outperforms both smaller T5 variants and the larger decoder-only Llama 2 on modality alignment. At the same time, the comparison between FLAN T5 Large and FLAN T5 XL shows that asking more questions is not sufficient on its own: curiosity without accurate modality targeting leads to poor alignment (Iji et al., 6 Aug 2025).

The paper does not report targeted ablations on variables such as question-chain length, gating strategies, or loss-term contributions. Instead, differences across model sizes and architectures serve as a coarse ablation on capacity and design. Qualitative exposition is likewise limited: the paper supplies canonical question types and a conceptual figure for the prompt-to-question-to-task-to-sensor pipeline, but does not enumerate extensive case studies or full reasoning traces (Iji et al., 6 Aug 2025).

5. Interpretability, adjacent paradigms, and terminological variants

The central conceptual contrast is with chain-of-thought. Chain-of-thought elicits intermediate textual steps but does not decide how to obtain new sensory evidence. CoQ adds an upstream stage in which the model formulates questions that determine what evidence to acquire and which modality to engage. In that sense, CoQ is both a reasoning framework and a control interface for perception (Iji et al., 6 Aug 2025).

The framework is also related to tool-usage prompting and planner-controller designs. Like a planner, CoQ determines what perceptual tools to invoke. Unlike a full planner-controller architecture, however, it does not define explicit planning policies, cost-aware tool selection, or hierarchical control. The chain consists of generated questions plus deterministic mappings to tasks and sensors. Its interpretability follows from this explicit structure: the question chain provides a visible rationale for why a modality was engaged, how observations relate to subgoals, and why the final answer is grounded in particular sensor-derived evidence. The paper argues that this structure can reduce hallucination, and that logging the chain yields transparent reasoning traces (Iji et al., 6 Aug 2025).

A frequent source of confusion is that the acronym CoQ is polysemous in recent arXiv literature. "Chain-of-Questions Training with Latent Answers for Robust Multistep Question Answering" uses CoQ for a text-centric multistep QA framework that generates sub-questions and latent sub-answers under QDMR supervision and optimizes them with a dynamic mixture of Hard-EM and MAPO (Zhu et al., 2023). "Generalizing Visual Question Answering from Synthetic to Human-Written Questions via a Chain of QA with a LLM" presents CoQAH, in which an LLM decomposes a human-written visual question into template-conformant sub-questions for a specialized VQA model, with an Existence and Uniqueness Handler enforcing logical validity (Kim et al., 2024). "V-REX: Benchmarking Exploratory Visual Reasoning via Chain-of-Questions" uses CoQ as an evaluation abstraction that separates planning from following in visual exploration with finite option sets at each step (Fan et al., 12 Dec 2025). By contrast, "Chain-of-Query: Unleashing the Power of LLMs in SQL-Aided Table Understanding via Multi-Agent Collaboration" explicitly uses CoQ to mean Chain-of-Query rather than Chain-of-Questions (Sui et al., 14 Aug 2025).

These works share a family resemblance—decomposition, intermediate decision points, and explicit reasoning traces—but they are not interchangeable. The multimodal curiosity framework of (Iji et al., 6 Aug 2025) is distinguished by its prompt-to-task-to-sensor pipeline and its emphasis on selective modality activation, rather than on latent-answer optimization, template bridging, benchmark design, or SQL clause generation.

6. Limitations, misconceptions, and open problems

Several limitations are explicit. If the early questions are poorly targeted, the framework may activate irrelevant modalities, collect noisy evidence, and compound errors in downstream reasoning. Because $Q = \{ q_1, q_2, \dots, q_k \}.$7 and $Q = \{ q_1, q_2, \dots, q_k \}.$8 are deterministic and not learned end-to-end, the framework cannot optimize task selection or sensor assignment for information gain or uncertainty reduction. The paper identifies this absence of formal optimization as a major open direction, noting that no policy or reward-driven mechanism quantifies curiosity, expected utility, or confidence reduction (Iji et al., 6 Aug 2025).

Scalability is also unresolved. Handling unseen modalities or tasks, coordinating multiple sensors, and operating in open-world environments are presented as open challenges. The reported experiments rely primarily on few-shot prompting; fine-tuning for more consistent question generation and better modality targeting is proposed but not carried out as the main experimental regime. The paper also does not provide code, and it does not quantify runtime beyond reporting GPU configurations, batching, and the additional overhead of beam search (Iji et al., 6 Aug 2025).

Two misconceptions are directly countered by the reported setup. First, CoQ should not be read as a reinforcement-learning solution for multimodal evidence selection: no such training objective is defined. Second, the benchmark results should not be interpreted as standard end-to-end QA performance, because the study is centered on the quality of generated questions and their modality alignment rather than on answer accuracy in the usual sense. A plausible implication is that downstream gains will depend heavily on the quality of the attached perception modules and aggregation design, both of which are left largely conceptual in the current formulation (Iji et al., 6 Aug 2025).

In that form, CoQ is best understood as a procedural layer for active multimodal inquiry. Its core contribution is to make modality choice explicit and inspectable by interposing a chain of targeted questions between the prompt and the final answer. That intervention is modest in formal terms, but it changes the locus of reasoning from passive fusion to selective evidence acquisition, and it defines a clear research agenda around learned question policies, adaptive modality engagement, and grounded multimodal control (Iji et al., 6 Aug 2025).