HowToDIV: Multimodal Expert-Novice Dialogue

• HowToDIV is a multimodal dataset of expert–novice dialogues constructed from instructional videos using an automatic LLM-based pipeline.
• It provides fine-grained procedural guidance with egocentric video alignment, enabling error correction and interactive assistance in AR settings.
• The dataset comprises 507 sessions, 6,636 dialogue turns, and 24 hours of video across domains like cooking, mechanics, and planting.

Searching arXiv for the HowToDIV papers and closely related versions. HowToDIV is a multimodal dataset of expert–novice conversations grounded in egocentric video clips and aligned to atomic procedural steps. It is constructed via a fully automatic, LLM-based pipeline that converts single-person instructional videos into two-person task-guidance conversations, with each turn linked to the user’s visual context through timestamped video segments. The dataset targets procedural task assistance in an augmented reality or wearable-agent setting, where a novice user requests help and an expert provides step-wise guidance, clarification, and corrective feedback during task execution. In its reported release, HowToDIV comprises 507 sessions, 6,636 dialogue turns, and approximately 24 hours of egocentric video spanning cooking, mechanical repair, and planting (Aggarwal et al., 15 Aug 2025).

1. Definition and research motivation

HowToDIV addresses the lack of scalable conversational datasets grounded in real-world procedural execution for AI task assistants. The motivating scenario is a novice equipped with an AR device or AI glasses with camera, microphone, and speaker, seeking help while performing a multi-step task. In this setting, the expert must guide the novice through atomic steps, answer clarifications, and correct mistakes while grounding responses in the novice’s surroundings (Aggarwal et al., 15 Aug 2025).

The dataset is positioned against two limitations in prior resources. First, major instructional video corpora such as HowTo100M, YouCook2, EPIC-KITCHENS, Ego4D, COIN, CrossTask, EGTEA, and Assembly101 are primarily monologic or step-labeled rather than dialogic. Second, interactive egocentric assistance datasets such as HoloAssist require intensive multimodal collection and manual annotation, making them costly to scale. HowToDIV’s central novelty is therefore the transformation of single-person instructional videos into expert–novice dialogues that are coherently aligned both to atomic procedural steps and to timestamped egocentric clips emulating the novice’s first-person stream (Aggarwal et al., 15 Aug 2025).

This design explicitly targets agents that can give step-wise guidance, answer clarifications, detect user mistakes, and recommend corrections. A plausible implication is that the dataset is intended not merely as a corpus of conversations, but as a benchmark substrate for long-horizon procedural reasoning, grounded dialogue generation, and corrective assistance in AR-like settings.

2. Source corpora, domains, and data model

HowToDIV is built atop two publicly available instructional video sources. The first is Narrated Instruction Videos (NIV), comprising 150 YouTube videos with expert narrations and ASR subtitles; for HowToDIV, egocentric-view videos were selected to align with the first-person objective. The second is Egocentric Procedural Error (EgoPER), consisting of 386 egocentric cooking recordings from HoloLens2 with RGB, depth, audio, gaze, hand pose, fine-grained step timestamps, and both normal and error trials (Aggarwal et al., 15 Aug 2025).

The covered domains are cooking, mechanics, and planting, with nine tasks overall. Reported examples include coffee, tea, oatmeal, pinwheels, quesadilla, jump-starting a car, changing tires, and re-potting plants. Sessions are categorized by speech style and action type: concise versus regular speech, and Following-Steps versus Making-Error executions (Aggarwal et al., 15 Aug 2025).

The dataset representation is formalized as a conversation sequence

$C = (c_1, \ldots, c_P),$

where each turn includes user and expert utterances, temporal grounding, and, in the later formulation, a grounded video segment:

$c_i = (u_i, e_i, v_i, t_{is}, t_{ie}).$

For narrated videos, the input consists of frames

$V = (I_1, I_2, \dots, I_N)$

and subtitle entries

$S_j = (s_j, t_{js}, t_{je}),$

while step-labeled silent videos use annotations

$Q_j = (q_j, t_{js}, t_{je}).$

The released session structure includes an ordered instruction list, turn-level dialogue, timestamps, aligned user-viewpoint clips, task and domain labels, and flags such as error detection or correction (Aggarwal et al., 15 Aug 2025).

3. Automatic monologue-to-dialogue conversion pipeline

The construction procedure is a three-stage Monologue-to-Dialogue Conversion pipeline. The first stage, Instruction Formation, extracts a complete, ordered set of atomic instructions. For NIV, transcripts are obtained from existing annotations or ASR, and an instruction-tuned LLM is prompted to infer concise step-wise instructions while enriching implicit procedural knowledge present in narration. For EgoPER, provided step annotations and timestamps are post-processed to merge duplicates, normalize descriptions, cluster semantically similar actions, enforce atomicity, and filter non-essential actions such as “open drawer” (Aggarwal et al., 15 Aug 2025).

The second stage, Dialogue Generation, maps one atomic instruction to one dialogue turn. The novice does not have access to the instruction list, whereas the expert does. The LLM produces multi-turn expert–novice interactions in concise and regular speech styles. For error scenarios derived from EgoPER, corrective steps are marked with special tokens such as <user error>, so that the generated expert responses identify mistakes and prescribe corrections rather than hallucinating unrelated actions. Procedural caveats and nuanced constraints present in narration are converted into clarification questions to enrich the interaction structure (Aggarwal et al., 1 Feb 2026).

The third stage, Video Localization, aligns each atomic step with the corresponding egocentric segment. For fine-grained step datasets such as EgoPER and EGTEA, provided timestamps are used directly, with aggregation when multiple actions merge into one instruction. For narrated NIV-style videos, subtitle timestamps from transcription are used to estimate segment boundaries, and the same LLM call that extracts instructions also infers temporal boundaries from narration segments. The papers explicitly note that more advanced temporal localization is left to future work (Aggarwal et al., 15 Aug 2025).

The pipeline is fully automatic and does not rely on manual annotation for core dataset creation. This is fundamental to HowToDIV’s positioning as a scalable, low-cost alternative to human-in-the-loop collection.

4. Dataset characteristics and internal organization

In the reported release, HowToDIV contains 507 conversations, 6,636 dialogue turns, and about 24 hours of egocentric video. The mean number of turns per session is about 13, with task-dependent ranges from 9.8 for simpler tasks to 16.4 for more complex tasks. Expert responses average 19.3 words, and clip durations average 12.5 seconds, with a distribution spanning from 1 second to more than 2 minutes (Aggarwal et al., 15 Aug 2025).

Speech-style statistics distinguish 180 concise sessions, averaging 3.4 words per user turn, from 327 regular sessions, averaging 10.7 words per user turn. Among regular sessions, 252 are Following-Steps sessions and 75 are Making-Errors sessions. Error examples include mismatches such as tearing a paper filter instead of folding it, or pouring sugar instead of honey, followed by corrective expert guidance (Aggarwal et al., 1 Feb 2026).

The train, validation, and test splits are stratified by task and user category. Reported sizes are 355, 44, and 108 sessions respectively, corresponding approximately to a 70/10/20 split. All published benchmark results are reported on the test split (Aggarwal et al., 15 Aug 2025).

A concise summary of the reported dataset scope is given below.

Aspect	Reported value
Sessions	507
Dialogue turns	6,636
Video duration	24 hours
Domains	Cooking, mechanics, planting
Tasks	9 tasks
Train/val/test	355 / 44 / 108

HowToDIV is explicitly presented as the first resource of its kind to combine instructions, dialogues, errors, egocentric clips, and step-level temporal alignment at low cost. Relative to instructional datasets, it adds expert–novice dialogue and egocentric grounding; relative to egocentric action datasets, it adds conversational guidance and corrective interaction; relative to HoloAssist, it trades richer manual multimodal annotation for breadth across domains and automatic construction (Aggarwal et al., 15 Aug 2025).

5. Quality control, benchmark setup, and reported results

The core benchmark task is expert response generation for procedural assistance dialogues: given the user query, and optionally the instruction list, the model must generate the appropriate next-step guidance. Published baselines evaluate this in text-only mode; multimodal response generation using video input is reserved for future work (Aggarwal et al., 15 Aug 2025).

Two inference-time prompting regimes are defined. In “Hint-only” or “History-only,” the model sees only the dialogue history and must infer the next step from internal domain knowledge. In “Hint + Steps” or “History + Steps,” the full instruction list is appended to the system prompt, providing explicit procedural structure. Dataset generation uses Gemma-3 27B instruction-tuned, baseline inference uses Gemma-3 4B, LLM-as-a-Judge uses Gemma-3 12B, and a later version additionally reports Qwen2.5-VL-7B baselines (Aggarwal et al., 1 Feb 2026).

Evaluation uses BLEU, ROUGE-1, ROUGE-2, ROUGE-L, and LLM-as-a-Judge. BLEU is reported as

$$\text{BLEU} = \text{BP} \cdot \exp\Big(\sum_{n=1}^{N} w_n \log p_n\Big),$$

and LLM-as-a-Judge assigns 1–5 scores for correctness, relevance, completeness, and helpfulness. For temporal localization visualization, the papers report IoU, precision, and accuracy (Aggarwal et al., 15 Aug 2025).

The main reported results show a strong effect of explicit procedural context. For Gemma-3 4B, Hint-only yields BLEU 0.321, ROUGE-1 0.325, ROUGE-2 0.125, ROUGE-L 0.270, and LLM-as-a-Judge 2.870, whereas Hint + Steps yields BLEU 0.457, ROUGE-1 0.489, ROUGE-2 0.268, ROUGE-L 0.429, and LLM-as-a-Judge 4.101. In the later benchmark, Qwen2.5-VL-7B achieves BLEU 0.281, ROUGE-L 0.220, and LLM-J 2.995 in History-only mode, improving to BLEU 0.441, ROUGE-L 0.356, and LLM-J 4.232 in History + Steps mode (Aggarwal et al., 1 Feb 2026).

Performance also varies by user category. Under Gemma-3 Hint-only, Regular-Follow outperforms Concise-Follow and Regular-Error; more generally, regular speech outperforms concise speech, and Following-Steps outperforms Making-Error. ROUGE-L declines as turn index increases, indicating long-horizon compounding error. A temporal alignment case study for changing a car tire reports IoU 0.392, precision 54.84%, and accuracy 57.95% (Aggarwal et al., 15 Aug 2025).

6. Applications, limitations, and research directions

The dataset is intended for training interactive assistants that provide step-wise guidance with grounded visual context. The authors also note its suitability for fine-grained action recognition, instruction-conditioned video generation, and retrieval tasks, including improving models such as VideoCLIP (Aggarwal et al., 15 Aug 2025).

Several limitations are explicitly identified. Current baselines are text-only, so multimodal response generation remains unbenchmarked. Instruction extraction and temporal localization rely on ASR and simple heuristics, especially for narrated videos where subtitle-based alignment may be coarse. Hint-only results reveal hallucinations and domain gaps, and benchmark performance degrades over long dialogues. Error coverage is partial: modifications and corrections are included, but omissions, additions, and slips from the full EgoPER taxonomy are not fully used (Aggarwal et al., 15 Aug 2025).

The later paper adds a sampled human review: two trained annotators judged 175 turns, defining a turn as usable if both annotators agreed it satisfied at least two criteria among instruction correctness, dialogue naturalness, and video-step alignment. This procedure reported 93.2% usable turns. Automatic filtering also removed duplicate utterances, degenerate responses, outliers in length, profanity, and failed temporal localization, with fewer than 4% of initially generated turns removed (Aggarwal et al., 1 Feb 2026).

Ethical concerns arise primarily from egocentric data. Although HowToDIV reuses public videos or controlled recordings, downstream live deployment must address personally identifiable information, bystander privacy, and safety implications in domains such as repair or maintenance. The published future directions include scaling to additional datasets such as EPIC-KITCHENS, CrossTask, YouCook2, EpicTent, and EGTEA; enriching proactivity, out-of-order steps, and error detection; augmenting data via image-conditioned video generation such as FramePack; fine-tuning LLMs on HowToDIV; and developing multimodal baselines and more robust evaluation protocols (Aggarwal et al., 15 Aug 2025).

In aggregate, HowToDIV constitutes a procedural-assistance benchmark organized around a specific representation choice: one atomic instruction, one dialogue turn, and one temporally grounded egocentric clip. This suggests a particularly useful role for studying long-horizon grounded dialogue, corrective interaction, and memory-sensitive reasoning in wearable or AR-centered assistants.