Streaming Interventions: Can Video Large Language Models Correct Mistakes as They Occur?

Abstract: Learning everyday skills, like cooking a dish, relies increasingly on instructional media such as online videos. This opens the door to the use of video (and multimodal) LLMs as task guidance assistants. A crucial capability for the real-world success of a prospective task guidance assistant is it's ability to intervene proactively as soon as a mistake is apparent in order to guide the user. To evaluate this crucial capability, we introduce Ego-MC-Bench (Mistake Corrections), a benchmark for evaluating reactive, step-by-step task guidance in realistic cooking scenarios. Extensive experiments show that Ego-MC-Bench is highly challenging for state-of-the-art video LLMs. We argue that a key reason is the limited availability of training data for fine-tuning models on this task. Although there exists a wide range of cooking video datasets, existing datasets lack examples of mistakes along with appropriately timed interventions. To help address this data limitation, we also introduce Ego-CoMist, a counterfactual synthetic dataset created by transforming non -interactive cooking videos into supervised training examples showing proactive interventions. We show that fine-tuning on Ego-CoMist yields performance gains especially for smaller and more efficient video LLMs that are well suited for delivering assistance on edge devices.

• The paper introduces Ego-MC-Bench, a benchmark for assessing real-time mistake detection and timely interventions in cooking tasks.
• The paper details a synthetic dataset, Ego-CoMist, that generates counterfactual cooking mistakes with precise feedback timestamps.
• The paper demonstrates that fine-tuning on Ego-CoMist significantly improves instruction completion accuracy and mistake correction precision.

Streaming Interventions in Video LLMs: Evaluating and Improving Mistake Correction

Problem Formulation and Benchmark Introduction

The paper addresses the critical capability of multimodal LLMs—specifically video LLMs—to deliver real-time, proactive task guidance by intervening at the moment a mistake is detected during sequential tasks, such as cooking. Prior benchmarks largely focused on conversational or turn-based data and lacked the necessary fidelity for evaluating real-time, stepwise task guidance. In response, the authors introduce Ego-MC-Bench (Mistake Corrections), a benchmark explicitly designed for reactive interaction in the cooking domain.

Ego-MC-Bench comprises approximately 10 hours of egocentric video, annotated with stepwise instructions and diverse, realistic errors that are addressed through appropriately timed feedback from an instructor. This setup enables direct evaluation of two key abilities: step completion recognition and timely mistake intervention.

Figure 1: Ego-MC-Bench features instructor interventions with feedback at the precise moment a mistake becomes apparent, actively guiding the user toward successful goal completion.

The benchmark differs fundamentally from prior datasets by featuring reactive participants and temporally aligned feedback, rather than passive or post-hoc correction. This enables challenging, high-fidelity assessment of video LLMs in realistic assistant–user interaction scenarios.

Synthetic Counterfactual Data: Ego-CoMist Dataset Construction

In addition to benchmarking, the authors tackle the data scarcity problem for training reactive intervention models by introducing Ego-CoMist. Ego-CoMist is a large, synthetic dataset generated from non-interactive cooking video datasets, transformed via a pipeline that:

1. Synthesizes counterfactual recipe steps by perturbing attributes (quantities, techniques, timing, preparation, temperature) to represent realistic mistakes.
2. Determines optimal feedback delivery timestamps by prompting state-of-the-art LLMs with temporally-aligned stepwise narrations of each video.

   Figure 2: Ego-CoMist process—counterfactual mistake annotation and timestamp inference—enables synthetic training examples with precisely timed feedback for each error type.

This two-stage pipeline ensures fine-grained annotation of mistake type, instruction, and intervention timing. Human evaluation demonstrates validity of the generated instruction–feedback pairs (87.1%), and timestamp inference demonstrates high temporal alignment (∼65% of annotations within 2.5 seconds of human-provided ground truth).

Ego-CoMist scales up reactive supervision by producing thousands of pairs across multiple error classes, facilitating supervised fine-tuning of video LLMs for streaming, proactive guidance.

Empirical Evaluation: Video LLMs on Streaming Interventions

The authors conduct extensive benchmarking of state-of-the-art video LLMs—both “turn-based” (prompted at fixed intervals) and streaming—with Ego-MC-Bench. Metrics are derived from prior reactive cooking datasets and include instruction completion accuracy (IC-Acc), mistake intervention precision/recall/F1, and fluency (BERTScore, ROUGE-L).

Findings reveal that all evaluated proprietary and open-source models struggle with mistake detection and intervention timing:

• Best-performing proprietary model (Gemini-3-Flash) achieves F1=0.18 on per-recipe steps; most open models achieve near-zero F1.
• Models frequently misinterpret step completion or fail to time interventions to observable mistakes.
• Streaming narration models converted to feedback via a helper LLM perform poorly due to lack of actionable feedback at the appropriate times.

  Figure 3: Streaming interventions—Gemini-3-Flash fails to timely correct user errors; Qwen3.5-2B fine-tuned with Ego-CoMist+ intervenes appropriately and guides successful instruction completion.

Strong improvement is observed when smaller (edge-suitable) models are fine-tuned on Ego-CoMist+ data:

• Qwen3.5-2B fine-tuned on Ego-CoMist+ achieves IC-Acc=37.1 and mistake F1=0.20, outperforming baseline models.
• Models trained on Ego-CoMist+ demonstrate both superior reactivity and fluency compared to those trained only on post-error datasets (e.g., QICD).
• Explicit multi-task prompts distinguishing intervention strategies further boost performance when mixing datasets.

Figure 4: Ego-MC-Bench streaming interventions: fine-tuned models on Ego-CoMist+ reactively correct diverse mistakes, offering precise and timely feedback.

Practical and Theoretical Implications

This work establishes both a benchmark and a scalable synthetic data pipeline for evaluating and training video LLMs in real-time, reactive assistive settings. The results demonstrate that off-the-shelf video LLMs are inadequate for mistake correction, even when queried at regular intervals, and that exposure to reactive, temporally aligned intervention data is pivotal for robust assistant behavior.

Practically, Ego-MC-Bench enables assessment of the temporal and semantic reasoning required for situated assistance and can serve as a testbed for diverse settings (e.g., fitness, assembly, robotics). Ego-CoMist provides scalable supervision, unlocking fine-tuning for smaller, efficient models suitable for edge deployment.

Theoretically, these findings underscore the need for multimodal, temporally-aware architectures capable of continuous progress monitoring, causal inference, and grounding in real-world procedural activities. The synthetic data pipeline for counterfactual mistakes could extend to other domains, propelling further research in proactive agentic interaction and contextual adaptation.

Speculation on Future Developments

Advancements in video LLMs informed by this benchmark and dataset are poised to:

• Enable edge-deployable, real-time assistive agents in diverse domains via improved efficiency and reactivity.
• Foster research into multimodal temporal attention mechanisms and action prediction for situated interaction.
• Motivate richer, causal feedback modeling to supplement current “observation-only” guidance.

Further, the synthetic counterfactual pipeline could be generalized and automated for other complex, multi-step activities, accelerating training for proactive and context-sensitive embodied AI.

Conclusion

The paper introduces Ego-MC-Bench—an interactive, mistake-centred benchmark—and Ego-CoMist—a scalable, counterfactual dataset—to address the gap in evaluating and training video LLMs for streaming, proactive task guidance and correction. Empirical analysis shows state-of-the-art models are inadequate for real-time intervention, but fine-tuning on synthetic counterfactual feedback yields significant gains, especially for efficient, edge-targeted architectures. The benchmark, dataset, and findings collectively represent a foundational advance for research in multimodal assistant evaluation, situated learning, and proactive agent development (2606.09547).