---
title: 'Demo-ICL-Bench: Procedural ICL in Videos'
url: https://www.emergentmind.com/topics/demo-icl-bench
type: topic
---

# Demo-ICL-Bench: Procedural ICL in Videos

Demo-ICL-Bench is a benchmark specifically constructed to evaluate the capacity of Multimodal Large Language Models (MLLMs) for in-context learning (ICL) in the domain of procedural knowledge acquisition from videos. The primary goal is to assess not a model’s static knowledge, but rather its ability to dynamically learn and adapt to novel procedural video contexts given few in-context demonstrations, encompassing both text (stepwise instructions) and video exemplars. Demo-ICL-Bench emerges from the broader need to move beyond traditional video understanding tasks, which typically measure retrieval or comprehension from static learned knowledge, toward a paradigm that assesses on-the-fly transfer and adaptation from in-context demonstrations [2602.08439].

## 1. Benchmark Construction and Data Curation

Demo-ICL-Bench is derived from the HowTo100M corpus, which comprises over 1.2 million narrated YouTube instructional videos spanning approximately 23,000 activities. For the benchmark, only English-language videos of moderate length (2–10 minutes) and high-quality ASR transcripts (HTM-AA/WhisperX) are retained. The data construction strategy involves three core demonstration modalities:

- **Text Demonstrations**: Each selected video’s ASR transcript is summarized into a sequence of numbered procedural steps. This is accomplished using a two-stage LLM pipeline: Qwen2.5-72B first generates coarse outlines, and Qwen2.5-VL-72B jointly uses sampled frames to refine and ground each step, producing concise, human-interpretable instructional sequences.
- **Video Demonstrations**: Pairs of videos depicting the same procedural task are constructed. Candidate pairs are first selected from metadata (ranking, title similarity) and further checked by comparing their LLM-generated stepwise instructions to ensure close alignment of demonstrated procedures.
- **Demonstration Selection**: For increased task difficulty, the benchmark includes a subset where the model must first select the correct demonstration from a distractor pool, before answering the procedural question.

The final benchmark comprises 500 text-demo questions, 500 video-demo questions, and 200 demonstration-selection items, totaling 1,200 manually curated test samples.

## 2. Task Definition and Evaluation Protocol

Demo-ICL-Bench formalizes three hierarchical tasks:

1. **Text-Demo ICL**: Given a video $V$ and its text demonstration $T$, the model must predict the next procedural step (“What comes next?”) at a chosen intermediate point in the instruction sequence.
2. **Video-Demo ICL**: Provided a target video $V$ and a reference demonstration video $V_D$, the model is required to generate the next step for $V$. Alignments between steps are rigorously mapped by LLMs, with human verification of answerability.
3. **Demonstration Selection (DS)**: Given $V$ and a set of four candidate demonstrations, the model must select the most suitable demonstration and then answer the next-step question conditioned on its selection.

Evaluation metrics are constructed as follows:

- **Demo. Acc**: Accuracy when using demonstrations.
- **w/o Demo**: Accuracy without using demonstrations (model relies on internal knowledge only).
- **Δ_ICL**: ICL gain, computed as the difference between performance with and without demonstrations.
- **S.Acc (DS)**: Success rate of demonstration selection.
- **Avg**: Arithmetic mean of accuracies across tasks.

## 3. Demo-ICL Model: Architecture and Training Regime

The Demo-ICL model is implemented atop Ola-Video, deploying the OryxViT vision encoder for dense video frame representation (up to 64 frames at resolutions 288×288–480×480) combined with Qwen2.5 as the LLM backbone, supporting a 16,384-token context. Input demonstrations are encoded as follows:

- **Text demonstrations**: Direct concatenation to the question prompt.
- **Video demonstrations**: Sampled frames and (optionally) ASR captions from the demo video are passed through the same visual encoder and input into the Transformer alongside the target video.

Demo-ICL training proceeds in two phases:

1. **Video-Supervised Fine-Tuning (VSF)**: The model undergoes large-scale supervised training on multimodal corpora aggregating millions of image–text and video–text pairs (e.g., from LLaVA-OneVision, VisualWebInstruct, LLaVA-Video, Oryx, COIN, Cross-Task), explicitly injecting demo-driven ICL samples. The cross-entropy loss is defined over the target response tokens, conditioned on both visual and demonstration context.
   
   $$
   \mathcal{L}_{\mathrm{VSF}} = -\sum_{i=1}^N \sum_{j=1}^{|y^{(i)}|} \log p(y_j^{(i)} \mid y_{<j}^{(i)}, V^{(i)}, D^{(i)})
   $$

2. **Information-Assisted Direct Preference Optimization (IADPO)**: This fine-tuning step leverages contrastive response pairs, augmented with auxiliary information (e.g., video timestamps or textual summaries) to supply preference signals. The reward model uses a Bradley–Terry preference formulation,
   
   $$
   p^{*}(R_c \succ R_r \mid x, I) = \sigma(r^*(x, I; R_c) - r^*(x, I; R_r))
   $$
   
   The policy is updated by maximizing the preference for chosen responses using the learned reward.

## 4. Experimental Results and Comparative Analysis

Demo-ICL-Bench exposes marked difficulty even for state-of-the-art models. Human upper bounds on the benchmark stand at 84.0% (text-demo), 80.4% (video-demo), and 88% (demonstration selection), with overall average 80.1%. Proprietary models demonstrate moderate performance, e.g., Gemini-2.5-Pro achieves 54.4% (text-demo) and 36.2% (video-demo), and open-source baselines such as Qwen2.5-VL (7B) obtain 32.8% and 28.0% respectively.

The full Demo-ICL (7B) model achieves:

| Task                    | Demo-ICL (full, 7B) |
|------------------------ |---------------------|
| Text-Demo Demo.Acc      | 43.4                |
| Text-Demo w/o Demo      | 29.4                |
| Δ_ICL                   | +14.0               |
| Video-Demo Demo.Acc     | 32.0                |
| Video-Demo w/o Demo     | 27.6                |
| Δ_ICL                   | +4.4                |
| DS S.Acc                | 58.0                |
| DS Acc                  | 24.0                |
| Avg                     | 33.1                |

A consistent, substantial gain ($\Delta_{ICL}$) is observed when demonstrations are provided, particularly for text-based demonstrations. Performance generalizes to standard video understanding and temporal tasks (e.g., VideoMMMU, VideoMME, MVBench, LongVideoBench, MLVU), matching or exceeding other 7B models and approaching 72B-scale open-source systems.

Ablation studies on video-demo accuracy confirm that increasing the number of provided frames, supplying ASR captions, and repeating the demo video enhance accuracy, underscoring the criticality of fine-grained temporal and semantic alignment.

## 5. Design Choices in Data Generation and Demonstration Curation

The construction of Demo-ICL-Bench incorporates rigorous selection and alignment protocols:

- **Automatic Summarization**: Multistage LLM-driven summarization of instructional transcripts ensures not only coverage of procedural steps but also semantic grounding in visual evidence via sampled frames.
- **Video Pair Mining and Alignment**: Candidate video pairs are mined using metadata and LLM-based semantic similarity, then filtered by alignment of their instructional decompositions. This ensures high-quality exemplars for procedural transfer.
- **Human Verification**: Key steps—especially for question answerability—are confirmed by annotators, mitigating annotation noise and enforcing robust testing of procedural reasoning.
- **Distractor Sampling**: In the demonstration selection task, close “hard negative” candidate videos are sampled to challenge discriminative procedural transfer.

## 6. Implications for Procedural In-Context Learning

Demo-ICL-Bench provides a test suite that surfaces several properties of current MLLMs:

- **Procedural Generalization**: Most models exhibit only modest benefit from procedural demonstrations, with clear performance gaps between open-source and proprietary systems, and substantial headroom to human-level performance.
- **Demonstration Modality Effects**: Text-based demonstrations yield consistently higher $\Delta_{ICL}$, suggesting that alignment and abstraction of procedural steps in text form are more tractable for current models than direct video-to-video transfer.
- **Granularity and Temporal Reasoning**: Adding temporally aligned ASR information substantially improves video-demo ICL, revealing that models benefit from explicit semantic–temporal cues.
- **Model and Training Scaling**: Larger context lengths, dense frame sampling, and deep video-text pretraining are essential but not sufficient; targeted procedural ICL training and specialized direct preference optimization yield additional but incremental improvements.

A plausible implication is that progress on ICL in procedural video domains will depend upon improved cross-modal grounding, richer alignment of temporally structured demonstration content, and the development of architectures tailored for multi-step, multi-modal sequence induction.

## 7. Broader Context and Future Directions

Demo-ICL-Bench sets a new standard for evaluating demo-driven, few-shot learning in procedural video understanding. It complements existing benchmarks such as VL-ICL Bench, which focuses on image-text ICL for perception and reasoning [2403.13164], by directly addressing the challenge of dynamic procedural transfer in temporally extended domains. The benchmark’s methodology—joint curation of textual and video procedural demonstrations, rigorous alignment, and hard distractor-based selection—serves as a template for future multimodal ICL evaluation.

Potential future research directions suggested by this work include the integration of multimodal chain-of-thought, expanding demonstration selection granularity, scaling to longer activity chains, and refining architectures for segment-level temporal compositionality.

Demo-ICL-Bench, together with the corresponding Demo-ICL model, advances the field by enabling systematic, fine-grained measurement of procedural in-context learning abilities for the next generation of video-capable MLLMs [2602.08439].

Source: https://www.emergentmind.com/topics/demo-icl-bench