---
title: Demo-Driven Video In-Context Learning
url: https://www.emergentmind.com/topics/demo-driven-video-in-context-learning-cae8a156-fdb9-4cce-82a8-7976bd64173b
type: topic
---

# Demo-Driven Video In-Context Learning

Demo-driven video in-context learning (VICL) refers to the class of methods that enable models to acquire, transfer, or generalize task knowledge from explicit demonstration examples—most critically, demonstration videos or video+text pairs—provided at inference time, rather than relying solely on parameters learned offline. This paradigm leverages the emergence of in-context learning abilities in large multimodal models, bridging the gap between static pre-trained knowledge and rapid adaptation to novel, often low-resource, domains.

## 1. Formalization and Motivating Tasks

In demo-driven VICL, the model receives as input a query (video segment and/or textual prompt) and a small set of demonstration exemplars, each itself a video or aligned sequence (possibly with associated text, actions, or labels). The task is to conditionally complete, classify, narrate, generate, or otherwise reason about the query by leveraging information distilled from the demos.

Different works instantiate the paradigm with formal objectives reflecting distinct capabilities. For example, Demo-ICL frames procedural acquisition as learning $P(A \mid V_{\text{test}}[0:t_1], Q, D; \theta)$, where $D = \{D_1, \ldots, D_k\}$ are video or text demonstrations, and $A$ is the desired output (e.g., what happens next in $V_{\text{test}}$) [2602.08439]. By contrast, VIOLA expresses VICL as prediction over $\mathcal{M}(x_{\text{test}}, \mathcal{C})$, with $\mathcal{C}$ a retrieved demo context, further emphasizing few-shot label efficiency [2601.15549].

Canonical tasks include:

- Few-shot video narration and procedural reasoning [2311.17041, 2602.08439]
- Video classification, captioning, or QA in new domains [2412.02186, 2601.15549]
- Computer-use/action trajectory imitation for agentic UI control [2511.04137, 2510.04673]
- Video-to-video or image-to-video generation via demo-driven control [2403.11535, 2412.09551, 2412.10783, 2407.07356]

This generality reflects the core motivation: to reduce annotation and re-training costs for specialized, rare, or rapidly-changing video domains, enabling models to adapt or generalize with only a handful of user-provided exemplars.

## 2. Key Methodological Elements in VICL Pipelines

While instantiations vary, most demo-driven VICL pipelines adhere to the following stages:

1. **Demonstration Pool Construction**: Assemble a candidate set of demos from various sources (curated video banks [2602.08439], online tutorial videos [2510.04673, 2511.04137], instructional datasets, or user-supplied clips).

2. **Demo Selection and Ranking**: Employ similarity-based retrieval using joint video-text embedding spaces—e.g., cosine similarity of video and/or text features, optionally with learned weighting [2412.02186, 2601.15549]. Confidence or label-reliability signals may influence selection [2601.15549, 2412.02186].

3. **Prompt Formatting and Context Construction**: Concatenate or otherwise interleave demonstrations and queries into a model-compatible prompt (video tokens and textual queries/answers) [2602.08439, 2407.07356, 2311.17041]. Modalities may be indicated via positional or special tokens.

4. **Model Conditioning and Inference**: Inject the demonstration context into the model, utilizing cross-attention, context tokens, or diffusion-based implicit conditioning (e.g., action latents in $\delta$-Diffusion [2412.09551], LoRA-based adapters [2412.10783], or action prism features [2403.11535]). The model produces outputs conditioned on both the demos and query.

5. **Iterative/Ensemble Refinement (as needed)**: For context windows too small for all demos, iterative retrieval and confidence-based refinement can extend effective demonstration count [2412.02186]. Ensemble schemes aggregate multiple pseudo-labels from ICL batches for consensus [2409.15867].

Distinct innovations include density-uncertainty-weighted sampling for annotation efficiency [2601.15549], hybrid pools of labeled and pseudo-labeled data with confidence modeling [2601.15549], preference-based optimization for demo utilization [2602.08439], and direct video trajectory extraction from web tutorials [2510.04673, 2511.04137].

## 3. Architectural and Training Considerations

VICL approaches are realized atop a range of architectures:

- **Autoregressive Transformers** trained on video token sequences (e.g., VQ-GAN compressed) support pure demonstration prefixing for zero-shot video imitation [2407.07356].
- **Encoder–Decoder Multimodal LLMs** combine frozen (or fine-tuned) visual frontends and large LMs with cross-modal attention or interleaved embedding fusion [2602.08439, 2311.17041, 2601.15549].
- **Diffusion-based models** condition sample generation on learned latent representations distilled from reference demos (e.g., action prism tokens [2403.11535], implicit action latents [2412.09551], panel-wise spatiotemporal blocks [2412.10783]).
- **UI/Workflow Agents** use action-labeled demonstration trajectories, sometimes segmented by VLM or LLM analysis, and inject both image and action sequence tokens into the context [2510.04673, 2511.04137].

Training strategies span pure self-supervision (for imitation ability to emerge; [2407.07356]), distribution-centric data curation (to elicit robust ICL abilities; [2311.17041]), in-context fine-tuning on demo-augmented corpora (e.g., SFT with hybrid demo-injected examples; [2602.08439]), preference optimization with demonstration-aware rewards [2602.08439], and LoRA-based lightweight adaptation to unlock cross-demo or in-context generation in massive models [2412.10783].

## 4. Evaluation Benchmarks and Quantitative Results

Research on demo-driven VICL proposes specialized benchmarks such as Demo-ICL-Bench, constructed from HowTo100M with rigorous annotation of stepwise instruction sequences and aligned video demonstrations [2602.08439]. Secondary evaluation is performed on SOP generation sets (e.g., WONDERBREAD “Gold Demo” [2409.15867]), UI agent tasks (OSWorld [2510.04673, 2511.04137]), and multiple domain-specific video-action/caption datasets (e.g., DriveAct, EgoSurgery, UCF-Crime, CapERA; [2601.15549, 2412.02186]).

Quantitative outcomes widely confirm the value of demonstration-driven context:
- VIOLA benchmarks demonstrate 19–54 point accuracy gains over zero-shot with tightly budgeted $B=20$ expert labels, and persistent advantage as $B$ scales [2601.15549].
- Demo-ICL yields 14.0% $\Delta_{\text{ICL}}$ for text-demo and 4.4% for video-demo tasks compared to demo-free baselines, with best open-source models below 30% on these challenging few-shot splits [2602.08439].
- In SOP generation, in-context ensemble aggregation (ICE) improves recall (+6.7%), precision (+1.7%), and time-ordering accuracy (+4.3%) over 8-shot ICL [2409.15867].
- Confidence-based iterative ICL (VideoICL) raises OOD classification by up to 33.2 percentage points relative to zero-shot [2412.02186].
- In controllable video generation, in-context concatenation and LoRA tuning support persistent role control, style transfer, and multi-scene coherence [2412.10783, 2412.09551].
- For UI agents, demo-driven in-context trajectories consistently lift success rate by 2–4% over text-based or frame-only baselines [2511.04137, 2510.04673].

## 5. Distinguishing Features and Innovations

Major technical advancements over classical fine-tuning or zero-shot approaches include:
- **Label-Efficient Demo Selection**: Density–uncertainty balancing (GMM + tokenwise entropy) for optimal expert annotation under strict budget [2601.15549].
- **Hybrid Labeled/Pseudo-Labeled Pools**: Confidence-aware retrieval and prompting to integrate uncertain pseudo-labels alongside ground-truth, and explicitly communicate reliability in prompt construction [2601.15549].
- **Iterative and Ensemble ICL**: Chunkwise demo selection with confidence-based early stopping (effective context extension) [2412.02186]; pseudo-label proposal/voting to realize context-efficient ensemble ICL [2409.15867].
- **Distributional Data Curation for Emergent ICL**: Training on “bursty,” skewed, and synonym-rich data distributions enables true few-shot transfer and semantic adaptation in VLMs [2311.17041].
- **Video-Driven Agentic Reasoning**: Pipelines for online demonstration scraping, trajectory segmentation, in-context injection at every agentic timestep, and adaptive demo selection [2511.04137, 2510.04673].

## 6. Limitations and Outstanding Challenges

Noted constraints across sources:

- **Demo Acquisition and Alignment**: Robustness depends on the quality and domain coverage of demonstration pools. Procedural alignment between demonstrations and query tasks remains challenging, especially for video-based demos [2602.08439].
- **Context Length and Compute**: Long video and text sequences stress context windows; ensemble and chunkwise strategies partially alleviate but do not abolish this bottleneck [2412.02186, 2409.15867].
- **Semantic Parsing and Transfer**: Especially in video-demo ICL, current MLLMs often fail to abstract fine-grained motion or transfer procedural semantics as effectively as with explicit textual steps [2602.08439].
- **Evaluation**: Some generation tasks lack standardized quantitative metrics (e.g., in-context video synthesis; [2412.10783]), or the best available metrics (e.g., FVD, CLIP similarity) may incompletely capture procedural alignment.
- **Hallucination and Consistency**: For multi-step outputs (SOPs, UI actions), models can hallucinate or reorder actions, and may default to stylistic templates rather than demo-specific reasoning [2409.15867].

## 7. Future Directions

The literature suggests several promising research avenues:
- **Architectural Innovations**: Dynamic demonstration memory, cross-modal indexing, and meta-RAG modules to enhance demo retrieval and usage efficiency [2602.08439].
- **Joint Multi-modal In-Context Learning**: Direct fusion of video, text, speech, diagrams, and tool outputs in a unified prompt to support realistic multimodal workflows [2602.08439].
- **Adaptive and Feedback-driven Prompting**: Learning to select, order, and weight demonstrations in-context, with user or agent feedback [2601.15549, 2412.02186].
- **Scaling and Open Resource**: Leveraging massively increased demonstration banks, learned domain adaptation priors, and persistent memory for cross-session VICL [2311.17041, 2510.04673].
- **Task-specific Extensions**: Beyond immediate imitation, applications may include meta-learning (learning to learn from demonstrations), long-horizon multi-step planning, and robust procedural transfer in real-world domains (e.g., surgery, manufacturing, UI manipulation).

## References

- VIOLA: Towards Video In-Context Learning with Minimal Annotations [2601.15549]
- Demo-ICL: In-Context Learning for Procedural Video Knowledge Acquisition [2602.08439]
- VideoICL: Confidence-based Iterative In-context Learning for Out-of-Distribution Video Understanding [2412.02186]
- Eliciting In-Context Learning in Vision-Language Models for Videos Through Curated Data Distributional Properties [2311.17041]
- Learning from Online Videos at Inference Time for Computer-Use Agents [2511.04137]
- Watch and Learn: Learning to Use Computers from Online Videos [2510.04673]
- Video Creation by Demonstration [2412.09551]
- Video Diffusion Transformers are In-Context Learners [2412.10783]
- Video In-context Learning: Autoregressive Transformers are Zero-Shot Video Imitators [2407.07356]
- In-Context Ensemble Learning from Pseudo Labels Improves Video-Language Models for Low-Level Workflow Understanding [2409.15867]
- AICL: Action In-Context Learning for Video Diffusion Model [2403.11535]

Source: https://www.emergentmind.com/topics/demo-driven-video-in-context-learning-cae8a156-fdb9-4cce-82a8-7976bd64173b