Video In-Context Learning

Updated 6 July 2026

Video ICL is a technique that uses contextual video demonstrations to guide inference-time adaptation without updating model weights.
It leverages paradigms such as demonstration-conditioned continuation and retrieval-based adaptation to achieve tasks like zero-shot imitation and procedural knowledge transfer.
Key mechanisms include selective retrieval, token compression, and iterative context presentation, enabling efficient domain adaptation and multimodal video editing.

Video in-context learning (Video ICL) denotes inference-time adaptation in which a model conditions on video demonstrations or other video-centered context to perform a task without task-specific weight updates. In the recent literature, the term covers several partially overlapping settings: zero-shot video imitation from demonstration clips, retrieval-based adaptation of frozen video MLLMs to out-of-distribution domains, demo-driven procedural knowledge transfer, and broader context-conditioned editing or generation in unified multimodal transformers. The common principle is that behavior is induced by contextualized video or multimodal examples rather than by conventional supervised retraining (Zhang et al., 2024, Kim et al., 2024, Dong et al., 9 Feb 2026, Ju et al., 24 Sep 2025).

1. Definitions and scope

A strict formulation appears in "Video In-context Learning: Autoregressive Transformers are Zero-Shot Video Imitators" (Zhang et al., 2024). There, the demonstration is a set of clips

$D^V = \{(s_i^1, \cdots, s_i^{n_i})\}_{i=1}^{k},$

the query is a partial video clip

$x^V = (s_q^1, \cdots, s_q^{n_q}),$

and the model generates a future clip

$y^V = (s_y^1, \cdots, s_y^{n_y})$

according to

$f_{\theta}(x^V) = P(y^V \mid x^V, D^V).$

The defining requirement is dual: the continuation should be perceptually coherent with the query scene while being semantically consistent with the demonstration. In this usage, Video ICL is a video-native analogue of in-context learning in LLMs, and the model is expected to infer what kind of future to generate from the demonstration itself rather than from explicit labels or action sequences (Zhang et al., 2024).

A second formulation emphasizes knowledge transfer rather than generation. "Demo-ICL: In-Context Learning for Procedural Video Knowledge Acquisition" (Dong et al., 9 Feb 2026) defines demo-driven Video ICL through a target instructional video $V_{Test}$ , either a text demonstration $T_D$ or a video demonstration $V_D$ , and a question whose answer depends on transferring procedural knowledge from the demonstration. The paper describes the task as using $V_D$ or $T_D$ to understand how a task should be completed and then, given a partial test video $V_{Test}[0,t_1]$ , predicting a subsequent action $x^V = (s_q^1, \cdots, s_q^{n_q}),$ 0. This shifts the emphasis from contextual evidence to contextual learning: the demonstration is meant to teach a procedure that is not fully recoverable from the target clip alone (Dong et al., 9 Feb 2026).

A third formulation treats Video ICL as test-time adaptation of frozen video LMMs under severe context and annotation constraints. "VideoICL: Confidence-based Iterative In-context Learning for Out-of-Distribution Video Understanding" (Kim et al., 2024) uses labeled video-text demonstrations for OOD video understanding, while "VIOLA: Towards Video In-Context Learning with Minimal Annotations" (Fujii et al., 22 Jan 2026) reframes Video ICL as label-efficient adaptation to a new target video domain with an unlabeled pool $x^V = (s_q^1, \cdots, s_q^{n_q}),$ 1, a strict annotation budget $x^V = (s_q^1, \cdots, s_q^{n_q}),$ 2, and no fine-tuning of the MLLM. In these settings, the problem is not only how to use demonstrations, but how to acquire, filter, and retrieve them when video context is expensive and fully labeled support pools are unrealistic (Kim et al., 2024, Fujii et al., 22 Jan 2026).

The literature also contains broader, architecture-centric uses of the term. EditVerse represents text, images, and videos as one interleaved token sequence, with one visual segment noised and reconstructed from the surrounding clean multimodal context, and treats this full-sequence conditioning as an in-context learning mechanism (Ju et al., 24 Sep 2025). MovieSeq, by contrast, is best understood as multimodal video context modeling rather than direct Video ICL: it serializes videos together with plots, subtitles, images, names, and history into interleaved prompts, but its context is primarily narrative evidence rather than few-shot demonstrations (Lin et al., 2024). This terminological spread is central to the field.

2. Major paradigms

Across the cited works, Video ICL is instantiated through several distinct paradigms. They share the use of contextualized video or multimodal examples, but differ in whether the target is future-frame generation, question answering, domain adaptation, editing, or dense prediction.

Paradigm	Representative papers	Core arrangement
Demonstration-conditioned continuation	(Zhang et al., 2024)	Demo clip(s) + query prefix $x^V = (s_q^1, \cdots, s_q^{n_q}),$ 3 future video
Retrieval-based frozen-MLLM adaptation	(Kim et al., 2024, Fujii et al., 22 Jan 2026)	Retrieved demonstrations + query $x^V = (s_q^1, \cdots, s_q^{n_q}),$ 4 answer, class, or caption
Demo-driven procedural transfer	(Dong et al., 9 Feb 2026)	Text/video demonstration + target instructional video + question
Unified editing/generation via context	(Ju et al., 24 Sep 2025, Fei et al., 2024, Shao et al., 6 May 2026)	Source/context segments + noised or masked target
Memory-based dense prediction	(Foster et al., 2023, Wahd et al., 21 Jun 2025)	Support image-mask pairs or keyframes as pseudo-video memory

In the autoregressive continuation paradigm, Video ICL emerges from generic causal video-token modeling. Vid-ICL converts videos into discrete visual tokens with a pretrained vector-quantized tokenizer, trains a decoder-only LLaMA-style transformer with next-token prediction, and at inference concatenates demonstration and query into a single causal context. In the main experiments, the model uses one demonstration video clip of 8 frames, a query prefix of 4 frames, and a generated continuation of 4 frames within a 16-frame, 4096-token context. The authors describe the resulting behavior as zero-shot video imitation because the model is not trained on explicit demonstration-query-response tuples for the target task (Zhang et al., 2024).

In retrieval-based frozen-MLLM adaptation, the central object is a demonstration pool rather than a generative prefix. Standard ICL is written in VIOLA as retrieving the top- $x^V = (s_q^1, \cdots, s_q^{n_q}),$ 5 demonstrations most similar to a test query from a fully annotated pool $x^V = (s_q^1, \cdots, s_q^{n_q}),$ 6, forming a context $x^V = (s_q^1, \cdots, s_q^{n_q}),$ 7, and predicting $x^V = (s_q^1, \cdots, s_q^{n_q}),$ 8. VIOLA rejects the assumption that $x^V = (s_q^1, \cdots, s_q^{n_q}),$ 9 is fully labeled and instead selects only $y^V = (s_y^1, \cdots, s_y^{n_y})$ 0 target-domain videos for expert annotation, pseudo-labels part of the remainder, merges them into a hybrid pool, and performs confidence-aware retrieval and prompting. VideoICL uses a different solution to the same bottleneck: it retrieves a ranked list of relevant examples, exposes only $y^V = (s_y^1, \cdots, s_y^{n_y})$ 1 at a time because videos are token-expensive, and iterates if the model’s confidence is low (Fujii et al., 22 Jan 2026, Kim et al., 2024).

Demo-driven procedural transfer occupies a different point in the design space. Demo-ICL-Bench contains three subsettings—text-demo ICL, video-demo ICL, and demonstration selection—and asks whether a model can learn procedural knowledge from demonstrations and transfer it to a new partial instructional video. The demonstrations can themselves be videos, and in the selection setting the model must first choose the relevant demonstration from a candidate pool. This makes retrieval and transfer jointly part of the task definition rather than mere prompt engineering (Dong et al., 9 Feb 2026).

Unified editing and generation works employ a broader notion of context. EditVerse serializes text, image, and video segments into one multimodal sequence, uses full self-attention over the entire sequence, and trains a flow-matching objective in which one image or video segment is noised and reconstructed from the others. "Video Diffusion Transformers are In-Context Learners" instead activates in-context generation by concatenating several related videos spatially or temporally into one composite video, jointly captioning the resulting multi-scene clip, and fine-tuning only a LoRA on a small curated dataset. "Lightning Unified Video Editing via In-Context Sparse Attention" then treats unified video editing itself as an ICL problem over source and context tokens, and optimizes the resulting long-sequence attention pattern rather than redefining the task (Ju et al., 24 Sep 2025, Fei et al., 2024, Shao et al., 6 May 2026).

Finally, segmentation papers reinterpret video memory as in-context learning. "Flexible visual prompts for in-context learning in computer vision" adapts XMem, a video object segmentation system, so that support image-mask pairs become a pseudo-video memory used to segment a query image. "Time-Contrastive Pretraining for In-Context Image and Video Segmentation" generalizes the same idea with a self-supervised prompt retriever and a SAM2-based VOS backbone, turning image and video segmentation into sequence-conditioned prediction where support examples are effectively earlier labeled frames (Foster et al., 2023, Wahd et al., 21 Jun 2025).

3. Core mechanisms

The most mature Video ICL systems are defined less by a single architecture than by a collection of mechanisms for selecting, compressing, and presenting context. Retrieval quality is a recurrent bottleneck. VideoICL ranks candidate demonstrations with a multimodal similarity

$y^V = (s_y^1, \cdots, s_y^{n_y})$ 2

using SentenceBERT for text and InternVideo2 for video, retrieves $y^V = (s_y^1, \cdots, s_y^{n_y})$ 3 examples, and consumes them in batches of $y^V = (s_y^1, \cdots, s_y^{n_y})$ 4. Confidence is computed from token probabilities as $y^V = (s_y^1, \cdots, s_y^{n_y})$ 5; if confidence falls below threshold, the model advances to the next batch and finally returns the answer with highest confidence across iterations. The same paper motivates this procedure with a concrete token-budget example: in LLaVA-Video, a 30-second $y^V = (s_y^1, \cdots, s_y^{n_y})$ 6 video sampled into 32 frames becomes about 5.5K tokens after visual encoding, so only about four videos total fit in a 32K context window (Kim et al., 2024).

VIOLA addresses the retrieval problem earlier in the pipeline, at demonstration acquisition time. It proposes density-uncertainty-weighted sampling under a strict annotation budget, pseudo-labels remaining samples with the frozen MLLM conditioned on trusted demonstrations, filters pseudo-labels by a 95th-percentile confidence threshold, and performs test-time retrieval with a similarity-confidence composite

$y^V = (s_y^1, \cdots, s_y^{n_y})$ 7

where $y^V = (s_y^1, \cdots, s_y^{n_y})$ 8 for expert-labeled samples. Prompting is also confidence-aware: human labels are phrased as ground truth, pseudo-labels are marked as predictions with numeric confidence, and pseudo-labeled demonstrations are placed at the beginning of the context, followed by labeled demonstrations. In the main experiments, “minimal annotations” means 20 labeled samples per dataset, with budget studies from 20 to 100 (Fujii et al., 22 Jan 2026).

Sequence construction is equally variable. Vid-ICL relies on the simplicity of causal token concatenation: no special tokens are inserted between frames, the model is trained on contiguous 16-frame clips, and test-time demonstration/query formatting is imposed only at inference. EditVerse uses a different sequence logic: text and vision are separately projected into a shared hidden dimension, visual segments are bounded by learnable “start of vision” and “end of vision” tokens, and a 4D RoPE allocates dimensions to height, width, sequential position, and temporal index. The target image or video segment is replaced by a noisy version and denoised from the remaining multimodal context with Flow Matching rather than autoregressive prediction (Zhang et al., 2024, Ju et al., 24 Sep 2025).

Several papers replace flat prompting with explicit memory. XMem-based visual prompting stores support image-mask pairs as memory entries and retrieves them pixelwise for query segmentation, effectively turning the support set into a pseudo-video memory. Temporal extends this pattern by pretraining a prompt retriever on unlabeled videos using a time-contrastive objective in which adjacent frames are positives and distant frames are negatives, then using the retrieved contexts inside a SAM2 VOS backbone. For video editing, LIVEditor observes that source tokens and context tokens play asymmetric roles and introduces In-context Sparse Attention: context Key/Value blocks are pre-pruned, and query blocks are routed either to full attention or to a 0-th order Taylor sparse attention branch according to a sharpness criterion that the paper formally relates to approximation error (Foster et al., 2023, Wahd et al., 21 Jun 2025, Shao et al., 6 May 2026).

A different compression strategy appears in SOP generation. The WONDERBREAD-based paper first uses standard Video ICL to produce multiple pseudo-label SOPs from different 8-shot batches, then feeds the test video together with those pseudo-labels into a second-stage VLM. The paper argues that this In-Context Ensemble Learning lets GPT-4o-mini “successfully see the equivalent of 24 videos as training examples,” even though direct prompting had supported only up to 8 high-resolution videos in early exploration (Xu et al., 2024).

4. Evaluation regimes and empirical evidence

Evaluation protocols differ sharply across the literature because the task families differ sharply. Vid-ICL separates visual quality from semantic control using PSNR, LPIPS, FID, classifier-based Video Accuracy (V-Acc), and Probing Accuracy (P-Acc), and supplements these with human evaluation of quality, alignment, and control on contrastive Something-Something v2 prompts (Zhang et al., 2024). VIOLA evaluates classification with accuracy and captioning with ROUGE-L across nine specialized datasets, while Demo-ICL reports demonstration accuracy gains, selection accuracy, and final answer accuracy on its benchmark of instructional videos. Segmentation work reports Dice or mIoU. EditVerseBench uses six metrics across four aspects, including VLM evaluation of editing quality, PickScore, CLIP or ViCLIP alignment, and temporal consistency. These metric choices reflect a general field-wide distinction between perceptual fidelity and successful use of context (Fujii et al., 22 Jan 2026, Dong et al., 9 Feb 2026, Wahd et al., 21 Jun 2025, Ju et al., 24 Sep 2025).

The benchmark landscape is correspondingly heterogeneous. Vid-ICL evaluates semantic imitation on SSv2 because its action semantics are especially contrastive (Zhang et al., 2024). VIOLA spans seven classification datasets—Drive&Act, EgoPet, EgoSurgery, ENIGMA, UCF-Crime, Xsports, and MammAlps—and two captioning datasets, Bora and CapERA, with no explicit QA benchmark in its reported experiments (Fujii et al., 22 Jan 2026). Demo-ICL-Bench is built from 1,200 instructional YouTube videos and is manually curated to 500 text-demo ICL questions, 500 video-demo ICL questions, and 200 demonstration selection questions (Dong et al., 9 Feb 2026). EditVerseBench contains 100 videos, split into 50 horizontal and 50 vertical, each paired with 2 editing prompts for 200 editing pairs spanning 20 distinct editing categories (Ju et al., 24 Sep 2025). WONDERBREAD’s “Gold Demo” subset contributes 724 workflow videos, split into 217 training videos and 507 testing videos, for SOP generation under a deliberately hard setting that excludes intent, action traces, and click coordinates (Xu et al., 2024).

Several results have become reference points. Vid-ICL reports that, for the 1.1B pretrain model, in-class demonstrations raise V-Acc and P-Acc from 22.9 and 29.6 with no demonstration to 24.7 and 36.7, while random demonstrations slightly harm semantics relative to no demonstration. The same paper reports monotonic improvement as model size increases from 300M to 700M to 1.1B, and interprets the emergence of semantic imitation as gradual rather than abrupt (Zhang et al., 2024). VIOLA reports strong low-budget gains with only 20 labeled samples per dataset; for Qwen2-VL, relative to zero-shot it improves Drive&Act by +19.2, EgoPet by +38.2, ENIGMA by +53.6, and CapERA by +0.119, while also showing that naive pseudo-labeling can be harmful—for example, Random+Pseudo Annotation on Xsports drops to 8.0, far below Random at 21.5 (Fujii et al., 22 Jan 2026).

Demo-ICL provides a complementary result: many current MLLMs benefit from text demonstrations but not reliably from video demonstrations. In its main table, Qwen2-VL 7B shows a text-demo gain of +7.2 but a video-demo gain of $y^V = (s_y^1, \cdots, s_y^{n_y})$ 9, Ola 7B shows +9.2 and $f_{\theta}(x^V) = P(y^V \mid x^V, D^V).$ 0, and Qwen2.5-VL 72B shows +20.8 and only +0.4. Demo-ICL 7B improves over its base Ola-Video 7B from 31.4 to 43.4 on text-demo ICL, from 25.0 to 32.0 on video-demo ICL, and from 18.0 to 24.0 on demonstration selection, but remains far below human performance, whose average is 80.1 (Dong et al., 9 Feb 2026). For dense prediction, Temporal reports 90.95% Dice for image segmentation and 92.45% Dice for video segmentation on MICCAI FLARE 2022, with the latter described as a 14.88% improvement over baselines (Wahd et al., 21 Jun 2025). In low-level workflow understanding, In-Context Ensemble Learning raises GPT-4o-mini from 42.62/78.13/32.93 zero-shot to 44.34/84.79/37.17 on Precision/Recall/Temporal Order, and raises Gemini-1.5-flash from 34.35/45.16/27.82 to 40.77/54.38/35.89 (Xu et al., 2024).

The editing literature adds a different type of evidence. EditVerse reports nontrivial video editing performance even without any video editing dataset—Editing Quality 3.62, Frame/text alignment 22.31, Video/text alignment 20.44, and Temporal consistency CLIP/DINO 93.48/90.27—which the paper treats as evidence of cross-modal transfer through unified context learning (Ju et al., 24 Sep 2025). LIVEditor then shows that the ICL formulation can itself become the optimization target: In-context Sparse Attention reduces attention-module latency by about 60% and yields a 1.47× speedup over full attention or FlashAttention-3 under default settings while slightly improving EditVerseBench scores over its own full-attention counterpart (Shao et al., 6 May 2026).

5. Interpretive disputes and recurring misconceptions

One persistent dispute concerns what should count as “in-context learning” in video. Under a strict reading, the context should function as demonstrations from which the model infers a task or procedural mapping at inference time, as in Vid-ICL and Demo-ICL (Zhang et al., 2024, Dong et al., 9 Feb 2026). Under a broader reading, any unified context-conditioned multimodal transformer may be described as in-context if it infers the appropriate transformation from a shared multimodal sequence, which is how EditVerse uses the term. EditVerse itself acknowledges that this is not GPT-style few-shot prompting with multiple labeled demonstrations and no explicit supervision, but rather a full-attention diffusion transformer whose sequence design enables context-based task inference and cross-task transfer (Ju et al., 24 Sep 2025). MovieSeq is even more explicit evidence that “context” and “ICL” can diverge: its interleaved multimodal sequences are highly relevant to prompt design, yet the paper is best read as narrative video context modeling rather than direct Video ICL (Lin et al., 2024).

A second misconception is that few-shot video QA or captioning automatically measure Video ICL. VL-ICL Bench argues, for multimodal ICL more broadly, that popular few-shot VQA and captioning evaluations often test output formatting or answer style rather than induction of a new mapping. Its image-based results show that exact-match evaluation can fabricate apparent ICL gains, that text variants of induction tasks scale much better with shots than image variants, and that even strong models often collapse when context becomes longer or more interleaved. The paper’s core lesson for video is methodological: a real Video ICL benchmark should test whether demonstrations teach a task, not merely whether they condition a familiar task (Zong et al., 2024).

A third misconception is that more context is always beneficial. The Video ICL literature repeatedly shows the opposite. Vid-ICL reports that random demonstrations can slightly harm semantics relative to no demonstration (Zhang et al., 2024). VIOLA emphasizes that in few-shot video prompting a single bad demonstration can strongly degrade prediction, especially if it is visually similar but mislabeled, and its ablations show why density-only or uncertainty-only annotation strategies are inferior to balancing both (Fujii et al., 22 Jan 2026). VideoICL and ICE both arise from the same negative observation: because video tokens are expensive, simply concatenating more demonstrations can be brittle or impossible, whereas retrieval plus iteration or pseudo-label compression can use a larger support pool without placing all exemplars in one prompt (Kim et al., 2024, Xu et al., 2024).

A fourth misconception is that video demonstrations are necessarily a stronger in-context substrate than text demonstrations. Demo-ICL’s results directly contradict that intuition: many models show positive gains from text demonstrations but weak, near-zero, or negative gains from video demonstrations. The paper attributes this asymmetry to the need for perception, temporal alignment, abstraction, and flexible transfer all at once, and its ablations show that replacing video demonstrations with ASR captions helps the most (Dong et al., 9 Feb 2026). This suggests that “video-native” ICL remains aspirational in many settings.

6. Constraints and likely directions

Several limitations recur across otherwise dissimilar papers. Vid-ICL is demonstrated on short continuations and does not establish long-horizon imitation; its tokenizer compresses spatially but not temporally, which constrains context length (Zhang et al., 2024). VIOLA depends on pretrained visual embeddings for both GMM-based selection and retrieval, and explicitly notes that severe domain shift can distort semantic density estimates and nearest-neighbor similarity; it also assumes an unlabeled target-domain pool is available before deployment (Fujii et al., 22 Jan 2026). Demo-ICL states that it introduces no specialized architecture for demo-driven video ICL, that video-demo ICL remains far below text-demo ICL, and that demonstration selection is difficult even when the correct demonstration is chosen (Dong et al., 9 Feb 2026). EditVerse notes that full self-attention over long multimodal sequences is computationally expensive and that its dataset is noisy, with average instruction length around 10 words and estimated automated edit success rate around 65% (Ju et al., 24 Sep 2025).

Computation is therefore not incidental. VideoICL pays extra latency for multiple inference rounds, though early stopping mitigates some of that cost (Kim et al., 2024). LIVEditor treats the resulting source-plus-context quadratic attention as the central systems problem of modern unified video editing and argues that Video ICL should exploit role asymmetry between source and context tokens rather than applying uniform full attention. Its ISA theorem gives an upper bound

$f_{\theta}(x^V) = P(y^V \mid x^V, D^V).$ 1

which the paper uses to justify sharpness-aware routing between full and approximate attention (Shao et al., 6 May 2026). This does not solve the entire long-context problem, but it shows that the structure of Video ICL can itself guide acceleration.

Benchmark design remains an open systems issue as much as a measurement issue. VL-ICL Bench’s critique of format-sensitive evaluation, Demo-ICL-Bench’s emphasis on procedural transfer rather than static knowledge, and VIOLA’s focus on low-resource target-domain adaptation all imply that future Video ICL evaluation will need to report not only task performance but also support-pool construction, retrieval policy, context budget, visual token count, and reliability modeling (Zong et al., 2024, Dong et al., 9 Feb 2026, Fujii et al., 22 Jan 2026). A plausible implication is that mature Video ICL systems will need to combine retrieval, compression, calibration, and architecture-aware sequence design rather than relying on any single prompting trick. The field has already produced workable exemplars of each component, but it has not yet converged on a unified formulation.