VidCtx: Training-Free Context-Aware VideoQA

• VidCtx is a training-free, context-aware VideoQA framework that integrates direct visual evidence with complementary distant, question-aware captions.
• It processes individual video frames independently, pairing each with a complementary caption and aggregating predictions via normalized L1 max pooling for efficient inference.
• Empirical results demonstrate competitive zero-shot performance on benchmarks like NExT-QA, IntentQA, and STAR, outperforming several open-model baselines.

VidCtx is a training-free, context-aware framework for multiple-choice Video Question Answering (VideoQA) that uses an image-conditioned Large Multimodal Model (LMM) to combine direct visual evidence from sampled frames with question-aware textual descriptions from distant frames. It was introduced to address two limitations of recent LMM/LLM-based video pipelines: computational and memory constraints caused by long-context attention over caption streams, and the loss of visual grounding in caption-only reasoning. VidCtx processes frames independently, pairs each frame with complementary context from a distant point in the video, and aggregates frame-level decisions with normalized max pooling, yielding competitive zero-shot performance on NExT-QA, IntentQA, and STAR among open-model approaches (Goulas et al., 2024).

1. Problem setting and design rationale

VidCtx is positioned against two common VideoQA strategies. The first relies on video-LLMs that combine a visual encoder with an LLM but remain constrained by input token limits and the number of frames processed per pass. The second converts videos into caption streams and asks an LLM to reason over text alone, which alleviates frame limits but removes direct access to visual evidence and often introduces repetitive descriptions of adjacent frames. VidCtx is designed to preserve visual grounding while avoiding long textual prompts.

The framework addresses these trade-offs through three explicit choices. First, it keeps the visual modality in the loop by feeding a frame image directly to an LMM at answer time. Second, it enriches that frame with question-aware captions from distant frames, rather than from nearby frames, so that the textual context is complementary rather than redundant. Third, it avoids attention over long caption sequences, thereby reducing memory and latency and enabling linear scaling in the number of frames.

This design is also a positioning statement within open-model VideoQA. VidCtx uses the same pre-trained LMM for both question-aware caption generation and multiple-choice answering, without any additional training or fine-tuning. The paper identifies its main contributions as the joint use of visual frames and question-aware textual context in a single LMM, a simple training-free frame-based architecture with distant-caption prompts and temporal specifiers, and lightweight aggregation via normalized max pooling over frame-level option scores (Goulas et al., 2024).

2. Formal pipeline and prompting

The end-to-end workflow begins by splitting an input video $V$ into $N$ non-overlapping segments and sampling the central frame from each segment, yielding frames $V = \{v_i, i = 1, \ldots, N\}$. VidCtx then performs question-aware caption extraction with a pre-trained LMM $\phi$:

$c_i = \phi(v_i, I_{cap})$

where $I_{cap}$ is the caption instruction concatenated with the question:

“Please provide a short description of the image, giving information related to the following question: <Q>”

Captions are generated at regular frame intervals defined by the segment boundaries, with one caption per sampled frame. In the reported experiments, $N = 64$ frames for NExT-QA and IntentQA, and $N = 32$ for STAR. The authors generate up to 200 tokens per caption.

For context selection, VidCtx pairs each frame with the caption from a distant frame exactly half the video apart:

$r(i) \equiv i + \frac{N}{2} \pmod N$

This deterministic rule is intended to maximize complementary temporal context and minimize redundancy. The prompt also includes an explicit temporal specifier—“earlier” or “later”—to clarify event order.

The same LMM is then prompted for per-frame QA inference:

$d_i = \phi\big(v_i, c_{r(i)}, I_{vqa}\big)$

The QA prompt contains the current frame, the question, the distant caption, and explicit answer options:

“Here is what happens earlier/later in the video: <Caption> Question: <Q> Option A: <A> Option B: <B> Option C: <C> Option D: <D> Option E: <E> Option F: No Answer. Considering the information presented in the caption and the video frame, select the correct answer in one letter from the options (A,B,C,D,E,F).”

The option set is $N$0, where $N$1 denotes “No Answer” when a frame lacks sufficient information. VidCtx employs a 4-bit quantized LLaVA-1.6-Mistral-7B, which processes a single image per forward pass. Captioning uses 200 generated tokens per frame, while answering scores only the first token, which substantially accelerates the second pass (Goulas et al., 2024).

3. Aggregation rule and computational profile

At the frame level, VidCtx records the normalized log-probability of the first-token answer letter for each option, denoted $N$2. Video-level prediction is then produced by max pooling over frames after per-frame $N$3 normalization, excluding the “No Answer” token:

$N$4

The paper emphasizes two points about this aggregation rule. First, normalization before pooling is empirically important; $N$5 normalization slightly outperforms softmax. Second, max pooling encourages the final answer to be determined by the most confident frame-context pair rather than by diffuse evidence across many frames. Because $N$6 is excluded from the final maximization, frames judged irrelevant can be effectively ignored.

The computational profile is a central part of the method’s appeal. Each frame is processed twice: once for caption generation and once for answer scoring. Since the second pass generates only the first token, the overall inference cost remains linear in the number of frames:

$N$7

This contrasts with caption-concatenation approaches, which incur the quadratic time and memory cost of long-context attention:

$N$8

Frames are processed independently, so VidCtx is not constrained by the LLM’s maximum input length and can scale to arbitrarily many frames. The reported experiments were run on a single RTX 4090 GPU (Goulas et al., 2024).

4. Benchmarks and empirical performance

VidCtx is evaluated zero-shot on three public VideoQA benchmarks. NExT-QA contains 5,440 videos and 48K multiple-choice question-answer pairs, with average length 44 seconds; the validation set contains 570 videos and 5K questions split into Causal, Temporal, and Descriptive categories. IntentQA is derived from NExT-QA and focuses on intent reasoning, with a test set of 2.1K question-answer pairs. STAR contains 60K situated questions across Interaction, Prediction, Sequence, and Feasibility, with a validation set of 7K questions.

Benchmark	Reported performance	Split details
NExT-QA	Causal 71.7, Temporal 65.1, Descriptive 79.2, All 70.7	Val: 570 videos, 5K questions
IntentQA	67.1	Test: 2.1K QAs
STAR	Interaction 53.9, Sequence 54.3, Prediction 51.4, Feasibility 44.7, Avg 51.1	Val: 7K questions

Against open baselines, the reported margins are specific. Q-ViD (12B) achieves 66.3 on NExT-QA All, 63.6 on IntentQA, and 45.7 average on STAR; VidCtx improves these by +4.4%, +3.5%, and +5.4%, respectively. The paper also states that VidCtx surpasses SeViLA (4B) and VideoChat2 (7B, extensively pre-trained on large-scale video data) on NExT-QA categories. On STAR, however, VideoChat2 retains the lead with an average of 63.8, although it performs poorly on NExT-QA relative to VidCtx and Q-ViD. The authors additionally note that VidCtx outperforms certain GPT-based pipelines, including LLoVi with GPT-3.5 on NExT-QA by +4.4%. Statistical significance is not reported (Goulas et al., 2024).

These results matter because VidCtx is explicitly training-free and uses a single open LMM for both captioning and answering. The reported competitiveness therefore derives from prompt design, context selection, and aggregation rather than from task-specific fine-tuning.

5. Ablations and qualitative behavior

The ablation study centers on whether context is useful, what kind of context is useful, and how frame-level scores should be fused. On NExT-QA All, the reported results are as follows:

Context design	Top-1 accuracy (%)
No Context	67.9
Concat 16 Captions (Q-Aware)	68.3
Current Caption (Q-Aware)	69.9
Distant Caption (Static)	69.5
Distant Caption (Q-Aware)	70.7

These numbers support three conclusions already emphasized by the authors: context helps, distant frames provide the most useful complementary information, and question-aware captions outperform static captions. A separate comparison against a captions-only baseline on NExT-QA with 32 frames reports 67.3 for captions-only reasoning and 70.3 for VidCtx, supporting the claim that joint visual-text processing is superior to caption-only reasoning.

Scaling with frame count is monotonic in the reported setting. On NExT-QA All, accuracy rises from 63.5 with 1 frame to 66.8 with 2 frames, 68.8 with 4, 69.2 with 8, 70.1 with 16, 70.3 with 32, and 70.7 with 64. This indicates that the framework can exploit denser temporal sampling without incurring the prompt-length bottleneck that caption-stream methods face.

Aggregation strategy also has measurable effects. The paper reports 69.7 for voting, 69.3 for mean pooling, 69.2 for max pooling without normalization, 70.2 for softmax plus mean pooling, 70.6 for softmax plus max pooling, and 70.7 for $N$9 normalization plus max pooling. The reported conclusion is that normalization before pooling is crucial, and that $V = \{v_i, i = 1, \ldots, N\}$0 plus max pooling slightly outperforms the alternatives.

The qualitative analysis focuses on temporal and causal disambiguation. Earlier/later context can sharpen decisions when a single frame is ambiguous with respect to action order or cause-effect relations. The paper also notes a failure-mitigation property of the aggregation rule: even when a specific frame-context pair is misleading because the caption contradicts visible evidence, max pooling over all frames allows strong, consistent pairs to dominate the final prediction (Goulas et al., 2024).

6. Limitations, reproducibility, and relation to adjacent work

VidCtx’s limitations are closely tied to its simplicity. Its effectiveness depends on caption quality and on alignment between the current frame and the selected distant caption; distant captions can sometimes be misleading. Context selection is based on a fixed heuristic—half a video apart—rather than on learned retrieval. The LMM processes one image per pass, so multi-frame dependencies may remain underexploited. The paper explicitly suggests future improvements through smarter context selection or temporal retrieval, integration of a localizer for frame relevance, and LMMs that accept multiple images to directly encode longer temporal spans. Domain shifts that degrade caption accuracy or prompt adherence could also reduce performance.

The implementation details are intentionally lightweight. VidCtx uses 4-bit quantized LLaVA-1.6-Mistral-7B; frames sampled per dataset are 64 for NExT-QA and IntentQA and 32 for STAR; captions generate 200 tokens; answers use single-token scoring; and experiments run on a single RTX 4090 GPU. The paper also states that frame rate, spatial resolution, inference temperature, and per-video latency are not specified. A repository is listed at https://github.com/IDT-ITI/VidCtx (Goulas et al., 2024).

Within the broader literature, VidCtx occupies a distinct point in the design space. “Visual Context Window Extension: A New Perspective for Long Video Understanding” extends the visual context window of pretrained LMMs without retraining on long video datasets, and introduces progressive pooling that reduces memory usage by approximately 45% in the 256-frame setting without performance loss (Wei et al., 2024). “ViTexQA: A Multi-Frame Temporal Perception Dataset for Video Text Question Answering” addresses a different but adjacent problem—video text understanding—by constructing a benchmark in which all QA pairs demand cross-frame text fusion and by training FrameThinker with CoT-Guided Supervised Fine-Tuning and Temporally-grounded Reinforcement Learning, reporting a ROUGE-L improvement of 6.3% over SOTA baselines on ViTexQA (Guo et al., 23 Jun 2026). This suggests that training-free distant-context prompting, visual context window extension, and explicitly supervised multi-frame reasoning are complementary strategies rather than mutually exclusive ones.