Reflective Dialogue between Teacher and Solver Agents for Video Question Answering
Abstract: Various approaches have been proposed to adapt Vision-LLMs (VLMs) to specialized domains for Video Question Answering, including fine-tuning and in-context learning. However, acquiring task-specific knowledge at the inference phase from only a small labeled support set without fine-tuning remains a challenge. In this paper, we propose a method that achieves adaptation solely through inference-time context injection. Our method first constructs a Reflective Dialogue (RD) -- a multi-turn conversation between two agents, in which Teacher poses each support question and delivers correctness feedback, and Solver answers and provides visual grounding explanations (or reflections) for both correct and incorrect answers. This dialogue history is then used as context at the inference phase. Experiments on the EgoCross benchmark demonstrate that our method outperforms both a baseline zero-shot setting and a standard in-context learning approach that passes support set examples directly, achieving 3rd place in the Open-source Track of the 1st Cross-Domain EgoCross Challenge at the CVPR 2026 EgoVis Workshop, for which this paper also serves as a technical report.
Summary
- The paper introduces Reflective Dialogue, an offline Teacher–Solver framework that converts feedback, error analysis, and visual grounding into reusable context for few-shot cross-domain Video QA.
- On the EgoCross benchmark, Reflective Dialogue reached 0.489 accuracy with Qwen3-VL-4B and 0.659 with Gemini 3.1 Pro plus timestamps, generally outperforming zero-shot and standard in-context learning.
- The method approaches fine-tuning performance without parameter updates, but increases input tokens roughly 6–7×, making dialogue compression, caching, and representative-example selection important for practical deployment.
Problem setting and motivation
Adapting large Vision-LLMs (VLMs) to specialized egocentric Video QA domains — surgery, industrial assembly, extreme sports, and animal behavior — is difficult when only a small support set of answered questions is available and fine-tuning is impractical, either because the model is proprietary or because compute is unavailable. The paper addresses this few-shot cross-domain setting using the EgoCross benchmark [li_AAAI2026_egocross], which contains 798 video clips and 957 four-way multiple-choice questions across 15 subtasks in four task categories (identification, localization, prediction, counting), with 20 answered questions per domain as a support set.
Two existing adaptation strategies fall short. Few-shot in-context learning (ICL) simply lists QA pairs as context, which the authors argue is insufficient for the model to internalize domain-specific visual patterns. Self-reflection methods such as Reflexion [shinn_NeurIPS2023_reflexion] and Self-Refine [madaan_NeurIPS2023_self_refine] improve reasoning through verbalized failure analysis, but require repeated trials on each test question, making them costly and ill-suited to few-shot settings where test answers are unavailable.
Method: reflective dialogue construction
The proposed Reflective Dialogue (RD) method transfers reflection from a per-question retry loop to an offline, reusable context. It proceeds in two phases:
Construction phase. Each support-set question is first assigned a question type (either given by metadata or predicted by an LLM; EgoCross has subtask types). For each domain–type pair , a Teacher agent and a Solver agent conduct a four-turn exchange per question:
- Question turn: the Teacher presents video frames and the question.
- Answer turn: the Solver produces an answer.
- Feedback turn: if correct, the Teacher prompts for visual evidence supporting the answer; if incorrect, it reveals the correct answer and asks for both the evidence supporting it and an explanation of why the wrong answer was chosen.
- Reflection turn: the Solver verbalizes visual grounding regardless of correctness.
These exchanges are concatenated sequentially into a single continuous conversation per domain–type pair, so that contrastive analyses of mistakes accumulate within the context.
Inference phase. For each test question, its domain and type are identified, the corresponding reflective dialogue is retrieved and prepended to the question (with a separator sentence such as "Warm-up complete. Now answer the following question"), and inference proceeds independently per question. Because dialogues are static, there is no per-test-question retry overhead, distinguishing RD from online self-reflection while retaining its feedback-rich context.
A practical detail with measurable impact: EgoCross frame filenames carry frame IDs but not absolute timestamps, so for temporal question types the authors prepend computed timestamp strings (e.g., "[Frame at 1.0s]") before each frame to enable temporal localization.
Experimental results
The evaluation covers three methods — zero-shot (domain-specific system prompt only), ICL (support-set QA pairs prepended), and RD — on Qwen3-VL-4B-Instruct (open-weight) and Gemini 3.1 Pro Preview / Gemini 3.1 Flash Image Preview (proprietary), evaluated by CloseQA accuracy.
Open-weight model
With Qwen3-VL-4B-Instruct, RD outperformed zero-shot overall and in every domain:
| Method | Animal | XSports | Industry | Surgery | Overall |
|---|---|---|---|---|---|
| Zero-shot | 0.541 | 0.386 | 0.331 | 0.463 | 0.424 |
| ICL | 0.618 | 0.411 | 0.441 | 0.445 | 0.468 |
| RD | 0.590 | 0.419 | 0.457 | 0.512 | 0.489 |
RD surpassed ICL overall and in three of four domains, with the largest gain over ICL in surgery (+0.067). The authors attribute this to complex procedural domains benefiting most from verbalizing "what was missed" on incorrect answers. Conversely, ICL beat RD in the animal domain, suggesting that when visual appearance directly cues identification, verbal reflection adds little — an explicit concession that RD's benefit is domain-dependent.
Proprietary models
Gemini 3.1 Pro Preview with RD plus timestamps achieved the best results across all models and all domains:
| Model / Method | Animal | XSports | Industry | Surgery | Overall | Cost (18.07 vs.$6.09). Context caching on Gemini 3.1 Pro reduced RD costs by approximately 49% (from ~$75 equivalent to$38.27), since cached tokens accounted for 25.2M of 37.7M input tokens.
Comparison with fine-tuningTo situate the training-free approach, the authors LoRA fine-tuned Qwen3-VL-4B-Instruct (rank 64, 6 epochs, AdamW, bfloat16) on each target domain's 20-question support set via LLaMA-Factory:
FT alone reached 0.514 overall, and combining FT with RD improved further to 0.528, indicating complementarity. Notably, training-free RD approached FT's overall performance (0.489 vs. 0.514) and actually exceeded FT in surgery (0.512 vs. 0.484), so the advantage of parameter updates is itself domain-dependent. With only 20 training examples per domain, FT's gains are also likely fragile, though the paper does not probe variance across seeds. Limitations and open questionsThe principal limitation, acknowledged by the authors, is token cost: prepending reflective dialogues inflates input tokens roughly 6–7× relative to zero-shot (e.g., 37.7M vs. 11.9M for Gemini 3.1 Pro), raising latency and API expense even with caching. They propose context compression or representative-question selection as future work but do not evaluate either. Additional open points include: the reliance on LLM-predicted question types when metadata is absent (no ablation of type-prediction errors); the sensitivity of RD's benefit to domain characteristics, which is observed empirically but not characterized predictively; and the absence of statistical significance testing or multiple runs, so differences such as Flash-RD exceeding Pro-zero-shot by 0.004 should be interpreted cautiously. ConclusionThis paper introduces Reflective Dialogue, a training-free inference-time adaptation method that converts offline Teacher–Solver conversations with correctness feedback and verbalized visual evidence into static ICL context for cross-domain Video QA. On EgoCross, RD consistently improves over zero-shot and generally over plain ICL, reaches 0.659 overall accuracy with Gemini 3.1 Pro Preview plus timestamps, allows a smaller proprietary model to match a larger one zero-shot, and approaches LoRA fine-tuning without any training. Its main cost is substantially increased input length, leaving efficient dialogue compression as the key unresolved question. Paper PromptsSign up for free to create and run prompts on this paper. Top Community PromptsSign up for free to explore the frontiers of research
Discover trending papers, chat with arXiv, and track the latest research shaping the future of science and technology.
Discover trending papers, chat with arXiv, and more.
|
|---|
