Interactive Multi-Turn Retrieval for Health Videos

Abstract: The growing availability of health-related instructional videos creates new opportunities for clinical training, patient rehabilitation, and health education, yet existing retrieval systems remain largely single-turn: a user submits one query and receives one ranked list. This interaction is brittle in health scenarios, where information needs are often vague at first and become clinically meaningful only after follow-up constraints such as posture, hand placement, contraindications, equipment, or patient condition are specified. We introduce interactive multi-turn semantic retrieval for health videos and construct MHVRC, a Multi-Turn Health Video Retrieval Corpus, by combining video-grounded descriptions from VideoChat-Flash with query refinements generated by DeepSeek. We further propose DATR, a Dialogue-Aware Two-Stage Retrieval framework. DATR first performs efficient coarse retrieval with a CLIP-style dual encoder and sparse frame sampling, then re-ranks the top candidates through multi-turn query fusion and a lightweight cross-encoder scoring module. Experiments on MHVRC show consistent gains over strong text-video retrieval baselines, while user studies indicate that refined multi-turn queries better capture fine-grained procedural semantics than single-turn annotations. The work establishes a benchmark and a scalable technical recipe for interactive health video retrieval.

• The paper's main contribution is DATR, a dialogue-aware two-stage retrieval framework that refines multi-turn queries for precise health video retrieval.
• It introduces the MHVRC corpus with 3,000 query triplets over 900 videos, enabling realistic evaluation of procedural search in health contexts.
• Experimental results show DATR outperforms baselines in Recall@1 and ranking metrics, underscoring its potential in clinical training and patient education.

Interactive Multi-Turn Retrieval for Health Videos: Technical Summary

Motivation and Background

The paper "Interactive Multi-Turn Retrieval for Health Videos" (2605.01409) addresses the limitations inherent in single-turn health video retrieval systems. Health information-seeking often requires iterative refinement: users initially pose broad queries that subsequently demand procedural detail, context, and constraints as seen in Figure 1.

Figure 1: Single-turn versus multi-turn health video retrieval; multi-turn enables retrieval with procedural and contextual constraints.

Typical vision-language retrieval systems—CLIP derivatives, hierarchical transformers—are optimized for broad content matching and lack explicit models for progressive, clinically contextual intent evolution. In health scenarios, semantic granularity matters: posture, safety, contraindications, and phase are critical retrieval filters not well-captured by static queries.

MHVRC Corpus Construction

The Multi-Turn Health Video Retrieval Corpus (MHVRC) is introduced to benchmark retrieval in realistic health information-seeking settings. For each video in MHVRC, VideoChat-Flash generates visually grounded procedural descriptions; DeepSeek then produces user-style query refinements based on those descriptions (see Figure 2).

Figure 2: MHVRC construction pipeline: video descriptions from VideoChat-Flash, query refinement via DeepSeek grounded in procedural content.

MHVRC comprises approximately 3,000 query triplets linked to 900 instructional health videos spanning over 90 hours. Each triplet $(q_1, d_v, q_2)$ pairs an initial coarse query, a rich procedural description, and a refined, contextually aware follow-up query. The generation pipeline supports realistic query evolution anchored in observable evidence and clinical relevance, mitigating typical synthetic annotation failure modes.

DATR: Dialogue-Aware Two-Stage Retrieval

The Dialogue-Aware Two-Stage Retrieval (DATR) framework is proposed as the technical recipe for scalable, precise, multi-turn health video retrieval. DATR decomposes retrieval into:

• Stage I: Dual encoder (CLIP-style), coarse retrieval using sparse frame sampling and bidirectional contrastive loss. Efficiently maps query and video into a common embedding space, selecting top candidates.
• Stage II: Fusion encoder combines multi-turn query embeddings (additive and multiplicative fusion), lightweight cross-encoder re-ranking applied only to Stage I candidates, capturing fine procedural constraints.

This architecture (see Figures 3 and 4) synergizes scalability and precision; the dual encoder ensures broad coverage while the cross-encoder enhances intent specificity and contextual relevance.

Figure 3: DATR overview—Stage I retrieves candidates, Stage II fuses queries and refines ranking.

Figure 4: Detailed two-stage retrieval architecture, illustrating the wide pipeline design for clarity.

Fusion strategies in Stage II are critical. Additive terms preserve overarching topic continuity; multiplicative terms emphasize newly introduced constraints by the refined query. Ablations confirm the necessity of both, and bidirectional contrastive losses outperform unidirectional alternatives.

Experimental Results and Analysis

Quantitative results on MHVRC demonstrate that DATR outperforms strong baselines (CLIP4Clip, Frozen-in-Time, CLIPBERT, HERO) in Recall@$K$ and rank metrics—most substantially at strict cutoffs (R@1, R@10). DATR achieves R@1 of 19.5 vs. HERO's 15.2, and smaller median and mean ranks, validating multi-turn fusion's utility for procedural health video retrieval.

Qualitative outputs illustrate DATR's ability to match refined health queries involving exercise, rehabilitation, and procedural demonstrations (see Figure 5).

Figure 5: DATR retrieves clinically relevant videos matching procedural, posture, and safety constraints in multi-turn queries.

User studies further corroborate query refinement and retrieval quality; clinically relevant refinements are consistently rated higher in specificity, clarity, and utility. DATR's top retrievals are close to ground truth, outperforming baselines in instructional clarity and procedural accuracy.

Ablation studies reveal:

• Transformer encoding outperforms recurrent alternatives.
• Second-stage re-ranking is essential; omitting it causes substantial degradation.
• Combined additive + multiplicative fusion yields best results.

Error analysis identifies topic confusions, temporal-phase mismatches, and over-specific refinements as prevalent challenges.

Practical and Ethical Implications

The practical impact of interactive multi-turn retrieval spans clinical training, patient education, remote rehabilitation, and misinformation mitigation. Systems must model evolving user intent and fine procedural granularity, not just global topical similarity.

However, limitations persist. MHVRC relies on generated annotations—human expert validation remains requisite for deployment. The current task models only two interaction turns; real sessions often involve more dialogic complexity. Future extensions could embed moment-level localization, multi-modal features (audio, pose), clinician-in-the-loop validation, and personalized retrieval sensitive to user capability.

Ethically, retrieved content must be contextualized as informational, not prescriptive. Systemic safety, privacy, and source credibility filtering are paramount. Generated refinements should be audited for correctness and clinical appropriateness.

Conclusion

This work establishes a benchmark and technical recipe for multi-turn, dialogue-aware health video retrieval. With the MHVRC corpus and DATR framework, it demonstrates substantial gains in procedural search quality while highlighting the necessity of modeling progressive intent evolution. The paper sets a foundation for research into adaptive health video systems capable of handling clinically meaningful, evolving user queries.