Vortex: Multi-Modal Fusion System for Intelligent Video Retrieval

Abstract: This paper presents Vortex, the multimodal video retrieval system developed by our team, FocusOnFun, for the Ho Chi Minh City AI Challenge 2025, designed to advance intelligent multimedia search and temporal reasoning. The system integrates adaptive keyframe extraction, multimodal metadata generation from vision-language and speech models, and a hybrid retrieval strategy that fuses CLIP and SigLIP2 embeddings through Reciprocal Rank Fusion to balance global and fine-grained semantics. To enhance interactivity, Vortex incorporates Rocchio-based relevance feedback and a multi-stage temporal search mechanism for sequential event alignment. Built on Milvus and Elasticsearch, the architecture enables scalable indexing and efficient retrieval. Evaluated in the official competition, our FocusOnFun team's system achieved a score of 79.6/88 (90.5\%) in the Preliminary Round and was further evaluated in the Final Round, achieving an Excellent' overall performance withOutstanding' results in the question-answering (QA) task. This demonstrating the complementary strengths of CLIP and SigLIP2 and confirming the effectiveness of the hybrid retrieval approach. The system establishes a robust foundation for future research in intelligent, context-aware, and interactive video retrieval.

• The paper introduces a dual-database backend integrating Milvus and Elasticsearch to optimize multimodal video retrieval workflows.
• It employs hybrid semantic search using CLIP and SigLIP2 embeddings with reciprocal rank fusion, enhanced by LLM-assisted query interpretation.
• Demonstrated on real-world datasets, Vortex achieves high retrieval accuracy with effective temporal event sequencing and user-driven feedback refinement.

Vortex: Multi-Modal Fusion System for Intelligent Video Retrieval

System Architecture and Key Components

Vortex leverages a dual-database backend integrating Milvus for vector search and Elasticsearch for indexed metadata, optimizing large-scale multimodal retrieval workflows. The architecture separates query processing—including text, image, temporal, and filter queries—from an interactive feedback loop. Keyframe extraction utilizes AutoShot combined with CLIP-based $L_2$-norm filtering for efficient, non-redundant visual sampling. Metadata enrichment is performed via Qwen2.5-VL for OCR and captioning, and Whisper for temporally aligned ASR, producing rich multimodal descriptors for each keyframe. The interface offers unified, responsive query input and result exploration across both search and feedback-driven refinement.

Figure 1: Overall system architecture depicts dual workflows for multimodal query processing and relevance feedback.

Figure 2: Data pre-processing pipeline involving adaptive keyframe extraction, multimodal feature generation, and embedding indexing.

Retrieval Methodology

The retrieval module implements hybrid semantic search by embedding queries using both CLIP and SigLIP2. These representations enable robust matching at both global semantic and fine-grained detail levels. Reciprocal Rank Fusion (RRF) is employed to merge independent candidate lists, optimizing ranking performance and reducing biases inherent in single-model retrieval. This fusion demonstrably improves retrieval accuracy, especially for queries that require both contextual and localized matching.

Figure 3: Retrieval Module utilizes CLIP and SigLIP2 embeddings, re-ranking candidates via Reciprocal Rank Fusion to select top-$k$ results.

For iterative search refinement, user feedback is utilized through Rocchio-based relevance optimization, systematically augmenting query vectors based on explicit relevance signals. This enables dynamic adaptation to user intent across successive queries.

Temporal Search and Sequential Retrieval

Vortex features a multi-stage temporal search mechanism designed for event sequence alignment in complex queries. Users specify three distinct event queries—Before, Now, After—which undergo independent retrieval. The system then performs temporal re-ranking to boost the scores of videos that contain the full described sequence, resulting in improved alignment for sequential event queries. This method achieves computational efficiency by eschewing traditional DP-based alignment for a lightweight, real-time ranking heuristic.

Figure 4: Temporal search mode interface demonstrates input fields for specifying temporal relations (Before, Now, After) in sequential retrieval tasks.

LLM-Assisted Query Interpretation

Vortex deploys LLMs as interactive query interpretation assistants rather than autonomous rewriters, mitigating risks of intent drift and hallucination. The LLM suggests semantically refined options for ambiguous queries, but leaves final authority explicitly to the user. This strategy synergizes pre-query semantic augmentation with post-query relevance feedback, providing robust user-controlled search refinement.

User Interface and Interactive Retrieval

The user interface is engineered to support seamless query management and iterative search. Ranked keyframes generated by multimodal retrieval are visualized alongside explicit relevance feedback controls. Temporal search workflows are facilitated through dedicated input fields for event sequencing, while OCR and semantic filtering are accessible for fine-grained control.

Figure 5: System user interface with integrated query bar and ranked keyframes, supporting multimodal and temporal search workflows.

Evaluation and Results

Vortex was evaluated against the official dataset from Ho Chi Minh City AI Challenge 2025, encompassing diverse video domains and queries. The system achieved a cumulative score of 79.6/88 (90.5%) in the Preliminary Round, with further “Excellent” ratings in the Final Round. Performance was particularly strong in Question Answering (Q{content}A) tasks, where multimodal metadata and sequential search mechanisms delivered “Outstanding” task-specific results. Progressive integration of Temporal Search and Relevance Feedback modules yielded marked improvement in retrieval quality across rounds.

Practical Usage Examples

Vortex was demonstrated across multiple query types, including textual Known-Item Search (KIS), video-based KIS, Q{content}A, and complex temporal alignment (TRAKE). Strategies employed included global OCR filtering for literal cues, semantic search leveraging ASR and generated captions, and coarse-to-fine search tactics combining temporal navigation and query-by-example refinement.

Figure 6: tkis-02: Global OCR filtering for literal textual cues exemplifies precision search in specialized scenarios.

Implications and Future Directions

Vortex exemplifies the integration of scalable multimodal retrieval, fine-grained event reasoning, and interactive feedback mechanisms in video search systems. The demonstrated complementary effectiveness of CLIP and SigLIP2 embeddings, fused via RRF, suggests further investigation into multi-model fusion strategies for multimedia retrieval. The architecture's support for user-driven query refinement points to expanded research in human-in-the-loop and explainable retrieval systems. The competition-driven results validate robust system design for real-world, context-aware search, providing a foundation for sophisticated temporal and semantic video understanding.

Conclusion

Vortex is a unified system for intelligent video retrieval, incorporating adaptive keyframe extraction, multimodal metadata generation, hybrid semantic retrieval, temporal event alignment, and relevance-driven query refinement. Empirical results confirm strong task-level performance, especially in content comprehension and event sequence queries. The system establishes a scalable and extensible basis for advancing interactive, multimodal, and context-aware video retrieval research.