Revisiting Uncertainty: On Evidential Learning for Partially Relevant Video Retrieval

Abstract: Partially relevant video retrieval aims to retrieve untrimmed videos using text queries that describe only partial content. However, the inherent asymmetry between brief queries and rich video content inevitably introduces uncertainty into the retrieval process. In this setting, vague queries often induce semantic ambiguity across videos, a challenge that is further exacerbated by the sparse temporal supervision within videos, which fails to provide sufficient matching evidence. To address this, we propose Holmes, a hierarchical evidential learning framework that aggregates multi-granular cross-modal evidence to quantify and model uncertainty explicitly. At the inter-video level, similarity scores are interpreted as evidential support and modeled via a Dirichlet distribution. Based on the proposed three-fold principle, we perform fine-grained query identification, which then guides query-adaptive calibrated learning. At the intra-video level, to accumulate denser evidence, we formulate a soft query-clip alignment via flexible optimal transport with an adaptive dustbin, which alleviates sparse temporal supervision while suppressing spurious local responses. Extensive experiments demonstrate that Holmes outperforms state-of-the-art methods. Code is released at https://github.com/lijun2005/ICML26-Holmes.

• The paper presents Holmes, a hierarchical evidential learning framework that models inter-video and intra-video uncertainty using Dirichlet distributions.
• It integrates multi-scale modalities with flexible optimal transport to establish dense, soft query-clip alignments, mitigating sparse supervision in untrimmed videos.
• Ablation studies demonstrate that Holmes achieves superior retrieval performance across benchmarks like ActivityNet Captions, Charades-STA, and TVR.

Hierarchical Evidential Learning for Uncertainty in Partially Relevant Video Retrieval

Introduction

The task of Partially Relevant Video Retrieval (PRVR) involves retrieving untrimmed videos in response to text queries that describe only partial or momentary content within a video. This introduces a pronounced asymmetry: queries are brief and ambiguous, while the associated videos exhibit rich, temporally complex content. As a result, ambiguity arises at two levels:

1. Inter-video semantic ambiguity: Queries may be under-determined (insufficient information), polysemous (multiple plausible matches), or precise.
2. Intra-video supervisory sparsity: Supervision is typically sparse, as only a small subset of temporal segments relate to the query; conventional multiple instance learning (MIL) further exacerbates this by focusing on maximally scoring segments and neglecting holistic understanding.

Addressing these challenges, "Revisiting Uncertainty: On Evidential Learning for Partially Relevant Video Retrieval" (2605.06083) proposes a unified, hierarchical evidential learning framework—referred to as Holmes—to explicitly model and utilize uncertainty at both inter- and intra-video levels. Holmes leverages multi-granular cross-modal evidence, modeled via Dirichlet distributions, to quantify different types of uncertainty and employs optimal transport for denser supervision, thus improving retrieval accuracy and capability to interpret query ambiguity.

Problem Formulation and Framework Overview

The goal is, given a text query and a collection of untrimmed videos, to retrieve those videos containing moments relevant to the query. Holmes constructs multi-scale representations: textual queries are encoded using RoBERTa and Transformers, while videos are encoded at both frame and clip level using CNN backbones and Gaussian attention blocks.

Similarity between a query and a video is computed as a weighted combination of maximum cosine similarities at frame and clip levels. The top-level architecture integrates:

• Evidential learning to model uncertainty at the inter-video level.
• Flexible Optimal Transport (FOT) to establish soft temporal alignments for intra-video uncertainty.

Inter-video Evidential Learning

Holmes treats cross-modal similarity scores as evidence, parameterizing them as Dirichlet distributions. This departs from deterministic similarity-based retrieval, enabling explicit estimation of uncertainty through the Dirichlet's belief mass and total evidence.

The three-fold principle guides the characterisation of uncertainty:

1. Epistemic uncertainty ($u$): Quantifies the confidence based on total evidence—high $u$ signals under-determined queries with weak evidence, but low $u$ alone does not guarantee semantic correctness.
2. Label consistency ($c$): Measures alignment between the most similar candidate and the ground-truth. Low $c$ indicates queries where the model fails to assign high similarity to the ground-truth; however, high $c$ does not exclude semantic ambiguity.
3. Aleatoric uncertainty ($\xi$): Estimated via entropy of the Dirichlet, it captures inherent ambiguity in the query: low $\xi$ suggests precision, while high $\xi$ signals polysemy.

By combining these measures, queries are adaptively categorized as precise, polysemous, or under-determined. The process is visualized in the following breakdown:

Figure 1: Different query types in PRVR, highlighting response margins for precise, under-determined, and polysemous queries, and demonstrating the vulnerability of MIL to noisy supervision.

A cross-scale partitioning fuses frame-level and clip-level uncertainty assessments based on a dominance ordering: under-determined $>$ polysemous $u$0 precise, reflecting increasing semantic ambiguity. Conflicts between scales are resolved by selecting the higher-dominance categorization.

For label calibration, Holmes preserves one-hot supervision for precise and under-determined queries. Polysemous queries receive soft labels derived from similarity scores, reducing over-penalization of non-unique ground truths and correcting for supervision noise.

The overall evidential loss is computed as the expected L2 distance (bias-variance decomposition) between the calibrated label and the Dirichlet's predicted class probability.

Figure 2: Overview of Holmes, illustrating multi-scale (frame and clip) encoding, dynamic evidential aggregation, uncertainty-guided identification, and the role of flexible optimal transport in temporal alignment.

Intra-video Evidential Learning via Flexible Optimal Transport

Conventional MIL-based clip-level supervision is sparse and susceptible to local noise. Holmes introduces a Flexible Optimal Transport (FOT) scheme to realize soft many-to-many query-clip alignments:

• The similarity matrix between query and video clips is used as the transport cost.
• A dustbin "bucket" is appended to absorb irrelevant/noisy segments in untrimmed videos; mass is transported here if clips are not pertinent to any query, mitigating the impact of video redundancy.

Compared to standard optimal transport, FOT enables partial, robust matching tailored to PRVR's partial relevance. The dustbin effect is visualized as follows:

Figure 3: Comparison of transport mass alignment. FOT's dustbin (Z) effectively absorbs alignment mass from irrelevant segments, improving supervision fidelity.

The intra-video evidential objective is formulated analogously to the inter-video objective, treating optimal transport assignments as soft targets for clip-level Dirichlet evidence modeling.

Qualitative impact of intra-video evidence is illustrated below:

Figure 4: Ablation on intra-video evidential learning: removing $u$1 leads to misranking, reinforcing the importance of dense temporal supervision.

Results and Ablation Studies

Holmes is evaluated on ActivityNet Captions, Charades-STA, and TVR, outperforming baselines and recently proposed methods across all retrieval metrics. Ablations demonstrate gains from each key component:

• Both frame- and clip-level evidence are vital; omitting either branch degrades SumR performance.
• Uncertainty-dominant partition fusion surpasses naive fusion/branch-trusting strategies.
• Query-specific label calibration delivers substantial improvements, especially for ambiguous polysemous queries.
• Flexible Optimal Transport yields higher recall than softmax or conventional OT.

Efficiency comparisons show Holmes operates at comparable computational cost to other state-of-the-art PRVR architectures despite superior effectiveness.

Qualitative Analysis

Holmes enables qualitative diagnosis of query categories and retrieval reliability. Representative identification results are provided below:

Figure 5: Query-wise examples with top-3 retrieved videos, exhibiting clear separation of precise, polysemous, and under-determined types. Correct and incorrect matches are outlined accordingly.

Parameter analysis indicates robust performance across hyperparameters—temperature $u$2 for evidence, threshold $u$3 for categorization, and dustbin ratio $u$4 for FOT—showing insensitivity to moderate tuning.

Figure 6: Parameter sensitivity analysis for temperature $u$5, validating robust retrieval under varying uncertainty aggregation settings.

Implications and Future Work

Holmes addresses the core challenges of PRVR by tightly coupling uncertainty quantification with both query understanding and supervision density. By moving beyond black-box matching, it fosters:

• Interpretable retrieval: Models can explicitly indicate lack of evidence or ambiguity, enabling trust and human verification.
• More robust training: Adaptive supervision counteracts annotation bias and under-constrained optimization.
• Foundations for uncertainty-aware multimodal models: The framework generalizes to settings with vague or noisy annotations and may benefit broader cross-modal retrieval tasks, including open-set and out-of-distribution scenarios.

Potential avenues for extension include end-to-end training with more powerful encoders (e.g., CLIP variants), adaptation to streaming or online retrieval tasks, and broader application to multi-modal event understanding.

Conclusion

Holmes advances PRVR by introducing a principled, hierarchical evidential learning approach that models inter-video and intra-video uncertainty using Dirichlet-based evidence aggregation and flexible optimal transport. Its explicit modeling of semantic ambiguity, adaptive label calibration, and robust temporal alignment yields consistent gains over state-of-the-art methods. The framework sets a methodological precedent for uncertainty-aware, interpretable retrieval systems in video and broader multimodal AI.