Debiased Evidential Learning for Moment Retrieval

• The paper introduces DEMR, a framework that integrates evidential learning, Reflective Flipped Fusion, and query reconstruction to address cross-modal imbalance and miscalibrated uncertainties.
• The methodology employs a progressive fusion strategy and a geometric regularizer to align video-text features and ensure lower prediction error with increased evidential support.
• Empirical results on benchmarks like QVHighlights and Charades-STA show that DEMR outperforms conventional models, achieving higher retrieval accuracy and enhanced robustness under debiased evaluations.

Debiased Evidential Learning for Moment Retrieval (DEMR) is a framework designed for robust temporal localization of moments in videos based on natural language queries. It resolves foundational challenges in moment retrieval—most notably, cross-modal imbalance and miscalibrated uncertainty estimation—by integrating evidential learning, a progressive fusion strategy, and an auxiliary query-grounding task. DEMR improves both retrieval accuracy and calibration under standard and debiased benchmarks, providing enhanced temporal-semantic robustness (Huang et al., 30 Nov 2025).

1. Formal Problem Setup and Task Definition

The objective of moment retrieval is to accurately predict temporal segments within a video $V = \{v_i\}_{i=1}^{L_v}$, where $v_i \in \mathbb{R}^D$, conditioned on a textual query $Q = \{q_j\}_{j=1}^{L_q}$ with $q_j \in \mathbb{R}^D$. The ground truth is a segment $m = [m^s, m^e]$ specified as clip indices with $1 \le m^s < m^e \le L_v$. The system generates candidate segments $$\{\tilde m_i = [\tilde m_i^s, \tilde m_i^e]\}_{i=1}^{L_v}$$, ranks them, and selects the top prediction by retrieval score.

The standard supervised loss comprises two terms over foreground clips ($f_i=1$): $\mathcal{L}_{mr} = \sum_{i=1}^{L_v} \mathds{1}_{f_i=1} \left[ \lambda_{L1}\, \mathcal{L}_{\mathrm{SmoothL1}}\bigl(\tilde d_i, d_i\bigr) + \lambda_{\mathrm{iou}}\, \mathcal{L}_{\mathrm{gIoU}}(\tilde m_i, m_i) \right],$ where $\tilde d_i = \tilde m_i^e - \tilde m_i^s$ and $d_i = m^e_i - m^s_i$.

2. Deep Evidential Regression Baseline and Its Limitations

The vanilla baseline for uncertainty-aware localization uses Deep Evidential Regression (DER), modeling the boundary $b_i \in \{m^s, m^e\}$ under a Normal-inverse-Gamma prior: $$p(\mu, \sigma^2 \mid \gamma, \upsilon, \alpha, \beta) = \mathcal{N}(\mu \mid \gamma, \tfrac{\sigma^2}{\upsilon})\, \Gamma^{-1}(\sigma^2 \mid \alpha, \beta),$$ with evidential parameters $(\gamma, \upsilon, \alpha, \beta)$ output by a linear head. The marginal likelihood is Student-$t$, yielding a negative log-likelihood training loss: $$\mathcal{L}_i^{\rm NLL} = -\log\left( \mathrm{St}(b_i; \gamma, \frac{\beta(1+\upsilon)}{\upsilon\alpha}, 2\alpha) \right)$$ and a heuristic uncertainty regularizer: $$\mathcal{L}^R_i = \Delta_i\,\Phi_i,\quad \Delta_i = |b_i-\gamma|, \quad \Phi_i = 2\upsilon + \alpha.$$ ALEATORIC and EPISTEMIC uncertainties are, respectively, $\mathbb{E}[\sigma^2]=\beta/(\alpha-1)$, $\mathrm{Var}[\mu]=\beta/(\upsilon(\alpha-1))$. The total baseline loss (for $N$ samples) is: $$\mathcal{L}_{\rm base} = \mathcal{L}_{mr} + \lambda_{\rm der} \frac{1}{N} \sum_{i=1}^N \left( \lambda_{\rm NLL} \mathcal{L}_i^{\rm NLL} + \lambda_{\rm Reg} \mathcal{L}_i^R \right).$$

However, two major issues emerge: (A) cross-modal (video-text) imbalance—where uncertainty is unduly sensitive to noise in the visual modality and (B) the uncertainty regularizer’s gradient depends solely on prediction error and ignores current evidence, causing pathological uncertainty assignments.

3. Core Methodological Advances in DEMR

3.1 Reflective Flipped Fusion (RFF) Block

The RFF block is a multi-layer cross-modal alignment module. Across $n$ layers, video and text branches alternate roles in a cross-attention (CA) operation followed by self-attention (SA). For layer $i$: $$\begin{aligned} &CA_{v\to q}^{(i)} = \mathrm{Softmax}\left(\frac{V^{(i)} Q^{(i)\top}}{\sqrt{d_k}}\right) Q^{(i)}, \ &SA^{(i)}_q(X) = \mathrm{Softmax}\left(\frac{X X^\top}{\sqrt{d_k}}\right) X, \ &Q^{(i+1)} = SA^{(i)}_q(CA_{v\to q}^{(i)}), \ &CA_{q\to v}^{(i)} = \mathrm{Softmax}\left(\frac{Q^{(i)} V^{(i)\top}}{\sqrt{d_k}}\right) V^{(i)}, \ &SA^{(i)}_v(Y) = \mathrm{Softmax}\left(\frac{Y Y^\top}{\sqrt{d_k}}\right) Y, \ &V^{(i+1)} = SA^{(i)}_v(CA_{q\to v}^{(i)}). \end{aligned}$$ By “flipping” the query-key-value structure between modalities at each layer, RFF progressively aligns video and text, mitigating over-reliance on a single modality.

3.2 Query Reconstruction (QR) Auxiliary Task

To enhance the linguistic grounding of the fusion module, DEMR incorporates a QR pretraining phase. Randomly masked noun tokens in $Q$ are reconstructed after RFF processing via a lightweight QR head: $$\mathcal{L}_{qr} = \mathbb{E}\left[ -\sum_{k=1}^l \log P(w_k \mid U, V) \right],$$ where $U$ is the set of unmasked tokens and $w_k$ the masked nouns. QR is trained for 30 epochs; the head is then frozen and the loss dropped.

3.3 Geometric Regularizer for Uncertainty Calibration

To address the structural limitation of the heuristic regularizer, DEMR normalizes the prediction error $\Delta_i$ and evidence $\Phi_i$ to $\bar\Delta_i, \bar\Phi_i \in [0,1]$ and introduces a geometric regularizer: $$\mathcal{L}^L_i = \|\bar\Phi_i + \bar\Delta_i - 1\|_2^2,$$ enforcing that increased evidence corresponds to low prediction error and vice versa. The evidential loss adopts: $$\mathcal{L}_i^e = \lambda_{\rm NLL} \mathcal{L}_i^{\rm NLL} + \lambda_{\rm geom} \mathcal{L}_i^L.$$

4. Composite Loss and Training Strategy

The complete DEMR loss function is: $$\mathcal{L} = \mathcal{L}_{mr} + \lambda_{\rm der} \frac{2}{N} \sum_{i=1}^N \left( \lambda_{\rm NLL} \mathcal{L}_i^{\rm NLL} + \lambda_{\rm geom} \mathcal{L}_i^L \right) + \mathcal{L}_{qr}.$$ Training proceeds in two stages: QR-only (30 epochs, learning rate $10^{-5}$), followed by full DEMR training with the QR head frozen. Features are pre-extracted from CLIP (ViT-B/32) and SlowFast (ResNet-50) backbones. RFF blocks use 4 layers ($d_k = D = 512$), with batch size 32 and AdamW optimizer.

5. Empirical Evaluation and Results

DEMR is evaluated on both standard and debiased datasets:

Dataset/Setting	[email protected]	[email protected]	mAP/[email protected]	IID–OOD Gap $\Delta$
QVHighlights (val)	65.0	49.4	43.0	—
Charades-STA (val)	60.2	38.0	—	—
TACoS (val)	37.3	19.4	—	—
Charades-CD (i.i.d)	62.20	43.29	—
Charades-CD (o.o.d)	52.46	30.97	—	9.74/12.32

On standard datasets, DEMR’s performance matches or exceeds strong baselines, while debiased benchmarks show a lower IID–OOD gap than prior methods, indicating improved robustness to distributional shift. In ablation studies, each architectural component incrementally reduces modality imbalance $\Delta_{\rm Var}$ and improves retrieval scores. Hyperparameter sweeps indicate best performance at $\lambda_{\rm geom} = 10^{-2}$, $\lambda_{\rm der} = 10^{-3}$, with QR optimal at 30–50 epochs. Calibration analysis shows DEMR yields monotonic uncertainty-error relationships and higher epistemic uncertainty in under-represented video regions.

6. Algorithmic Workflow and Code Availability

The DEMR pipeline is summarized as follows:

1. Extract $V$ and $Q$ features with CLIP+SlowFast.
2. Stage 1 (QR Pretrain): Mask one noun per query, apply RFF, train QR head via $\mathcal{L}_{qr}$ for 30 epochs.
3. Stage 2 (DEMR Training): Freeze QR head, run RFF, split into MR and evidential heads. Evidence head predicts evidential parameters and loss $\mathcal{L}^{\rm NLL} + \mathcal{L}^L$; MR head predicts offsets with $\mathcal{L}_{mr}$.
4. At inference, apply non-max suppression at threshold 0.7, rank by retrieval score (potentially by low aleatoric uncertainty), and output the top-1 segment.

Implementation code is available at https://github.com/KaijingOfficial/DEMR.

7. Significance and Impact

DEMR addresses two principal shortcomings of evidential approaches to moment retrieval—modality imbalance and structurally unsound regularization—by means of (1) a flip-flop cross-modal fusion mechanism, (2) targeted linguistic pretraining, and (3) an adaptive geometric regularizer. Empirical evaluation confirms significant improvements in effectiveness, calibration, and temporal-semantic robustness, particularly on debiased benchmarks where conventional models exhibit pronounced performance gaps (Huang et al., 30 Nov 2025). A plausible implication is that such debiasing strategies, especially geometric regularization for uncertainty, may generalize to other cross-modal and uncertainty-sensitive retrieval problems.