DreamPRVR: Diffusion-Guided Video Retrieval

• DreamPRVR is a method that retrieves segments from untrimmed videos by first generating global semantic registers through a coarse-to-fine diffusion process.
• It employs a probabilistic variational sampler and a text-supervised truncated diffusion model to mitigate query ambiguity and enhance cross-modal matching.
• Empirical evaluations on benchmarks like ActivityNet Captions, Charades-STA, and TVR demonstrate state-of-the-art retrieval performance with improved precision.

DreamPRVR is a method for Partially Relevant Video Retrieval (PRVR), introduced in “Imagine Before Concentration: Diffusion-Guided Registers Enhance Partially Relevant Video Retrieval,” that targets retrieval of an untrimmed video (V) from a text query (Q) describing only a partial event inside the video. It adopts a coarse-to-fine representation learning paradigm: it first generates global contextual semantic registers as coarse-grained highlights spanning the entire video, and then concentrates on fine-grained similarity optimization for precise cross-modal matching. The method is designed to address query ambiguity and incomplete global contextual perception, using a probabilistic variational sampler, a text-supervised truncated diffusion model, textual semantic structure learning, and register-augmented Gaussian attention blocks [2604.03653].

1. Task setting and motivating problem structure

PRVR differs from standard text-to-video retrieval (T2VR) because the query need not match the whole video. Instead, only a segment or moment in an untrimmed video must be relevant. This shifts the core difficulty from whole-video alignment to local moment matching under global video noise. In this setting, a short or general query can match the true moment in the target video while also spuriously aligning to visually similar clips in unrelated videos, producing local false positives. Existing PRVR models are described as often operating at clip or frame level and being prone to spiky local activations. Because they lack reliable global context, they may over-score irrelevant videos containing a few superficially matching clips [2604.03653].

DreamPRVR addresses these issues by introducing a global semantic stage before local similarity concentration. Its central premise is that retrieval benefits from first constructing a stable global context representation for the video and only then refining token-level representations for cross-modal matching. This suggests a deliberate separation between global semantic imagination and fine-grained concentration, rather than relying purely on MIL-style scoring of the best-matching clip.

2. Probabilistic formulation and coarse-to-fine retrieval logic

The model is formulated as a generative-plus-discriminative retrieval architecture. DreamPRVR represents the retrieval process as

$$
p_{\theta,\phi}(Q|V) = \int p_\theta(Q|V,r)\, p_\phi(r|V)\, dr
$$

where (p_\phi(r|V)) is the register generator conditioned on video and (p_\theta(Q|V,r)) is the retrieval model conditioned on both video and generated registers. In this formulation, the registers (r) serve as global contextual latent variables that mediate between coarse global understanding and fine-grained similarity estimation [2604.03653].

The paper gives a VAE-style ELBO:

$$
\log p_{\theta, \phi}(Q | V) \geq \mathbb{E}{q\varphi(r | Q_a)} \left[ \log p_\theta(Q | V, r) \right] - \mathbb{KL} \left[ q_\varphi(r | Q_a) \,|\, p_\phi(r | V) \right]
$$

with the corresponding optimization objective

$$
L_\text{DreamPRVR} = - \mathbb{E}{q\varphi(r | Q_a)} \left[ \log p_\theta(Q | V, r) \right] + \mathbb{KL} \left[ q_\varphi(r | Q_a) \,|\, p_\phi(r | V) \right].
$$

The interpretation given in the paper is explicit: the KL term pushes the video-generated registers to match text-induced semantic structure, while the likelihood term makes the registers useful for retrieval. This yields a two-stage internal logic. In the first stage, the model generates a small set of latent register tokens intended to represent the holistic semantics of the video. In the second stage, these registers are fused with frame and clip tokens to improve token-level representations for retrieval.

3. Global contextual semantic registers

DreamPRVR generates its global registers through a truncated diffusion process that begins from a video-centric probabilistic latent space rather than pure Gaussian noise. This stage is built from three coupled components: the Probabilistic Variational Sampler (PVS), the Textual Perturbation Sampler (TPS), and the Diffusion Register Estimator (DRE) [2604.03653].

Given video features (V_v \in \mathbb{R}^{N_v \times d}), extracted from a pretrained video backbone and refined by a lightweight encoder, PVS defines the initial register distribution as

$$
p(r_T|V_v) \sim \mathcal{N}(\boldsymbol\mu_v, \boldsymbol\sigma_v^2 I),
$$

where (\boldsymbol\mu_v) is produced by FC + LayerNorm + (l_2)-norm, (\boldsymbol\sigma_v) comes from a separate FC layer, and (r_T \in \mathbb{R}^{N_r \times d}) denotes the initial register tokens. Sampling uses the reparameterization form

$$
r_T = \boldsymbol\sigma_v \cdot \eta + \boldsymbol\mu_v, \qquad \eta \sim \mathcal{N}(0,I).
$$

This initialization is one of the method’s defining choices: the registers do not start from random Gaussian noise, but from a video-centric distribution, which gives them a semantic head start. PVS is regularized by

$$
L_{\text{pvs}} = \lambda_{kl} L_{kl},
$$

where (L_{kl} = \mathbb{KL}\left[p(r_T|V_v)\,|\,\mathcal{N}(0,I)\right]).

Text supervision is supplied by all queries associated with a video, denoted (Q_a). A video-level semantic text embedding (q_m) is obtained by averaging query embeddings, after which TPS samples uncertain but semantically aligned textual targets through

$$
\bar{q} = \frac{q_m - \mu_q}{\sigma_q},
$$

$$
\hat{q} = \alpha \cdot \bar{q} + \beta,
$$

with

$$
\alpha \sim \mathcal{N}(1, (\gamma \sigma_q)^2 I), \qquad \beta \sim \mathcal{N}(\mu_q, (\gamma \sigma_q)^2 I).
$$

TPS is intended to capture query uncertainty and provide richer supervision than a single deterministic query embedding.

The actual refinement of registers is performed by the DRE. The clean target is the sampled text latent (\hat{q}_0), and the forward noising process is

$$
q(\hat{q}_t|\hat{q}_0) = \mathcal{N}!\left(\hat{q}_t;\sqrt{\bar{\alpha}_t}\hat{q}_0,(1-\bar{\alpha}_t)I\right).
$$

The reverse process is

$$
p_{\phi}(\hat{q}{0:T}|\mathbf{c}) = p(\hat{q}_T)\prod{t=1}^{T} p_{\phi}(\hat{q}_{t-1}|\hat{q}_t,\mathbf{c}),
$$

with update

$$
\hat{q}{t-1} =
\frac{1}{\sqrt{\alpha_t}}
\left(
\hat{q}_t -
\frac{1-\alpha_t}{\sqrt{1-\bar{\alpha}_t}}
\boldsymbol{\epsilon}{\phi}(\hat{q}_t,\mathbf{c},t)
\right)
+ \sigma_t \mathbf{z},
$$

where (\boldsymbol{\epsilon}_{\phi}) is the learned noise predictor, (\mathbf{z}\sim\mathcal{N}(0,I)), and (\sigma_t) is predefined. The conditioning context is derived from the video by cross-attention:

$$
\mathbf{c} = \text{CA}(LP, V_v, V_v),
$$

where (LP) are learnable queries. The denoising objective is

$$
L_{\text{dre}} = \mathbb{E}{t,\hat{q}_t,\epsilon} \left[ \left| \epsilon - \epsilon{\phi}(\hat{q}_t, t, \mathbf{c}) \right|^2 \right].
$$

The paper characterizes this as truncated diffusion because the procedure starts from the PVS-produced video-centric latent (r_T), uses a small number of timesteps (T), and refines the registers efficiently.

4. Textual semantic structure learning and register-guided fusion

DreamPRVR does not treat textual supervision as an unstructured embedding source. It explicitly shapes the text latent space through textual semantic structure learning (TSSL) so that the supervision driving register generation is stable and semantically meaningful [2604.03653].

For each query, RoBERTa word features are extracted, projected down, passed through a Transformer encoder, and aggregated into a query embedding (q \in \mathbb{R}^d). TSSL combines two losses. The first is Query Diversity Loss:

$$
\ell(i,j) = (1 + \cos(q_i, q_j)) \log\left(1 + e^{\omega(\cos(q_i,q_j)+\delta)}\right),
$$

$$
L_{\text{div}} = \frac{2}{M_q(M_q-1)} \sum_{1 \le i,j \le M_q,\; i\ne j} \ell(i,j),
$$

which encourages queries associated with the same video to spread out and capture different semantic aspects. The second is Query Similarity Preservation:

$$
L_{\text{qsp}} = -\frac{1}{|V_q(i)|} \sum_{j \in V_q(i)} \log \frac{ \exp(\operatorname{sim}(q_i,q_j)/\tau) }{ \sum_{k \in \Omega}\exp(\operatorname{sim}(q_i,q_k)/\tau) }.
$$

Here, (V_q(i)) denotes queries from the same video as (q_i), (\Omega) is the set of all query indices, and (\tau) is a temperature. The combined objective is

$$
L_{\text{tssl}} = \lambda_d L_{\text{div}} + \lambda_q L_{\text{qsp}}.
$$

The intended effect is complementary: (L_{\text{div}}) enriches semantics by separating different query views, while (L_{\text{qsp}}) keeps same-video queries compact and discriminative.

After refinement, the registers (r_0) are fused with video tokens using Register-Augmented Attention Blocks (RAB). DreamPRVR uses two video branches, a frame-scale branch (V_f = {f_i}{i=1}^{M_f}) and a clip-scale branch (V_c = {c_i}{i=1}^{M_c}), unified as (V_o \in \mathbb{R}^{M\times d}). Fusion begins by concatenating tokens:

$$
x = \mathrm{Concat}([V_o, r_0]) \in \mathbb{R}^{(M+N_r)\times d}.
$$

Attention is then modified as

$$
\text{GA}(x) = \text{softmax}\left( \mathcal{M}{r} + \left( \mathcal{M}^{g}{\sigma} \odot \frac{x^q (x^k)^\top}{\sqrt{d_h}} \right) \right)x^v,
$$

where (x^q, x^k, x^v) are linear projections, (\mathcal{M}^{g}_{\sigma}) is a Gaussian matrix over video features, and (\mathcal{M}_r) is an asymmetric mask. The attention pattern is asymmetric by design: video queries can attend to both video tokens and registers, whereas register queries attend only to video tokens. The paper describes this asymmetry as making the registers global context providers, not noisy self-updating tokens. Multiple RABs are stacked, and outputs are aggregated via MAIM.

5. Similarity function, optimization, and experimental protocol

DreamPRVR computes retrieval similarity after register-augmented refinement by max-pooling over frame and clip similarities. The frame and clip scores are

$$
S_f(Q,V)=\max{\cos(q,f_1),\dots,\cos(q,f_{M_f})},
$$

$$
S_c(Q,V)=\max{\cos(q,c_1),\dots,\cos(q,c_{M_c})},
$$

and the final similarity is

$$
S(Q,V)=\alpha_f S_f(Q,V)+\alpha_c S_c(Q,V), \qquad \alpha_f+\alpha_c=1.
$$

The practical similarity loss is

$$
L_{\text{sim}} = L_c^{\text{trip}} + L_f^{\text{trip}} + \lambda_c L_c^{\text{nce}} + \lambda_f L_f^{\text{nce}},
$$

and the total training objective is

$$
L_{\text{total}} = L_{\text{sim}} + L_{\text{tssl}} + L_{\text{pvs}} + \lambda_{dre} L_{\text{dre}}.
$$

The method is evaluated on three standard PRVR benchmarks: ActivityNet Captions, Charades-STA, and TVR. ActivityNet Captions contains about 20K YouTube videos, with average duration 118 seconds and about 3.7 moments per video. Charades-STA contains 6,670 videos and 16,128 sentence descriptions, with average 2.4 moments per query/video setting. TVR contains 21.8K clips from six TV shows, with 5 natural language descriptions per clip, and moment annotations unavailable for this evaluation setup. Metrics are rank-based retrieval metrics (R@1), (R@5), (R@10), (R@100), and SumR [2604.03653].

Important implementation details are explicit. For ActivityNet Captions and Charades-STA, the model uses provided I3D features; for TVR, it uses 3072-d video features combining ResNet152 frame-level + I3D segment-level. Text features are RoBERTa features, with 1024-d in some settings and 768-d for TVR. The latent dimension is (d=384), the number of attention heads is 4, the number of register-augmented blocks is (N_a=8), and the diffusion timesteps are (T=10). The number of registers is dataset-specific: (N_r=6) for Charades-STA, (N_r=4) for ActivityNet Captions, and (N_r=8) for TVR. Training uses Adam, a single Nvidia A100-40G GPU, and batch size 128. The paper notes that the method is slightly slower than HLFormer because of iterative diffusion, but that the retrieval overhead is still acceptable, especially in offline settings with cached video features. Code is released by the authors [2604.03653].

6. Empirical performance, ablations, limitations, and naming scope

The reported results position DreamPRVR as the best method among the methods listed in the paper’s comparison tables. On ActivityNet Captions, it achieves (R@1 = 8.7), (R@5 = 27.5), (R@10 = 40.3), (R@100 = 79.5), and SumR = 156.1, which is the best reported SumR in the table. On Charades-STA, it reports (R@1 = 2.6), (R@5 = 8.7), (R@10 = 14.5), (R@100 = 54.2), and SumR = 80.0. On TVR, it reports (R@1 = 17.4), (R@5 = 39.0), (R@10 = 50.4), (R@100 = 86.2), and SumR = 193.1 [2604.03653].

Dataset	DreamPRVR	Note
ActivityNet Captions	(R@1 = 8.7), (R@5 = 27.5), (R@10 = 40.3), (R@100 = 79.5), SumR (= 156.1)	Best reported SumR
Charades-STA	(R@1 = 2.6), (R@5 = 8.7), (R@10 = 14.5), (R@100 = 54.2), SumR (= 80.0)	Outperforms prior methods in the table
TVR	(R@1 = 17.4), (R@5 = 39.0), (R@10 = 50.4), (R@100 = 86.2), SumR (= 193.1)	Best among reported methods

The comparison includes PRVR baselines such as MS-SL, MS-SL++, PEAN, LH, BGM-Net, GMMFormer, ProtoPRVR, DL-DKD, ARL, MGAKD, GMMFormerV2, and HLFormer, as well as non-PRVR baselines such as RIVRL, DE++, CLIP4Clip, Cap4Video, ReLoCLNet, XML, CONQUER, and JSG. The ablation studies isolate the roles of the register-generation pipeline and the loss design. Removing registers hurts performance; replacing the generative mechanism with adaptive pooling (w/ AP) is worse than diffusion-based registers; removing DRE degrades performance; and removing PVS shows that random Gaussian initialization is inferior to video-centric initialization. Loss ablations show that (L_{\text{sim}}) only is worst, and that removing (L_{\text{pvs}}), (L_{\text{dre}}), or (L_{\text{tssl}}) lowers performance. Sensitivity studies indicate that too few registers lack capacity, too many can be redundant, performance is robust around (N_r \in [4,8]), performance improves up to (T=10), and declines for (T>10), which the paper attributes to over-refinement or overfitting.

The paper also states several practical limitations. Diffusion introduces overhead relative to non-diffusion retrieval models. The choice of (N_r) and (T) matters, and the method uses a multi-loss objective whose balance is consequential. At inference time, TPS and forward diffusion are omitted; the system uses the trained reverse generation starting from PVS-produced registers. The registers are discarded after fusion, functioning as auxiliary global context carriers rather than final retrieval outputs. A common naming confusion is that DreamPRVR should not be conflated with DreamForge-World 0.1 Preview, where “DreamPRVR” does not appear as a named system [2606.30292], or with DreamVAR, whose paper explicitly states that the correct name is DreamVAR, not “DreamPRVR” [2601.22507].