Diffusion Latent Beam Search (DLBS)

• DLBS is an inference-time algorithm that enhances text-to-video generation by framing conditional generation as a discrete search over the latent denoising trajectory.
• It employs a light-weight beam search in latent space with short short-horizon deterministic lookahead, achieving improved alignment to prompt semantics without re-training.
• DLBS outperforms traditional methods by balancing beam width, candidate count, and lookahead depth, leading to significant gains on video generation benchmarks.

Diffusion Latent Beam Search (DLBS) is an inference-time algorithm for optimizing the perceptual and semantic quality of outputs from diffusion models, with a primary application in text-to-video generation. The approach frames conditional generation as a discrete search problem over the latent denoising trajectory, combining light-weight beam search in the latent space with lookahead reward estimation. Unlike traditional sampling or greedy inference, DLBS enables effective alignment to prompt semantics and perceptual criteria without model retraining, utilizing a calibrated, potentially non-differentiable reward function for scoring and selection (Oshima et al., 31 Jan 2025).

1. Mathematical Framework for Diffusion Latent Beam Search

DLBS operates over the reverse-time Markov chain defined by a diffusion model with $T$ denoising steps $t=T,T-1,...,1,0$. At each $t$, the algorithm maintains $B$ beams, each representing a latent $z^j_t\in\mathbb{R}^d$. The primary objective is to maximize a given reward function $R$ (possibly non-differentiable), evaluated on the fully denoised latent $z_0$ and text prompt $c$: $$\max_{\{z^j_T \sim \mathcal{N}(0,I)\}} \max_{\text{reverse paths } z_t\rightarrow z_{t-1}} R(z_0(z_T),c)$$ At each denoising step, every beam expands to $K$ candidates, producing $t=T,T-1,...,1,0$0 new partial trajectories. Candidates are ranked by a "lookahead-augmented" score: $t=T,T-1,...,1,0$1 where $t=T,T-1,...,1,0$2 is computed via Tweedie’s formula (posterior mean after one denoising step), $t=T,T-1,...,1,0$3 is a short deterministic DDIM chain lookahead estimator (of length $t=T,T-1,...,1,0$4), and $t=T,T-1,...,1,0$5 controls the influence of lookahead (default $t=T,T-1,...,1,0$6).

The reward function $t=T,T-1,...,1,0$7 is calibrated as a non-negative weighted sum of $t=T,T-1,...,1,0$8 base metrics, reflecting alignment to text, subject consistency, motion, aesthetics, and image fidelity: $t=T,T-1,...,1,0$9 The optimal weights $t$0 are chosen to maximize Pearson correlation with feedback from large vision-LLMs (VLMs) such as GPT-4o or Gemini.

2. Algorithmic Structure and Pseudocode

The DLBS procedure is as follows:

• Initialization: Sample $t$1 for $t$2.
• Denoising Loop (for $t$3 down to $t$4):
  • For each beam, compute DDIM posterior mean for the next latent.
  • For $t$5, for each beam, sample $t$6 candidate children via Gaussian perturbation.
  • For each candidate, compute:
  • One-step posterior-mean estimate $t$7
  • Optionally, run $t$8-step deterministic DDIM to obtain lookahead endpoint $t$9
  • Score using the sum (or only) of rewards at these endpoints.
  • Retain the top $B$0 candidates by score for the next step.
• Output: For the final $B$1 denoised latents $B$2, select the one maximizing $B$3 after VAE decoding.

The overall computational budget per step is $B$4 denoising samples; deeper lookahead ($B$5) increases reward-fidelity at the cost of more compute. Empirically, $B$6 to $B$7 provides significant benefit.

3. Lookahead Estimation and Search-Efficiency Tradeoffs

DLBS’s core innovation is its efficient, short-horizon lookahead estimator. For each candidate latent at step $B$8, a deterministic DDIM chain of length $B$9 projects the latent to a near-terminal state, producing a sharper estimate of eventual reward. This contrasts with purely greedy approaches, which are prone to suboptimal local decisions especially for video or sequential domains.

Resource allocation is governed by trades between beam width $z^j_t\in\mathbb{R}^d$0, candidates per beam $z^j_t\in\mathbb{R}^d$1, and lookahead depth $z^j_t\in\mathbb{R}^d$2. Larger $z^j_t\in\mathbb{R}^d$3 increases exploration but at fixed computational budget requires smaller $z^j_t\in\mathbb{R}^d$4. Increasing $z^j_t\in\mathbb{R}^d$5 typically offers better reward-fidelity per beam than increasing $z^j_t\in\mathbb{R}^d$6, with diminishing returns beyond $z^j_t\in\mathbb{R}^d$7.

4. Reward Calibration and Metric Weighting

DLBS employs a calibrated reward $z^j_t\in\mathbb{R}^d$8, constructed from six base VBench metrics: Subject Consistency ($z^j_t\in\mathbb{R}^d$9), Motion Smoothness ($R$0), Dynamic Degree ($R$1), Aesthetic Quality ($R$2), Imaging Quality ($R$3), and Text–Video Consistency ($R$4).

Weights $R$5 are selected via grid search from $R$6 to maximize the correlation between $R$7 and model or human-proxy (VLM) preferences. This calibration substantially increases alignment between perceived quality and computed reward over any single metric, with statistical significance ($R$8) (Oshima et al., 31 Jan 2025).

5. Empirical Results and Comparative Performance

DLBS achieves state-of-the-art inference-time perceptual alignment on multiple video generation benchmarks (MSRVTT-test, DEVIL-high/medium/static), consistently outperforming both Best-of-N (BoN) sampling and Greedy Search (GS). Quantitative improvements (normalized 0–1 scale with Gemini-calibrated reward on MSRVTT-test):

Method	KB=8	KB=16	KB=32
BoN	0.42	0.45	0.47
GS	0.40	0.44	0.46
DLBS	0.48	0.52	0.55

Ablation studies demonstrate that even minimal lookahead ($R$9) yields ~5% improvement over no lookahead, and optimal performance is achieved when $z_0$0. Calibration weights yield significant Pearson $z_0$1 increases relative to individual base metrics. Additionally, VLM-evaluated quality scales nearly linearly with $z_0$2.

6. Deployment Guidelines and Recommendations

Effective use of DLBS follows this priority for compute allocation:

1. Allocate initial resources to short-horizon lookahead ($z_0$3–$z_0$4) to stabilize reward estimates.
2. Increase total search budget ($z_0$5) to $z_0$6–$z_0$7 for coverage.
3. Scale total DDIM steps $z_0$8 only when additional compute is available beyond the above.

Robust hyperparameters are $z_0$9–$c$0, $c$1–$c$2 (for $c$3), $c$4, and DDIM stochasticity $c$5. Default guidance scale ($c$6) and reward lookahead weight ($c$7) are stable across tasks.

7. Relationship to Related Beam Search Variants

DLBS represents a static-beam-width, static-tree-width beam search with deterministic lookahead reward evaluation. In contrast, algorithms such as Dynamic Search for Diffusion (DSearch) introduce dynamic beam-width scheduling, adaptive search-timestep selection, and multi-particle lookahead to further improve search efficiency and reward maximization (Li et al., 3 Mar 2025). Other work, such as BeamDiffusion for image sequences, employs cross-attention–based scoring and beam search at the sequence level, optimizing for joint consistency across image sets (Fernandes et al., 26 Mar 2025). DLBS’s efficiency and calibration approach make it a general-purpose inference-time optimizer for conditional generation in diffusion models.

---

References:

• "Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Search" (Oshima et al., 31 Jan 2025)
• "Dynamic Search for Inference-Time Alignment in Diffusion Models" (Li et al., 3 Mar 2025)
• "Latent Beam Diffusion Models for Decoding Image Sequences" (Fernandes et al., 26 Mar 2025)