SMC-ITA: Sequential Monte Carlo Inference-Time Alignment for Video-to-Audio Generation

Abstract: Video-to-audio (V2A) generation must jointly satisfy audiovisual alignment, semantic consistency, temporal synchronization, and perceptual quality. While prior work has mainly focused on model architecture, multimodal conditioning, and training objectives, inference-time alignment for V2A remains underexplored. In this paper, we study inference-time alignment for flow-matching-based V2A generation and formulate it as a search problem. We propose Sequential Monte Carlo Inference-Time Alignment (SMC-ITA), which combines lookahead-based reward estimation and sequential Monte Carlo resampling to reallocate computation adaptively using multi-dimensional cross-modal rewards. SMC-ITA improves over naive single-trajectory sampling, achieving a 55.67% relative reduction in DeSync, a 20.23% improvement in IB-score, and a 15.44% improvement in Audio Quality. Under matched NFE budgets, it also achieves the best overall trade-off among the compared search baselines, outperforming Best-of-N and Beam Search. Ablation studies further show that lookahead improves the reliability of intermediate reward estimates and that systematic resampling is a strong practical default for V2A inference-time alignment.

• The paper presents a search-based approach combining Sequential Monte Carlo resampling with lookahead reward estimation to improve inference-time alignment in video-to-audio generation.
• It demonstrates significant performance gains, including a 55.67% reduction in temporal misalignment and improvements in audiovisual and quality metrics over baseline methods.
• The method adapts computational resources effectively through diversity preservation and multi-dimensional reward optimization, suggesting potential applications in other cross-modal generation tasks.

SMC-ITA: Sequential Monte Carlo Inference-Time Alignment for Video-to-Audio Generation

Problem Formulation and Motivation

Video-to-audio (V2A) generation is tasked with synthesizing high-quality, temporally synchronized, and semantically aligned audio from silent video input, optionally conditioned on text. While architectural and training advances have been extensively explored in prior works, inference-time alignment—specifically during the generative process—remains notably under-addressed. Flow-matching-based generative models (e.g., conditional flow matching, CFM [lipman2023flow]) face significant challenges in optimizing for multiple cross-modal constraints at inference. The search space is vast and rewards at initial timesteps are noisy, making trajectory selection both difficult and crucial for desirable outcomes.

SMC-ITA Methodology

SMC-ITA addresses inference-time alignment via a search-based approach, combining Sequential Monte Carlo (SMC) resampling with lookahead-based reward estimation to steer V2A generation trajectories. The method maintains a population of audio trajectories, introducing diversity through an SDE solver (as opposed to deterministic ODE), and periodically intervenes using predefined search steps. At each intervention, lookahead is used to perform fast rollouts toward final states, with rewards measured using a composite of semantic, audiovisual, temporal, and perceptual metrics. High-rewarded trajectories are replicated while low-rewarded ones are pruned, reallocating the fixed sampling budget (NFE) adaptively to promising regions.

Figure 1: Overview of SMC-ITA: candidate trajectories are scored via lookahead on multi-dimensional rewards and iteratively resampled for alignment during generation.

This methodology offers a softer and more flexible alternative to hard pruning or delayed selection (as in Best-of-N or Beam Search), enabling effective search and allocation of computational resources under noisy and multi-objective reward signals.

Reward Functions and Scoring

Alignment is assessed through four normalized reward functions:

1. Semantic Reward: LAION-CLAP cosine similarity between input text and generated audio.
2. AV-align Reward: ImageBind cosine similarity between video and generated audio.
3. Temporal Reward: Synchformer-based video-audio synchronization.
4. Quality Reward: Audiobox-Aesthetics Product Quality score.

Aggregated reward is calculated via z-normalization of each function, supporting multi-dimensional reward-guided search. Ablations confirm that each reward contributes to its target metric, and combining all yields the best overall tradeoff across evaluation dimensions.

Lookahead and Resampling

Lookahead facilitates accurate reward estimation at search steps by rolling out trajectories to their final audio prediction, mitigating early-stage reward noise and yielding more predictive scoring for resampling. Systematic resampling is empirically validated as the default due to its efficiency and effectiveness in preserving diversity of high-rewarded trajectories.

Figure 2: Correlation analysis: lookahead increases the predictive power of intermediate reward estimates with respect to the final reward.

Experimental Results

SMC-ITA was evaluated using MMAudio-S-16kHz [cheng2025mmaudio] on the VGGSound test set. Under fixed NFE budgets, SMC-ITA dominated naive and baseline sampling approaches, achieving:

• 55.67% relative reduction in DeSync (temporal misalignment),
• 20.23% improvement in IB-score (AV alignment),
• 15.44% improvement in Audio Quality.

These improvements were consistent across multiple high-level perceptual, semantic, and synchronization metrics. Beam Search proved a strong baseline but its gains were limited by early pruning and lack of adaptive trajectory preservation. SMC-ITA demonstrated superior performance, especially as NFE budget scaled.

Figure 3: Pairwise human evaluation: SMC-ITA is preferred over Naive Sampling, Best-of-N, and Beam Search by raters.

Figure 4: Scaling analysis: SMC-ITA delivers consistent performance improvements as NFE budgets increase, outperforming other search-based methods.

Practical and Theoretical Implications

SMC-ITA shifts the paradigm toward reward-guided inference-time search for V2A generation, highlighting the value of adaptive allocation, trajectory diversity, and multi-dimensional reward optimization. Practically, this enables more robust, aligned audio generation without retraining or architectural changes, making it applicable to a wide range of base generative models. Theoretically, SMC-ITA demonstrates that particle filtering combined with lookahead is highly effective in multimodal generative tasks where early reward signals are unreliable and hard selection is suboptimal.

This framework is extensible to other cross-modal generation tasks (e.g., text-to-video, text-to-audio) where inference-time rewards can be formulated, and suggests future directions in learning adaptive search schedules or reward configurations, potentially leveraging reinforcement learning for further efficiency.

Conclusion

SMC-ITA establishes an effective inference-time alignment methodology for video-to-audio generation, achieving superior performance across alignment, synchronization, and perceptual quality metrics compared to prior search-based baselines. The gains are attributed to adaptive search enabled by SMC resampling and lookahead-based reward estimation, making intermediate reward signals significantly more informative. This work demonstrates the importance of inference-time search strategies for multimodal generative models, and points toward promising directions for further optimizations in computation allocation and reward design in cross-modal synthesis.