AVP-RPO: Video-to-Audio Preference Optimization

• The paper introduces AVP-RPO, a post-training alignment stage that adapts Direct Preference Optimization to conditional flow matching diffusion models for video-to-audio generation.
• It leverages SF-CAVP as a ranking reward model to score generated audios and construct winner–loser pairs, optimizing a normalized combination of flow-matching losses.
• Empirical evaluations reveal significant improvements in semantic-temporal alignment, audio quality, and distribution matching in multi-event video scenarios.

Searching arXiv for the cited papers to ground the article in current research. to=arxiv_search.search 娱乐开号անչanted ＿老司机_json {"query":"(Yang et al., 24 Sep 2025) OR (Cao et al., 29 Sep 2025) OR (Liao et al., 2024) OR (Sun et al., 31 Jan 2025) OR (Oi et al., 10 Feb 2026) OR (Zhang et al., 14 Feb 2025)","max_results":10,"sort_by":"relevance"} _日本毛片免费视频观看 to=arxiv_search.search code 天天中彩票公司_json {"query":"(Yang et al., 24 Sep 2025)", "max_results": 5, "sort_by": "relevance"} 神彩争霸 to=arxiv_search.search code 大发快三怎么json {"query":"(Cao et al., 29 Sep 2025)", "max_results": 5, "sort_by": "relevance"} {Jsii to=arxiv_search.search code 】!【json {"query":"(Liao et al., 2024)", "max_results": 5, "sort_by": "relevance"} I found the core papers and will use them as the basis for the article, especially "MultiSoundGen: Video-to-Audio Generation for Multi-Event Scenarios via SlowFast Contrastive Audio-Visual Pretraining and Direct Preference Optimization" (Yang et al., 24 Sep 2025), which introduces AVP-RPO, together with related alignment frameworks and ranking-based preference optimization methods (Cao et al., 29 Sep 2025, Liao et al., 2024, Sun et al., 31 Jan 2025, Oi et al., 10 Feb 2026, Zhang et al., 14 Feb 2025). AVP-Ranked Preference Optimization (AVP-RPO) is a post-training alignment stage for video-to-audio (V2A) generation introduced in the MultiSoundGen framework for multi-event video scenarios. It adapts Direct Preference Optimization (DPO) to conditional flow matching (CFM) diffusion models, uses SlowFast Contrastive Audio-Visual Pretraining (SF-CAVP) as a reward/scoring model, ranks generated audios for each video using SF-CAVP scores, constructs winner–loser preference pairs, and optimizes a normalized combination of a DPO-style flow-matching loss and a winner-anchoring flow-matching loss. In the published formulation, AVP-RPO is designed to improve semantic-temporal alignment, audio quality, and distribution matching while preserving the pretrained generator’s generative quality through normalization and partial fine-tuning (Yang et al., 24 Sep 2025).

1. Definition and system role

AVP-RPO is presented as the paper’s adaptation of DPO to the V2A domain, specialized for multi-event video scenarios involving multiple sound sources, sound events, or transitions. Its stated purpose is to improve semantic-temporal alignment between video and generated audio, as well as audio quality and distribution matching, after pretraining a base V2A generator, MMAudio (Yang et al., 24 Sep 2025).

The method is structurally inseparable from SF-CAVP. SF-CAVP serves as the reward model in AVP-RPO, and AVP-RPO serves as the alignment mechanism that injects SF-CAVP’s audio-visual alignment signals into the generator. The authors motivate this design by noting that existing V2A methods rely heavily on text-mediated pretraining such as CLIP and CLAP, which lack fine temporal synchronization and multi-event expressiveness, and that prior DPO-style methods in audio generation are language–audio rather than video–audio (Yang et al., 24 Sep 2025).

Component	Role	Specification
SF-CAVP	Reward/scoring model	SlowFast contrastive audio-visual pretraining model
Ranking stage	Preference construction	Rank generated audios by SF-CAVP score
AVP-RPO objective	Post-training alignment	Normalized sum of DPO-FM and FM-win

A central point is that the ranking in AVP-RPO is used to construct preference pairs rather than to define a full listwise objective. For each video, multiple generated audios are scored and ranked, but the optimization stage uses a DPO-style winner–loser formulation adapted to CFM diffusion models, with the winner fixed to the ground-truth audio and the loser chosen as the lowest-scoring generated audio (Yang et al., 24 Sep 2025).

2. Objective formulation and relation to DPO

AVP-RPO starts from a DPO-Diffusion formulation made compatible with CFM. In the underlying CFM model, the interpolation and target velocity are

$x_t = t\,x_1 + (1 - t)\,x_0,$

and

$u_t = x_1 - x_0,$

with the CFM objective

$$\mathbb{E}_{t,\,g(x),\,q(x),\,C} \,\bigl\lVert v_\theta(t, C, x_t) - u_t \bigr\rVert_2^2.$$

AVP-RPO then defines preference optimization in terms of winner and loser samples under this flow-matching parameterization (Yang et al., 24 Sep 2025).

The winner and loser are defined asymmetrically. The winner $x^{w}$ is the ground-truth audio paired with the video. The loser $x^{\ell}$ is the lowest-scoring generated audio for that video under SF-CAVP ranking. This differs from formulations that use the highest-scoring generated sample as the winner; the paper explicitly states that using ground truth as winner is a critical conceptual difference and that ablation shows using the highest-scoring generated sample as winner yields weaker improvements and IB-score deterioration (Yang et al., 24 Sep 2025).

The DPO-style term is the DPO-FM loss, described in the paper as comprising winning loss, losing loss, winning reference loss, and losing reference loss under a logistic sigmoid. It encourages the new model to reduce the winning loss relative to the losing loss more than the reference model does. AVP-RPO then adds an explicit winner-only flow-matching term,

$$L_{\text{FM-win}} = \mathbb{E}_{t,x^{w},C} \bigl\lVert v_\theta(t, C, x^{w}) - u^{w} \bigr\rVert_2^2,$$

and combines both terms through per-batch normalization:

$$L_{\text{AVP-RPO}} = \mathcal{N}\!\bigl(L_{\text{DPO-FM}}\bigr) + \mathcal{N}\!\bigl(L_{\text{FM-win}}\bigr).$$

Here $\mathcal{N}(\cdot)$ rescales each term to $[0,1]$ per batch (Yang et al., 24 Sep 2025).

This formulation retains the standard DPO idea of anchoring to a reference model, but the anchoring is implicit rather than expressed as an explicit KL term. The paper states that KL regularization is implicit in the use of reference-model losses inside $L_{\text{DPO-FM}}$, and that AVP-RPO follows the standard DPO spirit of staying close to a reference policy without introducing a separate KL penalty (Yang et al., 24 Sep 2025).

3. SF-CAVP as reward model and ranked signal source

SF-CAVP is an audio–video contrastive encoder with a unified SlowFast design for both modalities. The video encoder uses a ResNet-50 backbone with a slow pathway that samples fewer frames with stride $u_t = x_1 - x_0,$0 and high channel capacity $u_t = x_1 - x_0,$1, and a fast pathway with higher temporal resolution and lower channel capacity. The audio encoder applies the same SlowFast design to log-mel spectrograms. After global pooling and concatenation, each segment representation is 2304-dimensional for both visual and audio streams (Yang et al., 24 Sep 2025).

The SF-CAVP objective is segment-level InfoNCE contrastive learning. Positive pairs are audio and visual segments from the same video and same time interval. Negative pairs include segments from different time intervals within the same video and segments from other videos in the batch. The directional loss from audio to video is

$u_t = x_1 - x_0,$2

with the reverse $u_t = x_1 - x_0,$3 defined analogously, and the full loss

$u_t = x_1 - x_0,$4

After training on VGGSound, the resulting embeddings are used to score alignment between candidate audio and video segments (Yang et al., 24 Sep 2025).

AVP-RPO derives its ranked signal from SF-CAVP through segment-wise cosine similarities. For a video $u_t = x_1 - x_0,$5 and candidate audio $u_t = x_1 - x_0,$6, both are split into $u_t = x_1 - x_0,$7 aligned segments, segment embeddings are computed, and cosine similarities are obtained for each segment:

$u_t = x_1 - x_0,$8

These similarities are sorted, and the final score for candidate audio $u_t = x_1 - x_0,$9 is defined as the mean of the lowest quarter:

$$\mathbb{E}_{t,\,g(x),\,q(x),\,C} \,\bigl\lVert v_\theta(t, C, x_t) - u_t \bigr\rVert_2^2.$$0

The use of the lowest quarter is a deliberate design choice intended to penalize poorly aligned segments and increase discriminability between candidates in multi-event scenes (Yang et al., 24 Sep 2025).

The ranking stage then orders generated audios by $$\mathbb{E}_{t,\,g(x),\,q(x),\,C} \,\bigl\lVert v_\theta(t, C, x_t) - u_t \bigr\rVert_2^2.$$1. The lowest-scoring generated audio becomes the loser. The winner is the ground-truth audio rather than the highest-scoring generation. This makes the ranked stage a mechanism for pair construction rather than a full listwise optimization procedure (Yang et al., 24 Sep 2025).

4. Training procedure and implementation regime

AVP-RPO training combines a pretrained reward model, a pretrained generator, iterative candidate generation, preference construction, and partial parameter updates. SF-CAVP is trained on VGGSound, while AVP-RPO fine-tuning uses VGG-Sound Source (VGG-SS), with 4.4k video–audio pairs for training, 120 for validation, and 500 for test. The base generator is MMAudio, described as a multimodal diffusion transformer with a CFM objective (Yang et al., 24 Sep 2025).

The iterative loop operates as follows. For each training video, the current generator samples $$\mathbb{E}_{t,\,g(x),\,q(x),\,C} \,\bigl\lVert v_\theta(t, C, x_t) - u_t \bigr\rVert_2^2.$$2 audio candidates. SF-CAVP scores each candidate, the lowest-scoring candidate is selected as loser, the paired ground-truth audio is selected as winner, and AVP-RPO fine-tuning is then performed using $$\mathbb{E}_{t,\,g(x),\,q(x),\,C} \,\bigl\lVert v_\theta(t, C, x_t) - u_t \bigr\rVert_2^2.$$3 and $$\mathbb{E}_{t,\,g(x),\,q(x),\,C} \,\bigl\lVert v_\theta(t, C, x_t) - u_t \bigr\rVert_2^2.$$4. Each iteration runs for 1000 training steps, and the paper reports five iterations overall, while also stating that the first iteration already gives good improvements before metric saturation (Yang et al., 24 Sep 2025).

The optimization setup is explicitly specified: learning rate $$\mathbb{E}_{t,\,g(x),\,q(x),\,C} \,\bigl\lVert v_\theta(t, C, x_t) - u_t \bigr\rVert_2^2.$$5, weight decay $$\mathbb{E}_{t,\,g(x),\,q(x),\,C} \,\bigl\lVert v_\theta(t, C, x_t) - u_t \bigr\rVert_2^2.$$6, linear warmup over the first 100 steps followed by cosine annealing, gradient accumulation of 2, gradient clipping with norm capped at 1.0, hardware consisting of 2 × NVIDIA H800, and a model size of 157M parameters (Yang et al., 24 Sep 2025).

A decisive implementation detail is the partial fine-tuning strategy. The authors state that full fine-tuning of MM-DiT leads to severe degradation, including noise in outputs and metric drops. Instead, they freeze earlier single-modal transformer blocks and lower multimodal transformer blocks, and fine-tune the last single-modal transformer layer for the audio stream, Adaptive LayerNorm layers, and 1D convolution layers. The stated purpose is to preserve pretrained diffusion and denoising mechanisms while allowing AVP-RPO to reshape final audio latents and temporal alignment (Yang et al., 24 Sep 2025).

5. Empirical behavior, ablations, and reported gains

The empirical claim made for AVP-RPO is that MultiSoundGen achieves state-of-the-art performance in multi-event scenarios across distribution matching, audio quality, semantic alignment, and temporal synchronization. On the multi-event subset VGG-SS-M, distribution matching measured by FD improves by up to 10.3% versus base MMAudio, IS increases from 6.295 to 6.354 on multi-event, IB-score improves slightly from 0.342 to 0.343, and DeSync improves by up to 5.3%, where lower is better (Yang et al., 24 Sep 2025).

The paper also reports that in single-event VGG-SS-S and out-of-distribution Kling-Audio-Eval, the method maintains top-2 performance across most metrics. This is presented as evidence that the alignment procedure does not merely overfit to multi-event distributions (Yang et al., 24 Sep 2025).

The ablation results isolate three design choices. First, SF-CAVP matters as a reward model: replacing it with Segment AVCLIP causes degradation across iterations in FD, IS, IB, and DeSync. Second, the full AVP-RPO objective matters: using only $$\mathbb{E}_{t,\,g(x),\,q(x),\,C} \,\bigl\lVert v_\theta(t, C, x_t) - u_t \bigr\rVert_2^2.$$7 yields some improvements but is inferior to the normalized sum with $$\mathbb{E}_{t,\,g(x),\,q(x),\,C} \,\bigl\lVert v_\theta(t, C, x_t) - u_t \bigr\rVert_2^2.$$8, with the added term improving all metrics except DeSync, where it is on par. Third, the freeze-bottom/optimize-top fine-tuning strategy is described as essential, because full fine-tuning drastically harms performance (Yang et al., 24 Sep 2025).

The paper’s conceptual reading of these results is that AVP-RPO improves multi-event V2A generation through three coupled mechanisms: segment-wise SlowFast alignment via SF-CAVP, preference optimization anchored to ground truth, and preservation of global quality through FM-win and partial fine-tuning. This suggests that the method is not simply ranking outputs by an external reward model; it is using that ranking signal under architectural and optimization constraints meant to preserve the pretrained generator’s distribution (Yang et al., 24 Sep 2025).

6. Position within ranked preference optimization research

AVP-RPO belongs to a broader movement away from binary, response-level preference optimization toward richer preference structures, but it occupies a specific point in that design space. Tree Preference Optimization (TPO), for example, formulates alignment as a Preference List Ranking problem over full preference trees, uses all trajectories and their relative rewards, and introduces a LambdaRank-style Preference Ranking Loss together with Adaptive Step Reward for multi-step reasoning (Liao et al., 2024). Relative to TPO, AVP-RPO is more narrowly pairwise at optimization time: it uses ranking to identify a loser but trains with a DPO-style winner–loser objective rather than a listwise loss over the full ranked set.

Robust Preference Optimization recasts preference optimization as a meta-framework for noisy human preference data, treating true preferences as latent variables, using an Expectation–Maximization procedure to infer posterior correctness, and reweighting training examples accordingly (Cao et al., 29 Sep 2025). This does not define AVP-RPO, but it indicates one path by which a ranked V2A alignment method could be made noise-aware if reward-model scores or preference labels were treated as uncertain. A plausible implication is that AVP-RPO’s current winner–loser construction could, in principle, be coupled to explicit uncertainty modeling rather than fixed pair selection.

Reward-Aware Preference Optimization provides a mathematical framework in which preference optimization methods can be expressed as distances between implicit and explicit reward differences, including multi-response objectives over sets of candidates (Sun et al., 31 Jan 2025). This suggests a formal lens for AVP-RPO: SF-CAVP scores play the role of explicit reward signals, while the generator induces an implicit preference structure through its flow-matching losses. The paper introducing AVP-RPO does not present it in those terms, but the connection is structurally suggestive.

Autoregressive DPO argues that introducing the autoregressive assumption before applying Bradley–Terry yields prefix-wise objectives with the summation outside the log-sigmoid, and it distinguishes token length from feedback length as separate design variables (Oi et al., 10 Feb 2026). That discussion is centered on autoregressive LLMs rather than diffusion-based V2A generation, yet it sharpens a general distinction between response-level and finer-grained preference supervision. AVP-RPO currently applies preference optimization at the sample level, while SF-CAVP’s segment-level scoring already provides a finer temporal signal on the reward-model side.

A separate line of analysis argues that grounded evaluation of preference learning should be understood through win rate, and distinguishes win-rate optimization from non-win-rate-optimization objectives such as DPO and SFT on preferred samples (Zhang et al., 14 Feb 2025). AVP-RPO is formulated as a DPO-style diffusion objective rather than as a direct win-rate objective. This suggests that, within that framework, its practical success should be interpreted primarily through its surrogate optimization design, reward-model quality, and stability mechanisms rather than through a claim of direct win-rate optimality.

The limitations acknowledged for AVP-RPO are correspondingly specific. SF-CAVP operates on pooled features per segment and does not yet leverage fine-grained temporal matching at frame or sub-segment level. AVP-RPO uses a relatively small dataset for alignment, metrics may saturate or fluctuate across iterations, the method depends on the quality and bias of SF-CAVP, and training requires repeated candidate generation and scoring. The paper identifies more fine-grained AVP, scaling to larger models, and richer reward signals as future directions (Yang et al., 24 Sep 2025).