Rerank-DPO: Ranking-Aware Preference Optimization

• Rerank-DPO is a family of methods that extends traditional DPO by integrating ranking signals from candidate lists, external scores, and sequence-level feedback.
• It transforms pairwise preference losses into listwise, ranking-aware objectives that directly optimize metrics like NDCG to improve model ranking performance.
• Applications span in-context ranking for LLMs, text-to-image synthesis, and recommendation systems, demonstrating measurable gains over standard approaches.

Rerank-DPO is an informal label for preference-optimization methods that use reranking signals—candidate lists, external reranker scores, listwise judges, or sequence-level feedback—to train a policy or scoring model. In current literature, the term does not denote a single canonical algorithm. Instead, it refers to a family of DPO-derived procedures that replace or augment pairwise chosen-versus-rejected supervision with ranking-aware supervision, including listwise in-context ranking for LLMs, reranker-supervised query generation, ranking-based DPO for text-to-image models, and sequence-level post-training for generative recommendation rerankers (Wu et al., 21 Apr 2025, Coelho et al., 25 May 2025, Karthik et al., 2024, Lin et al., 29 Oct 2025).

1. Definition and scope

In its narrowest sense, Rerank-DPO denotes a DPO-style objective applied to a reranking problem: a model receives a query or context together with a candidate set, and training encourages the model to assign higher probability or score to the better-ranked candidates or sequences. In a broader and more common usage, it describes any preference-optimization pipeline in which reranking signals are the source of supervision rather than raw human pairwise labels.

The literature uses this idea in several distinct ways. In "In-context Ranking Preference Optimization," the term is an informal description of IRPO, which generalizes DPO from one preferred/dispreferred pair to a full ranked list and optimizes listwise metrics such as NDCG by aggregating positional pairwise preferences within a single forward pass (Wu et al., 21 Apr 2025). In web query generation, "re-rank then DPO" describes a pipeline in which an external cross-encoder or GPT-3.5 judge ranks synthetic queries for a document, and those rankings are converted into DPO preference pairs for the generator (Coelho et al., 25 May 2025). In text-to-image generation, "Rerank-DPO" corresponds precisely to RankDPO, a ranking-based extension of diffusion DPO trained on ranked synthetic image sets (Karthik et al., 2024). In recommendation, GReF uses "Rerank-DPO" for sequence-level post-training of a generative reranker from exposure logs and user feedback (Lin et al., 29 Oct 2025).

Related methods broaden the same design space even when they do not use the exact term. RankPO adapts DPO to bi-encoder similarity scoring for job–talent matching (Zhang et al., 13 Mar 2025). RaCT and its Chain DPO stage optimize chain-of-thought ranking sequences rather than plain responses (Liu et al., 2024). DRPO treats alignment explicitly as a learning-to-rank problem and replaces pairwise preference loss with differentiable NDCG optimization (Zhou et al., 2024). A plausible unifying description is that Rerank-DPO is DPO-style alignment under ranking supervision, with the ranked list, reranked candidate pool, or sequence order as the primary training object.

2. From pairwise DPO to reranking objectives

Standard DPO begins with a Bradley–Terry preference model over pairs and an implicit reward of the form

$$r(x,y)=\beta \log \frac{\pi_\theta(y\mid x)}{\pi_{\text{ref}}(y\mid x)} + \text{constant},$$

which yields the familiar pairwise objective

$$\mathcal{L}_{\text{DPO}} = -\mathbb{E}\left[ \log \sigma\left( \beta \log \frac{\pi_\theta(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)} - \beta \log \frac{\pi_\theta(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)} \right) \right].$$

This form is retained exactly in reranker-supervised query generation, where the only novelty is how the pairs $(q^+,q^-)$ are constructed from reranker judgments rather than direct human comparisons (Coelho et al., 25 May 2025).

Listwise Rerank-DPO variants modify the training object itself. IRPO treats a ranked list as the basic object and defines a positional preference probability for the item placed at rank $i$, followed by DCG-style positional aggregation. Its loss is

$$\mathcal{L}_{\text{IRPO}} = -\mathbb{E}\left[ \sum_{i=1}^{n} w(i)\log \sigma(z_i) \right], \qquad w(i)=\frac{2^{y_{\tau(i)}-1}}{\log_2(1+i)},$$

where each $z_i$ is a log-sum-exp margin against all competing items in the list (Wu et al., 21 Apr 2025). This retains DPO’s implicit reward construction but replaces a single pairwise margin by a position-weighted, listwise surrogate aligned with NDCG and related ranking metrics.

RankDPO for text-to-image models keeps the pairwise diffusion-DPO inner term but weights every pair $(i,j)$ by the DCG change induced by swapping their ranked positions: $$\Delta_{i,j} = |G_i-G_j| \cdot \left| \frac{1}{D(\tau(i))}-\frac{1}{D(\tau(j))} \right|.$$ The resulting loss is a LambdaLoss-style, listwise generalization of diffusion DPO (Karthik et al., 2024). DRPO pushes the same logic further by discarding pairwise logistic alignment as the primary target and directly optimizing a differentiable approximation of NDCG through a sorting network (Zhou et al., 2024).

A separate trajectory is sequence-level reranking. In GReF, the “responses” in DPO are entire recommendation sequences $Y_w$ and $Y_l$, and the sequence probability under the autoregressive reranker replaces response probability: $$\mathcal{L}_{\text{DPO}} = -\mathbb{E}\left[ \log \sigma\left( \beta \log \frac{\pi_\theta(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)} - \beta \log \frac{\pi_\theta(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)} \right) \right].$$0 Here reranking is not over isolated items but over permutations of exposed items (Lin et al., 29 Oct 2025).

3. Supervision and data construction

Rerank-DPO methods differ most sharply in how they construct ranking supervision. One common pattern is reranker-induced pair construction. For web query generation, a baseline generator produces $$\mathcal{L}_{\text{DPO}} = -\mathbb{E}\left[ \log \sigma\left( \beta \log \frac{\pi_\theta(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)} - \beta \log \frac{\pi_\theta(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)} \right) \right].$$1 queries per document, and a pointwise cross-encoder reranker, bge-reranker-v2-m3, or a listwise GPT-3.5 judge selects better and worse queries. Those judgments are then converted into a DPO preference dataset $$\mathcal{L}_{\text{DPO}} = -\mathbb{E}\left[ \log \sigma\left( \beta \log \frac{\pi_\theta(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)} - \beta \log \frac{\pi_\theta(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)} \right) \right].$$2 or $$\mathcal{L}_{\text{DPO}} = -\mathbb{E}\left[ \log \sigma\left( \beta \log \frac{\pi_\theta(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)} - \beta \log \frac{\pi_\theta(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)} \right) \right].$$3 over $$\mathcal{L}_{\text{DPO}} = -\mathbb{E}\left[ \log \sigma\left( \beta \log \frac{\pi_\theta(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)} - \beta \log \frac{\pi_\theta(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)} \right) \right].$$4 triples (Coelho et al., 25 May 2025).

A second pattern is listwise relevance annotation. IRPO assumes an in-context ranking scenario with a query $$\mathcal{L}_{\text{DPO}} = -\mathbb{E}\left[ \log \sigma\left( \beta \log \frac{\pi_\theta(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)} - \beta \log \frac{\pi_\theta(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)} \right) \right].$$5, a candidate set $$\mathcal{L}_{\text{DPO}} = -\mathbb{E}\left[ \log \sigma\left( \beta \log \frac{\pi_\theta(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)} - \beta \log \frac{\pi_\theta(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)} \right) \right].$$6, and sparse relevance labels $$\mathcal{L}_{\text{DPO}} = -\mathbb{E}\left[ \log \sigma\left( \beta \log \frac{\pi_\theta(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)} - \beta \log \frac{\pi_\theta(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)} \right) \right].$$7. It treats each ranked list as one training instance and uses graded or binary labels such as: conversational recommendation with labels $$\mathcal{L}_{\text{DPO}} = -\mathbb{E}\left[ \log \sigma\left( \beta \log \frac{\pi_\theta(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)} - \beta \log \frac{\pi_\theta(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)} \right) \right].$$8, generative retrieval with supporting contexts labeled $$\mathcal{L}_{\text{DPO}} = -\mathbb{E}\left[ \log \sigma\left( \beta \log \frac{\pi_\theta(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)} - \beta \log \frac{\pi_\theta(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)} \right) \right].$$9 and distractors $(q^+,q^-)$0, and multiple-choice QA reranking with the correct answer labeled $(q^+,q^-)$1 and others $(q^+,q^-)$2 (Wu et al., 21 Apr 2025).

A third pattern is synthetic ranked-set generation. RankDPO builds Syn-Pic by generating images for each prompt from multiple text-to-image models, scoring them with five reward models, aggregating pairwise wins into

$(q^+,q^-)$3

and sorting by $(q^+,q^-)$4 to obtain a ranked list. The gains $(q^+,q^-)$5 then feed the RankDPO weights (Karthik et al., 2024).

A fourth pattern is logged sequence reconstruction. In GReF, a logged exposure sequence $(q^+,q^-)$6 is transformed into a preferred sequence $(q^+,q^-)$7 by sorting items with the personalization score

$(q^+,q^-)$8

where $(q^+,q^-)$9 is the exposure position and $i$0 is user feedback. The original exposure order becomes the less-preferred sequence $i$1 (Lin et al., 29 Oct 2025).

Other ranking-oriented adaptations use narrower supervision. RankPO samples candidate pairs from the top-100 retrieved profiles for a job description and asks DeepSeek-V3 to choose the better match, yielding AI-curated pairwise preferences for a bi-encoder reranker (Zhang et al., 13 Mar 2025). RaCT generates chain-of-thought ranking traces and defines chosen and rejected continuations by how long they overlap with teacher rankings before divergence (Liu et al., 2024). In video–text alignment, ranking supervision is generated by progressively degrading captions so that each later caption introduces exactly one additional error, yielding totally ordered caption chains (Blume et al., 26 Mar 2026).

4. Representative formulations and reported results

Domain	Mechanism	Selected result
In-context LLM ranking	IRPO listwise objective	CommonsenseQA with Llama3: NDCG@1 = 68.6, Recall@5 = 97.5
Web query generation	Ranker-DPO / GPT-DPO	MS MARCO Doc MRR@100 = 0.3727 / 0.3795
Text-to-image	RankDPO on SDXL	DPG-Bench: DSG 79.26, VQAScore 87.52, Q-Align 0.81
Recommendation reranking	GReF post-training via Rerank-DPO	AUC 0.7387, NDCG 0.7498
Job–talent matching	RankPO	alignment 0.647 with nDCG@20 = 0.367
Video–text alignment	PL ranking over caption chains	PE-Video: Rel 5.14, Descr 4.69, Temp Cons 5.42, Flu 8.69

IRPO reports gains specifically on top-of-list retrieval behavior. On CommonsenseQA, Llama3 with IRPO reaches NDCG@1 $i$2 and Recall@5 $i$3, and the paper states that IRPO consistently outperforms SFT, standard pairwise DPO, and S-DPO across conversational recommendation, generative retrieval, and QA reranking (Wu et al., 21 Apr 2025).

In query generation for dense retrieval, Ranker-DPO and GPT-DPO produce retrievers that outperform all baselines on MS MARCO Document Retrieval, MS MARCO Passage Retrieval, and TREC-DL19. The main table reports MARCO Doc MRR@100 of $i$4 for Ranker-DPO and $i$5 for GPT-DPO, with MARCO Passage nDCG@10 of $i$6 and $i$7, respectively (Coelho et al., 25 May 2025).

For text-to-image generation, RankDPO improves both prompt following and judged visual quality. On DPG-Bench, SDXL rises from DSG $i$8, VQAScore $i$9, and Q-Align $$\mathcal{L}_{\text{IRPO}} = -\mathbb{E}\left[ \sum_{i=1}^{n} w(i)\log \sigma(z_i) \right], \qquad w(i)=\frac{2^{y_{\tau(i)}-1}}{\log_2(1+i)},$$0 to $$\mathcal{L}_{\text{IRPO}} = -\mathbb{E}\left[ \sum_{i=1}^{n} w(i)\log \sigma(z_i) \right], \qquad w(i)=\frac{2^{y_{\tau(i)}-1}}{\log_2(1+i)},$$1, $$\mathcal{L}_{\text{IRPO}} = -\mathbb{E}\left[ \sum_{i=1}^{n} w(i)\log \sigma(z_i) \right], \qquad w(i)=\frac{2^{y_{\tau(i)}-1}}{\log_2(1+i)},$$2, and $$\mathcal{L}_{\text{IRPO}} = -\mathbb{E}\left[ \sum_{i=1}^{n} w(i)\log \sigma(z_i) \right], \qquad w(i)=\frac{2^{y_{\tau(i)}-1}}{\log_2(1+i)},$$3; on GenEval, SDXL mean score rises from $$\mathcal{L}_{\text{IRPO}} = -\mathbb{E}\left[ \sum_{i=1}^{n} w(i)\log \sigma(z_i) \right], \qquad w(i)=\frac{2^{y_{\tau(i)}-1}}{\log_2(1+i)},$$4 to $$\mathcal{L}_{\text{IRPO}} = -\mathbb{E}\left[ \sum_{i=1}^{n} w(i)\log \sigma(z_i) \right], \qquad w(i)=\frac{2^{y_{\tau(i)}-1}}{\log_2(1+i)},$$5 (Karthik et al., 2024).

In recommendation, GReF isolates the contribution of Rerank-DPO through ablation. On Kuaishou, pre-training only yields AUC $$\mathcal{L}_{\text{IRPO}} = -\mathbb{E}\left[ \sum_{i=1}^{n} w(i)\log \sigma(z_i) \right], \qquad w(i)=\frac{2^{y_{\tau(i)}-1}}{\log_2(1+i)},$$6 and NDCG $$\mathcal{L}_{\text{IRPO}} = -\mathbb{E}\left[ \sum_{i=1}^{n} w(i)\log \sigma(z_i) \right], \qquad w(i)=\frac{2^{y_{\tau(i)}-1}}{\log_2(1+i)},$$7, post-training only yields $$\mathcal{L}_{\text{IRPO}} = -\mathbb{E}\left[ \sum_{i=1}^{n} w(i)\log \sigma(z_i) \right], \qquad w(i)=\frac{2^{y_{\tau(i)}-1}}{\log_2(1+i)},$$8 and $$\mathcal{L}_{\text{IRPO}} = -\mathbb{E}\left[ \sum_{i=1}^{n} w(i)\log \sigma(z_i) \right], \qquad w(i)=\frac{2^{y_{\tau(i)}-1}}{\log_2(1+i)},$$9, and pre-training plus post-training yields $z_i$0 and $z_i$1. This shows a small but measurable gain from Rerank-DPO on top of a strong pretrained generator, while also showing that post-training alone is inadequate (Lin et al., 29 Oct 2025).

RankPO provides a reranking-specific DPO adaptation for bi-encoders. The paper emphasizes the trade-off between alignment with AI preferences and preservation of rule-based retrieval quality: at alignment $z_i$2, RankPO with sigmoid loss reports nDCG@20 $z_i$3, whereas SFT at similar alignment $z_i$4 reports nDCG@20 $z_i$5 (Zhang et al., 13 Mar 2025).

In video–text alignment, listwise ranking over caption chains outperforms binary DPO. On PE-Video with PerceptionLM under GPT-OSS judging, Ranking reports Relevance $z_i$6, Descriptiveness $z_i$7, Temporal Consistency $z_i$8, and Fluency $z_i$9, versus DPO at $(i,j)$0, $(i,j)$1, $(i,j)$2, and $(i,j)$3 (Blume et al., 26 Mar 2026).

5. Optimization pathologies and related corrections

A substantial body of recent work argues that pairwise DPO can behave poorly even before listwise reranking is introduced. BPO attributes this to Degraded Chosen Responses, the possibility that DPO increases the relative margin $(i,j)$4 while still decreasing the chosen likelihood. It replaces the relative margin by the balanced reward margin

$(i,j)$5

and reports gains such as $(i,j)$6 on Llama-3.1-8B-Instruct and $(i,j)$7 on Qwen2.5-Math-7B (Sun et al., 4 Jun 2025). This suggests that reranking-oriented DPO methods may need explicit control over absolute reward levels, not only order.

BDPO makes a related diagnosis: rejected responses can dominate the DPO gradient because the derivative with respect to $(i,j)$8 scales as $(i,j)$9. Its remedy is to replace the rejected probability by the mixture

$$\Delta_{i,j} = |G_i-G_j| \cdot \left| \frac{1}{D(\tau(i))}-\frac{1}{D(\tau(j))} \right|.$$0

which bounds negative influence while preserving the DPO structure (Cho et al., 15 Jun 2025). For reranking with many negatives, this directly addresses the tendency to optimize chiefly by crushing losers.

FocalPO departs in another direction. It argues that DPO emphasizes misranked pairs but rarely corrects them, and proposes

$$\Delta_{i,j} = |G_i-G_j| \cdot \left| \frac{1}{D(\tau(i))}-\frac{1}{D(\tau(j))} \right|.$$1

to down-weight heavily misranked pairs and emphasize pairs already ranked correctly. On AlpacaEval 2.0, it reports length-controlled win rate $$\Delta_{i,j} = |G_i-G_j| \cdot \left| \frac{1}{D(\tau(i))}-\frac{1}{D(\tau(j))} \right|.$$2 versus DPO’s $$\Delta_{i,j} = |G_i-G_j| \cdot \left| \frac{1}{D(\tau(i))}-\frac{1}{D(\tau(j))} \right|.$$3 for Llama-3-Instruct-8B (Liu et al., 11 Jan 2025). A plausible implication is that Rerank-DPO systems should not assume that hard-negative emphasis is always desirable.

DRPO generalizes the critique from pairwise optimization to ranking accuracy itself. It reports that current preference alignment techniques underperform expectations, with ranking accuracies below $$\Delta_{i,j} = |G_i-G_j| \cdot \left| \frac{1}{D(\tau(i))}-\frac{1}{D(\tau(j))} \right|.$$4 on standard datasets, and replaces pairwise DPO with diffNDCG plus an Adaptive Rank Policy Score (Zhou et al., 2024). In this sense, DRPO functions as a direct learning-to-rank alternative to pairwise Rerank-DPO.

6. Limitations and open problems

Rerank-DPO methods inherit a common structural requirement: they need candidate sets and some form of ranking signal. IRPO assumes that ranked lists can be formed and that at least sparse relevance labels are available; it also notes that the listwise log-sum-exp over all items induces $$\Delta_{i,j} = |G_i-G_j| \cdot \left| \frac{1}{D(\tau(i))}-\frac{1}{D(\tau(j))} \right|.$$5 interactions per list, with importance sampling offered as a mitigation (Wu et al., 21 Apr 2025). Query-generation Rerank-DPO depends on the quality of the reranker or GPT judge, remains single-document conditioned, and improves retrieval only indirectly because NDCG or MRR are not directly optimized during generator training (Coelho et al., 25 May 2025).

Multimodal variants add further dependencies. RankDPO relies entirely on off-the-shelf reward models for labeling, uses only prompts from Pick-a-Pic v2, and still requires multiple large text-to-image models plus multiple reward models to build Syn-Pic (Karthik et al., 2024). In video–text alignment, ranking optimization outperforms binary DPO, but the results also show that finetuning of the vision encoder is required, which challenges the view of DPO as purely a language-reweighting process (Blume et al., 26 Mar 2026).

System-level Rerank-DPO also remains sensitive to initialization and pipeline design. GReF shows that post-training only with Rerank-DPO performs poorly relative to pre-training plus post-training, so the method functions best as a refinement stage atop a strong generative reranker rather than as a standalone learning principle (Lin et al., 29 Oct 2025). In academic matching, RankPO improves the trade-off between semantic alignment and rule retention but does not eliminate catastrophic forgetting; increasing alignment still degrades nDCG on the original rule-based task (Zhang et al., 13 Mar 2025).

These limitations make two open directions especially prominent. One is better coupling of generation and reranking: several papers explicitly or implicitly treat candidate generation and reranking as separate modules, and aligning them end-to-end remains unresolved (Wu et al., 21 Apr 2025, Coelho et al., 25 May 2025). The other is more faithful ranking supervision: many current systems depend on rerankers, reward models, or LLM judges whose preferences may diverge from human utility, user intent, or multimodal faithfulness (Karthik et al., 2024, Blume et al., 26 Mar 2026). As a result, Rerank-DPO is best understood not as a single stable algorithm, but as an active family of ranking-aware preference-optimization methods whose central design question is how to convert richer listwise feedback into robust policy updates.