CTR-Driven Preference Optimization

• CTR-driven preference optimization is a method that converts click-through rates into explicit signals for optimizing ranking, recommendation, and generative tasks.
• It employs formulations like binary supervision, pairwise comparisons, and weighted re-scaling to mitigate issues such as sparse feedback, exposure bias, and personalization challenges.
• Applications include cold-user recommendations, search ranking, ad creative generation, and conversational query suggestions with validated performance gains.

Searching arXiv for the cited work and adjacent papers on CTR-driven preference optimization. CTR-driven preference optimization denotes a family of methods that converts click-through signals into explicit optimization targets for prediction, ranking, recommendation, or generation. In the cited literature, CTR is variously treated as a direct binary supervision signal, a pairwise preference relation, a calibrated reward, a relevance-conditioned probability, or a sample-level weight used to reshape alignment objectives. The resulting methods span cold-user recommendation, search ranking, advertising image generation, advertising text generation, conversational query suggestion, on-demand food delivery CTR prediction, and on-device assistant query recommendation, with recurring concerns around sparse feedback, exposure bias, personalization, calibration, and relevance preservation (Shen et al., 2022, Chen et al., 5 Feb 2025, Cao et al., 24 Feb 2026, Abdolmaleki et al., 2024, Min et al., 5 Jul 2025, Jiang et al., 2023, Wei et al., 2022, Chen et al., 27 Jul 2025, Luo et al., 7 Jun 2026).

1. Scope of the concept

CTR-driven preference optimization is not a single algorithmic recipe. In recommender and search systems it may appear as a predictive decomposition of click probability, a personalized incentive multiplier, or a meta-learning procedure for cold-start users. In generative systems it more often appears as pairwise preference alignment, where generated candidates are ranked by observed or predicted CTR and then used to optimize a policy with DPO, KTO, contrastive objectives, or EM-style objectives. In local-service and assistant settings it is also tied to operational constraints such as latency, grounding, and tool executability (Shen et al., 2022, Chen et al., 5 Feb 2025, Cao et al., 24 Feb 2026, Min et al., 5 Jul 2025, Luo et al., 7 Jun 2026).

Setting	CTR-derived preference object	Representative work
Cold-user CTR prediction	Shared global click propensity plus user-specific residuals	RESUS (Shen et al., 2022)
Advertising image generation	Pairwise CTR preference between two images of the same product	CAIG (Chen et al., 5 Feb 2025)
Search ranking	Relevance-conditioned click probability modulated by $\tau(x)$	PRECTR-V2 (Cao et al., 24 Feb 2026)
Conversational search	Response-level aggregated predicted CTR with diversity-aware pairs	GQS (Min et al., 5 Jul 2025)
Advertising text generation	Within-item winners from online A/B or A/B/n feedback	CREATER (Wei et al., 2022), CTOP (Chen et al., 27 Jul 2025)
On-device assistants	Dual-calibrated click logs optimized with weighted KTO	ToolRec (Luo et al., 7 Jun 2026)

This breadth has a methodological consequence: “preference” may refer to user-specific latent taste, pairwise superiority of one creative over another, calibrated click desirability under a reference policy, or a relevance incentive that reweights a final rank score. The common denominator is that CTR is used to induce an ordering or weighting over alternatives rather than serving only as a downstream metric.

2. Construction of preference signals from clicks

Several works start from the explicit definition $$\mathrm{CTR} = \frac{\text{clicks}}{\text{impressions}}$$, but differ in how that quantity is converted into usable supervision. In cold-user CTR prediction, RESUS operates at the instance level: click labels remain binary, while the residual preference target for a support instance is defined by subtracting the shared predictor’s probability, $\Delta y^S = y^S - \sigma(\Psi(x^S))$. This turns preference learning into residual correction around a global baseline. In the EM-based PMPO formulation, click is mapped to a Bernoulli success variable $S \in \{0,1\}$, so that click logs can be partitioned into accepted positives and rejected negatives and optimized either jointly or separately (Shen et al., 2022, Abdolmaleki et al., 2024).

In generative advertising, preference construction is commonly pairwise and within-item. CAIG generates two candidate backgrounds for the same product, renders two images, and uses a multimodal reward model to decide which image has higher CTR. The pairwise classifier is complemented by a pointwise CTR regression branch, but the optimization signal used for preference alignment is the within-product comparison. CTOP likewise defines a winning ad title as a generated candidate whose CTR exceeds the item’s human-crafted title in online A/B/n, then weights that pair by normalized CTR gain and a confidence term derived from a parallel AA group. CREATER uses professional editors’ alternative ad texts for the same ad context, labels the higher-CTR text as $y^+$ and the lower-CTR text as $y^-$, and filters pairs with a Z-test, insufficient impressions, or anomalously high CTR (Chen et al., 5 Feb 2025, Chen et al., 27 Jul 2025, Wei et al., 2022).

Conversational and assistant settings add further structure. GQS predicts $\hat p(q_n \mid x)$ for each suggested query and scores a full response by $r(Y)=\sum_j \hat p(q_j \mid x)$; preference pairs are then formed from high-scoring and low-scoring responses, with separate diversity-aware pairs when CTR is similar but semantic diversity differs. ToolRec uses set-level labels: a recommendation slate is positive if any displayed query is clicked and negative otherwise. Its raw click logs are then reweighted by user activity and by whether a clicked slate invokes a system tool, so the preference signal is already calibrated before optimization (Min et al., 5 Jul 2025, Luo et al., 7 Jun 2026).

Search ranking introduces a different construction. PRECTR-V2 defines relevance score levels $rsl \in \{1,2,3,4\}$ and decomposes click probability as

$$P(\mathrm{click}=1\mid x)=\sum_{i=1}^{k} P(\mathrm{click}=1\mid rsl=i, x)\cdot P(rsl=i\mid x),$$

after which a personalized relevance preference incentive $$\mathrm{CTR} = \frac{\text{clicks}}{\text{impressions}}$$0 modulates the final ranking score. Here preference is not merely “clicked versus not clicked,” but an incentive term learned from user and cross-user relevance behavior under a specific query category (Cao et al., 24 Feb 2026).

3. Objective families and optimization principles

One major family uses decomposition. RESUS predicts

$$\mathrm{CTR} = \frac{\text{clicks}}{\text{impressions}}$$1

where $$\mathrm{CTR} = \frac{\text{clicks}}{\text{impressions}}$$2 is a shared predictor that captures basis preferences and $$\mathrm{CTR} = \frac{\text{clicks}}{\text{impressions}}$$3 is a residual inferred from a few user-specific interactions. The outer-loop objective is a CTR loss on query instances, while the residual module is instantiated either by softmax-weighted nearest neighbors or by closed-form ridge regression. PRECTR-V2 also uses decomposition, but at the level of search relevance and conditional CTR; the composed click probability is then multiplied by $$\mathrm{CTR} = \frac{\text{clicks}}{\text{impressions}}$$4 to obtain $$\mathrm{CTR} = \frac{\text{clicks}}{\text{impressions}}$$5 (Shen et al., 2022, Cao et al., 24 Feb 2026).

A second family uses pairwise preference alignment with a frozen reference policy. CAIG applies DPO to background descriptions for advertising images:

$$\mathrm{CTR} = \frac{\text{clicks}}{\text{impressions}}$$6

It then augments this with Product-Centric Preference Optimization, which contrasts the same positive description under correct product inputs and mismatched product inputs. GQS keeps the DPO form but multiplies each pair by a CTR-derived weight $$\mathrm{CTR} = \frac{\text{clicks}}{\text{impressions}}$$7 and adds a diversity-aware DPO term, yielding $$\mathrm{CTR} = \frac{\text{clicks}}{\text{impressions}}$$8 with $$\mathrm{CTR} = \frac{\text{clicks}}{\text{impressions}}$$9. CTOP also stays within DPO, but its pair weights are the product of normalized CTR gain and AA-based confidence, so pair importance reflects both effect size and reproducibility (Chen et al., 5 Feb 2025, Min et al., 5 Jul 2025, Chen et al., 27 Jul 2025).

A third family addresses unpaired or asymmetric feedback. PMPO derives an EM-based objective that can use positives only, negatives only, or both:

$\Delta y^S = y^S - \sigma(\Psi(x^S))$0

The forward KL is central for stable negative-only learning. ToolRec uses weighted KTO rather than DPO because its assistant logs are unpaired. It defines a sample-level weight $\Delta y^S = y^S - \sigma(\Psi(x^S))$1 from user-side and system-side calibration, then optimizes $\Delta y^S = y^S - \sigma(\Psi(x^S))$2, where the prospect-theoretic value function depends on the implicit reward $\Delta y^S = y^S - \sigma(\Psi(x^S))$3 (Abdolmaleki et al., 2024, Luo et al., 7 Jun 2026).

A fourth family is contrastive rather than DPO-based. CREATER optimizes teacher-forced likelihood on both $\Delta y^S = y^S - \sigma(\Psi(x^S))$4 and $\Delta y^S = y^S - \sigma(\Psi(x^S))$5 and adds either a hinge loss on the log-likelihood gap or an InfoNCE objective on encoder–decoder representations. The paper explicitly notes that it does not use a DPO-style logistic pairwise loss. In this formulation, CTR induces the positive/negative partition, but the optimizer remains contrastive sequence modeling rather than policy alignment against a reference model (Wei et al., 2022).

4. Representation learning and architectural patterns

A persistent motif is the separation of robust shared structure from volatile personalized deviation. RESUS formalizes this most directly by decoupling basis preferences learned from pooled interactions and residual preferences learned from few-shot supports. PRECTR-V2 addresses the same problem in search by mining query-category-conditioned relevance preferences from similar users, attending over both the target user’s sparse sequence and cross-user sequences, and then combining them with a Mixture-of-Experts gate to produce $\Delta y^S = y^S - \sigma(\Psi(x^S))$6. Both works treat cold-start not as the absence of preference, but as a regime in which personal preference must be inferred relative to stronger global regularities (Shen et al., 2022, Cao et al., 24 Feb 2026).

A second motif is context-conditioned preference activation. CSPM constructs a spatiotemporal activation representation from query, location, and time features, then uses it in a Spatiotemporal Preference Extractor to query user history and in a Spatiotemporal Information Filter to gate latent context features. The same historical sequence therefore yields different activated preferences under different search states. GQS uses an analogous contextual expansion in conversational search: current query, assistant response, conversation history, user profile, co-occurring queries, prior generated queries, and display position are fused by shared BERT encoders, cross-attention, and an MLP to estimate fine-grained CTR (Jiang et al., 2023, Min et al., 5 Jul 2025).

Multimodal and grounded variants extend the same logic. CAIG pretrains an MLLM on 1.2M multimodal e-commerce samples, uses a reward model with CLIP ViT-L/14-336, Vicuna, and a pairwise CTR classifier plus regression head, and renders final creatives with Stable Diffusion, ControlNet, and inpainting. ToolRec grounds generation in SysToolKit, a repository of 708 system tools, retrieves relevant tools with Qwen-3-embedding and Qwen-3-reranker, and then aligns recommendations with weighted KTO so that CTR optimization remains tied to executable actions (Chen et al., 5 Feb 2025, Luo et al., 7 Jun 2026).

Generative ad text systems emphasize controlled conditioning and candidate diversity. CREATER prepends an aspect term to the source review, uses aspect-controlled masking during pre-training, and fine-tunes a Transformer encoder–decoder on CTR-labeled pairs. CTOP instead externalizes diversity into retrieval: it retrieves exemplars with offline chain-of-thought reasoning, builds one-shot prompts, and generates one candidate per exemplar, so diversity is induced by exemplar variation rather than by high-temperature sampling (Wei et al., 2022, Chen et al., 27 Jul 2025).

5. Sparsity, bias, and calibration

A central difficulty is that raw clicks are neither abundant nor clean. Cold-start users in RESUS have only a handful of interactions, so direct user-specific estimation is brittle; the shared predictor provides a reliable baseline when query items are far from sparse supports, while nearest-neighbor or ridge-regression residual inference supplies user-specific correction without inner-loop fine-tuning. PRECTR-V2 faces a related problem for low-activity and new users, and therefore augments sparse sequences with clicks from 50 globally sampled similar-category users, followed by text-similarity filtering and target attention (Shen et al., 2022, Cao et al., 24 Feb 2026).

Exposure bias and policy shift are handled quite differently across works. PRECTR-V2 synthesizes hard negatives by assigning a lower fake relevance label and injecting Gaussian noise into text embeddings, then optimizes a bounded-margin pairwise debias loss with $\Delta y^S = y^S - \sigma(\Psi(x^S))$7 and a dynamic truncation threshold of $\Delta y^S = y^S - \sigma(\Psi(x^S))$8 to preserve PCOC calibration. GQS models position bias with learned position embeddings and retrains its CTR model under policy shift with clipped likelihood-ratio importance weights between the current and initial policies. PMPO presents propensity weighting as the natural extension when logs are off-policy, defining $\Delta y^S = y^S - \sigma(\Psi(x^S))$9 with $S \in \{0,1\}$0 and recommending normalization to control variance (Cao et al., 24 Feb 2026, Min et al., 5 Jul 2025, Abdolmaleki et al., 2024).

Another recurring issue is that naive CTR optimization can damage relevance or integrity. CAIG explicitly frames this as a risk of CTR-only optimization that ignores product-context match, and addresses it with PCPO, visual-aware masking, textual-aware attribute replacement, ControlNet-guided structure preservation, and safety checking. GQS counters diversity collapse by constructing auxiliary diversity pairs. ToolRec prevents the model from retreating to generic but safe recommendations by assigning $S \in \{0,1\}$1 to unclicked tool-invoking negatives, while still amplifying clicked tool-invoking positives according to tool-frequency percentile. CREATER and CTOP rely on within-ad or within-item online experiments, significance or confidence filtering, and equal random traffic allocation to mitigate confounding without explicit inverse propensity correction (Chen et al., 5 Feb 2025, Min et al., 5 Jul 2025, Luo et al., 7 Jun 2026, Wei et al., 2022, Chen et al., 27 Jul 2025).

A common misconception is that CTR-driven preference optimization is necessarily “CTR-only.” The literature shows the opposite: several systems explicitly combine CTR with relevance, diversity, product integrity, or executable-action grounding. Another misconception is that absolute CTR is always the optimization target. In practice, multiple works favor relative comparisons within the same product or prompt because category-level or item-level baseline CTR varies widely (Chen et al., 5 Feb 2025, Cao et al., 24 Feb 2026, Min et al., 5 Jul 2025).

6. Empirical performance and deployment

Reported results show that CTR-driven preference optimization is not confined to offline gains. On cold-user CTR prediction, RESUS consistently achieves the best or second-best AUC across Movielens-1M, Frappe, and Taobao. Averaged across coldness stages, RESUS_RR improves relative AUC versus the best baseline by about 2.9% on Movielens; RESUS_NN improves relative AUC by about 1.8% on Frappe and about 5.0% on Taobao. Test-time latency for RESUS_NN and RESUS_RR is reported as about 10s on Movielens, compared with about 104s for MAMO in the reported setting (Shen et al., 2022).

In multimodal advertising generation, CAIG reports Pair Accuracy of 58.6% on commercial data and 56.2% on public data for CTR preference prediction. In a one-week production evaluation with more than 10M impressions, it reports a 7.4% overall CTR improvement over using the pre-trained MLLM alone, with category-level gains such as Beauty +9.5% and Fashion +7.6%. An additional A/B test reports +2% CTR with over 60M impressions (Chen et al., 5 Feb 2025).

PRECTR-V2 reports offline AUC of 0.7674 and GAUC of 0.6933 in a nine-day experiment, corresponding to relative improvements of +5.61% and +5.28% versus a Wide&Deep base. In online A/B testing with more than 20% traffic in the experimental group, it reports +1.39% per-capita orders and +3.18% GMV, and notes deployment in Xianyu search. It also reports a PCOC deviation of 1.7% versus 2.3% for the baseline (Cao et al., 24 Feb 2026).

On conversational search, GQS reports substantial gains over strong baselines. For general query suggestion, DPO_ctr achieves CTR +60.15%, while GQS reaches CTR +70.36%, Relevance 94.60, and Diversity 86.04. For creative-writing suggestion, DPO_ctr achieves CTR +25.51%, while GQS reaches CTR +30.72%, Relevance 98.11, and Diversity 82.09. The paper attributes the additional gains to CTR weighting, diversity-aware pairing, and iterative CTR calibration (Min et al., 5 Jul 2025).

In local-service CTR prediction, CSPM achieves AUC 0.8605 on the Industrial dataset and 0.7481 on the Real-OFD Public Dataset, outperforming the cited baselines. A one-month online A/B test on Ele.me reports +0.88% CTR lift and +1.0% purchase rate per impression versus the production baseline, with the largest gains in highly spatiotemporal segments (Jiang et al., 2023).

For advertising text generation, CREATER reports the best offline automatic metrics among its baselines, including BLEU-4 54.56, ROUGE-1 65.93, ROUGE-2 47.44, and ROUGE-L 59.77. In a one-week online A/B test with more than 10M impressions and over 12,000 advertisers, it reports +6.9% CTR and −6.1% CPC relative to advertiser-provided base creatives (Wei et al., 2022).

CTOP extends this line with weighted DPO from online A/B/n. In a two-week online experiment using 5% traffic and 1M evaluation items, Prompt-only reports CTR −9.83%, Top-1 sampling reports −0.6%, Top-1 SFT reports +0.92%, unweighted DPO reports +3.48%, and full CTOP reports +4.76%, with win rate 60.2%. Long-term deployment since late 2024 reports +1.11% CTR and +1.02% RPM relative to the human-crafted baseline (Chen et al., 27 Jul 2025).

On OPPO Xiaobu, ToolRec reports the strongest online results among the compared methods. In the main A/B study, Base has CTR 0.3095 and 1,063,499 clicks, Vanilla KTO has CTR 0.3167 and 1,100,807 clicks, and ToolRec has CTR 0.3198 and 1,113,871 clicks, corresponding to +3.32% CTR and +4.74% clicks relative to Base. It also reports a +1.44% relative increase in the share of tool-invoking recommendations (Luo et al., 7 Jun 2026).

7. Limitations, trade-offs, and future directions

The surveyed works make clear that CTR-driven preference optimization inherits the limitations of click data. CAIG notes that aggregated CTR preferences may underrepresent minority segments. PRECTR-V2 reports a slight increase in irrelevant item rate among top-10 results, attributing it to personalization adjustments. ToolRec observes that increasing $S \in \{0,1\}$2 raises CTR but can flatten or reduce relevance. GQS notes that residual bias may persist, including position effects and delayed feedback, and that iterative optimization increases computational cost (Chen et al., 5 Feb 2025, Cao et al., 24 Feb 2026, Luo et al., 7 Jun 2026, Min et al., 5 Jul 2025).

Method-specific limitations are also prominent. RESUS highlights non-stationary preferences as users warm up over time. CREATER depends on historical A/B data and still exhibits occasional faithfulness issues. CTOP requires a curated exemplar repository, online A/B/n infrastructure, and a parallel AA group. ToolRec depends on SysToolKit coverage and retrieval quality. PRECTR-V2 acknowledges that its synthetic negatives rely on Gaussian noise and label reconstruction heuristics, and that some encoder hyperparameters are not specified for reproducibility (Shen et al., 2022, Wei et al., 2022, Chen et al., 27 Jul 2025, Luo et al., 7 Jun 2026, Cao et al., 24 Feb 2026).

The literature nonetheless points to a coherent set of extensions. Explicitly proposed directions include dynamic $S \in \{0,1\}$3 or hierarchical temporal models for RESUS, sequence-aware modeling and contextual bandit feedback, end-to-end differentiable neighbor retrieval, personalized preference optimization for CAIG, adaptive noise magnitudes or adversarial hard-negative generation for PRECTR-V2, and inverse propensity weighting or causal adjustment as complements to pairwise debiasing. A plausible implication is that future CTR-driven preference optimization will remain multi-objective: systems that optimize clicks alone increasingly add constraints for relevance, calibration, safety, diversity, or tool executability rather than abandoning CTR as a signal (Shen et al., 2022, Chen et al., 5 Feb 2025, Cao et al., 24 Feb 2026, Abdolmaleki et al., 2024, Luo et al., 7 Jun 2026).

Across these formulations, CTR-driven preference optimization is best understood as a design pattern for turning interaction data into structured preferences under real-world constraints. Its mature forms do not merely chase higher click counts; they reshape the click signal so that optimization remains personalized, robust to bias and sparsity, and compatible with the semantic or operational requirements of the target system.