Semi-Supervised Preference Optimization

• Semi-Supervised Preference Optimization is a framework that integrates scarce paired preference data with abundant unlabeled data to generate additional supervisory signals.
• It employs methodologies such as reward-threshold pseudo-labeling, confidence filtering, and debiased risk estimation to mitigate label noise and improve training robustness.
• Empirical studies demonstrate SSPO’s efficiency in reducing human labeling costs while achieving competitive results in applications like language modeling, reinforcement learning, and medical segmentation.

Searching arXiv for the core SSPO paper and closely related semi-supervised preference optimization work to ground the article in current literature. Semi-Supervised Preference Optimization (SSPO) denotes a class of preference-learning methods that use a small labeled preference dataset together with a larger unlabeled or weakly labeled pool to optimize a policy or preference model. In the formulation of “Semi-Supervised Preference Optimization with Limited Feedback,” SSPO assumes paired labeled preference data $$D_L = \{(x^{(i)}, y_w^{(i)}, y_l^{(i)})\}_{i=1}^{n_L}$$ and unpaired unlabeled data $D_U = \{(x_u^{(j)}, y_u^{(j)})\}_{j=1}^{n_U}$, with $n_L \ll n_U$, and uses a learned reward threshold to pseudo-label unlabeled responses as likely winning or losing before joint optimization (Lee et al., 28 Oct 2025). More broadly, the contemporary literature uses closely related mechanisms—pseudo-labeling, confidence filtering, synthetic annotators, reward-model scoring, or automatically induced preference pairs—to reduce dependence on expensive human pairwise feedback in language modeling, diffusion alignment, preference-based reinforcement learning, and domain-specific applications such as medical segmentation (Park et al., 2022, Pham et al., 5 Feb 2026, Liu et al., 27 Apr 2026, Konwer et al., 6 Mar 2025).

1. Definition and problem setting

SSPO addresses the regime in which pairwise preference supervision is scarce but unpaired responses, unlabeled comparison candidates, or weak automatic feedback are abundant. In the canonical language-model setting, standard preference optimization methods such as DPO, ORPO, and SimPO assume access to tuples $(x, y_w, y_l)$, where $x$ is a prompt, $y_w$ is a preferred response, and $y_l$ is a dispreferred response. SSPO relaxes that assumption by supplementing a small paired set with a large unlabeled or weakly labeled set and then deriving additional supervisory signal from that larger pool (Lee et al., 28 Oct 2025).

In “Semi-Supervised Preference Optimization with Limited Feedback,” the learning problem is written in terms of a policy model $\pi_\theta(y \mid x)$, a reward/scoring function $r_\theta(x,y)$, and a probabilistic preference classifier $f_\theta(x,y,y')$. The labeled data are

$D_U = \{(x_u^{(j)}, y_u^{(j)})\}_{j=1}^{n_U}$0

and the unlabeled data are

$D_U = \{(x_u^{(j)}, y_u^{(j)})\}_{j=1}^{n_U}$1

The central motivation is that human preference data are expensive, whereas large stores of SFT-style prompt-response data already exist and may contain latent preference information that ordinary preference optimization ignores (Lee et al., 28 Oct 2025).

A related but older formulation appears in SURF, which studies preference-based reinforcement learning with a small labeled comparison set $D_U = \{(x_u^{(j)}, y_u^{(j)})\}_{j=1}^{n_U}$2 and a large unlabeled comparison set $D_U = \{(x_u^{(j)}, y_u^{(j)})\}_{j=1}^{n_U}$3. There the unlabeled object is a pair of trajectory segments $D_U = \{(x_u^{(j)}, y_u^{(j)})\}_{j=1}^{n_U}$4, and pseudo-labels are assigned when the current preference predictor is sufficiently confident (Park et al., 2022). This establishes a general SSPO pattern that later work inherits: a trusted labeled subset, a much larger weakly supervised or unlabeled pool, and a mechanism that converts model confidence or auxiliary scores into additional preference supervision.

This suggests that SSPO is best understood not as a single loss function but as a family of pipelines that combine three ingredients: a supervised preference component, a pseudo-labeling or weak-labeling mechanism, and a strategy for controlling noise in the automatically induced labels.

2. Core mathematical formulation

A common entry point is to cast preference learning as binary classification. In the SSPO formulation of (Lee et al., 28 Oct 2025), the Bayes-optimal classifier is

$D_U = \{(x_u^{(j)}, y_u^{(j)})\}_{j=1}^{n_U}$5

with binary cross-entropy

$D_U = \{(x_u^{(j)}, y_u^{(j)})\}_{j=1}^{n_U}$6

Using a Bradley–Terry-style parameterization, the classifier is modeled as

$D_U = \{(x_u^{(j)}, y_u^{(j)})\}_{j=1}^{n_U}$7

where $D_U = \{(x_u^{(j)}, y_u^{(j)})\}_{j=1}^{n_U}$8 is the reward score and $D_U = \{(x_u^{(j)}, y_u^{(j)})\}_{j=1}^{n_U}$9 is the sigmoid (Lee et al., 28 Oct 2025).

For the labeled preference dataset $n_L \ll n_U$0, responses are ordered as winner then loser, so $n_L \ll n_U$1, yielding

$n_L \ll n_U$2

The paper instantiates this with the SimPO reward

$n_L \ll n_U$3

leading to the supervised risk

$n_L \ll n_U$4

SSPO is therefore a semi-supervised extension of a direct preference objective rather than a reward-model-plus-RL pipeline in the RLHF sense (Lee et al., 28 Oct 2025).

An important neighboring formulation appears in DeDPO, which recasts DPO for diffusion models as binary classification on unordered pairs. For an unordered pair $n_L \ll n_U$5 under condition $n_L \ll n_U$6, a binary label $n_L \ll n_U$7 indicates which element is preferred, and DPO becomes

$n_L \ll n_U$8

with binary cross-entropy $n_L \ll n_U$9. That binary-classification view is what enables semi-supervised and debiasing machinery to be inserted directly into the preference objective (Pham et al., 5 Feb 2026).

SURF uses an analogous Bradley–Terry likelihood for preference-based RL. A segment is

$(x, y_w, y_l)$0

and the probability that $(x, y_w, y_l)$1 is preferred over $(x, y_w, y_l)$2 is

$(x, y_w, y_l)$3

The supervised reward-learning objective is

$(x, y_w, y_l)$4

with

$(x, y_w, y_l)$5

This shows that SSPO-style learning can be phrased either as direct policy preference optimization or as semi-supervised preference model learning inside a larger optimization loop (Park et al., 2022).

3. Pseudo-labeling mechanisms and weak supervision sources

The defining operation in SSPO is the conversion of unlabeled or weakly labeled examples into additional preference supervision. In (Lee et al., 28 Oct 2025), this is done with a reward threshold. For unlabeled data, only $(x, y_w, y_l)$6 is observed. The method introduces a conceptual imaginary baseline response $(x, y_w, y_l)$7, replaces its unobservable reward with a threshold $(x, y_w, y_l)$8, and pseudo-labels each unlabeled response by

$(x, y_w, y_l)$9

If the reward exceeds the estimated threshold, the response is pseudo-labeled as winning; otherwise it is pseudo-labeled as losing (Lee et al., 28 Oct 2025).

The threshold is estimated from the labeled set using KDE on winner and loser rewards: $x$0

$x$1

and then

$x$2

with

$x$3

This is a one-sided pseudo-labeling scheme: unpaired responses become pseudo-winning or pseudo-losing relative to a learned separator (Lee et al., 28 Oct 2025).

SURF uses confidence-thresholded pseudo-labeling on unlabeled comparison pairs. For an unlabeled pair $x$4, the hard pseudo-label is

$x$5

The pair is kept only if the predicted winning probability exceeds a threshold $x$6: $x$7 The resulting semi-supervised objective is

$x$8

Here the unlabeled object is itself a comparison candidate, not a single response, which makes SURF a prototypical pairwise pseudo-labeling method (Park et al., 2022).

DeDPO uses synthetic annotators to fill in missing pairwise labels. With a labeled set $x$9 and unlabeled set $y_w$0, a synthetic annotator $y_w$1 provides pseudo preferences for unlabeled pairs. The naive mixture

$y_w$2

is biased when synthetic feedback differs systematically from human feedback, which motivates the debiased correction in DeDPO (Pham et al., 5 Feb 2026).

Semi-DPO offers another pseudo-labeling design, specialized to noisy diffusion preference datasets. It first partitions a dataset into a clean labeled subset and a noisy unlabeled subset by multi-reward consensus: $y_w$3

$y_w$4

The current DPO model then acts as an implicit classifier on the noisy subset, using the timestep-specific logit $y_w$5 both to orient the pair and to determine confidence. A pseudo-label is accepted only if

$y_w$6

This is a self-training paradigm over preference pairs rather than over single examples (Liu et al., 27 Apr 2026).

Another weak-supervision route is automatic preference-pair construction from model samples and scalar rewards. “Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization” studies the case where a policy model $y_w$7 generates $y_w$8 on-policy responses $y_w$9 per prompt, a reward model scores them, and a chosen/rejected pair is constructed for DPO. The conventional rule is

$y_l$0

The paper shows that this max-vs-min heuristic can degrade as $y_l$1 increases and that rejected responses near $y_l$2 are more effective than the minimum reward sample. Its practical rule—sample 5 responses first, choose the minimum among those 5 as the rejected response, then continue sampling to $y_l$3 and choose the global maximum as positive—can be read as a pseudo-labeling heuristic for automatically inducing informative preference pairs (Xiao et al., 24 Feb 2025).

4. Robustness, debiasing, and label-noise control

Because SSPO depends on automatically induced labels, robustness to pseudo-label error is a central design problem. The literature treats this in several distinct ways.

The most explicit debiasing result appears in DeDPO. The method replaces naive mixing with the objective

$y_l$4

Equivalently, it can be written as BCE on corrected targets $y_l$5, where labeled examples receive an amplified correction

$y_l$6

The paper proves unbiasedness,

$y_l$7

and gives an orthogonality-style excess-risk bound

$y_l$8

indicating robustness to first-order pseudo-label error when the synthetic annotator is independent of the final learner (Pham et al., 5 Feb 2026).

Semi-DPO addresses a different noise source: binary preference labels that collapse multi-dimensional human judgments. Its theory shows that compressing multi-dimensional preferences into single winner/loser labels yields conflicting gradient signals. For a preference dimension $y_l$9, aligned and conflicting subsets are

$\pi_\theta(y \mid x)$0

and the variance of the alignment quantity is lower-bounded by

$\pi_\theta(y \mid x)$1

This motivates treating consensus-clean pairs as labeled data and conflicting pairs as noisy unlabeled data, rather than training on all observed preferences as if they were equally reliable (Liu et al., 27 Apr 2026).

SURF uses aggressive confidence thresholding rather than formal debiasing. Pseudo-labels are retained only when the winning probability exceeds $\pi_\theta(y \mid x)$2, with $\pi_\theta(y \mid x)$3 for Window Open, Sweep Into, and Cheetah and $\pi_\theta(y \mid x)$4 for other state-input tasks. The paper explicitly warns that low-confidence pseudo-labels can significantly degrade preference-based learning, citing B-Pref for noise sensitivity. It also reports that too-low threshold $\pi_\theta(y \mid x)$5 hurts performance, and that an unlabeled batch ratio $\pi_\theta(y \mid x)$6 can overfit to unlabeled data (Park et al., 2022).

The reward-threshold SSPO method (Lee et al., 28 Oct 2025) controls pseudo-label noise differently: through the threshold itself, through EMA stabilization, and through a curriculum over the labeled and pseudo-labeled losses. The unlabeled pseudo-risk is

$\pi_\theta(y \mid x)$7

and the total objective is

$\pi_\theta(y \mid x)$8

with

$\pi_\theta(y \mid x)$9

This scheduler ensures that training begins almost entirely on reliable labeled comparisons and only gradually increases the influence of pseudo-labeled unpaired data (Lee et al., 28 Oct 2025).

A plausible synthesis is that modern SSPO methods fall into three robustness families: confidence filtering as in SURF, threshold-based pseudo-label calibration as in SSPO with limited feedback, and bias-corrected risk estimation as in DeDPO. Semi-DPO adds a fourth, namely clean/noisy partitioning plus iterative self-training.

5. Representative algorithms and empirical findings

The empirical literature shows that SSPO-style methods can substantially reduce dependence on labeled preferences, but the gains depend on the quality of the weak supervision source.

A compact comparison of representative methods is useful.

Method	Weak supervision source	Core mechanism
SSPO with Limited Feedback	Unpaired prompt-response data	Reward-threshold pseudo-labeling and joint loss (Lee et al., 28 Oct 2025)
SURF	Unlabeled trajectory-segment pairs	Confidence-thresholded pseudo-labeling + temporal cropping (Park et al., 2022)
DeDPO	Synthetic AI feedback on unlabeled image pairs	Debiased DPO objective with human calibration (Pham et al., 5 Feb 2026)
Semi-DPO	Noisy preference pairs	Consensus clean/noisy split + iterative self-training (Liu et al., 27 Apr 2026)
PFPO	Pseudo verification for reasoning	Test-case-derived scores converted to DPO/pDPO pairs (Jiao et al., 2024)
Medical SAM preference alignment	Virtual annotator from IoU	Preference ranking over segmentation candidates (Konwer et al., 6 Mar 2025)

In (Lee et al., 28 Oct 2025), SSPO trained with Llama3-8B-Instruct on just 1% of UltraFeedback is reported to surpass strong baselines trained on 10% of UltraFeedback. On Mistral-7B with UltraFeedback, 1% labels gives LC win rate 26.7 for SSPO, compared with 18.2 for SPA, 16.4 for KTO, 17.0 for DPO, and 13.2 for SimPO; with 10% labels, SSPO reaches LC 30.0 (Lee et al., 28 Oct 2025). The paper further shows robustness to prior choice, with $r_\theta(x,y)$0 performing best in the reported UltraFeedback Mistral setting, and reports that adaptive scheduling improves over fixed $r_\theta(x,y)$1 (Lee et al., 28 Oct 2025).

SURF reports large feedback-efficiency gains in control. A highlighted result is Window Open, where with 400 preference queries, SURF reaches performance comparable to SAC, whereas PEBBLE reportedly needs 2,500 queries, about a 6× reduction in labels. The method also shows that pseudo-labeling alone improves PEBBLE, temporal cropping alone improves PEBBLE, and combining them yields the best results (Park et al., 2022).

DeDPO shows that with only 25% human labels and 75% synthetic labels, diffusion-model preference alignment can match or slightly exceed fully human-labeled DPO. On FiFA-5K with SD1.5, DeDPO reports PickScore 21.91 and HPSv2 27.80, compared with 21.88 and 27.79 for DPO with 100% human labels, while naive DPO with synthetic preferences degrades to 21.71 and 27.39. DeDPO also remains stable as unlabeled data scale increases, whereas naive DPO peaks and then degrades (Pham et al., 5 Feb 2026).

Semi-DPO reports that on Pick-a-Pic V2 with SD1.5, Semi-DPO achieves ImageReward 0.801, HPS v2 0.288, PickScore 21.524, Aesthetic 5.801, and MPS 11.030, improving over Diffusion-DPO and Diffusion-KTO across nearly all metrics. The paper also reports that its iterative refinement helps from Iter0 to Iter1 and again to Iter2, though gains saturate after two rounds (Liu et al., 27 Apr 2026).

PFPO demonstrates that pseudo verification can substantially improve reasoning models. On Mathstral-7B-v0.1, the method improves MATH from 58.3 to 68.6, GSM8K from 85.6 to 90.3, and College Math from 34.3 to 42.2 or 42.3 depending on iteration. On LiveCodeBench, Deepseek-coder-7B-v1.5 rises from 21.1 to 24.6, surpassing Claude-3-Haiku in the reported comparison (Jiao et al., 2024). The paper also finds that frontier-LLM pseudo feedback is better for bootstrapping, while self-consistency can later become comparable (Jiao et al., 2024).

The medical-image segmentation paper (Konwer et al., 6 Mar 2025) is more domain-specific, but it demonstrates the same pattern: with 10% labeled + 10% unannotated data, the method reaches Dice 78.87 on Chest X-ray and mean Dice 77.69 on AMOS-CT, outperforming several supervised/SAM-family baselines under the same low-label regime. Its preference stage is driven by a virtual annotator that rates or ranks thresholded candidate masks using IoU against hidden ground truth, and ranking performs best among the preference schemes considered (Konwer et al., 6 Mar 2025).

These results collectively suggest that SSPO is most effective when unlabeled or weakly labeled data are abundant and semantically aligned with the target task, and when the pseudo-labeling mechanism is good enough to preserve useful preference structure.

6. Scope, related concepts, and terminology

The terminology around SSPO is unstable, and several neighboring papers use similar acronyms or mechanisms without being canonical instances of Semi-Supervised Preference Optimization.

The most direct usage is “Semi-Supervised Preference Optimization with Limited Feedback,” where SSPO explicitly means semi-supervised use of paired labeled data and unpaired unlabeled data (Lee et al., 28 Oct 2025). DeDPO does not use the term SSPO, but its setup—25% labeled and 75% unlabeled image pairs with synthetic preferences—closely matches the concept (Pham et al., 5 Feb 2026). Semi-DPO likewise reframes preference optimization under noisy labels as a semi-supervised learning problem, with a clean labeled subset and a noisy unlabeled subset (Liu et al., 27 Apr 2026). SURF predates modern LLM alignment but instantiates the same semi-supervised pattern for preference-based RL (Park et al., 2022).

By contrast, “S-SPPO: Semantic-Calibrated Self-Play Preference Optimization” is not semi-supervised in the standard labeled/unlabeled sense. It uses prompts only, self-generated responses, and a preference oracle, then calibrates targets based on semantic similarity and latent repulsion. The paper itself explicitly distinguishes this from standard SSPO and frames it instead as self-play-based preference optimization with self-supervised semantic calibration (Chen et al., 1 Jun 2026).

Likewise, “SSPO: Self-traced Step-wise Preference Optimization for Process Supervision and Reasoning Compression” uses the acronym SSPO for Self-traced Step-wise Preference Optimization, not Semi-Supervised Preference Optimization. Its step-wise preference signals are derived from the model’s own reasoning trajectories and rule-based correctness, which makes it self-/semi-supervised in a broad sense, but terminologically it is a different method family (Xu et al., 18 Aug 2025).

Several additional papers are relevant as neighboring building blocks rather than canonical SSPO methods. “Soft Preference Optimization” emphasizes global distribution-level regularization and a softness parameter $r_\theta(x,y)$2, but it does not use unlabeled data or pseudo-labeling in the standard semi-supervised sense (Sharifnassab et al., 2024). “Learning from negative feedback, or positive feedback or both” develops a probabilistic-inference objective that can learn from unpaired accepted samples, unpaired rejected samples, or both, which is directly relevant to asymmetric or partially labeled supervision but is not framed as unlabeled-data SSPO (Abdolmaleki et al., 2024). “Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularization” studies automatic preference-data construction and budget-controlled regularization; its iterative pairwise ranking mechanism is naturally interpretable as pseudo-label generation for unlabeled prompts, though the paper itself does not present a mixed labeled/unlabeled SSPO training study (Chen et al., 2024).

This suggests a useful taxonomy. Canonical SSPO methods explicitly combine labeled preference data with unlabeled or weakly labeled data in a joint objective. Weak-supervision neighbors automatically construct preference pairs or pseudo labels but may not have a separate labeled/unlabeled split. Self-play/self-supervised neighbors generate preferences from the model’s own outputs without human labels at training time. The boundaries are porous, but the distinction matters when interpreting theoretical assumptions and empirical claims.

7. Limitations and open questions

A recurring limitation is dependence on the quality and calibration of the pseudo-labeling mechanism. In (Lee et al., 28 Oct 2025), the reward-threshold method assumes that winner and loser rewards become sufficiently separable under a single threshold. The theory proves that if rewards for winners and losers are i.i.d. sub-Gaussian with means $r_\theta(x,y)$3, then for suitable $r_\theta(x,y)$4 there exists

$r_\theta(x,y)$5

such that

$r_\theta(x,y)$6

This justifies pseudo-labeling in a stylized setting, but it does not establish full policy-consistency guarantees under finite-sample KDE threshold estimation or under domain shift (Lee et al., 28 Oct 2025).

DeDPO depends critically on the synthetic annotator being informative and preferably independent of the final learner. The theory explicitly says that sample splitting / independence is violated by self-training, and the paper states that this is why external VLM judges are theoretically preferred over self-labeling (Pham et al., 5 Feb 2026). Semi-DPO, in turn, depends on the quality of the clean subset selected by consensus and on the DPO model being strong enough to produce useful pseudo-labels on the noisy subset (Liu et al., 27 Apr 2026). SURF assumes that high-confidence pseudo-labels are usually correct but does not model calibration error beyond thresholding (Park et al., 2022).

A second limitation is computational overhead. SSPO often reduces annotation cost at the expense of additional sampling, scoring, or synthetic-label generation. SURF requires unlabeled-to-labeled minibatch ratios such as $r_\theta(x,y)$7 for state inputs and confidence filtering over many replay-buffer pairs (Park et al., 2022). DeDPO requires synthetic annotation over large unlabeled image-pair pools (Pham et al., 5 Feb 2026). PFPO relies on test-case synthesis, multiple sampled reasoning traces, and sometimes frontier-LLM supervision (Jiao et al., 2024). Automatic preference-pair construction by repeated model sampling and reward scoring, as in (Xiao et al., 24 Feb 2025), can scale to 200 or 400 samples per prompt, which is expensive even when it improves pair quality.

A third open question is how to combine gold and pseudo-labeled preferences optimally. DeDPO addresses this with a debiased objective (Pham et al., 5 Feb 2026), and (Lee et al., 28 Oct 2025) uses an adaptive mixture schedule, but broader questions remain: confidence-weighted mixing, prompt-conditional thresholds, curriculum over pseudo-label hardness, and methods for detecting when unlabeled data are semantically mismatched to the preference task.

A fourth limitation concerns the structure of the supervision source. PFPO works well because math and code have verifier-like signals; that may not transfer to open-ended domains without executable or exact-match criteria (Jiao et al., 2024). The medical segmentation paper uses a virtual annotator based on IoU, which is highly specific to segmentation (Konwer et al., 6 Mar 2025). SURF’s temporal cropping relies on trajectory semantics that do not obviously transfer outside sequential control (Park et al., 2022). A plausible implication is that SSPO methods are strongest in domains where there exists either a useful scalar verifier, a reliable synthetic annotator, or a task-specific augmentation that preserves preference semantics.

Finally, the field still lacks a unified theory connecting pseudo-label quality, confidence calibration, and downstream preference optimization error across these different regimes. Existing theory is fragmented: threshold existence in (Lee et al., 28 Oct 2025), orthogonal debiasing in (Pham et al., 5 Feb 2026), gradient-conflict analysis in (Liu et al., 27 Apr 2026), and empirical saturation/instability analyses in (Park et al., 2022) and (Xiao et al., 24 Feb 2025). This suggests that SSPO is currently better characterized as a convergent methodological family than as a settled formal framework.

Semi-Supervised Preference Optimization is therefore best regarded as the systematic extension of preference optimization from fully labeled pairwise datasets to mixed-supervision regimes. Its central question is how to exploit abundant unlabeled or weakly labeled data without allowing pseudo-label error to dominate training. Current answers include reward-threshold pseudo-labeling (Lee et al., 28 Oct 2025), confidence-filtered self-training (Park et al., 2022, Liu et al., 27 Apr 2026), debiased synthetic-feedback integration (Pham et al., 5 Feb 2026), verifier-derived pseudo preferences (Jiao et al., 2024), and automatically constructed preference pairs from reward distributions (Xiao et al., 24 Feb 2025). The shared lesson is that preference supervision can be scaled beyond direct human comparisons, but only when pseudo-label construction, calibration, and optimization are treated as first-class components of the alignment pipeline.