Self-Correcting Preference Learning (SPL)

• SPL is a dynamic framework where models automatically generate and use their own preference signals to correct outputs.
• In radiology, SPL constructs disease-specific preference datasets, filters inconsistent descriptions, and synthesizes refined reports.
• SPL extends to reasoning and multimodal tasks, enabling self-correction that improves performance with minimal human oversight.

Self-correcting Preference Learning (SPL) denotes a class of preference-learning procedures in which a model uses its own outputs, or multiple noisy observations of those outputs, to construct preference signals and then uses those signals to filter, relabel, edit, or optimize subsequent generations. In the radiology-reporting formulation introduced within clinically aligned Evidence-aware Self-Correcting Reinforcement Learning (ESC-RL), SPL is a disease-aware, fully automatic preference-learning procedure that constructs a disease-level preference dataset from multiple noisy generated reports and a ground-truth report, trains a preference predictor, filters unreliable disease-level descriptions, and re-integrates reliable ones into a refined report (Zhou et al., 15 Apr 2026). In a broader formulation, SPL is the idea that a model can generate its own preference signals—typically “this is better than that” judgments between its own outputs—and then use those signals to train itself to correct and improve over time, ideally with little or no external supervision (Ding et al., 28 May 2025).

1. Definition and conceptual scope

In its most specific form, SPL is a procedure with three recurrent elements: automatic construction of preference data, learning or applying a preference mechanism, and a correction step that feeds improved outputs back into training or inference. The radiology formulation makes this explicit at disease granularity: it constructs a disease-level preference dataset from multiple noisy generated reports and a ground-truth report, learns a preference predictor that estimates whether a generated disease-specific description is consistent with the ground-truth, filters unreliable descriptions, and re-integrates the reliable ones into a refined report (Zhou et al., 15 Apr 2026).

A broader abstraction appears in multimodal reasoning. Sherlock describes SPL as the regime in which a model generates its own preference signals from original and corrected reasoning trajectories and then trains itself to correct and improve over time with minimal external supervision (Ding et al., 28 May 2025). S$^2$COPE instantiates the same pattern without human labels: candidate textual concepts are generated from images, scored by a self-supervised reward, converted into pairwise preferences, and used in DPO to update the concept-generation policy (Xiang et al., 12 Jun 2026). RISE uses the same logic at the level of subtle reasoning errors by constructing preference pairs between correct steps and minimally self-edited incorrect variants (Xu et al., 2024).

The preference object in SPL is therefore not fixed. Depending on the system, it can be a disease-specific description, a reasoning suffix, a textual concept, an unbiased continuation, or a step in a process-based MDP. This suggests that SPL is best treated as a methodological family organized around self-generated or automatically induced preference signals, rather than as a single loss function.

A terminological complication is that the acronym “SPL” is also used for “Swap-guided Preference Learning” in personalized RLHF. That work is formally about variational preference learning with fictitious swap annotators rather than self-correction in the radiology or reasoning sense, although its latent-space mirroring mechanism is explicitly described as self-regularizing and interpretable as self-correcting with respect to posterior collapse (Kim et al., 13 Mar 2026).

2. Canonical disease-aware SPL in radiology report generation

The most explicit end-to-end SPL pipeline appears in ESC-RL for radiology report generation. Given an image $\mathbf{I}$, the current policy $\psi_p$ samples $N$ candidate reports,

$\{O_n\}_{n=1}^N,$

and also has access to a ground-truth report $R^*$ (Zhou et al., 15 Apr 2026). These candidate reports are “multiple noisy observations” because they come from a partially trained policy and may contain hallucinations, omissions, or inconsistent phrasing.

SPL then performs disease-aware decomposition with CheXbert. Reports are mapped into status vectors

$Y \in \{0,1,2,3\}^K,$

with entries corresponding to blank, positive, negative, and uncertain, and the textual content is decomposed into disease-specific descriptions

$$\mathbf{O}_n = \{O_n^k\}_{k=1}^K, \qquad \mathbf{R}^* = \{R^{*k}\}_{k=1}^K.$$

For each image, candidate report, and disease, SPL constructs pairs $(O_n^k, R^{*k})$ and assigns a two-dimensional preference label $\tilde{o}_n^k \in \{(1,0),(0,1),(0.5,0.5)\}$ with an LLM. The labels denote consistent, inconsistent, or indistinguishable/uncertain, and the resulting disease-aware preference dataset is

$\mathbf{I}$0

The preference predictor $\mathbf{I}$1 is a BERT-base model plus classifier. It takes the concatenation of $\mathbf{I}$2 and $\mathbf{I}$3 and outputs a 2D probability distribution over consistent versus inconsistent. Predictor training uses cross-entropy,

$\mathbf{I}$4

Reliability is then estimated by KL divergence, and only pairs with divergence between a lower and upper threshold are retained: $\mathbf{I}$5 Following this selection step, an LLM receives the full candidate reports $\mathbf{I}$6 and the trustworthy subset $\mathbf{I}$7 and synthesizes a refined report $\mathbf{I}$8 that removes inconsistent content, keeps supported descriptions, and re-integrates reliable disease-level information.

This is the “self-correcting” step in a strict sense: the current policy generates the raw material, the preference mechanism identifies which disease descriptions should survive, and the refined report becomes a corrected target for later reward computation and policy improvement. In ESC-RL, this refined report is passed to GEAR, and the full training objective is

$\mathbf{I}$9

SPL is therefore a side-channel preference-learning mechanism: it does not directly define a Bradley–Terry reward over policy outputs, but it changes the reward landscape through refined reports and through the learned predictor (Zhou et al., 15 Apr 2026).

3. Preference objects, supervision sources, and correction operators

A central distinction across SPL systems is the unit over which preference is defined. In ESC-RL, the unit is the disease-specific description. The supervision source is a ground-truth report plus LLM labeling of consistency, and the correction operator is LLM-guided re-integration constrained by a trustworthy subset of disease-level descriptions (Zhou et al., 15 Apr 2026).

In Sherlock, the unit is a reasoning trajectory suffix. The system models a trajectory as

$\psi_p$0

and its trajectory-level self-correction objective trains the model to revise only the erroneous suffix of a trajectory, conditioned on a presumably correct prefix. Preference data are constructed by visual perturbation and truncation in the offline stage, and later by self-consistency across multiple rounds of self-correction in the online stage. Dynamic $\psi_p$1 is used to scale preference updates according to truncation depth and perturbation strength (Ding et al., 28 May 2025).

In S$\psi_p$2COPE, the preference object is a short natural-language concept. A VLLM generates candidate concepts $\psi_p$3 for image $\psi_p$4, and a frozen CLIP model supplies the self-supervised reward

$\psi_p$5

which favors invariance across views of the same image and specificity relative to other images in the batch. Pairwise preferences are then turned into DPO updates of the VLLM itself (Xiang et al., 12 Jun 2026).

In RISE, the preference object is a reasoning step or a full solution. The system samples correct and incorrect solutions, then self-edits correct steps to inject subtle errors such as numerical calculation errors, numeric or symbolic substitution errors, omission of calculation terms, errors in calculation order, and errors in the use of calculation symbols. Training combines a step-wise DPO term on self-edited pairs, a full-solution DPO term, and a conditional NLL stabilization term (Xu et al., 2024).

In Self-Debias, the preferred object is an unbiased reasoning suffix over a biased one. The resource margin is

$\psi_p$6

and the system adds a fairness regularizer based on Jain’s index to equalize margins across samples, thereby upweighting hard cases during debiasing (Feng et al., 9 Apr 2026).

In SPPD, preferences are defined over actions in a step-level MDP. The model compares preferred and dispreferred next steps at the same state and introduces a dynamic value margin

$\psi_p$7

so that preference optimization depends not only on local action likelihoods but also on the long-term value difference between successor states (Yi et al., 19 Feb 2025). Across these systems, SPL is unified less by a shared architecture than by a recurring template: generate candidates, induce pairwise preference structure, and use the resulting signal to alter future generations.

4. Representative systems and empirical characterization

Representative SPL-style systems span radiology report generation, multimodal reasoning, self-supervised concept discovery, mathematical reasoning, debiasing, and process-level self-training.

System	Preference unit	Selected reported result
ESC-RL SPL	Disease-specific descriptions	GREEN 0.344 to 0.394 on MIMIC-CXR
Sherlock	Reasoning trajectory suffixes	64.1 direct, 65.4 after self-correction
S$\psi_p$8COPE	Textual concepts	Up to a 24-point absolute improvement in downstream top-1 classification accuracy
RISE	Self-edited reasoning steps	+3.0% on GSM8K and +7.9% on MATH with only 4.5K training samples
Self-Debias	Unbiased over biased suffixes	81.7 direct, 82.1 with self-correction
SPPD	Step-level actions in an MDP	Qwen2.5-7B-Base: MATH500 60.0 to 72.2, GSM8k 82.3 to 90.3 after SPPD-Stage2

In ESC-RL, SPL alone improves GREEN from 0.344 to 0.356, and the full RL+GEAR+SPL system reaches 0.394 on MIMIC-CXR; on the same benchmarks, ESC-RL improves CheXbert F1 from 0.592 to 0.608 on MIMIC-CXR and from 0.273 to 0.295 on IU-Xray (Zhou et al., 15 Apr 2026). Sherlock reports that self-correction becomes beneficial only after training for it: existing reasoning models often show no gain or degraded performance under repeated self-correction, whereas Sherlock Iter2 reaches 64.1 with direct generation and 65.4 after self-correction across eight benchmarks while using only 20k randomly sampled annotated data (Ding et al., 28 May 2025).

S$\psi_p$9COPE shows that self-supervised preference optimization can reshape a VLLM into an amortized concept generator. Using only 1,300 unlabeled images from iNaturalist-mini, the method reports gains such as BloodMNIST 55.26 to 79.73, OrganCMNIST 68.06 to 89.17, and iNaturalist 71.15 to 85.00, with consistent improvements across 2B, 4B, and 8B VLLMs (Xiang et al., 12 Jun 2026). RISE demonstrates that self-generated hard negatives aimed at subtle error tokens can produce targeted reasoning gains: on Qwen2-7B-Instruct, GSM8K rises from 85.4 to 88.4 and MATH from 52.2 to 59.9, while larger models obtain smaller but still positive gains (Xu et al., 2024).

Self-Debias presents SPL as a debiasing mechanism rather than a factuality mechanism. With merely 20k annotated samples, the system reports average performance rising from 80.6 after Stage I to 81.7 direct and 82.1 with self-correction after Iter2, while preserving or improving GSM8K and ARC-C performance (Feng et al., 9 Apr 2026). SPPD provides a process-level variant in which self-sampled trees, PRM scoring, sentence-level DPO, and dynamic-margin step-level preference optimization are combined; the second self-training stage further improves both in-domain and out-of-domain mathematical benchmarks (Yi et al., 19 Feb 2025).

These results indicate that SPL is not tied to a single modality. What changes across domains is the preference object, the verifier or reward source, and the mechanism by which corrected outputs are reinjected into learning.

5. Relation to RLHF, RLAIF, process supervision, and personalization

SPL overlaps with RLHF but is not identical to it. A standard RLHF pipeline collects human-labeled preference pairs, trains a reward model, and optimizes a policy against that reward. The radiology formulation explicitly contrasts itself with such pipelines: SPL uses an LLM and ground-truth reports rather than human annotators, operates at disease-level rather than whole-report granularity, and trains a predictor used for filtering and editing rather than a scalar reward model used directly for policy optimization (Zhou et al., 15 Apr 2026). Sherlock likewise uses self-generated preference pairs and optimizes the policy directly with a self-correction loss plus DPO, rather than training a separate reward model (Ding et al., 28 May 2025).

The relation to RLAIF is closer in systems that use automatic or model-based preference signals. ESC-RL states that its automatic preference labeling makes it closer to “RL from AI feedback” than RLHF, while S$N$0COPE uses a frozen CLIP-derived reward from unlabeled images and a DPO objective over generated concepts (Zhou et al., 15 Apr 2026, Xiang et al., 12 Jun 2026). Self-Debias also fits this pattern: self-generated biased and corrected trajectories become preference pairs, and online self-improvement proceeds without continuous external oversight (Feng et al., 9 Apr 2026).

A second axis of differentiation concerns process supervision. Step-level or trajectory-level SPL methods do not merely rank final outputs. Sherlock learns to correct only erroneous suffixes; RISE constructs self-edited step pairs that differ by a few error tokens; and SPPD builds a process-based MDP in which a dynamic value margin weights step preferences by downstream value difference (Ding et al., 28 May 2025, Xu et al., 2024, Yi et al., 19 Feb 2025). This process orientation is one reason SPL is frequently connected to self-correction rather than only to preference alignment.

The acronym overlap with personalized RLHF is instructive. Swap-guided Preference Learning introduces user-specific latent variables, identifies posterior collapse in Variational Preference Learning, and combats collapse with swap-guided base regularization, Preferential Inverse Autoregressive Flow, and adaptive latent conditioning (Kim et al., 13 Mar 2026). Although its official expansion is different, it shows that “SPL” can also denote methods that use internal structural constraints—here, fictitious swap annotators and mirroring—to repair weaknesses in preference learning itself. A plausible implication is that self-correction in preference learning can target either outputs or the latent representations that organize preference information.

6. Limitations, misconceptions, and open directions

Across current systems, a first recurrent limitation is dependence on external models. ESC-RL depends on CheXbert, GPT-5, and MAVL; S$N$1COPE depends on Qwen3-VL and frozen CLIP; SPPD depends on a fixed PRM; and Self-Debias relies on GPT-4o-generated fair CoTs during cold start (Zhou et al., 15 Apr 2026, Xiang et al., 12 Jun 2026, Yi et al., 19 Feb 2025, Feng et al., 9 Apr 2026). These dependencies introduce computational overhead, model-specific biases, and the possibility that verifier errors or reward noise will be fed back into training.

A second limitation is that self-generated preference signals are not guaranteed to be correct. Sherlock notes that self-consistency is not the same as truth, and its perturbation-based ordering can be noisy if visual corruption does not reliably degrade quality. RISE depends on a hand-designed taxonomy of subtle errors and on ground-truth final answers to identify correct solutions. S$N$2COPE reports that hard negatives amplify reward noise and that there is no formal convergence theory. ESC-RL states that preference labels are weakly supervised, that some errors may remain after KL-based filtering, and that ground-truth reports may themselves be noisy (Ding et al., 28 May 2025, Xu et al., 2024, Xiang et al., 12 Jun 2026, Zhou et al., 15 Apr 2026).

A common misconception is that SPL always means training a scalar reward model and then optimizing a policy against it. In fact, the provided systems realize preference learning in materially different ways. ESC-RL uses a consistency classifier and LLM-guided report synthesis rather than a direct RL-type preference objective. Sherlock embeds the preference signal inside the policy with a self-correction loss plus DPO. S$N$3COPE keeps the reward model fixed and optimizes only the generator. Self-Debias adds a fairness regularizer over resource margins, and SPPD derives a dynamic value margin from a Bellman-optimality analysis (Zhou et al., 15 Apr 2026, Ding et al., 28 May 2025, Xiang et al., 12 Jun 2026, Feng et al., 9 Apr 2026, Yi et al., 19 Feb 2025).

Domain restriction is another recurring issue. ESC-RL is validated on chest X-ray report generation; Self-Debias focuses on social-bias benchmarks and selected reasoning tasks; RISE is primarily math-centric; and SPPD is evaluated on mathematical reasoning with a fixed PRM (Zhou et al., 15 Apr 2026, Feng et al., 9 Apr 2026, Xu et al., 2024, Yi et al., 19 Feb 2025). Several papers explicitly suggest extensions to other modalities, richer forms of human feedback, cheaper or distilled external models, and broader multimodal settings. This suggests that the central open problem is not whether self-correcting preference learning can work in isolated domains, but how to make self-generated preference signals reliable enough, portable enough, and stable enough to support continual self-improvement at scale.