---
title: Self-Consistency Preference Optimization (ScPO)
url: https://www.emergentmind.com/topics/self-consistency-preference-optimization-scpo
type: topic
---

# Self-Consistency Preference Optimization (ScPO)

Self-Consistency Preference Optimization (ScPO) is a self-alignment framework for language models that leverages self-consistency at the training stage rather than solely at inference time. The core idea is to prefer solutions that are most consistent across multiple samples for a given reasoning task and to use these internal consistency signals to guide preference-based finetuning in the absence of gold annotation. This approach enables fully unsupervised or semi-supervised training for multi-step reasoning and logic problems, and has demonstrated substantial gains over reward-model-based optimization, closing much of the gap with supervised preference training on tasks such as GSM8K, MATH, and ZebraLogic [2411.04109].

## 1. Motivation and Theoretical Rationale

Self-alignment for reasoning tasks is inherently challenging due to the difficulty for models to provide reliable self-judgment of correctness, especially for multi-step problems. In standard practice, inference-time self-consistency—where multiple chain-of-thought (CoT) samples per prompt are generated, and the most common answer is selected—has empirically boosted accuracy [2411.04109]. The key hypothesis of ScPO is to transfer this powerful inference-time signal into the training paradigm: use the model's own vote-based consistency per query as a preference indicator between model responses, rather than relying on noisy external reward models or expensive human labels.

Objectives of ScPO include:
1. Bootstrapping high-quality training data from unlabeled reasoning problems, including synthetic problems generated by the model itself.
2. Forming preference pairs by ranking model-sampled solutions to a problem according to the frequency of their answer.
3. Optimizing a preference-based loss weighted by the degree of intra-sample agreement, directly increasing the model’s likelihood of producing self-consistent outputs.
4. Leveraging gold-labeled (supervised) data only when available, but not as a prerequisite.

## 2. Formalism and Optimization Framework

Given an unlabeled reasoning question $x$, and a current policy $M_t$, ScPO samples $k$ completions $\{y_1,\dots,y_k\}$ using CoT prompting. For each solution $y$, the final answer is extracted as $\mathrm{ans}(y)$. The self-consistency vote for each $y$ is defined as the number of completions that produce the same final answer:
$$
\mathcal{V}(y) = \sum_{m=1}^k \mathbf{1}(\mathrm{ans}(y_m) = \mathrm{ans}(y))
$$
Preference pairs are constructed by taking $y^+ = \arg\max_{y} \mathcal{V}(y)$ (the most consistent answer) and $y^- = \arg\min_{y} \mathcal{V}(y)$ (the least consistent answer), and only including $(x, y^+, y^-)$ as a training instance if $\mathcal{V}(y^+) \geq \tau$, a minimum consistency threshold. Each pair is weighted by
$$
w(x) = \frac{\mathcal{V}(y^+) - \mathcal{V}(y^-)}{k}
$$
The ScPO loss builds upon Direct Preference Optimization (DPO) by optimizing:
$$
\mathcal{L}_{\text{ScPO}}(x; y^+, y^-) = -w(x)\,\log \sigma\Bigg(
\beta \log \frac{M_\theta(y^+|x)}{M_t(y^+|x)} - \beta \log \frac{M_\theta(y^-|x)}{M_t(y^-|x)}
\Bigg) - \frac{\alpha w(x)}{|y^+|}\log M_\theta(y^+|x)
$$
where $\sigma$ is the sigmoid, $\alpha$ and $\beta$ are hyperparameters, and $|y^+|$ is the token length of $y^+$. When supervised labels are available, $y^+$ is a gold solution and $w(x)=1$.

The following table summarizes essential notation:

| Symbol             | Description                                          |
|--------------------|-----------------------------------------------------|
| $x$                | Unlabeled reasoning problem                         |
| $M_t$              | Model at iteration $t$                              |
| $y_1,\dots,y_k$    | CoT-sampled solutions                               |
| $\mathcal{V}(y)$   | Vote count for answer in sample set                 |
| $y^{+}$, $y^{-}$   | Most-consistent, least-consistent solutions         |
| $w(x)$             | Preference pair weight ($\in [0,1]$)                |
| $\tau$             | Consistency threshold for filtering                 |

## 3. Algorithmic Procedure

The canonical ScPO procedure is an iterative self-bootstrapping loop over $T$ iterations:

1. For each training round, augment the query set by generating new problems using few-shot prompting, discarding queries where $\max_y \mathcal{V}(y) < \tau$.
2. For each problem, sample $k$ solutions, compute vote counts, and form weighted preference pairs as above.
3. Aggregate these into a preference dataset for the round.
4. Train a new model copy $M_\theta$ on the ScPO loss using these weighted pairs.
5. Replace $M_t$ with $M_\theta$ and repeat.

The process is formalized as follows:

```latex
\begin{algorithm}[H]
\caption{Self-Consistency Preference Optimization (ScPO)}
\begin{algorithmic}[1]
\Require seed model \(M_0\), seed queries \(\mathcal D_0\), iterations \(T\), votes \(k\), threshold \(\tau\)
\For{\(t=0\) to \(T-1\)}
    \State \(\mathcal D_{t+1}\leftarrow \mathcal D_t\)
    \State Generate new problems, filter where \(\max_y\mathcal V(y)<\tau\)
    \State \(\mathcal D_{t+1}\cup=\{\text{new queries}\}\)
    \State Initialize \(\mathcal P_t\)
    \For{each \(x\in\mathcal D_{t+1}\)}
      \State Sample \(\{y_1,\dots,y_k\}\sim M_t(\cdot\mid x)\)
      \State Compute \(\mathcal V(y_i)\) for all \(i\)
      \State \(y^+\leftarrow\arg\max_i\mathcal V(y_i)\),
             \(y^-\leftarrow\arg\min_i\mathcal V(y_i)\)
      \If{\(\mathcal V(y^+)\ge\tau\)}
        \State \(w\leftarrow(\mathcal V(y^+)-\mathcal V(y^-))/k\)
        \State Add \(\bigl(x,y^+,y^-,w\bigr)\) to \(\mathcal P_t\)
      \EndIf
    \EndFor
    \State Train \(M_\theta\) by minimizing \(\sum_{(x,y^+,y^-,w)\in\mathcal P_t}\mathcal L_{ScPO}(x;y^+,y^-)\)
    \State \(M_{t+1}\leftarrow M_\theta\)
\EndFor
\end{algorithmic}
\end{algorithm}
```
Empirically, two iterations are sufficient for convergence and further iterations yield diminishing returns.

## 4. Empirical Results and Benchmarks

Experiments span math and logic domains, with principal evaluation on GSM8K (math word problems), MATH (complex math questions), and ZebraLogic (logic grid puzzles) [2411.04109]. Training uses Llama-3 (8B) as the base, with larger models as comparative baselines. The protocol includes both purely unsupervised training (using only model-generated preference pairs) and a semi-supervised variant (using available labeled data).

Key results are summarized in the tables below:

**GSM8K Zero-Shot Exact-Match Accuracy (%)**

| Method                     | Train data (K)         | Greedy | SC 8-way |
|----------------------------|------------------------|-------:|---------:|
| Seed \(M_0\)               | –                      | 41.17  | 51.80    |
| IRPO$_\mathrm{RM}$ $M_2$   | seed 4.4 + gen –       | 50.11  | 61.25    |
| ScPO$_\mathrm{Unsup.}$ $M_2$| seed 1.4 + gen 5.1   | 63.91  | 71.11    |
| IRPO$_\mathrm{Gold}$ $M_2$ | seed 5.7 + gen –       | 64.29  | 72.56    |
| ScPO$_\mathrm{Semi}$ $M_2$ | seed 5.7 + gen 4.5     | 66.64  | 74.75    |

**MATH Zero-Shot Exact-Match Accuracy (%)**

| Method                      | Train data (K)        | Greedy | SC 8-way |
|-----------------------------|-----------------------|-------:|---------:|
| Seed \(M_0\)                | –                     | 14.46  | 18.20    |
| IRPO$_\mathrm{RM}$ $M_2$    | seed 6.5 + gen –      | 18.08  | 22.64    |
| ScPO$_\mathrm{Unsup.}$ $M_2$| seed 1.2 + gen 2.5   | 19.72  | 24.58    |
| IRPO$_\mathrm{Gold}$ $M_2$  | seed 3.0 + gen –      | 20.32  | 26.88    |
| ScPO$_\mathrm{Semi}$ $M_2$  | seed 3.0 + gen 2.2    | 20.48  | 26.92    |

**ZebraLogic Logic Grid Puzzle—Cell Acc. (%)**

| Model                | Train seed K + gen K | Puzzle ↑ | Cell ↑ |
|----------------------|---------------------|----------|--------|
| Llama-3 70B          | –                   | 17.2     | 42.9   |
| Gemma-2 27B          | –                   | 16.3     | 41.2   |
| Claude-3 Haiku       | –                   | 14.3     | 37.9   |
| $M_0$ Llama-3 8B     | –                   | 11.6     | 39.1   |
| IRPO$_\mathrm{RM}$   | seed 1.0            | 11.3     | 42.1   |
| ScPO$_\mathrm{Unsup.}$| seed 0.4 + gen 2.2 | 18.1     | 45.2   |

Statistical significance was not explicitly reported, but margins (2–8 pp on GSM8K/MATH; 6 pp on ZebraLogic) are well outside normal random variation.

## 5. Analysis, Advantages, and Limitations

Quantitative ablations reveal that weighting the loss by the degree of consistency yields 1–2 pp accuracy improvement over unweighted variants. The consistency threshold $\tau$ governs precision–recall trade-off in preference generation; $\tau \approx 0.5k$ yields optimal results [2411.04109].

Theoretical insights:
- No formal convergence guarantees, but empirical evidence shows saturation after two rounds.
- Consistency (vote share) is strongly correlated with ground truth accuracy (Somers’ D ≈ 0.8 for GSM8K, 0.68 for MATH, 0.92 for ZebraLogic), which justifies self-consistency as a proxy for correctness.
- ScPO acts as a distillation of the empirical “consistency distribution” into the model’s base prediction distribution, boosting accuracy and pseudo-likelihood of correct outputs.

Key limitations:
- Requires the seed model to initially exhibit non-trivial self-consistency; on extremely difficult or under-specified tasks, bootstrapping may cover only a minority of samples.
- Currently designed for single-answer reasoning; adaptation to open-ended or generative tasks remains nontrivial.
- No formal or theoretical convergence proof; empirically, performance gains saturate after two or three rounds.

## 6. Practical Recommendations for Implementation

ScPO can be used with any LLM capable of chain-of-thought generation; instruction tuning is helpful but not mandatory. For effective deployment:
- Use $k=8$ samples per prompt (16 for broad output spaces) to estimate consistency, with temperature $\approx 0.7$ and top-$p=0.9$ for chosen solutions, temperature $\approx 1.2$ to diversify rejected ones.
- Set $\tau \approx 0.5k$ initially, raising to $0.6k-0.7k$ in later rounds as model consistency improves.
- Optimize with $\beta=0.5$, $\alpha=1$; hyperparameter tuning is recommended if validation data are available.
- Two full ScPO iterations are typically sufficient; a third pass can be applied on held-out queries if data privacy allows.
- Computational requirements are similar to ordinary preference-based finetuning; ScPO does not increase inference-time complexity and is compatible with concurrent inference-time self-consistency.
- If any gold solutions are available, include them as labeled preference pairs with $w(x)=1$ for additional gain.

ScPO enables robust, annotation-free finetuning for multi-step reasoning and logic tasks, successfully translating the inference-time self-consistency signal into a direct and effective training signal [2411.04109].

Source: https://www.emergentmind.com/topics/self-consistency-preference-optimization-scpo