---
title: Self-Consistency Rate in LLMs
url: https://www.emergentmind.com/topics/self-consistency-rate
type: topic
---

# Self-Consistency Rate in LLMs

Self-consistency rate denotes a family of agreement measures for stochastic language-model outputs. In the canonical chain-of-thought setting, it is the fraction of sampled reasoning paths whose extracted final answers coincide with the selected modal answer, and it functions simultaneously as a decoding rule, a confidence surrogate, and a diagnostic of reasoning stability [2203.11171]. Subsequent work formalizes the same underlying notion as probability mass on the model’s own majority answer under stochastic decoding, probability that finite-budget majority vote matches an infinite-budget or population majority, or accuracy of majority-vote recovery of the answer-mode; adjacent literatures use related self-consistency rates for evaluator repeatability, cross-context agreement, and repeated-error persistence [2509.15172][2602.16745][2510.17472][2412.00543][2505.17656].

## 1. Canonical definition in chain-of-thought decoding

The original formulation introduces an unobserved reasoning path $r$ alongside the final answer $a$, with joint model score $p(r,a\mid x)$ for input $x$. The answer marginal is

$$
p(a\mid x)=\sum_r p(r,a\mid x).
$$

Because this sum is intractable, self-consistency approximates it by sampling $T$ reasoning paths $\{r_i\}_{i=1}^T$ and then, in practice, replacing probability weighting with an unweighted count over extracted final answers. If $a_i$ is the final answer extracted from sample $i$, the selected answer is

$$
a^*=\arg\max_a \sum_{i=1}^T \mathbf{1}[a_i=a],
$$

and its self-consistency rate is

$$
SC(a^*\mid x)=\frac{1}{T}\sum_{i=1}^T \mathbf{1}[a_i=a^*].
$$

Operationally, self-consistency replaces greedy chain-of-thought decoding by a sample-and-marginalize procedure: prepend few-shot chain-of-thought examples, draw $T$ independent samples under a temperature or top-$p$/top-$k$ scheme, parse each generation into a reasoning path and final answer, tally answer counts, and select the most frequent answer. The paper also gives an optional weighting by normalized log-probability, but reports that simple majority voting already suffices [2203.11171].

This decoding rule was introduced as a replacement for naive greedy decoding in chain-of-thought prompting and was reported to improve performance on multiple arithmetic and commonsense reasoning benchmarks, including GSM8K, SVAMP, AQuA, StrategyQA, and ARC-challenge [2203.11171].

## 2. Probabilistic interpretations of the rate

Later work makes explicit that the empirical agreement fraction is a Monte Carlo estimator of a latent answer-level probability mass. In the MACA formalization, a reasoning trajectory $y$ is sampled from $\pi_{\theta,\tau}(y\mid x)$ under temperature $\tau$, a deterministic function $A(y)$ extracts the answer, and the induced answer distribution is

$$
P_{\theta,\tau}(a\mid x)=\sum_{y:A(y)=a}\pi_{\theta,\tau}(y\mid x).
$$

The model’s internal consensus answer is

$$
a^*_{\theta,\tau}(x)=\arg\max_a P_{\theta,\tau}(a\mid x),
$$

and the true self-consistency mass is

$$
S^+_{\theta,\tau}(x)=P_{\theta,\tau}(a^*_{\theta,\tau}(x)\mid x).
$$

Since this quantity is intractable, it is estimated from $t$ sampled trajectories by the empirical majority answer $\hat a(x)$ and the single-agent sampling consistency

$$
s_t^{\theta,\tau}(x)=\frac{1}{t}\sum_{i=1}^t \mathbf{1}[A(y_i)=\hat a(x)].
$$

Averaged over a test set $X$ of $N$ prompts, this yields

$$
SCR=\frac{1}{N}\sum_{x\in X}s_t^{\theta,\tau}(x).
$$

As $t\to\infty$, $s_t^{\theta,\tau}(x)\to S^+_{\theta,\tau}(x)$ [2509.15172].

A distinct but closely related interpretation appears in PETS, where each question induces an answer distribution $\theta_i$ over categories. The infinite-budget majority label is

$$
y_i^\infty:=\arg\max_{y\in[M]}\theta_{i,y},
$$

and the self-consistency rate at budget $B$ is

$$
SC(q_i;B)=P(Y_i^{Maj}(B)=y_i^\infty).
$$

Here the target is not ground-truth correctness but agreement with the infinite-sampling consensus. PETS states that $SC(q_i;B)$ is nondecreasing in $B$ and approaches $1$ as $B\to\infty$ if $\theta_i$ has a unique maximum [2602.16745].

A third formalization, in certified self-consistency, treats majority vote as an estimator of the mode of the model’s terminal answer distribution. If $X_1,\dots,X_n\sim \mathrm{Cat}(\mathbf p)$ are terminal answers, $c^\star=\arg\max_j p_j$ is the Bayes-optimal answer under $0$–$1$ loss, and $\widehat c_n=\arg\max_j \hat p_{n,j}$ is the majority-vote estimate, then

$$
\mathrm{SC}_n=\Pr[\widehat c_n=c^\star].
$$

Under $p_{c^\star}>\max_{j\neq c^\star}p_j$, the paper derives finite-sample concentration bounds on $\Pr[\widehat c_n\neq c^\star]$ and an anytime-valid Martingale Majority Certificate for adaptive stopping [2510.17472].

## 3. Empirical behavior and relation to correctness

In the original chain-of-thought study, increasing the number of samples raises both average self-consistency rate and answer accuracy. On GSM8K with PaLM-540B, greedy chain-of-thought achieved $56.5\%$ accuracy, while self-consistency with $T=40$ reached $74.4\%$, a gain of $+17.9\%$; the mean self-consistency rate for correct predictions was substantially higher than for incorrect ones, and a plot of consistency versus error rate showed a strong negative correlation [2203.11171].

The same pattern extends beyond symbolic reasoning. On MMLU, a paper specifically targeted at encyclopedic knowledge recall defines the agreement score

$$
s=\frac{\max_{a\in A}\sum_{i=1}^n \mathbf{1}[a_i=a]}{n},
$$

with majority-vote answer $\hat a=\arg\max_{a\in A}\sum_{i=1}^n \mathbf{1}[a_i=a]$. Using GPT-4o, direct answering on the MMLU test achieved $83.26/75.45/85.56$ on All/Reasoning/Knowledge, zero-shot chain-of-thought with $n=1$ achieved $87.86/90.38/87.12$, and chain-of-thought plus self-consistency achieved $88.64/91.32/87.85$ for $n=5$ and $88.93/91.94/88.04$ for $n=20$; on prototypical benchmarks, GSM8K moved from $84.23$ to $84.46$ and MedMCQA from $76.76$ to $77.67$ when self-consistency was added to chain-of-thought [2604.19395]. The same study reports a Pearson correlation of approximately $0.40$–$0.46$ between the agreement-based confidence score and correctness on MMLU [2604.19395].

MACA reports that post-training can raise single-agent SCR itself. With $t=20$ samples at $\tau=1.0$, Qwen-2B on GSM8K increased from approximately $26.4\%$ to approximately $54.0\%$, Qwen-2B on MATH from approximately $12.0\%$ to approximately $27.8\%$, and Llama-3B on MathQA from approximately $31.2\%$ to approximately $49.4\%$. These SCR gains strongly correlated with accuracy improvements, with Pearson $r>0.86$ [2509.15172].

Taken together, these results support the use of self-consistency rate as an intrinsic reliability signal, but only within a task-dependent decoding regime. The literature does not treat the rate as a substitute for ground-truth evaluation; rather, it uses agreement structure to filter stochastic variation and to estimate whether the model’s internal answer distribution is concentrated.

## 4. Estimation error, convergence, and efficiency-oriented variants

The principal theoretical critique of vanilla self-consistency is its slow Monte Carlo convergence. In the RPC analysis, if $\hat y$ is a candidate answer and $p(\hat y\mid x)$ is its true answer-level probability, the self-consistency estimator is

$$
\hat p^{(Sc)}(\hat y\mid x)=\frac{1}{n}\sum_{i=1}^n \mathbf{1}\{\tilde y_i=\hat y\},
$$

with total squared error

$$
\mathcal E\bigl(\hat p^{(Sc)}\bigr)
=
\underbrace{\frac{1}{n}\,p(\hat y\mid x)\bigl(1-p(\hat y\mid x)\bigr)}_{E_{\rm est}}
+
\underbrace{\bigl(p(\hat y\mid x)-\mathbf 1\{\hat y=y\}\bigr)^2}_{E_{\rm model}}.
$$

Accordingly,

$$
E_{\rm est}^{(Sc)}=O(1/n),
$$

and the paper argues that self-consistency therefore exhibits high estimation error relative to methods whose estimation error decays exponentially in $n$ [2502.00511]. A parallel theoretical treatment gives the same decomposition and bound, emphasizing that the variance term can remain large for modest budgets and motivating hybrid estimators such as Perplexity Consistency and Reasoning Pruning [2510.15444].

A broad line of work modifies the aggregation rule to lower sampling cost or improve ranking among sampled answers. Confidence-Informed Self-Consistency (CISC) replaces uniform voting with confidence-weighted voting using scores such as response probability, verbal confidence, or $P(\mathrm{True})$, followed by softmax normalization. It defines

$$
â_{SC}=\arg\max_a \sum_{i=1}^m \mathbf{1}[a_i=a]
$$

for vanilla self-consistency and

$$
â_{CISC}=\arg\max_a \sum_{i=1}^m \mathbf{1}[a_i=a]\cdot \tilde c_i
$$

for the weighted version. Across nine models and four datasets, the paper reports that CISC reduces the required number of reasoning paths by over $40\%$ on average; with $P(\mathrm{True})$, macro-averaged cost reduction was $41\%$ at budget $5$ and $46\%$ at budget $10$ [2502.06233].

Soft Self-Consistency replaces majority counting by a continuous score derived from token probabilities of each candidate solution. On interactive tasks with many distinct valid outputs, it was reported to require half as many samples as self-consistency for comparable or better performance, with absolute success-rate gains of $1.3\%$ on Bash, $6.6\%$ on WebShop, and $4.7\%$ on ALFWorld at fixed $k=10$ [2402.13212].

Reasoning-Aware Self-Consistency (RASC) adds a sufficiency score $SS_i\in[0,1]$ to each sampled reasoning-answer pair, performs weighted majority voting

$$
\hat a=\arg\max_{a\in\mathcal A}\sum_{i:a_i=a} SS_i,
$$

and stops sampling once a buffer of sufficiently high-quality samples reaches a target size. The paper reports approximately $70\%$ sample reduction while maintaining accuracy in the abstract, and elsewhere summarizes empirical savings as approximately $70$–$80\%$ fewer samples while matching self-consistency’s accuracy [2408.17017].

RISC reformulates answer selection as a ranking problem with a lightweight LambdaRank model using five features: answer length, ratio-to-best, semantic centrality, worst-step coherence, and shared-checkpoints count. On PopQA, HotpotQA, and MATH500, it was reported to achieve a better accuracy-efficiency trade-off than standard self-consistency and strong baselines; on PopQA, RISC with only $18$ samples surpassed self-consistency at $99$ samples [2606.05054].

Finally, PETS shifts the optimization target from per-question majority frequency to agreement with the infinite-budget majority. In offline and online allocation settings, it reports reductions in sampling budget by up to $75\%$ and $55\%$, respectively, relative to uniform allocation, while achieving perfect self-consistency on GPQA in both settings [2602.16745].

## 5. Alternative meanings in adjacent literatures

The expression “self-consistency rate” is not uniform across the broader literature. It denotes related but non-identical quantities depending on the object being sampled, compared, or certified.

| Setting | Representative definition | Source |
|---|---|---|
| Chain-of-thought reasoning | Fraction of sampled paths agreeing on the modal answer | [2203.11171] |
| Stochastic trajectory consensus | Average over prompts of sampled agreement with the empirical majority answer | [2509.15172] |
| Finite-budget allocation | Probability that finite-budget majority matches infinite-budget majority | [2602.16745] |
| Certifiable inference | Probability that majority vote recovers the mode of the answer distribution | [2510.17472] |
| LLM evaluators | Krippendorff’s $\alpha$ over $K$ replicate sampled scores | [2412.00543] |
| Self-consistent errors | Fraction of incorrect instances whose greedy answer is semantically repeated across all stochastic samples | [2505.17656] |
| Ambiguous completion/explanation | Cross-context agreement rate $R=(1/|D|)\sum_{S\in D}\delta(S)$ | [2310.13439] |
| Multi-step reasoning | Hypothetical consistency rate $C_h$ and compositional consistency rate $C_c$ | [2305.14279] |
| Generator-evaluator pipelines | Chance-adjusted agreement between generated outputs and self-evaluations | [2606.30653] |

In the evaluator literature, self-consistency is explicitly detached from human agreement and defined as intra-model repeatability. Lee et al. sample $K=5$ replicate scores from the same evaluator model at temperature $1.0$ and compute $\alpha(\{s_i\}_{i=1}^K)$. They report that Mistral-Instruct typically achieves $\alpha>0.90$ on interval scales and remains in the $0.80$–$0.95$ range on 5-point Likert and binary scales, whereas several other models degrade sharply on non-numeric scales [2412.00543].

In ambiguity studies, the statistic becomes cross-context coherence rather than answer-frequency agreement. On ambiguous integer sequence completion, observed self-consistency rates were $66.9\%$ for text-davinci-003, $74.7\%$ for gpt-3.5-turbo, and $82.2\%$ for gpt-4, all substantially above the corresponding random baselines [2310.13439]. In multi-step reasoning, hypothetical and compositional consistency rates diagnose whether a model preserves its own sub-answers under prompt transformation or substitution. Even GPT-4 remained below $65\%$ compositional consistency on the reported tasks [2305.14279].

A different extension appears in generator-evaluator self-consistency. There, the model first produces an output and then judges whether that output satisfies the invoked concept. Chance-adjusted consistency $\widehat C^{\rm adj}(f,c,t)$ is averaged across tests and concepts, yielding approximate model-wide scores from $0.28$ for Claude Sonnet 4.5 up to $0.84$ for Gemini 3 Pro [2606.30653].

## 6. Failure modes, misconceptions, and controversies

A recurrent misconception is that high self-consistency implies correctness. The literature repeatedly rejects that equivalence. The MACA study notes that greedy decoding as $\tau\to0$ can achieve near-$100\%$ consistency by collapse while often yielding suboptimal answers, and explicitly states that high self-consistency does not automatically imply the best reasoning [2509.15172]. The original chain-of-thought paper likewise identifies spurious consensus as a failure mode: if sampling is systematically biased toward a common but wrong heuristic, self-consistency can reinforce error rather than correct it [2203.11171].

This limitation becomes acute in the study of self-consistent errors. There, an error is self-consistent when the greedy response is wrong and all stochastic samples are semantically equivalent to that same wrong answer. Across SciQ and TriviaQA and nine checkpoints, inconsistent-error frequency fell from approximately $40$–$50\%$ at small scales to approximately $5$–$10\%$ at the largest scales, while self-consistent-error frequency remained approximately $15$–$25\%$ or even increased slightly [2505.17656]. Detection methods struggled sharply on these cases: on Llama3.1-8B and SciQ, semantic entropy achieved AUROC $0.88$ on inconsistent errors but only approximately $0.46$ on self-consistent errors, and even the best supervised out-of-domain probe dropped from $0.91$ to $0.73$ [2505.17656].

A related controversy concerns whether self-consistency is intrinsically desirable in self-evaluating pipelines. In the “consistency dilemma” study, higher generator-evaluator self-consistency was associated with greater vulnerability to physician-validated clinical mistakes even after controlling for benchmark accuracy; the reported coefficient on self-consistency was $\hat\beta_2=1.116$ with standard error $0.224$ and $p<10^{-6}$, with the authors interpreting the effect as a between-model pattern [2606.30653]. This suggests that stable application of an internal criterion can coexist with systematic error if the criterion itself is flawed.

Other literatures make the same point in different forms. In ambiguity tasks, models were often miscalibrated when judging their own consistency, with over-confidence in some models and under-confidence in gpt-4 [2310.13439]. In evaluator studies, strong proprietary models were not necessarily the most self-consistent evaluators [2412.00543]. In multi-step reasoning, correctness could exceed compositional consistency, indicating that a model can arrive at a correct final answer without being internally reusable or transformation-stable [2305.14279].

A plausible implication is that self-consistency rate is best treated as a structural statistic of a model’s answer distribution or decision process, not as a direct synonym for truthfulness. The surveyed literature consistently uses it to characterize concentration, stability, and agreement under sampling, while supplementing it with external verification, task accuracy, or certified stopping criteria when reliability is the objective.

Source: https://www.emergentmind.com/topics/self-consistency-rate