Self-Consistency Rate in LLMs

• Self-Consistency Rate is a metric that quantifies the fraction of sampled reasoning paths that agree on the modal answer, serving as a decoding rule.
• It functions as a surrogate for confidence and a diagnostic tool, linking internal decision concentration with improved accuracy in language models.
• Variants such as confidence-weighted and efficiency-boosted methods enhance sample efficiency, reduce error, and refine the estimation of agreement probabilities.

Self-consistency rate denotes a family of agreement measures for stochastic language-model outputs. In the canonical chain-of-thought setting, it is the fraction of sampled reasoning paths whose extracted final answers coincide with the selected modal answer, and it functions simultaneously as a decoding rule, a confidence surrogate, and a diagnostic of reasoning stability (Wang et al., 2022). Subsequent work formalizes the same underlying notion as probability mass on the model’s own majority answer under stochastic decoding, probability that finite-budget majority vote matches an infinite-budget or population majority, or accuracy of majority-vote recovery of the answer-mode; adjacent literatures use related self-consistency rates for evaluator repeatability, cross-context agreement, and repeated-error persistence (Samanta et al., 18 Sep 2025, Liu et al., 18 Feb 2026, Cordero-Encinar et al., 20 Oct 2025, Lee et al., 2024, Tan et al., 23 May 2025).

1. Canonical definition in chain-of-thought decoding

The original formulation introduces an unobserved reasoning path $r$ alongside the final answer $a$, with joint model score $p(r,a\mid x)$ for input $x$. The answer marginal is

$p(a\mid x)=\sum_r p(r,a\mid x).$

Because this sum is intractable, self-consistency approximates it by sampling $T$ reasoning paths $\{r_i\}_{i=1}^T$ and then, in practice, replacing probability weighting with an unweighted count over extracted final answers. If $a_i$ is the final answer extracted from sample $i$, the selected answer is

$a^*=\arg\max_a \sum_{i=1}^T \mathbf{1}[a_i=a],$

and its self-consistency rate is

$a$0

Operationally, self-consistency replaces greedy chain-of-thought decoding by a sample-and-marginalize procedure: prepend few-shot chain-of-thought examples, draw $a$1 independent samples under a temperature or top-$a$2/top-$a$3 scheme, parse each generation into a reasoning path and final answer, tally answer counts, and select the most frequent answer. The paper also gives an optional weighting by normalized log-probability, but reports that simple majority voting already suffices (Wang et al., 2022).

This decoding rule was introduced as a replacement for naive greedy decoding in chain-of-thought prompting and was reported to improve performance on multiple arithmetic and commonsense reasoning benchmarks, including GSM8K, SVAMP, AQuA, StrategyQA, and ARC-challenge (Wang et al., 2022).

2. Probabilistic interpretations of the rate

Later work makes explicit that the empirical agreement fraction is a Monte Carlo estimator of a latent answer-level probability mass. In the MACA formalization, a reasoning trajectory $a$4 is sampled from $a$5 under temperature $a$6, a deterministic function $a$7 extracts the answer, and the induced answer distribution is

$a$8

The model’s internal consensus answer is

$a$9

and the true self-consistency mass is

$p(r,a\mid x)$0

Since this quantity is intractable, it is estimated from $p(r,a\mid x)$1 sampled trajectories by the empirical majority answer $p(r,a\mid x)$2 and the single-agent sampling consistency

$p(r,a\mid x)$3

Averaged over a test set $p(r,a\mid x)$4 of $p(r,a\mid x)$5 prompts, this yields

$p(r,a\mid x)$6

As $p(r,a\mid x)$7, $p(r,a\mid x)$8 (Samanta et al., 18 Sep 2025).

A distinct but closely related interpretation appears in PETS, where each question induces an answer distribution $p(r,a\mid x)$9 over categories. The infinite-budget majority label is

$x$0

and the self-consistency rate at budget $x$1 is

$x$2

Here the target is not ground-truth correctness but agreement with the infinite-sampling consensus. PETS states that $x$3 is nondecreasing in $x$4 and approaches $x$5 as $x$6 if $x$7 has a unique maximum (Liu et al., 18 Feb 2026).

A third formalization, in certified self-consistency, treats majority vote as an estimator of the mode of the model’s terminal answer distribution. If $x$8 are terminal answers, $x$9 is the Bayes-optimal answer under $p(a\mid x)=\sum_r p(r,a\mid x).$0–$p(a\mid x)=\sum_r p(r,a\mid x).$1 loss, and $p(a\mid x)=\sum_r p(r,a\mid x).$2 is the majority-vote estimate, then

$p(a\mid x)=\sum_r p(r,a\mid x).$3

Under $p(a\mid x)=\sum_r p(r,a\mid x).$4, the paper derives finite-sample concentration bounds on $p(a\mid x)=\sum_r p(r,a\mid x).$5 and an anytime-valid Martingale Majority Certificate for adaptive stopping (Cordero-Encinar et al., 20 Oct 2025).

3. Empirical behavior and relation to correctness

In the original chain-of-thought study, increasing the number of samples raises both average self-consistency rate and answer accuracy. On GSM8K with PaLM-540B, greedy chain-of-thought achieved $p(a\mid x)=\sum_r p(r,a\mid x).$6 accuracy, while self-consistency with $p(a\mid x)=\sum_r p(r,a\mid x).$7 reached $p(a\mid x)=\sum_r p(r,a\mid x).$8, a gain of $p(a\mid x)=\sum_r p(r,a\mid x).$9; the mean self-consistency rate for correct predictions was substantially higher than for incorrect ones, and a plot of consistency versus error rate showed a strong negative correlation (Wang et al., 2022).

The same pattern extends beyond symbolic reasoning. On MMLU, a paper specifically targeted at encyclopedic knowledge recall defines the agreement score

$T$0

with majority-vote answer $T$1. Using GPT-4o, direct answering on the MMLU test achieved $T$2 on All/Reasoning/Knowledge, zero-shot chain-of-thought with $T$3 achieved $T$4, and chain-of-thought plus self-consistency achieved $T$5 for $T$6 and $T$7 for $T$8; on prototypical benchmarks, GSM8K moved from $T$9 to $\{r_i\}_{i=1}^T$0 and MedMCQA from $\{r_i\}_{i=1}^T$1 to $\{r_i\}_{i=1}^T$2 when self-consistency was added to chain-of-thought (Hoshino et al., 21 Apr 2026). The same study reports a Pearson correlation of approximately $\{r_i\}_{i=1}^T$3–$\{r_i\}_{i=1}^T$4 between the agreement-based confidence score and correctness on MMLU (Hoshino et al., 21 Apr 2026).

MACA reports that post-training can raise single-agent SCR itself. With $\{r_i\}_{i=1}^T$5 samples at $\{r_i\}_{i=1}^T$6, Qwen-2B on GSM8K increased from approximately $\{r_i\}_{i=1}^T$7 to approximately $\{r_i\}_{i=1}^T$8, Qwen-2B on MATH from approximately $\{r_i\}_{i=1}^T$9 to approximately $a_i$0, and Llama-3B on MathQA from approximately $a_i$1 to approximately $a_i$2. These SCR gains strongly correlated with accuracy improvements, with Pearson $a_i$3 (Samanta et al., 18 Sep 2025).

Taken together, these results support the use of self-consistency rate as an intrinsic reliability signal, but only within a task-dependent decoding regime. The literature does not treat the rate as a substitute for ground-truth evaluation; rather, it uses agreement structure to filter stochastic variation and to estimate whether the model’s internal answer distribution is concentrated.

4. Estimation error, convergence, and efficiency-oriented variants

The principal theoretical critique of vanilla self-consistency is its slow Monte Carlo convergence. In the RPC analysis, if $a_i$4 is a candidate answer and $a_i$5 is its true answer-level probability, the self-consistency estimator is

$a_i$6

with total squared error

$a_i$7

Accordingly,

$a_i$8

and the paper argues that self-consistency therefore exhibits high estimation error relative to methods whose estimation error decays exponentially in $a_i$9 (Zhou et al., 1 Feb 2025). A parallel theoretical treatment gives the same decomposition and bound, emphasizing that the variance term can remain large for modest budgets and motivating hybrid estimators such as Perplexity Consistency and Reasoning Pruning (Zhou et al., 17 Oct 2025).

A broad line of work modifies the aggregation rule to lower sampling cost or improve ranking among sampled answers. Confidence-Informed Self-Consistency (CISC) replaces uniform voting with confidence-weighted voting using scores such as response probability, verbal confidence, or $i$0, followed by softmax normalization. It defines

$i$1

for vanilla self-consistency and

$i$2

for the weighted version. Across nine models and four datasets, the paper reports that CISC reduces the required number of reasoning paths by over $i$3 on average; with $i$4, macro-averaged cost reduction was $i$5 at budget $i$6 and $i$7 at budget $i$8 (Taubenfeld et al., 10 Feb 2025).

Soft Self-Consistency replaces majority counting by a continuous score derived from token probabilities of each candidate solution. On interactive tasks with many distinct valid outputs, it was reported to require half as many samples as self-consistency for comparable or better performance, with absolute success-rate gains of $i$9 on Bash, $a^*=\arg\max_a \sum_{i=1}^T \mathbf{1}[a_i=a],$0 on WebShop, and $a^*=\arg\max_a \sum_{i=1}^T \mathbf{1}[a_i=a],$1 on ALFWorld at fixed $a^*=\arg\max_a \sum_{i=1}^T \mathbf{1}[a_i=a],$2 (Wang et al., 2024).

Reasoning-Aware Self-Consistency (RASC) adds a sufficiency score $a^*=\arg\max_a \sum_{i=1}^T \mathbf{1}[a_i=a],$3 to each sampled reasoning-answer pair, performs weighted majority voting

$a^*=\arg\max_a \sum_{i=1}^T \mathbf{1}[a_i=a],$4

and stops sampling once a buffer of sufficiently high-quality samples reaches a target size. The paper reports approximately $a^*=\arg\max_a \sum_{i=1}^T \mathbf{1}[a_i=a],$5 sample reduction while maintaining accuracy in the abstract, and elsewhere summarizes empirical savings as approximately $a^*=\arg\max_a \sum_{i=1}^T \mathbf{1}[a_i=a],$6–$a^*=\arg\max_a \sum_{i=1}^T \mathbf{1}[a_i=a],$7 fewer samples while matching self-consistency’s accuracy (Wan et al., 2024).

RISC reformulates answer selection as a ranking problem with a lightweight LambdaRank model using five features: answer length, ratio-to-best, semantic centrality, worst-step coherence, and shared-checkpoints count. On PopQA, HotpotQA, and MATH500, it was reported to achieve a better accuracy-efficiency trade-off than standard self-consistency and strong baselines; on PopQA, RISC with only $a^*=\arg\max_a \sum_{i=1}^T \mathbf{1}[a_i=a],$8 samples surpassed self-consistency at $a^*=\arg\max_a \sum_{i=1}^T \mathbf{1}[a_i=a],$9 samples (Marina et al., 3 Jun 2026).

Finally, PETS shifts the optimization target from per-question majority frequency to agreement with the infinite-budget majority. In offline and online allocation settings, it reports reductions in sampling budget by up to $a$00 and $a$01, respectively, relative to uniform allocation, while achieving perfect self-consistency on GPQA in both settings (Liu et al., 18 Feb 2026).

5. Alternative meanings in adjacent literatures

The expression “self-consistency rate” is not uniform across the broader literature. It denotes related but non-identical quantities depending on the object being sampled, compared, or certified.

Setting	Representative definition	Source
Chain-of-thought reasoning	Fraction of sampled paths agreeing on the modal answer	(Wang et al., 2022)
Stochastic trajectory consensus	Average over prompts of sampled agreement with the empirical majority answer	(Samanta et al., 18 Sep 2025)
Finite-budget allocation	Probability that finite-budget majority matches infinite-budget majority	(Liu et al., 18 Feb 2026)
Certifiable inference	Probability that majority vote recovers the mode of the answer distribution	(Cordero-Encinar et al., 20 Oct 2025)
LLM evaluators	Krippendorff’s $a$02 over $a$03 replicate sampled scores	(Lee et al., 2024)
Self-consistent errors	Fraction of incorrect instances whose greedy answer is semantically repeated across all stochastic samples	(Tan et al., 23 May 2025)
Ambiguous completion/explanation	Cross-context agreement rate $a$04	(Bartsch et al., 2023)
Multi-step reasoning	Hypothetical consistency rate $a$05 and compositional consistency rate $a$06	(Chen et al., 2023)
Generator-evaluator pipelines	Chance-adjusted agreement between generated outputs and self-evaluations	(Mancoridis et al., 16 Jun 2026)

In the evaluator literature, self-consistency is explicitly detached from human agreement and defined as intra-model repeatability. Lee et al. sample $a$07 replicate scores from the same evaluator model at temperature $a$08 and compute $a$09. They report that Mistral-Instruct typically achieves $a$10 on interval scales and remains in the $a$11–$a$12 range on 5-point Likert and binary scales, whereas several other models degrade sharply on non-numeric scales (Lee et al., 2024).

In ambiguity studies, the statistic becomes cross-context coherence rather than answer-frequency agreement. On ambiguous integer sequence completion, observed self-consistency rates were $a$13 for text-davinci-003, $a$14 for gpt-3.5-turbo, and $a$15 for gpt-4, all substantially above the corresponding random baselines (Bartsch et al., 2023). In multi-step reasoning, hypothetical and compositional consistency rates diagnose whether a model preserves its own sub-answers under prompt transformation or substitution. Even GPT-4 remained below $a$16 compositional consistency on the reported tasks (Chen et al., 2023).

A different extension appears in generator-evaluator self-consistency. There, the model first produces an output and then judges whether that output satisfies the invoked concept. Chance-adjusted consistency $a$17 is averaged across tests and concepts, yielding approximate model-wide scores from $a$18 for Claude Sonnet 4.5 up to $a$19 for Gemini 3 Pro (Mancoridis et al., 16 Jun 2026).

6. Failure modes, misconceptions, and controversies

A recurrent misconception is that high self-consistency implies correctness. The literature repeatedly rejects that equivalence. The MACA study notes that greedy decoding as $a$20 can achieve near-$a$21 consistency by collapse while often yielding suboptimal answers, and explicitly states that high self-consistency does not automatically imply the best reasoning (Samanta et al., 18 Sep 2025). The original chain-of-thought paper likewise identifies spurious consensus as a failure mode: if sampling is systematically biased toward a common but wrong heuristic, self-consistency can reinforce error rather than correct it (Wang et al., 2022).

This limitation becomes acute in the study of self-consistent errors. There, an error is self-consistent when the greedy response is wrong and all stochastic samples are semantically equivalent to that same wrong answer. Across SciQ and TriviaQA and nine checkpoints, inconsistent-error frequency fell from approximately $a$22–$a$23 at small scales to approximately $a$24–$a$25 at the largest scales, while self-consistent-error frequency remained approximately $a$26–$a$27 or even increased slightly (Tan et al., 23 May 2025). Detection methods struggled sharply on these cases: on Llama3.1-8B and SciQ, semantic entropy achieved AUROC $a$28 on inconsistent errors but only approximately $a$29 on self-consistent errors, and even the best supervised out-of-domain probe dropped from $a$30 to $a$31 (Tan et al., 23 May 2025).

A related controversy concerns whether self-consistency is intrinsically desirable in self-evaluating pipelines. In the “consistency dilemma” study, higher generator-evaluator self-consistency was associated with greater vulnerability to physician-validated clinical mistakes even after controlling for benchmark accuracy; the reported coefficient on self-consistency was $a$32 with standard error $a$33 and $a$34, with the authors interpreting the effect as a between-model pattern (Mancoridis et al., 16 Jun 2026). This suggests that stable application of an internal criterion can coexist with systematic error if the criterion itself is flawed.

Other literatures make the same point in different forms. In ambiguity tasks, models were often miscalibrated when judging their own consistency, with over-confidence in some models and under-confidence in gpt-4 (Bartsch et al., 2023). In evaluator studies, strong proprietary models were not necessarily the most self-consistent evaluators (Lee et al., 2024). In multi-step reasoning, correctness could exceed compositional consistency, indicating that a model can arrive at a correct final answer without being internally reusable or transformation-stable (Chen et al., 2023).

A plausible implication is that self-consistency rate is best treated as a structural statistic of a model’s answer distribution or decision process, not as a direct synonym for truthfulness. The surveyed literature consistently uses it to characterize concentration, stability, and agreement under sampling, while supplementing it with external verification, task accuracy, or certified stopping criteria when reliability is the objective.