Skip to content
sinc-LLMBook a 15-minute call

Self-Consistency Prompting: How It Works, an Example and When It Pays

Self-consistency samples several reasoning paths and keeps the most common answer. What the paper measured, a worked example and when the extra calls pay.

On this page

By Mario Alexandre · November 19, 2026 · 5 min read

The short answer: self-consistency prompting asks the model the same question several times with sampling turned on, lets each run reason its own way to an answer, and keeps the answer that appears most often. It works when the task has one correct answer from a fixed set, such as a number, a label or a yes or no. In the original paper it raised GSM8K math accuracy for PaLM 540B from 56.5% to 74.4%. It also costs one model call per sample, so use it where a wrong answer costs more than the extra calls.

What self-consistency is

Self-consistency was introduced by Wang and colleagues at Google Research in "Self-Consistency Improves Chain of Thought Reasoning in Language Models", published at ICLR 2023. Chain-of-thought prompting asks a model to write out its reasoning before answering. Normally the model's single most likely reasoning path is used. Self-consistency replaces that with what the authors call a "sample-and-marginalize" procedure:

  1. Prompt the model with chain-of-thought, as usual.
  2. Sample several outputs with temperature above zero, so each run can reason differently.
  3. Read only the final answer of each run.
  4. Return the answer that most runs agree on.

The intuition in the paper is that a hard problem usually has several valid ways of reasoning toward its one correct answer, while wrong reasoning paths are less likely to arrive at the same answer.

What the paper measured

The authors compared self-consistency with ordinary chain-of-thought on arithmetic and commonsense reasoning benchmarks across several models, sampling 40 outputs per question and averaging results over 10 runs. Their abstract reports gains on GSM8K (+17.9%), SVAMP (+11.0%), AQuA (+12.2%), StrategyQA (+6.4%) and ARC-challenge (+3.9%). These are absolute gains in percentage points, not relative improvements. The models tested (PaLM 540B, LaMDA 137B, GPT-3 and UL2) were published between 2020 and 2022, so gains on current models can be smaller or larger: measure on your own task.

Model Benchmark Chain-of-thought Self-consistency (40 paths)
PaLM 540B GSM8K 56.5% 74.4%
PaLM 540B AQuA 35.8% 48.3%
PaLM 540B SVAMP 79.0% 86.6%
LaMDA 137B GSM8K 17.1% 27.7%

Three findings matter most for production use:

  • Agreement predicts correctness. The share of samples that agree with the final answer was highly correlated with accuracy. The authors say low consistency can be read as low confidence, so the method gives the model some ability to "know when it doesn't know".
  • Most of the gain comes early. The paper names the extra computation as a limitation and suggests trying "a small number of paths (e.g., 5 or 10)" as a starting point, because performance saturates quickly.
  • It needs a fixed answer. The paper states that self-consistency "can be applied only to problems where the final answer is from a fixed answer set". Free text needs a separate way to decide whether two answers agree.

A worked example with a vote and an abstain rule

The example is fictional. An accounts payable team uses a prompt to decide whether an invoice line is covered by an existing purchase order. The answer is one of three labels: covered, not covered or needs review. A wrong "covered" lets an unapproved payment through.

The team runs the prompt 5 times at temperature 0.7 for each line and applies two rules: take the majority label, and if fewer than 4 of the 5 samples agree, send the line to a person whatever the majority says.

Invoice line The 5 sampled answers Majority Agreement Action
Line A: 40 laptop chargers, PO 1182 lists 40 covered ×5 covered 5 of 5 Pay
Line B: 12 monitors, PO 1182 lists 10 not covered ×4, covered ×1 not covered 4 of 5 Hold for a new PO
Line C: "installation service", PO wording unclear covered ×3, needs review ×2 covered 3 of 5 Send to a person

A single call on line C might have returned "covered" and paid it. The split vote is the signal that the input is ambiguous, which is exactly the case the team wants a person to see.

When the extra calls are worth it

Five samples cost roughly five times the output tokens and usually five calls' worth of rate limit, even when they run in parallel. Run the numbers for your own task before switching it on everywhere:

  • Use it where the answer is a label, number or choice, where a wrong answer is expensive, and where your tests show single-call accuracy is not good enough.
  • Use the agreement rate as a routing signal even when accuracy is already high: unanimous answers go through, split answers go to a person.
  • Skip it for free-form writing, for tasks where single calls are already near your error target, and where the model is consistently wrong in the same way. If all five samples share the same mistake, voting will not fix it. Our analysis of whether majority voting improves LLM reliability looks at that failure case.
  • Measure before and after on your own labelled cases: accuracy at 1, 3, 5 and 10 samples, cost per item, and how many items the abstain rule sends to people.

Decide the sample count on your own task mix

Multi-Shot Reliability Layer is a scoped project that adds this kind of sampling policy to your system: when voting helps, when its assumptions fail, which sample count to use and when to hand an item to a person, validated on your own task mix. The published project price is $15,000, with scope and timing confirmed in the proposal.

To check one prompt first, our guide on how to test a prompt shows how to build the labelled cases this decision depends on.

Book a 15-minute call about sampling and voting · Request an estimate by email

Frequently asked questions about self-consistency

Is self-consistency the same as asking the model to check its own answer?

No. Self-checking asks one model run to review its answer. Self-consistency runs the question several times independently and compares the final answers. It does not need the model to recognize its own mistake; it relies on wrong reasoning paths disagreeing with each other.

Does self-consistency work at temperature 0?

Not usefully. At temperature 0 the samples are usually identical, so the vote adds cost without adding information. The paper sampled with temperature 0.7 for PaLM 540B and GPT-3, and reported that results were robust across a range of sampling settings.

Can I use self-consistency for summaries or other free text?

Only with an extra step that decides when two texts agree, for example by extracting the key facts from each and comparing those. The original method assumes a fixed set of possible answers.

Sources

The method, benchmark results, sampling settings and limitations come from the original paper. The invoice example and its rules are fictional. Wang et al., Self-Consistency Improves Chain of Thought Reasoning in Language Models, ICLR 2023 (arXiv:2203.11171)