Improve LLM reasoning by asking for step-by-step thinking before the final answer.
Published September 21, 2026
A language model generates its answer one token at a time, and each token is conditioned on everything written before it. If a question needs several steps (a calculation, a plan, a diagnosis) and the model jumps straight to the answer, all that intermediate work has to happen "silently" within a single token prediction, which often fails. Chain-of-thought (CoT) prompting asks the model to write out its intermediate steps first. Each written step becomes context the next steps can build on, which markedly improves accuracy on multi-step problems.
Q: A cafe sells coffee at ₹120 and sandwiches at ₹180. A team orders 7 coffees and 4 sandwiches
and has a 15% discount coupon. What do they pay?
Answered directly, a model may produce a plausible but wrong total. Asked to reason first:
Coffees: 7 × 120 = 840
Sandwiches: 4 × 180 = 720
Subtotal: 840 + 720 = 1,560
Discount: 15% of 1,560 = 234
Total: 1,560 − 234 = 1,326
Answer: ₹1,326
Every step is simple, and the final answer follows from the ones written above it.
Zero-shot CoT: simply ask for it: "Think through this step by step, then give the final answer on the last line as Answer: …." Separating the reasoning from a clearly marked final answer makes the answer easy to parse in code.
Few-shot CoT: include one or two worked examples whose answers show the reasoning. This is more reliable than an instruction alone when you want a particular style of reasoning, for example "list the constraints, then check each option against them".
Structured reasoning: give the steps yourself when you know how the problem should be approached:
Review this method for bugs. Work in this order:
1. State in one sentence what the method is supposed to do.
2. Trace it with input [] and with input [3, 3].
3. List edge cases it doesn't handle.
4. Only then list bugs, each with a fix.
Tracing concrete inputs before judging is especially effective for code review and debugging, because it forces the model to actually simulate the code instead of pattern-matching on how it looks.
Newer "reasoning" or "thinking" models are trained to do an extended internal chain of thought before answering, often with a configurable reasoning budget. With them:
Self-consistency: sample several independent reasoning chains (temperature above 0) and take the most common final answer. Different chains make different mistakes, so the majority is right more often than any single chain. It costs N times the tokens, so use it for high-stakes answers with a single checkable result (a number, a label).
Map<String, Long> votes = IntStream.range(0, 5)
.mapToObj(i -> extractAnswer(chat.prompt().user(question + "\nThink step by step, then end with 'Answer: <value>'.")
.options(OpenAiChatOptions.builder().temperature(0.7).build())
.call().content()))
.collect(Collectors.groupingBy(a -> a, Collectors.counting()));
String answer = Collections.max(votes.entrySet(), Map.Entry.comparingByValue()).getKey();
Decomposition: for large tasks, first ask for a plan (the list of sub-questions), then solve each sub-question in its own call, and combine the results. It's easier to check and debug than one enormous chain.
Verify-then-revise: after an answer, ask the model (or a second pass) to check it against the constraints and fix any violations. It's useful for code, where you can also run tests, and for structured outputs.
| Helps | Doesn't help much |
|---|---|
| Arithmetic and multi-step word problems | Simple fact lookups ("What port does HTTPS use?") |
| Logic puzzles, planning, scheduling | Classification with obvious signals |
| Debugging and tracing code | Short creative writing |
| Comparing design options against constraints | Extraction of fields from a document |
For the right-hand column, reasoning mostly adds length, latency and cost.
Answer: …, or a JSON field), so your code doesn't have to search free text.Q: Why does asking for step-by-step reasoning improve accuracy? A: The model predicts each token from everything before it. Writing intermediate results turns one hard prediction into several easy ones, each conditioned on correct earlier steps. It also lets the model catch inconsistencies as it goes.
Q: Do reasoning models still need chain-of-thought prompts? A: Generally no. They're trained to reason internally before answering. Give them a clear objective, constraints and output format, and adjust their reasoning effort or budget if the API exposes one, rather than scripting their thinking.
Q: What is self-consistency, and when is it worth the cost? A: Generating several reasoning paths at non-zero temperature and returning the majority answer. It's worth it when correctness matters more than cost or latency, and when answers can be compared exactly (numbers, labels, short strings).
Q: How do you get a clean final answer out of a long reasoning response?
A: Instruct the model to finish with a fixed marker (Answer: …) or to return JSON with separate reasoning and answer fields, then parse only the answer field. Use structured-output features where available.
Q: Can chain-of-thought make things worse? A: For simple tasks it adds cost and latency with no benefit. It can occasionally talk itself into a wrong answer through a plausible but flawed chain. That's why high-stakes outputs should be verified independently, not trusted because the reasoning looks convincing.