Reasoning & Planning

The Quest for Deeper Thinking in AI

Early large language models were impressive pattern matchers, but they struggled with tasks that required genuine multi-step reasoning — mathematical proofs, complex code debugging, strategic planning, and scientific analysis. The development of reasoning models represents a fundamental shift: instead of generating answers in a single forward pass, these models are trained to "think" — to break problems into steps, explore solution paths, and verify their own work before producing a final answer.

This module explores how reasoning models work, the techniques behind them, and where the frontier of AI reasoning stands in 2026.

Chain-of-Thought Prompting: Where It Started

The reasoning revolution began with a deceptively simple observation. In 2022, Wei et al. at Google showed that adding the phrase "Let's think step by step" to a prompt dramatically improved LLM performance on math and logic problems. This technique — chain-of-thought (CoT) prompting — doesn't change the model at all. It simply encourages the model to generate intermediate reasoning steps before arriving at an answer.

Why does this work? When a model generates tokens step by step, each step becomes part of the context for the next step. The model essentially gets to "use paper" — externalizing its reasoning into text that it can then reference. Without chain-of-thought, the model must solve the entire problem in the latent computation of a single forward pass, which has limited depth.

Beyond Linear Reasoning: Tree-of-Thought

Chain-of-thought is linear — the model follows one reasoning path. But many problems benefit from exploring multiple paths and backtracking when a path leads to a dead end. Tree-of-Thought (ToT), introduced by Yao et al. in 2023, addresses this by having the model generate multiple possible next steps at each point, evaluate them, and pursue the most promising paths.

Think of it as the difference between walking down a single corridor and exploring a maze. ToT enables the model to:

• Generate several candidate reasoning steps
• Evaluate which steps are most promising
• Backtrack from dead ends and try alternatives
• Combine insights from different branches

ToT dramatically improves performance on problems that require search and planning, such as puzzle-solving, game strategy, and creative problem-solving. However, it is significantly more computationally expensive than linear chain-of-thought.

Reasoning Models: The Test-Time Compute Revolution

The biggest leap in AI reasoning came with the realization that you can trade inference-time compute for better answers. Instead of making a model smarter by making it larger (more parameters, more training), you make it smarter by letting it think longer at inference time. This is the core idea behind reasoning models.

The Frontier Reasoning Models

How Reasoning Models Work

Reasoning models differ from standard LLMs in several fundamental ways:

1. Extended internal chain-of-thought: When given a problem, the model generates a long internal reasoning trace — often thousands of tokens — before producing its final answer. This trace includes hypothesis formation, step-by-step work, self-correction, and verification.
2. Test-time compute scaling: Harder problems automatically trigger longer reasoning chains. The model learns to allocate more "thinking time" to problems that require it. This is fundamentally different from standard models, which spend the same compute on easy and hard problems.
3. Training with process supervision: Rather than only rewarding the final answer (outcome supervision), reasoning models are trained with process reward models (PRMs) that evaluate each step of the reasoning chain. This teaches the model to reason correctly, not just get lucky with the right answer.
4. Self-verification: The best reasoning models learn to check their own work — re-reading the problem, verifying intermediate steps, and catching errors before committing to a final answer.

Process Reward Models (PRMs)

Process reward models are a critical training innovation for reasoning systems. Traditional reward models in RLHF evaluate only the final output — did the model get the right answer? PRMs instead evaluate each step of the reasoning process.

Consider a math problem where the correct answer is 42. A model might arrive at 42 through correct reasoning or through a series of errors that coincidentally cancel out. Outcome-based reward would give both approaches the same score. A PRM would correctly reward the sound reasoning and penalize the lucky errors.

Training with PRMs produces models that:

• Make fewer reasoning errors at each step
• Are more reliable on problems outside their training distribution
• Generate reasoning chains that humans can actually follow and verify
• Self-correct more effectively, because they have learned what correct reasoning looks like

Monte Carlo Tree Search Applied to LLMs

Monte Carlo Tree Search (MCTS) — the technique that powered AlphaGo's victory over world champion Go players — is being adapted for language model reasoning. The idea is to treat reasoning as a search problem:

1. Selection: Choose a promising reasoning path to explore further, guided by a value function that estimates how likely the current partial solution is to lead to a correct answer.
2. Expansion: Generate the next reasoning step, creating new nodes in the search tree.
3. Simulation: Roll out the reasoning to completion (potentially with a cheaper/faster model) to estimate the quality of this path.
4. Backpropagation: Update the value estimates of all nodes along the path based on the outcome.

This approach allows reasoning models to systematically explore the space of possible solutions rather than committing to a single reasoning chain. It is particularly powerful for problems with clear verification criteria — such as math (check the answer), code (run the tests), and formal logic (verify the proof).

The Frontier of AI Reasoning

Reasoning models are pushing into domains previously considered out of reach for AI:

Mathematical Proofs

AI systems are increasingly capable of solving competition-level mathematics and assisting with research-level proof development. Systems like AlphaProof (Google DeepMind) have made headway on International Mathematical Olympiad problems. The combination of language models for intuition with formal proof assistants like Lean for verification is a particularly promising direction.

Code Generation and Debugging

Reasoning models have dramatically improved code generation quality. By thinking through the problem, planning the approach, writing the code, and then mentally tracing through test cases, reasoning models achieve pass rates on coding benchmarks that far exceed standard models. They are particularly strong at debugging — reasoning about why code fails and systematically identifying the root cause.

Scientific Discovery

Reasoning models are being applied to scientific tasks: analyzing experimental data, generating and evaluating hypotheses, and synthesizing findings across papers. While they are not yet making independent discoveries, they are increasingly valuable as research assistants that can process and reason about vast amounts of scientific literature.

Limitations of Current Reasoning Approaches

Despite impressive progress, important limitations remain:

• Faithfulness of reasoning chains: The reasoning trace a model generates may not accurately reflect its actual internal computation. Models can produce plausible-sounding reasoning that leads to a conclusion they were going to reach anyway — a form of post-hoc rationalization rather than genuine reasoning.
• Compositional generalization: Reasoning models struggle with problems that require combining known concepts in truly novel ways. They can solve problems that resemble their training data, but novel problem structures remain challenging.
• Cost and latency: Reasoning models can be 10-100x more expensive than standard models due to the extended thinking tokens. For applications requiring real-time responses, this is a significant constraint.
• Planning horizon: Current models struggle with tasks requiring very long-horizon planning — sequences of dozens or hundreds of dependent steps. They can plan a few steps ahead reliably but degrade over longer horizons.
• Brittleness: Small changes in problem phrasing can dramatically affect reasoning performance. Models can solve a problem presented one way but fail when the same problem is rephrased.

Resources