Research guide

What Do LLMs Know About Their Own Reasoning? What the Internals Show

By Aojie (Justin) Yuan · October 2026 · draws on 3 papers

TL;DR — More than they say, and less than you can use. Hidden states predict whether a chain-of-thought is wrong with 0.95 AUROC while the model still sounds confident; the same reasoning concept written as prose, code or math lands in a shared 10-dimensional subspace; and that subspace is not just the output readout in disguise. But these signals are diagnostic: steering on them did not fix the errors.

Chain-of-thought gives us text that looks like reasoning. Three of my papers look underneath it, at hidden states, and ask what the model represents about its own reasoning and whether that representation can be trusted or used.

Models often know when their reasoning is going wrong

A linear probe on hidden states predicts whether a chain-of-thought trace is correct with 0.95 AUROC, and already 0.79 at the first reasoning step. The model's spoken confidence barely moves: 4.55/5 on wrong traces versus 4.87/5 on correct ones, and a classifier that reads only the generated text reaches 0.59. The effect holds across Qwen, Llama and Phi from 1.5B to 72B parameters and in RL-trained reasoners such as DeepSeek-R1 (0.852 AUROC). Read: Do LLMs know when their chain-of-thought is wrong?.

But knowing is not fixing

The obvious next step is to use that signal to correct the model. Four interventions were tested: activation steering, probe-guided best-of-N, self-correction and activation patching. All four failed, and patching destroyed output coherence. The signal reads out the quality of the computation; it is not a lever that redirects it. The practical use is as an alarm, for example to route a likely-wrong answer to a verifier or a human.

One concept, three notations

Across five LLMs (1.6B–8B), the same reasoning concept written as English prose, Python code or math notation lands in a shared Format-Agnostic Reasoning Subspace (FARS) in the middle layers. A 10-dimensional basis amplifies concept structure about 3× while suppressing form; swapping only those 10 dimensions across forms preserves 90–96% of model output, versus 44–56% for full activation replacement. The biggest split is not language versus formal notation but declarative versus procedural: prose and math are far more compatible with each other than either is with code. Read: Do LLMs reason the same way in English, code, and math?.

Is that just the logit lens in disguise?

A subspace that changes behavior could simply be aligned with the directions the model uses to emit tokens. A two-sided, weights-only test measures how much of a subspace overlaps the dominant directions of the unembedding matrix, calibrated against readout-oriented controls. Across 26 models, activation-derived concept estimators carry only 0.38–0.80% mean energy in the top-ten readout span, while a next-token control carries roughly 13× more than FARS. The representation sits upstream of the readout. Read: Is a concept subspace just the logit lens in disguise?.

What this means in practice

Do not read chain-of-thought as a faithful report. The written reasoning can sound confident while the hidden state already signals an error.
Use internal signals as monitors, not as controls. A cheap probe can flag likely failures; it did not repair them.
Separate “where it is” from “what it does”. Geometry locates a representation and interventions test behavioral effects; neither alone shows causal sufficiency.

Frequently asked questions

Is chain-of-thought a faithful view of how an LLM reasons?

Not reliably. In this research, hidden states predicted that a reasoning trace was wrong with 0.95 AUROC while the model's spoken confidence stayed high (4.55/5 on wrong traces versus 4.87/5 on correct ones), and a text-only classifier reached just 0.59.

Can you detect LLM reasoning errors from hidden states?

Yes. A linear probe on hidden states predicted chain-of-thought correctness with 0.95 AUROC, 0.79 from the first reasoning step, across Qwen, Llama and Phi models and in DeepSeek-R1.

Can you fix LLM reasoning errors by steering hidden states?

Not in this study. Activation steering, probe-guided best-of-N, self-correction and activation patching all failed; the error signal is diagnostic, not causal.

Do LLMs represent code, math and natural language the same way?

Largely yes for reasoning concepts: they share a 10-dimensional Format-Agnostic Reasoning Subspace in the middle layers. Prose and math are much more compatible with each other than either is with code.

Papers covered in this guide

Written by Aojie (Justin) Yuan, USC Fortis Lab. Every claim comes from the papers listed above; each section links to the detailed explainer for its paper.