Reasoning Structure of Large Language Models

Two models can solve the exact same logic puzzle. They might use the same number of tokens and achieve 100% accuracy. Yet, one performs a surgical strike of logical deduction while the other wanders through a thicket of irrelevant thoughts. Currently, we evaluate Large Reasoning Models (LRMs) by looking only at the finish line—did they get the answer right?—or by counting the steps taken. This is a shallow way to judge intelligence.

In LRM evaluation, we are moving from simple completion tasks to complex reasoning. Models use long Chain-of-Thought (CoT) traces (sequences of intermediate reasoning steps) to deliberate before answering. Traditionally, we have judged these traces using two crude proxies: final-answer accuracy and total token count. However, these metrics fail to distinguish between a model that follows a tight, logical path and one that "overthinks" by generating vast amounts of redundant reasoning. This paper introduces a way to map the actual "logic steps" an AI takes. It turns unstructured text into a flowchart of verifiable claims to see if the model is thinking directly or just meandering aimlessly.

The Problem

The status quo in LRM evaluation is dangerously reductive. When we benchmark models on complex reasoning, we typically collapse their entire cognitive process into a single scalar: accuracy. If the model hits the right answer, we call it a success. If we want to measure "effort," we use token count.

As the authors demonstrate, these metrics conflate fundamentally different behaviors. A model might arrive at a correct solution through a highly efficient chain of five logical steps. Alternatively, it might arrive at the same solution after a thousand tokens of "diffuse exploration" (scattered, non-essential reasoning). This involves generating hundreds of tangential claims, redundant restatements, and unnecessary verifications. Because both models show the same accuracy and similar token budgets, current benchmarks treat them as equals. This masks critical failure modes like "overthinking" or "drift." We lack a way to measure the topology (the geometric structure) of reasoning—the actual shape and density of the logical flow.

How It Works

The researchers propose shifting the unit of analysis from the token to the "claim." They introduce a pipeline that transforms free-form textual traces into verifiable reasoning graphs. These are Directed Acyclic Graphs (DAGs)—structures where nodes represent claims and edges represent deductive dependencies without forming loops.

The methodology follows a three-stage architecture:

1. Claim Extraction: The pipeline uses a hybrid approach to pull individual assertions out of the text. It combines deterministic pattern matching with LLM-augmented extraction to ensure high recall. It segments the trace into chunks and identifies "atomic claims"—discrete, verifiable statements like "there is a tree at (4,1)." As shown in, this process turns messy, human-readable text into a structured set of nodes.

2. Rule Extraction: Once the claims are identified, the system attempts to recover the "edges" of the graph. An LLM identifies the specific rule application that connects a premise to a conclusion. This builds the deductive links that define the graph's structure.
3. Claim Verification: To ensure the graph is grounded in reality, every extracted claim is checked against an executable environment. In this study, they used 21 deterministic grid puzzles. This allows the researchers to label nodes as correct, incorrect, or unverifiable.

The centerpiece of the paper is the reasoning-flow efficiency metric ($\eta$). The authors model the reasoning process as an absorbing Markov chain (a mathematical model describing a sequence of possible events) on the claim graph. By calculating the structural entropy ($H_{str}$, a measure of randomness or dispersion) of the logical mass, they quantify how concentrated the model's focus is. A high $\eta$ indicates a model that moves directly from observations to the solution. A low $\eta$ indicates a model whose logical mass is scattered across many peripheral, non-essential claims.

Numbers

The most striking result is that token count is a useless proxy for reasoning quality. The authors report that reasoning-flow efficiency ($\eta$) is essentially uncorrelated with token count ($r = -0.05$, $p = 0.64$). This means that simply asking a model to "think longer" does not inherently improve the structural integrity of its logic. Increasing the token budget does not guarantee better reasoning.

Instead, the metric $\eta$ provides a much sharper signal. The paper finds that $\eta$ is positively correlated with both claim accuracy ($r = 0.33$) and the depth at which the first error occurs ($r = 0.28$), as shown in .

This means higher efficiency usually points to more accurate and more stable reasoning. Crucially, the authors show that as puzzle difficulty increases, models suffer a massive accuracy collapse. For example, GPT-5 drops from 83.8% to 5.7%. This happens even as they consume significantly more tokens.

Furthermore, the authors demonstrate that extra tokens primarily translate into "verification overhead." They report a strong correlation between token count and the ratio of verification nodes to solution nodes ($|V_{ver}|/|V_{sol}|$, $r = 0.53$). This implies that longer traces are often just models repeatedly checking their own work. They are spending compute on auxiliary reasoning rather than advancing the core logical chain.

What's Missing

While the framework is mathematically elegant, there are several gaps that a practitioner should consider.

First, the extraction pipeline itself is not "pure." It relies on high-end LLMs (specifically GPT-5 and GPT-5-mini) to perform the claim and rule extraction. While the authors include an ablation study showing that the choice of extractor has minimal impact on the final $\eta$ value, the system is still subject to the biases of the extractor LLM. If the extractor fails to capture a subtle logical nuance, the resulting graph will be wrong.

Second, the verification logic is currently tied to "verifiable state transitions." The authors admit they exclude "state-dependent" claims. These are claims that reference a changing partial board state. Tracking those is non-trivial and not robust in their current setup. This means the current implementation is optimized for puzzles with rigid, executable rules. It might struggle with more fluid, semantic reasoning tasks where the "state" is not easily captured by a simulator.

Finally, the paper focuses on a controlled suite of 21 logic puzzles. While these are excellent for scaling studies, they differ from the messy reasoning required in real-world software engineering. We don't yet know if $\eta$ holds up when the "ground truth" is not a deterministic grid.

Should You Prototype This

Yes, but as a diagnostic tool, not a replacement for standard evals.

If you are building or fine-tuning models that rely heavily on long-context reasoning, you should look into this. The ability to distinguish between a model that is truly reasoning and one that is just "looping" or "meandering" is invaluable. If you can identify that a model's extra tokens are purely verification overhead, you have a clear target for optimization. You can teach the model to be more decisive rather than more verbose.

The code is reportedly available at https://github.com/ETH-DISCO/llm-reasoning-efficiency. Start by applying it to a narrow, verifiable domain like math or code execution. Build your own verifier for that domain. Don't expect it to work out-of-the-box for general chat. However, for specialized reasoning agents, this is a real step toward measuring how they think.

Figures from the paper