Formalizing Latent Thoughts: Four Axioms of Thought Representation in LLMs

The Hidden Failure of Latent Reasoning

Researchers aim to move LLM reasoning from discrete, expensive Chain-of-Thought (CoT) tokens into continuous, efficient latent representations. The goal is to compress the "thinking" process into hidden vectors. These vectors allow the model to process information without the overhead of generating thousands of text tokens. However, we currently lack ways to tell if these latent vectors are truly useful. We cannot easily distinguish if they facilitate reasoning or if the model is merely getting lucky on benchmarks.

Current evaluations are fundamentally flawed. They conflate representation quality with model capacity (the ability of the model to process information). We judge a "thought" by whether the model eventually gets the right answer. This does not tell us if the thought itself was accurate, efficient, or distinct. A model might achieve high accuracy through brute-force pattern matching. Meanwhile, its internal "reasoning" vectors might be essentially useless. This paper introduces a way to audit those vectors directly. This audit is independent of downstream accuracy. The results are sobering: current latent reasoning methods fail to distinguish between different questions within the same task.

The accuracy trap

The status quo in continuous reasoning research relies on heuristic proxies. These include things like token budgets, step counts, or imitating explicit CoT. Many papers report accuracy gains on complex reasoning benchmarks using "soft tokens" (weighted combinations of token embeddings) or "latent thinking" (recurrent hidden state updates). However, these metrics are deceptive. As the authors note, existing evaluations almost exclusively use downstream task accuracy. This usage masks representational failures.

If a model maintains high accuracy but its internal representations collapse, we won't know immediately. We might only notice when the model hits a distribution shift (a change in the statistical properties of the input data). The authors argue that we must decouple the quality of the representation from the capability of the decoder (the part of the model that turns vectors back into text). Without this, we cannot distinguish whether a failure stems from a poor latent representation or a decoder that failed to interpret a good one. This creates a blind spot. We might be optimizing for surface-level mimicry rather than functional reasoning.

Four axioms for functional thought

To solve this, the authors propose an axiomatic framework. It defines a "thought" by what it does rather than its form. They formalize four functional properties that a valid thought representation $T$ must satisfy, as visualized in :

1. Causality: The representation must functionally substitute for explicit reasoning tokens. If you replace the reasoning prefix with $T$, the predicted distribution for the answer should remain essentially unchanged.
2. Minimality: Following the Information Bottleneck principle, $T$ should compress the input $X$ while retaining maximum relevance to the output $Y$. It should filter out "nuisance variables" (irrelevant context that does not affect the answer).
3. Separability: The latent space must have enough topological structure to distinguish between semantically different outputs. If two questions have different answers, their representations should be resolvable by a simple linear classifier.
4. Stability: The representation should be robust to surface-level lexical variations (different words with the same meaning). It should also reflect the model's uncertainty. If the model is confused, $T$ should encode that entropy (a measure of randomness or uncertainty).

The authors quantify these using specific mathematical measures. They use KL substitution error for Causality. They use an IB residual gap ($\Delta_{IB}$) for Minimality. They use discriminator accuracy for Separability. Finally, they use a Distributional Consistency Score (DCS) for Stability. This allows for an intrinsic audit of the model's internal state without needing to retrain the entire system.

A structural collapse in the latent space

The empirical audit covers five open-weight LLMs and 23 reasoning tasks. The results reveal a massive gap between benchmark performance and representational integrity.

The most striking result is the "Separability collapse." While the models excel at distinguishing between different tasks (e.g., math versus biology), they fail to distinguish between different questions within the same task. As shown in, across-task accuracy stays near saturation.

However, within-task accuracy drops to near the random baseline for almost every candidate except for direct output embeddings. The authors use geometric analysis in to confirm this is not a failure of the probes. Instead, it is a structural reality. The within-task geometry has simply collapsed. This means the vectors for different questions are clustered too tightly to be pulled apart.

Furthermore, the representations are surprisingly redundant. The authors find that most candidates encode little information beyond what is already present in the initial input embedding ($IE$). Regarding Causality, all candidates perform better than random noise. Yet, none consistently outperform the prompt embedding itself. Essentially, the "thinking" steps are not adding new causal information. They are primarily reshuffling the prompt.

What the audit misses

While this framework is a significant step, there are clear boundaries to its current scope.

First, the study focuses on "extractable" representations. These are obtainable from a pre-trained LLM without additional training. The authors concede that achieving all four axioms simultaneously might require specifically designed extraction layers. These layers would need to be trained explicitly to meet these functional requirements. If you seek a "plug-and-play" solution for existing models, this paper suggests you may not find one.

Second, the measurement cost is non-trivial. Unlike a standard accuracy benchmark, this protocol requires generating multiple beams (multiple potential output paths) per problem. It also requires training several specialized probes. For a practitioner, this means auditing a new architecture is more compute-intensive than running a standard test. Finally, the study is limited to English-language, open-weight models. It is unclear if these representational collapses manifest differently in highly optimized, closed-source models or in multilingual contexts.

The verdict: stop chasing accuracy

The verdict is clear: do not trust downstream accuracy as a proxy for reasoning quality.

If you are building or fine-tuning models for continuous latent reasoning, the current state of the art is hitting a structural wall. The "thinking" occurring in the latent space is currently a shallow transformation of the input prompt. It is not yet a robust, separable, and minimal distillation of the problem.

The authors provide a roadmap for improvement. Treat Causality, Minimality, Separability, and Stability as explicit optimization targets. If you develop new latent-thinking architectures, you should measure these four metrics during your validation loops. Relying on a single accuracy number is insufficient. It tells you nothing about whether the internal representations are actually performing the reasoning they are meant to drive.

Code and the full evaluation pipeline are available at https://fard-lab.github.io/formalize-thoughts.

Figures from the paper