Latent Reasoning with Normalizing Flows

Instead of making AI models write out every step of their thinking in text, this method lets them "think" in a compact, continuous mathematical space. This makes the reasoning process much faster and more efficient. It also allows the model to find many different ways to solve a problem.

Large language models (LLMs) have recently unlocked sophisticated reasoning through Chain-of-Thought (CoT) prompting. This technique generates intermediate text tokens—essentially a "scratchpad" of logical steps—before reaching a final answer. Researchers have seen massive performance jumps in math and coding tasks using this method. However, this reliance on explicit text creates a fundamental bottleneck.

Current paradigms force reasoning through a discrete, serial, and communication-oriented token stream. Even when an underlying semantic update is subtle, the model must verbalize it into specific words. This is akin to a mathematician being forced to narrate every micro-movement of their pen aloud. Such a process is verbose and computationally expensive. It also ties the "thought" to the limitations of surface-level language.

The bottleneck of verbalized logic

Existing attempts to bypass textual CoT fall into two main camps. Both carry significant trade-offs. Some methods, like Coconut, recycle the model's hidden states (continuous numerical vectors representing meaning) as latent thoughts. While efficient, these thoughts are often deterministic. This means they do not allow the model to explore different reasoning paths through probabilistic sampling.

Other approaches use diffusion models. These models iteratively "denoise" a signal to create a representation. As shown in, diffusion-based methods like LaDiR offer stochasticity (the ability to sample different outcomes).

However, they require expensive, iterative denoising steps. This breaks the natural left-to-right flow of language models. It also makes integration with the standard KV-cache (a memory mechanism for reusing past computations) difficult. Users must choose between the slow verbosity of text or the cumbersome mechanics of diffusion.

Integrating flows into the causal stream

The authors propose NF-CoT to bridge this gap. They treat continuous thoughts with the same status as text tokens. Instead of diffusion, the paper introduces a scalable normalizing flow inside the LLM backbone. A normalizing flow is an invertible mathematical transformation. It maps a complex data distribution to a simple one. This is like stretching and folding dough until it matches a specific shape. Because this transformation is invertible, the model moves between "raw" and "structured" thoughts without losing information.

The architecture, shown in, operates in a unified causal stream.

The process involves three main stages:

1. Reparameterization: During training, the model uses a frozen VAE (an encoder-decoder that compresses data) to turn text CoT into continuous targets. Shallow flow blocks then transform these into a specialized "$u$-space" optimized for the LLM.
2. Joint Modeling: The LLM backbone uses two specialized "heads." At continuous-thought positions, an NF head predicts the parameters of a Gaussian distribution (mean and variance). At text positions, a standard LM head predicts the next word.
3. Autoregressive Sampling: At inference, the model skips the heavy training machinery. It samples continuous thoughts left-to-right from the learned distribution. It then seamlessly transitions to decoding text tokens.

This design ensures the KV-cache built during "thinking" is reused for "answering." This preserves the efficiency of standard autoregressive decoding.

Faster reasoning with higher diversity

The empirical results suggest that continuous reasoning provides both speed and intelligence. On code-generation benchmarks, the authors report that the "Unified" version of NF-CoT improves the average pass@1 (the probability that a single sampled solution is correct) from 55.8% to 68.8% on a Qwen3-8B-Base model. This 13.0% absolute improvement means the model is significantly more likely to solve a coding problem correctly on the first try.

The authors highlight that these gains come from modeling the distribution of reasoning paths. As shown in, NF-CoT demonstrates superior pass@k scaling.

This means that as you allow the model to sample more potential solutions, its success rate climbs more effectively than previous methods. Unlike token-space reinforcement learning, which can cause "collapse" (where a model only finds one way to solve a problem), NF-CoT maintains diversity. shows that applying reinforcement learning in the latent space preserves this upward scaling trend.

This ensures the model remains a versatile problem solver.

Efficiency is a major benefit for engineers. The paper reports that NF-CoT (Unified) is 1.92× faster overall and 2.48× cheaper in terms of FLOPs (a measure of computational work) compared to the diffusion-based LaDiR baseline. By replacing iterative denoising with a single left-to-right pass, the authors substantially reduce the cost of intermediate reasoning.

Limits of the latent window

NF-CoT is not a universal replacement for text-based reasoning. The authors note several important caveats. First, the validation focuses primarily on code generation. It remains to be seen if this translates to nuanced linguistic reasoning.

Second, the model's reasoning is limited by a fixed latent budget. The authors use $N=64$ latent slots to represent what might have been hundreds of words of text. This provides a 6.0× compression rate. However, a fixed budget might not fit extremely complex, multi-step problems. Finally, because reasoning happens in a continuous mathematical space, the thoughts are not human-readable. We can see the model is "thinking," but we cannot easily audit its specific logic. This poses challenges for safety and interpretability.

The verdict: A new standard for latent thought?

If you prioritize reasoning latency and token cost, NF-CoT is a compelling direction. It marries the stochastic flexibility of diffusion with the efficient architecture of autoregressive LLMs. The ability to perform reinforcement learning directly on continuous trajectories is a significant technical win. It offers a cleaner path to optimizing "internal" thought processes.

However, for tasks requiring high interpretability, the current fixed-length bottleneck remains a hurdle. The work is ready for researchers pushing the boundaries of efficient agentic workflows. For mission-critical systems where "why" matters, the lack of human-readable traces requires caution. Code is reportedly available; see the project page at https://nf-cot.vercel.app for the canonical link.

Figures from the paper