ThoughtFold: Folding Reasoning Chains via Introspective Preference Learning

Large reasoning models (LRMs) often "overthink." They generate long, rambling chains-of-thought (CoT)—the intermediate reasoning steps—filled with self-repetition and tangents. While these models eventually arrive at the correct answer, the extra tokens are pure waste. This drives up inference costs and latency without adding intelligence.

The current state of the art relies on Reinforcement Learning with Verifiable Rewards (RLVR). In this paradigm, the model explores various reasoning paths. We provide a binary reward based on whether the final answer is correct. The problem is that RLVR is a blunt instrument. If a model takes a 1,000-token path to solve a problem that only requires 200 tokens, RLVR reinforces the entire sequence. It treats vital logic and redundant "noise" identically.

The authors of this paper argue that this creates a feedback loop. Models begin to memorize inefficient, circular reasoning. They propose ThoughtFold. This framework is designed to "fold" these reasoning chains into concise, direct paths.

The failure of outcome-only reinforcement

Standard RLVR approaches suffer from a coarse credit assignment problem. This occurs because the reward is tied only to the final outcome. The model cannot distinguish between a crucial logical deduction and a redundant loop. As shown in, vanilla RLVR uniformly reinforces every step in a correct trajectory.

This causes the model to memorize the "hit-or-miss" exploration phases alongside the actual signal.

Previous attempts to fix this have mostly relied on trajectory-level penalties. These essentially add a "tax" on length. While this nudges the model toward brevity, it does not solve the underlying issue. The model is still being trained on noisy, redundant data. These methods lack the granularity to penalize specific, useless steps within a successful chain. The result is often a model that tries to be short but loses its reasoning depth. This can lead to hallucinations (the generation of false or nonsensical information) as the model attempts to bypass necessary logic.

Identifying and folding redundant steps

ThoughtFold moves from outcome-based rewards to fine-grained preference learning. It uses an "introspective" strategy. The process begins by taking a verified, correct reasoning trajectory. The system then attempts to prune it to see if the model can still reach the right answer. This happens in two distinct phases, as illustrated in .

1. Tail Truncation: The system performs a binary search on the prefix length. This removes trailing self-repetition. This is the tendency of models to keep talking after they have already solved the problem.
2. Internal Folding: The system identifies "low-utility" steps. It looks at the model's own attention distribution (the mechanism that determines which tokens are most relevant to others). Specifically, it calculates an importance score $I(s_t)$ [Equation 7]. This score averages the attention weights from the final answer tokens back to the tokens within a reasoning step. Steps with low attention are candidates for removal.

Once these candidate sub-trajectories are generated, the framework builds a preference dataset. If a pruned, shorter trajectory still reaches the correct answer, it is a "Concise Success" ($z_{cand} \succ z_{best}$). If pruning breaks the logic, it is an "Over-Simplified Failure" ($z_{best} \succ z_{cand}$).

To train the model, the authors use a Dynamic Mask Strategy. This is part of a Masked Direct Preference Optimization (MDPO) objective. Instead of penalizing an entire rejected trajectory, the masks allow selective penalization. The model penalizes redundant steps. Simultaneously, it encourages "Fold Anchors." These are the specific steps that bridge the gap left by the removed redundancy. This allows the model to learn to jump directly between essential logical segments.

Efficiency gains without accuracy loss

The empirical results suggest that ThoughtFold restructures how the model thinks. The authors report that for a DeepSeek-R1-Distill-Qwen-7B model, ThoughtFold reduces average token usage by approximately 56.1%. This reduction helps lower the computational cost of running the model. Crucially, it does so while maintaining, and in some cases improving, state-of-the-art accuracy [Table 1].

On the AIME benchmarks, the efficiency gains are particularly striking. The authors use a metric called Minimum Average Length@k (ML@k). This estimates the expected minimum sequence length given a budget of $k$ rollouts. ThoughtFold shows a lower bound for how concisely a model can solve a problem .

Unlike previous length-regularized methods, ThoughtFold's reasoning topology undergoes a qualitative shift. As visualized in, the reasoning goes from a chaotic web of backtracking and loops to a highly linear, sequential structure.

Furthermore, the authors demonstrate that these gains generalize. Even on out-of-domain scientific tasks like GPQA Diamond, the model maintains its efficiency and accuracy. This suggests the method learns robust reasoning structures. It does not just memorize shorter ways to answer specific math problems.

Complexity and credit assignment risks

There are technical trade-offs to consider. The "introspective" phase requires additional computation during the training stage. This involves a prune-and-verify loop. The authors describe this overhead as "negligible." However, it still requires multiple rollouts to ensure a pruned path is truly valid.

There is also a delicate balancing act involving the $\lambda$ coefficient. This constant controls the trade-off between the standard GRPO (accuracy-focused) objective and the MDPO (efficiency-focused) objective [Table 3]. If $\lambda$ is too high, the model prioritizes brevity too aggressively. This can cause accuracy to drop. If $\lambda$ is too low, the model reverts to the overthinking behavior seen in vanilla RLVR.

Finally, the effectiveness of the internal folding relies on the quality of the attention-based importance score. The authors justify this by showing that middle-layer attention provides the most useful signal [Table 4]. However, the system's ability to "fold" is limited by the model's own attention mechanism. It must accurately identify which steps are actually redundant.

The verdict

ThoughtFold is a significant step forward for managing the high inference costs of Large Reasoning Models. It attacks the structural cause of "overthinking." Rather than just applying a length penalty, it reshapes the reasoning path. This achieves massive token reductions—up to 56%—without the usual accuracy tax.

Is it ready for your production pipeline? If you struggle with the massive KV cache (a memory component used to store previous tokens) requirements of long-chain CoT models, this is worth a prototype. The methodology is grounded in how transformers actually attend to reasoning steps. The transition from looping to linear topologies is a compelling piece of evidence. The code is reportedly available; see the paper for the canonical link.

Figures from the paper