ESPO: Early-Stopping Proximal Policy Optimization

When an AI is solving a complex math problem and makes a mistake early on, it usually keeps typing useless nonsense until it hits a predefined token limit. This wastes massive amounts of GPU compute. Worse, it pollutes the training signal with "post-failure noise"—tokens that follow a mistake but carry no useful information for learning.

Current reinforcement learning (RL) paradigms for large language models (LLMs) struggle with this "rollout continuation problem." Standard algorithms like PPO (Proximal Policy Optimization) force the model to finish its entire thought process. They do this even if the reasoning path branched into an irrecoverable error halfway through.

The Problem

In long-horizon reasoning tasks, a single incorrect step renders the rest of the trajectory doomed. Under standard PPO, the model continues generating until it hits $T_{max}$ (the maximum rollout horizon). Because these subsequent tokens are part of a failed attempt, they receive zero or negative rewards. Yet, they are still factored into the advantage estimates (the calculation used to determine how much better an action was than expected).

This creates a dual inefficiency. First, there is the raw compute waste. You are burning FLOPs on tokens that will never contribute to a positive reward. Second, there is a signal-to-noise problem. Including these post-failure tokens in the gradient update misdirects the learning process. The model tries to optimize for tokens that are essentially hallucinations or irrelevant filler. Existing solutions like Process Reward Models (PRMs) attempt to solve this with step-level feedback. However, they require expensive, manual human annotation for every intermediate step.

How It Works

ESPO avoids the need for external reward models. It derives a failure signal directly from the actor's own logits (the raw output scores before softmax) and the critic's value estimates. The mechanism relies on four core components:

1. Per-step Surrogate Regret: At each decoding step, the algorithm calculates a "regret" value ($g_t$). This is the logit gap between the token actually sampled and the token the policy most preferred. A high regret indicates the model is deviating sharply from its own highest-confidence path.
2. EMA Normalization: The scale of these logit gaps shifts during training. The authors use an Exponential Moving Average (EMA) of the batch mean and variance to normalize the regret signal. This keeps the signal comparable to the critic's value estimates.
3. Value-Gated Stopping: This is the critical architectural choice. Rather than stopping purely on regret, ESPO compares the accumulated, smoothed regret ($z_t$) against the critic's predicted value ($V_\phi(s_t)$). If the regret significantly exceeds the predicted value, the rollout is terminated. This "gate" prevents the model from killing trajectories that are high-uncertainty but still have high potential for success.
4. Terminal Failure Penalty: When a trajectory is truncated, it is treated as an "absorbing failure state" (a state from which no recovery is possible). The authors assign a terminal penalty ($r_{fail}$) at the exact step where the stopping criterion fired. This concentrates the negative temporal-difference (TD) error at the point of failure. This allows the gradient to propagate the mistake backward more effectively.

To prevent a poorly trained critic from causing premature terminations, the authors implement an adaptive critic warmup period. As seen in, they monitor the critic loss to ensure it has stabilized before the stopping rule is activated.

Numbers

The primary value proposition here is the simultaneous improvement of accuracy and compute efficiency. Testing on a DeepSeek-R1-Distill-Qwen-7B model, the authors report that ESPO outperforms standard PPO across several rigorous mathematical benchmarks. Specifically, on AIME 2024, ESPO achieved 46.28% accuracy compared to PPO's 45.25%. On AMC 2023, the delta was even larger: 85.83% for ESPO versus 82.94% for PPO.

Crucially for engineers managing GPU budgets, the authors measure a significant reduction in training overhead. ESPO saved more than 20% of cumulative rollout tokens compared to PPO. As illustrated in, ESPO maintains a higher accuracy lead through the training steps. It does this while consuming far fewer tokens. Even at the smaller 1.5B scale, the authors find that ESPO consumes roughly 13% fewer tokens than PPO while maintaining superior average accuracy.

What's Missing

While the results are compelling, the paper has a few blind spots that a practitioner should note.

First, there is a vulnerability to "confident errors." If the model is highly confident in an incorrect reasoning path, the logit gap (regret) will be near zero. In these cases, ESPO may fail to detect the error. The model will then continue to waste compute on a doomed trajectory.

Second, the method introduces a trade-off in the form of "false positives." The authors admit that approximately 2.7% of trajectories that could have eventually reached a correct answer are prematurely truncated. While they argue the benefit of noise reduction outweighs this cost, the impact on extremely complex reasoning remains unquantified.

Finally, the system relies on several sensitive hyperparameters. These include the $\beta$ multiplier for the stopping threshold and the target termination rate. The authors use a proportional controller to adjust $\beta$ during training. This adds another layer of complexity to the RL pipeline that must be tuned for different model architectures.

Should You Prototype This

Yes, if you are currently training long-horizon reasoning models via PPO or GRPO.

The implementation is remarkably lightweight. It doesn't require a new reward model or extra human labels. It reuses the logits and value estimates you are already computing during a standard forward pass. The "orthogonal" nature of the method means you can layer it onto existing frameworks like DAPO or GRPO. Since the implementation is reportedly compatible with the verl library, the barrier to entry is low. If you are hitting a wall with compute costs or finding that your reasoning models are "wandering" into nonsense after a mistake, ESPO is a high-leverage optimization worth a weekend of prototyping.

Figures from the paper