A Gradient Perspective on RLVR Stability and Winner Advantage Policy Optimization

Why Reinforcement Learning Often Collapses Into Nonsense

Training AI to reason using verifiable rewards often fails. The model might start repeating nonsense or rambling. Researchers have found that these failures—known as "collapse"—are frequently driven by the very updates meant to improve the model. A new study from Layer 6 AI proposes a new solution. The key to stability isn't managing how far a model drifts from its previous version. Instead, it involves controlling which specific tokens influence the gradient (the mathematical direction used to update the model).

The researchers report that identifying specific types of token updates can trigger instability. They designed a method called Winner Advantage Policy Optimization (WAPO). This method only learns from successful attempts. This approach significantly improves training stability across various mathematical and question-answering benchmarks.

The Mechanics of Reasoning Collapse

The core problem exists within Reinforcement Learning with Verifiable Rewards (RLVR). In this framework, a model solves tasks with objective outcomes. For example, a math problem has a single correct numerical answer. To improve, the model generates multiple attempts (rollouts). It compares them against the ground truth. It then uses the relative success of each attempt—called the "advantage"—to update its parameters.

However, this process is notoriously fragile. During training, models often undergo "collapse." This is a state where the model's output degenerates. As shown in, this takes two forms.

High-entropy collapse occurs when the model produces a chaotic mix of irrelevant languages and code fragments. Low-entropy collapse occurs when the model settles into repetitive, malformed loops of nonsense.

The Background You Need

To understand this, one must understand policy gradients. In RLVR, researchers typically use algorithms like GRPO (Group Relative Policy Optimization). GRPO samples a group of completions for a single prompt. It calculates how much better or worse each completion is compared to the group average. This relative score is the "advantage."

The goal is to increase the probability of completions with a positive advantage. Simultaneously, the goal is to decrease the probability of those with a negative advantage. Most existing research focuses on "trust regions." These are mechanisms designed to ensure the new policy doesn't stray too far from the old one. Think of a trust region like a leash on a dog. If the dog tries to run too far from its owner, the leash pulls it back.

While these "leashes" help, the authors argue they don't address the root cause. They suggest that instability isn't just about the distance traveled. It is about the local effect of individual token updates on the model's probability distribution.

A Taxonomy of Unstable Updates

The authors propose a way to categorize how a single token update changes a model's "entropy." Entropy is a measure of how uncertain or diverse the model's next-token predictions are. They introduce a reference level, $C(p)$. This is the sum of the squares of all token probabilities in the current distribution. This value serves as a threshold to classify tokens as "peaks" (high probability) or "valleys" (low probability).

By combining this with the sign of the advantage, the authors derive a four-part taxonomy:

1. Pos-peak: Reinforcing a highly likely successful token. This is stable but conservative.
2. Pos-valley: Reinforcing a low-probability successful token. This encourages exploration but can increase entropy.
3. Neg-peak: Suppressing a highly likely unsuccessful token.
4. Neg-valley: Suppressing a low-probability unsuccessful token.

The paper's central insight is that certain combinations are inherently destructive. The authors find that Neg-peak and Pos-valley updates tend to increase entropy. This drives the model toward the chaotic states seen in . Conversely, Neg-valley updates can lead to a "low-entropy collapse." In this state, the model becomes overconfident in repetitive, incorrect patterns.

The researchers tested these dynamics by masking different parts of the update. They used the SmolLM3-3B model. As demonstrated in, training on only "Neg-peak" or "Pos-valley" tokens led to rapid collapse.

Meanwhile, "Pos-peak" training was stable but plateaued early.

Learning Only from Winners

The authors propose Winner Advantage Policy Optimization (WAPO). WAPO simply ignores non-positive advantage updates. It masks the policy gradient so the model only updates based on completions with a positive advantage.

In an idealized setting with binary rewards (correct or incorrect), WAPO behaves predictably. It performs the same task as a standard policy gradient. However, it uses an adaptive scaling factor $(1 - q_x)$. This factor accounts for the current success rate ($q_x$). This allows the model to prioritize harder prompts where success is rare.

The empirical results are significant. WAPO matched or outperformed existing baselines like GRPO and GSPO across multiple model families. This includes Qwen3, SmolLM3, and Gemma3. In many cases, WAPO provided stability where other methods collapsed .

For instance, on multi-hop question-answering tasks, WAPO showed strong improvements. Furthermore, the authors report that WAPO preserves "pass@k" performance. This means the model retains its ability to explore diverse solution paths .

Where The Edges Are

WAPO has notable trade-offs. The $(1 - q_x)$ adaptive factor can make updates more conservative. This may happen during the early stages of training. Consequently, it might lead to slower initial progress compared to other methods.

The framework is primarily optimized for verifiable rewards. The current taxonomy relies on clearly distinguishing "winners" from "losers." Selecting specific, helpful tokens under very coarse or noisy reward signals remains a difficult challenge. This specific selection problem lies outside the scope of the current study.

Figures from the paper