Not only where, But when: Temporal Scheduling for RLVR

When training AI to reason, we usually focus on "where" to give credit for a correct answer. We try to identify which specific tokens in a long chain of thought actually drove the success. This paper suggests that when we give that credit matters just as much. The authors propose a curriculum. It starts by focusing on the end of a thought process (the final answer). It then gradually moves backward to the beginning (the reasoning scaffolding). This builds more stable and logical steps.

The Problem

In Reinforcement Learning with Verifiable Rewards (RLVR), a model generates a long sequence of tokens. It then receives a single scalar reward at the very end. The challenge is credit assignment (attributing success to specific actions). How do you tell the model which part of its reasoning was brilliant and which part was a hallucination? Current approaches use "credit allocation." They use proxies like token entropy (a measure of prediction uncertainty) to reweight token importance.

The problem is that these allocation criteria are typically "stagnant." Once you pick a method, you stick with it for the entire training run. The authors argue this creates a fundamental tension. If you optimize heterogeneous behaviors simultaneously, you sacrifice policy entropy (the model's capacity for exploration). This happens because the optimizer tries to satisfy conflicting behavioral requirements at once .

How It Works

The core innovation shifts from a spatial problem (which tokens?) to a temporal one (at which training stage?). The authors introduce a framework that schedules the credit allocation criteria over the course of optimization.

1. Defining the Proxy: The system identifies token importance using a proxy $\mu_{i,t}$. This can be a continuous value or a discrete mask (selecting only certain tokens for updates).
2. The Schedule Function: They introduce a monotonically decreasing schedule function $S(\tau)$. Here, $\tau$ is the current training progress. This function acts as a gatekeeper. It determines how strictly the allocation criteria are applied.
3. Temporal Expansion: Instead of optimizing everything at once, the process starts by targeting specific, high-signal tokens. As training progresses, the schedule "attenuates." This means the strictness of the filter relaxes. It eventually incorporates all tokens into the optimization loop.

A standout implementation is TP-Schedule (Trajectory Percentile Scheduling). The authors use the Trajectory Percentile Score (TP-Score) to categorize tokens by their position in a response. Different parts of a trajectory exhibit different behaviors. Early tokens act as "scaffolding" while later tokens realize the "answer" .

They schedule optimization to move from later percentiles to earlier ones. This ensures the model stabilizes its answer-generation before refining complex reasoning steps.

Numbers

The authors demonstrate that temporal scheduling raises reasoning ceilings. Testing on the Qwen3-4B model, TP-Schedule outperforms vanilla GRPO by 2.2% on mathematical benchmarks. It also improves general reasoning benchmarks by 2.7% [Table 1]. For the larger Qwen3-8B model, they report a consistent ~1% improvement over vanilla GRPO.

Crucially, temporal scheduling preserves policy entropy better than stagnant methods. It can preserve policy entropy by up to 33.9% compared to entropy-based reweighting .

This prevents the premature convergence that often plagues RL runs. The gradient norm analysis in supports this.

While standard GRPO has diluted signals, temporal scheduling provides more concentrated, stable optimization signals during early training.

What's Missing

While the results are compelling, some gaps remain: * Task Domain Narrowness: The experiments focus on mathematical and logical reasoning. It is unclear if this "backward-to-forward" curriculum works for creative writing or conversational tasks. * Hyperparameter Sensitivity: The method relies on the temporal range ($\epsilon_{high}$). The authors suggest a "sweet spot" between 0.6 and 0.8. However, they do not provide an exhaustive search regarding interactions with different learning rates. * Complexity Overhead: The scheduling is a "drop-in" replacement. However, the paper does not explicitly quantify the extra computational time added to the training wall-clock.

Should You Prototype This

Yes, if you are struggling with policy collapse in RLVR. If your reasoning models hit a performance plateau, this is a useful lever. If you notice policy entropy cratering, this method addresses that directly. It is designed as a drop-in replacement for existing GRPO pipelines. It adjusts the weights of existing gradients. Therefore, it should not significantly impact your VRAM footprint or throughput.

Code is reportedly available at https://github.com/Jinghaoleven/RLVR-Schedule. If you are running large-scale reasoning post-training, a one-day prototype is a high-upside bet.

Figures from the paper