Filter, Then Reweight: Rethinking Optimization Granularity in On-Policy Distillation

Teaching AI Through Selective Attention

When teaching a smaller AI model using a larger one, not all examples or words are equally helpful. Some "stories"—the complete sequences of thought a model generates—are low-quality or confusing. Other words within those stories are trivial. Some are critical for learning.

A new study proposes a method called FiRe-OPD (Filter, then Reweight). The researchers suggest we should not treat every word and sequence with equal importance. Instead, they propose throwing away the worst "stories" (trajectories) first. Then, they assign more weight to the most useful "words" (tokens) within the remaining good ones. This dual-layer approach aims to make the learning process much smoother and more efficient.

Refining the Signal in On-Policy Distillation

The core challenge addressed here is On-Policy Distillation (OPD). In the context of Large Language Models (LLMs), distillation is the process of transferring intelligence from a massive "teacher" model to a smaller "student" model. "On-policy" means the student learns from its own attempts at solving problems. This helps prevent "distribution mismatch" (a gap between the training data and real-world usage).

However, the authors point out a flaw in standard OPD. It applies uniform supervision. It treats every token (the basic units of text, like words) and every trajectory (the entire sequence of steps) as equally important. This is problematic. Not all student attempts are worth learning from. Furthermore, not all words in a successful attempt carry meaningful information.

The Logic of Dual-Granularity Optimization

The researchers argue that current methods suffer from "granularity isolation." This means they focus on either the whole sequence or individual words. They rarely focus on both. To fix this, they propose a two-stage pipeline, as shown in .

First, the framework performs trajectory-level filtering. The authors observe that if a teacher model assigns a very low probability to a student's reasoning path, a massive gap exists. Trying to force the student to follow a path the teacher finds unlikely can introduce "noisy" (unreliable) signals. These signals can actually hurt learning. Instead of using external rewards, the paper uses the teacher's own log-probability as a proxy for reliability. The authors discard the bottom $p\%$ of these trajectories. This ensures the student only learns from paths within the teacher's "competence region."

Second, the method applies token-level soft reweighting. Rather than using "hard selection" (simply deleting unhelpful words), the authors use a continuous weighting mechanism. They define an importance weight ($w_t$) by multiplying two factors: teacher confidence and student confusion.

As shown in, the model identifies that "transition tokens" carry the highest weight.

These include words like "Therefore," "So," or "However." These are moments where the teacher is certain of the direction, but the student is uncertain of the logic. Conversely, as seen in, procedural words or simple numbers receive very low weights.

They do not represent a strategic decision point. By using "soft" weights rather than binary "keep or discard" rules, the authors report they can preserve subtle gradient information (the mathematical signal used to update the model). Hard selection would otherwise destroy this information.

Scaling Across Complexity and Teachers

The effectiveness of this approach is demonstrated across several difficult scenarios. In "strong-to-weak" distillation (transferring knowledge from a 30B parameter model to a 4B model), the authors report a 60.83% average accuracy. This outperforms the standard OPD baseline by 2.13 percentage points [Table 2]. The gains were most significant on high-level competition math. For example, there was a 6.25-point improvement on the AIME 2024 benchmark.

The method also performs well in "multi-teacher" settings. In this case, a student learns from two different specialized experts, such as one for math and one for code. The authors find that FiRe-OPD helps the student integrate these different types of knowledge. They report a 18.81-point improvement on the MinervaMath benchmark compared to standard OPD [Table 4]. This suggests the filtering mechanism helps resolve conflicts when two teachers provide different guidance.

Where The Edges Are

The authors acknowledge several limitations to the FiRe-OPD framework. Currently, the model treats every token as an independent unit. It does not account for how an error early in a reasoning chain might degrade later tokens. This is known as a lack of "prefix-aware" weighting.

Furthermore, the research focuses on token-level and trajectory-level granularity. The authors note they have not yet explored intermediate granularities. These would include "step-level" or "segment-level" weighting. Such methods might align more naturally with "chain-of-thought" (step-by-step reasoning) structures. Finally, the optimal amount of trajectory filtering ($p=20\%$) and the sensitivity of hyperparameters remain areas for further refinement.

Figures from the paper