GDSD: Reinforcement Learning as Guided Denoiser Self-Distillation for Diffusion Language Models

Current AI models that generate text using "diffusion"—processes that gradually refine noise into coherent sequences—struggle with reinforcement learning (RL) because calculating their exact probability is mathematically intractable. While autoregressive models can easily track likelihoods step-by-step, diffusion models face a massive hurdle when trying to optimize for rewards. Most practitioners currently rely on approximations, but these workarounds often introduce significant errors.

The status quo involves replacing the impossible-to-calculate policy likelihood with an Evidence Lower Bound (ELBO). The ELBO is a mathematical surrogate that provides a lower limit on the true likelihood. It is typically estimated by randomly masking parts of a sequence. While this is computationally efficient and aligns with pre-training, it creates a fundamental flaw: Training-Inference Mismatch (TIM). Because the ELBO is only an approximation, the importance-sampling ratios (weights used to adjust updates based on how much a new policy deviates from an old one) become biased. This bias can cause the entire training process to collapse. Furthermore, the way these models actually generate text during inference—using iterative re-masking—differs from the distribution used during ELBO-based training.

The Problem

The core issue is that we are optimizing a surrogate that does not represent the reality of the model's generation process. In standard RL, we use the ratio between the new policy and the old policy to determine how much to update. When we plug the ELBO into this ratio, we are not using the true likelihood. The authors note that this gap between the ELBO estimate and the true likelihood biases the importance ratio. This can lead to catastrophic training collapse.

As illustrated in, existing methods attempt to bridge this gap using importance sampling based on the ELBO.

However, this creates two distinct pathologies. First, there is the inherent likelihood gap between the ELBO and the true density. Second, there is a sampler-policy gap. The iterative re-masking decoders used in production produce a sampling distribution that does not match the training policy. Trying to correct for these mismatches with a flawed surrogate is inefficient.

How It Works

The authors propose Guided Denoiser Self-Distillation (GDSD), which reframes RL for diffusion models. Instead of estimating likelihoods and correcting for bias, they treat RL as a likelihood-free self-distillation problem.

The mechanism works in three primary stages:

1. Deriving the Teacher: Under a reverse-KL regularized RL framework, the optimal policy can be expressed in closed form. This optimal policy acts as an "advantage-guided self-teacher." The teacher is a version of the model's current logits (raw, unnormalized scores) shifted by the advantage (the reward relative to the mean).
2. Direct Denoiser Distillation: Rather than using importance sampling to adjust a likelihood ratio, GDSD matches the student model's denoiser logits directly to the teacher's logits. This bypasses the need to calculate the ELBO or the intractable partition function (the normalization constant required to turn logits into probabilities) entirely.
3. Normalization-Free Optimization: A major technical hurdle in logit matching is the partition function. Computing this is computationally expensive. It requires summing over the entire vocabulary space. To solve this, the authors introduce Token-level Logit Centralization (TLC). By leveraging the translation invariance of the Softmax operator—meaning $\text{Softmax}(y) = \text{Softmax}(y + c)$—they can center the logits. This effectively cancels out the need for the normalization constant.

This approach moves the optimization from the realm of probabilistic likelihood ratios to the realm of squared-error logit matching.

Numbers

The results suggest that removing the reliance on the ELBO provides a boost to both stability and accuracy. Testing on the Dream-7B model, the authors report that GDSD achieves test-accuracy improvements of up to +19.6% on Sudoku tasks compared to prior state-of-the-art ELBO-based methods [Table 1]. This means the model becomes significantly better at solving logical puzzles.

On the LLaDA-8B model, the performance gains are more modest but highly consistent. The paper finds that GDSD outperforms previous methods across planning, math, and coding benchmarks. Gains range from +0.6% to +5% [Table 2]. Perhaps more importantly for engineers managing long training runs, the reward dynamics are noticeably smoother. As seen in, GDSD demonstrates much more stable convergence on GSM8K, Countdown, and coding tasks.

The ablation study in confirms that increasing the guidance coefficient $\psi$ (the strength of the reward signal) leads to higher training rewards.

This proves the distillation target effectively captures the reward signal.

What's Missing

While the results are strong, there are a few gaps for a practitioner to consider:

• The Generalization Gap in TLC: The authors admit that while Token-level Logit Centralization (TLC) is theoretically more faithful, it sometimes leads to degraded test performance compared to "direct matching." They hypothesize that TLC might make the update focus too strongly on relative logit differences. This may amplify overfitting to training-specific signals.
• Scaling to Massive Parameter Counts: Most experiments were conducted on 7B and 8B parameter models. The computational cost of generating multiple completions per prompt for the "self-teacher" could become a bottleneck for larger models.
• Sensitivity to Reward Quality: The method is entirely dependent on the advantage $A(x_0)$. If the reward function is poorly specified, the "teacher" will distill those errors into the student.

Should You Prototype This

Yes, if you are moving toward non-autoregressive or diffusion-based generation for reasoning tasks. The ability to bypass the ELBO solves a frustrating aspect of training dLLMs: the tendency for training to diverge due to the training-inference mismatch. The implementation is straightforward. The authors provide code at https://github.com/GaryBall/GDSD. If you are seeing oscillating reward curves or sudden collapses in your diffusion RL runs, switching to a logit-matching distillation objective is a high-leverage move. Start with the "direct matching" version before moving to TLC. This is because TLC carries a slight risk of overfitting.

Figures from the paper