BlockPilot: Instance-Adaptive Policy Learning for Diffusion-based Speculative Decoding

When using AI to speed up text generation, most systems use a fixed "block size" for parallel processing. This is akin to a factory assembly line that moves at a single, unchanging speed. It moves the same way whether it is assembling simple bolts or complex engines. If the speed is too high, mistakes accumulate and the line jams. If it is too low, the factory wastes energy and time.

Large Language Models (LLMs) are traditionally slow. They generate text one token at a time. This sequential process is called autoregressive decoding. To break this bottleneck, researchers use "speculative decoding." In this setup, a tiny, fast "draft model" guesses several tokens at once. A large "target model" then verifies them in parallel. Recent advancements use diffusion-based drafting. This allows the draft model to propose entire blocks of tokens simultaneously. This significantly increases throughput.

A new study from Alibaba Group introduces BlockPilot. This method moves away from the industry standard of using a static block size. The researchers argue that the ideal amount of parallel work depends on the specific input. By learning to predict the optimal block size for every individual sentence, BlockPilot achieves significant speedups. It does this without sacrificing the quality of the output.

The mismatch in fixed-block drafting

Current state-of-the-art diffusion-based speculative decoding relies on a fixed block size ($B$). This size is typically inherited from the model's training phase. The logic is simple. Use the same settings for inference that were used during training to ensure consistency. However, the authors argue that this assumption is fundamentally flawed.

As illustrated in, the draft model proposes a block of tokens.

The target model then verifies the block. The efficiency of this process depends on the "acceptance length." This is the number of tokens the target model actually keeps. If the block size is too large for a difficult prompt, the draft model will likely make errors. This causes the verification step to reject most of the block. Such errors waste computational cycles. Conversely, if the block size is too small for a predictable prompt, the system fails to use the parallelism of diffusion models.

The study finds that a single, fixed block size is almost never optimal for all inputs. Through a systematic sweep of different sizes, the authors report in [Figure 3a] that only a subset of samples performs best under the default training configuration. Many samples clearly prefer different block sizes to maximize their acceptance length.

Predicting the optimal window

Instead of searching for the best block size during every inference step, the authors propose a lightweight classification problem. They call this approach BlockPilot.

The mechanism works in three primary stages:

1. Feature Extraction: After the target model completes the "prefilling" stage (the initial pass where it processes the user's prompt), researchers extract the predictive probability distribution of the last token. This distribution is a list of probabilities for all possible next tokens. Because this token has attended to the entire preceding context, it summarizes the prompt's complexity.
2. Policy Prediction: This distribution is fed into a lightweight Multi-Layer Perceptron (MLP). An MLP is a simple type of neural network. The authors found that the optimal block size isn't scattered randomly. It tends to cluster in a narrow range around the original training size. This "local structure" [Figure 3b, 3c] allows them to limit the predictor's search to a small, discrete set of candidates.
3. Adaptive Execution: The predictor selects the best block size ($\hat{B}$). This size is then used for the rest of that specific generation task. As shown in, this happens only once after prefilling.

Therefore, the decision-making overhead is minimal.

The authors justify this approach using mathematical theory. They note that the "regret"—the loss in efficiency from an incorrect prediction—is bounded. This bound depends on how close the predicted size is to the true optimum. Thus, the predictor does not need to be perfect. It only needs to find the right neighborhood.

Gains in speed and acceptance

The experimental results suggest that an adaptive policy provides a measurable boost to throughput. The authors evaluated BlockPilot across several model scales. These included Qwen3-4B and Qwen3-8B. They used diverse benchmarks covering mathematics, coding, and general conversation.

The paper reports that on the Qwen3-4B model, BlockPilot achieves an average speedup of 4.20× under a temperature of $T=1$. Temperature $T=1$ refers to a setting that introduces more randomness into text generation. This outperforms the best fixed-block baseline, DFlash(16). That baseline achieved a 3.80× speedup under the same conditions. This means BlockPilot is significantly faster than previous methods in stochastic settings. Crucially, the authors note that this speedup includes a higher average acceptance length ($\tau$) of 5.92 tokens. This tells us the draft model is successfully proposing more valid content per cycle.

The benefits appear robust across different tasks. For example, in the "Chat" category (MT-Bench), BlockPilot showed significant advantages. It effectively balances speed with the uncertainty of conversational dialogue. Even when generation becomes more stochastic, the predictor adjusts the block size to maintain efficiency.

Computational trade-offs

The implementation of BlockPilot involves specific costs that practitioners must consider.

First, there is a one-time "offline" cost during data construction. To train the predictor, the authors ran many different block sizes for every training sample. This was done to find the ground truth. The paper notes that for a 32B parameter model, constructing one training sample could take roughly 25 seconds. This does not affect real-time inference. However, it is a hurdle for teams training custom predictors for massive models.

Second, the predictor introduces a small amount of additional memory and latency. According to the overhead analysis in Table 1, the predictor adds a millisecond-level delay. For a Qwen3-4B backbone, this delay is approximately 7.34 ms. It also adds a minor memory footprint. However, the authors argue this is a favorable trade-off. The millisecond-scale cost is much smaller than the multi-fold speedup gained during generation.

Finally, the paper does not explore how this adaptive strategy interacts with extremely long-context windows. It also does not examine interactions with specialized hardware that relies on rigid memory patterns.

The verdict

BlockPilot is a highly effective "plug-and-play" optimization for diffusion-based speculative decoding. It solves a genuine inefficiency. It addresses the misalignment between fixed block sizes and the varying predictability of different prompts. It does so without requiring changes to the underlying LLM or draft model.

If you are running large-scale inference, this is a strong option. The minimal latency penalty is justified by the consistent speedup gains. This applies across math, code, and chat tasks. Code for the implementation is reportedly available; see the paper for the canonical link.

Figures from the paper