Sumi: Open Uniform Diffusion Language Model from Scratch

Most large language models work like a typewriter. They predict one token (a unit of text) at a time, moving strictly from left to right. This autoregressive (AR) approach is the industry standard. However, it is inherently rigid. Once a token is generated, it is locked in.

A newer class of models called diffusion language models offers a different way to work. Instead of sequential typing, they treat text generation like an image diffusion process. They start with noise and gradually refine it into coherent text. Specifically, uniform diffusion language models (UDLMs) allow any token in a sequence to be updated at any step. This theoretically enables more flexible generation and self-correction. While masked diffusion models (which only fill in gaps) have been scaled to 8B parameters, no one has yet pretrained a UDLM from scratch at a significant scale.

The Sumi paper changes that. The authors introduce Sumi, a 7B-parameter UDLM pretrained from scratch on 1.5T tokens. They are releasing the weights, checkpoints, and the full training recipe. This provides a clean reference point for how uniform diffusion scales in the data-rich regime.

The limitations of sequential and masked generation

Current state-of-the-art modeling is bifurcated. On one side, you have autoregressive models. These are excellent at maintaining coherence but struggle with non-linear refinement. On the other, you have masked diffusion models (MDLMs), such as LLaDA. These can fill in blanks but suffer from a fundamental rigidity. Once an MDLM fills a masked token, that token can never be revised. This creates a "one-shot" bottleneck in the middle of the generation process.

Uniform diffusion aims to break this. It allows the model to revisit any part of the "canvas"—the allocated buffer of token positions—at any time. However, the community has lacked a large-scale, scratch-pretrained baseline. Without this, we cannot see if this flexibility translates to better performance or if the overhead makes it impractical.

Architecture and the GIDD objective

Sumi is built as a 7B-parameter time-agnostic bidirectional Transformer. Unlike AR models that use causal masking to prevent looking at future tokens, Sumi is bidirectional. This means every token can attend to every other token in the sequence.

The core mechanism relies on the Generalized Interpolating Discrete Diffusion (GIDD) framework. It utilizes an improved formulation that reparameterizes the Evidence Lower Bound (ELBO)—a mathematical objective used to approximate the likelihood of the data—in terms of the signal-to-noise ratio (SNR). The training process follows these stages:

1. Pre-training: The model is trained on 1.3T tokens. This mixture is heavily weighted toward educational content and code.
2. Mid-training Stage 1: A 130B token run focused on domain-specific data.
3. Mid-training Stage 2: A 120B token run with an extended sequence length of 4,864 to improve context handling.

To ensure stability, the authors used a LLaMA-style architecture. This includes SwiGLU MLPs and Grouped-Query Attention (GQA). GQA reduces the memory footprint of the Key-Value (KV) cache (the model's short-term memory of previous tokens). They also implemented an "off-by-one" softmax in the attention mechanism. This helps mitigate "attention sinks"—a phenomenon where models over-rely on the first few tokens to maintain stability.

Competitive performance and the canvas constraint

The authors report that Sumi-7B performs competitively with AR models trained on similar token budgets. On general knowledge (MMLU: 51.1), Sumi shows strong capabilities. On coding (HumanEval: 22.6), it outperforms several baseline AR models like Falcon-7B and Llama 2-7B. However, there is a clear trade-off. The model significantly underperforms on commonsense reasoning benchmarks like PIQA and HellaSwag. The authors suggest this is due to their "education-heavy" data mixture. This mixture prioritizes logic and code over casual web text.

One critical finding involves "canvas length." This is the number of token positions allocated at the start of generation. As shown in, Sumi's fluency is highly sensitive to this setting.

If the canvas is too short or too long relative to the training distribution, perplexity (a measure of how "surprised" the model is by the text) spikes. The model is most stable within a specific "fluent band" around its trained sequence lengths.

Furthermore, the authors explored whether the theoretical advantage of uniform diffusion—parallelism—holds up. In, they show that for coding tasks, you can commit up to $k=4$ tokens per denoising step without a major hit to accuracy.

However, for mathematical tasks like GSM8K, accuracy drops immediately if you attempt to move faster than one token per step.

Unresolved questions in self-correction

Despite the promise of "revision," the paper reveals a challenging reality regarding self-correction. Because UDLMs can overwrite tokens, one might assume extra denoising steps would allow the model to "fix" mistakes.

The authors tested this by providing an explicit revision budget. This lets the model take multiple passes at the same sequence. As detailed in [Table 2], the model frequently overwrites tokens. In some tasks, this happens 100% of the time. However, these edits are mostly "A $\rightarrow$ B $\rightarrow$ A" round trips. The model changes its mind, then changes it back. This results in almost zero net change to the final answer. Consequently, accuracy remains unchanged. This suggests that naive over-denoising is not a viable strategy for error correction.

Additionally, the authors note that confidence-based sampling induces a structured order of token commitment .

This sampler picks tokens the model is most uncertain about. Even though the model is technically order-agnostic, its actual "thinking" during inference follows a specific pattern.

The verdict

Is Sumi ready for production? Probably not.

As a base model, it lacks instruction tuning and safety alignment. More importantly, the inference dynamics are finicky. You cannot simply throw a variable-length prompt at it. You must carefully manage the canvas length to avoid the fluency collapses seen in . The lack of effective self-correction also means you aren't getting the "thinking twice" benefit often expected from diffusion models.

However, for researchers, this is a goldmine. The authors have released the weights, checkpoints, and the full training recipe. They also included the specific data mixture. If you want to understand how to scale uniform diffusion, this is the first real sandbox available.

Code is available at https://github.com/tohoku-nlp/sumi and model weights at https://huggingface.co/collections/tohoku-nlp/sumi.

Figures from the paper