What Gets Unmasked First? Trajectory Analysis of Diffusion Models for Graph-to-Text Generation

Unmasking the Trajectory: How Diffusion Models Generate Text from Graphs

Researchers have been trying to solve the problem of turning structured data, like knowledge graphs, into natural language sentences. Most people use "autoregressive" models—the standard LLM architecture that predicts one token at a time from left to right. However, a new study looks at a different approach: masked diffusion language models (MDLMs). These models work by starting with a sequence of "mask" tokens (essentially blanks) and iteratively filling them in.

While standard LLMs are forced to commit to a linear sequence immediately, MDLMs have the freedom to choose which parts of the sentence to resolve first. This paper investigates whether that flexibility actually helps when the input is a complex graph. The researchers discovered that while these models naturally prioritize important information like entities, the process of fine-tuning them for specific tasks actually breaks this helpful behavior. This causes them to "give up" and end sentences prematurely.

The Problem

The status quo for graph-to-text generation relies on autoregressive LLMs. These models generate text sequentially. This forces them to commit to a specific linearization (a text-based ordering) of a graph's structure from the very first token. This "left-to-right" constraint is a fundamental bottleneck. If the model makes a poor choice early in the sequence regarding how to represent a relationship, it cannot go back and revise it. This often leads to information omission or factual hallucinations, especially as the number of graph triplets increases.

The researchers identified a specific, previously undocumented failure mode that occurs when you try to fix these models using Supervised Fine-Tuning (SFT). While SFT is intended to improve accuracy, it disrupts the natural generation trajectory of MDLMs. Specifically, the models begin to anchor structural tokens—like periods or end-of-sentence (EOS) markers—too early in the denoising process. As shown in, this premature commitment effectively fixes the output length before the model has had a chance to resolve the actual content.

This results in sentences that are either truncated or filled with redundant, hallucinated filler to satisfy the prematurely fixed structure.

How It Works

The authors' approach centers on two main interventions: a way to fix the decoding process and a way to better ingest graph data.

1. $\lambda$-scaled Structural Decoding: This is an inference-time modification designed to prevent the premature commitment mentioned above. Instead of letting the model pick the most confident token at every step, the researchers introduce a scalar $\lambda \in (0, 1]$. During the denoising process, they multiply the predicted confidence of any structural token (punctuation, EOS, etc.) by this $\lambda$. This effectively downweights the "certainty" of structural tokens. This forces the model to resolve semantic content—like entities and relations—before it is allowed to finalize the sentence structure.

2. Graph-LLaDA Architecture: To move beyond simple text-based linearization, the authors propose Graph-LLaDA. As illustrated in, this architecture uses a dual-stream approach.

A Graph Transformer encoder processes the relational topology of the graph. It produces structure-aware node representations. These representations are then injected into the LLaDA decoder using a per-node placeholder strategy. Rather than compressing the whole graph into a single token—which creates an information bottleneck—they use a sequence of tokens delimited by <g_start> and <g_end> markers. This allows the language model to attend to individual entity representations directly.

The training is conducted in two stages. First, the LLM is frozen to align the GNN (Graph Neural Network) output with the LLM's embedding space. Second, they use LoRA (Low-Rank Adaptation, a technique for efficient fine-tuning) to adapt the LLM to these structural signals.

Numbers

The impact of these changes is measurable and significant. The authors report that applying $\lambda$-scaled structural decoding to an SFT-tuned LLaDA-8B model recovers +9.4 BLEU-4 points on the WebNLG benchmark [Table 2]. This metric measures lexical overlap with reference text. Looking closer at the timing, the researchers found that without $\lambda$-scaling, the proportion of content tokens generated before the first period was a mere 17.8%. With $\lambda$-scaling, this jumped to 89.9% [Table 7]. This means the model successfully prioritizes substance over structure.

In terms of generalization, the paper demonstrates that while specialized supervised models (like GAP) achieve high scores on specific datasets, they fail when moved to new domains. In contrast, the Graph-LLaDA approach shows much higher robustness. On the LAGRANGE benchmark—a large-scale dataset used here for out-of-domain testing—the authors' models ranked in the top two for pairwise LLM-based evaluations [Table 4]. Notably, the Graph-LLaDA model, despite having a much smaller parameter count than LLaMA 3.3-70B, was able to close much of the performance gap in in-domain scenarios [Table 9].

What's Missing

There are several gaps in this research that a practitioner should consider before moving to production.

First, the validation of the "premature commitment" failure mode and the effectiveness of $\lambda$-scaling is currently limited to the LLaDA backbone. While the authors replicated the "entity-first" trajectory pattern on the Dream-7B model, they did not test if the SFT failure mode or the $\lambda$-scaling fix applies to other MDLM architectures. If you are using a different diffusion backbone, you cannot assume this fix will work out of the box.

Second, the experiments rely exclusively on single-token-per-step decoding. The authors admit that multi-token decoding—which is much faster for inference—causes a substantial degradation in quality in their current setup. For a real-world deployment where throughput is a primary concern, the current "slow and steady" diffusion approach might be a dealbreaker.

Finally, the evaluation is strictly limited to English. Knowledge graphs are often global in scope. The paper provides no insight into how these structural encodings or decoding strategies behave in multilingual contexts.

Should You Prototype This

Yes, but only if your priority is faithfulness over throughput.

If you are building a system where omitting a single fact from a database is catastrophic (e.g., medical or legal data extraction), the Graph-LLaDA architecture and the $\lambda$-scaling trick are highly promising. The ability to inject explicit relational topology via a Graph Transformer provides a level of grounding that standard autoregressive LLMs struggle to maintain.

However, keep the following risks in mind. First, if you need high-concurrency, low-latency text generation, the current single-token requirement makes this an expensive way to generate text. Second, the authors used 1x–8x H100 GPUs for training and inference. You should account for this significant compute footprint in your budget. Finally, there is no official code repository listed in the paper. Code is reportedly available via the authors' canonical links, so verify availability before starting your implementation.

Figures from the paper