Initialization is Half the Battle: Generating Diverse Images from a Guidance Potential Posterior

Initialization is Half the Battle

Most AI image generators suffer from a frustrating lack of variety. Even when you change the random seed, the model tends to spit out the same few compositions, colors, or styles. This phenomenon, known as mode collapse (the tendency of a model to produce a limited set of outputs), occurs because the model gets stuck in a handful of "dominant" ways to interpret a prompt. Current industry standards attempt to fix this by intervening during the generation process itself. They essentially try to steer the ship while it is already moving.

A new paper argues that we are looking at the wrong end of the pipeline. The authors suggest that the problem begins at the very first step: the selection of the initial noise. Because standard Gaussian initialization is agnostic to the specific prompt, it frequently drops the starting point into "high-potential" regions. These regions act like gravity wells. They pull every subsequent trajectory into the same predictable modes.

The Problem

The status quo in generative modeling assumes that drawing initial noise $x_T$ from a standard isotropic Gaussian distribution $\mathcal{N}(0, I)$ is a sufficient starting point. However, the authors identify a critical oversight. This initialization is blind to the guidance potential landscape created by the user's prompt.

Geometrically, dominant modes in a model create deep basins of attraction. These are characterized by high local sharpness (a measure of how quickly the probability density changes). As shown in, a standard prior often lands in these sharp, high-potential hills.

Once the generative process begins, these regions induce a rapid contraction of probability volume. This forces diverse stochastic trajectories to merge into a single, stereotypic output. The authors demonstrate this via a toy experiment in .

Here, standard Gaussian initialization only recovers 5 out of 9 possible modes. In contrast, a more informed start recovers all of them. Essentially, if you start in a trap, you stay in a trap.

How It Works

The authors propose Diversity-inducing Initialization (DivIn). It replaces the fixed Gaussian prior with a "guidance potential posterior." Instead of picking a random starting point, they re-weight the prior to favor "low-potential" regions. These are areas where the landscape is flatter and trajectories are less likely to collapse.

The mechanism works in three primary stages:

1. Defining the Potential: The core innovation is the use of a generalized Tweedie potential, $U(x_T, c)$. This is calculated as the Euclidean distance between the conditional and unconditional single-step denoising estimates. This compares how the model sees the image with the prompt versus without it. By using this "Tweedie" proxy, the method remains robust across different inference schedules. It works for both diffusion and flow matching models.
2. Targeting Diversity: They formulate a new target distribution, $p_{diverse}(x_T | c)$. This mathematically balances the standard Gaussian prior with an energy term. This term penalizes high potential. It effectively steers the initialization toward the "flat" basins seen in .
3. Langevin Navigation: To sample from this complex new distribution, they use Langevin dynamics (a stochastic iterative process). This involves an update rule that uses the gradient of the potential. It pushes the noise away from collapsing regions. Simultaneously, it uses a restorative force to anchor the noise to the valid data manifold.

The authors note that while multiple steps can be taken, a single gradient update ($K=1$) provides significant gains. This makes it a lightweight addition to the inference pipeline.

Numbers

From a production standpoint, DivIn is highly efficient. It is also remarkably orthogonal (independent) to existing methods. The authors report that for a single step ($K=1$), the additional wall-clock time is only approximately 3%. In their tests, generation time increased from 0.754 seconds to 0.779 seconds per image.

The performance deltas are significant. In class-to-image generation on ImageNet using Stable Diffusion v1.4, DivIn achieves a Vendi score of 4.688. The Vendi score quantifies the effective number of modes in a distribution. This outperforms standalone trajectory-based baselines like PG, CADS, and IG. Furthermore, DivIn does not just trade quality for diversity. It actually expands the "Pareto frontier." This is the optimal trade-off curve between quality and variety. As seen in, combining DivIn with existing trajectory-based methods moves the entire frontier outward.

This means you get more diversity for any given level of image fidelity.

What's Missing

While the results are compelling, there are gaps that a practitioner should consider:

• Hyperparameter Sensitivity: The method relies on a temperature hyperparameter $\tau$ to balance prior adherence and diversity. Deploying this on entirely new architectures may require empirical tuning to find the optimal $\tau$.
• Conditional Limitation: The current formulation is exclusively applicable to conditional models. Because the objective relies on the delta between conditional and unconditional guidance, it cannot natively support unconditional generation.
• Variance in Stochasticity: Because DivIn uses Langevin dynamics, it introduces higher metric variance across independent runs. This is compared to the deterministic nature of a standard fixed Gaussian seed.

Should You Prototype This

Yes, specifically if you are building applications where "creative variety" is a core requirement. Use this if you are already using diffusion or flow matching models. Because DivIn is a "plug-and-play" inference-time modification, you do not need to retrain anything. The 3% latency hit is negligible compared to the jump in mode coverage shown in and .

If you are currently fighting mode collapse by tweaking guidance scales, stop. Try implementing this at the initialization step first. The code is reportedly available at https://github.com/South7X/divin.

Figures from the paper