LISA: Likelihood Score Alignment for Visual-condition Controllable Generation

Can We Explicitly Teach the "Helper" in Diffusion?

When teaching AI to follow visual instructions—like a sketch, a depth map, or a human pose—it usually takes a massive amount of training time and compute. Current state-of-the-art methods rely on a "dual-branch" setup. This uses a massive, frozen "main" model that knows how to draw beautiful images. It also uses a smaller "side" network that learns how to translate a specific condition (like a pose) into instructions the main model understands. But we don't really know what the side network is actually doing. We also do not know why it takes so long to get it right.

The authors of this paper argue that we can solve this by being more mathematically explicit. Instead of letting the side network learn its job implicitly, they propose LISA (Likelihood Score Alignment). This method forces the side network to focus specifically on the "gap" between a generic image and the conditioned one.

The missing objective in dual-branch training

The central question the authors investigate is: why is the training of side-networks in the dual-branch paradigm so inefficient? Can we provide a better supervisory signal to guide them?

In the standard setup, we take a pretrained diffusion model (the "main net") and freeze it. This preserves its ability to generate high-quality, realistic textures. We then train a "side net" to encode a condition $c$ (the visual instruction). This network then injects its features into the main net. While this works, the authors observe that the side network's role is essentially a mathematical residue. If the main net provides the "unconditional score"—the direction toward the general manifold of natural images—then the side net must provide the "likelihood score." This score steers the generation toward the specific condition. Currently, the side network is only supervised by the final output of the combined model. This means it must figure out this complex, decomposed role entirely through trial and error.

Cracks in the implicit learning paradigm

Before this work, the field largely treated the side network as a black-box adapter. As shown in the conceptual breakdown in, the dual-branch paradigm assumes the main network handles perceptual quality.

Simultaneously, the side network handles control. However, because the side network is only trained via the standard diffusion loss (minimizing the distance between the predicted noise and the actual noise), its internal features lack explicit direction. They are never explicitly told to represent the difference between the conditional and unconditional distributions.

Existing attempts to fix this, such as REPA (Representation Alignment), involve aligning the side network's features with a pretrained semantic encoder like DINOv2. The authors note a significant crack here. These methods bound the performance of the generator to the capabilities of the external encoder. If the encoder does not understand a specific spatial nuance, the generator won't either. This creates a bottleneck where the "helper" is only as smart as the teacher it is being aligned with.

Aligning with the likelihood residue

The authors' investigation proceeds by turning this theoretical decomposition into a concrete training objective. They realize that while the true conditional score is intractable, they can construct a highly effective proxy.

As detailed in, the researchers use a two-step forward pass during training.

First, they perform a forward pass with the main network alone to get the unconditional score. Second, they use the known denoising target (the actual noise added to the image) to calculate an "approximated likelihood score." This is essentially the vector difference between the target and the unconditional prediction.

To turn this into a training signal, they "hook" the intermediate features from a specific layer of the side network. They then pass them through a very lightweight decoder ($D_\psi$). This decoder's sole job is to project those features into the same latent score space as the score. The core of the LISA method is a regularization loss ($\mathcal{L}_{LISA}$). This loss penalizes the distance between the decoder's output and this approximated likelihood score. By optimizing this alongside the standard diffusion loss, the side network is explicitly coached. It learns to represent the conditional "delta" required to move from an unconditional image to a conditioned one.

Faster convergence and better composition

The findings are quite striking, particularly regarding training efficiency. The authors report that LISA can achieve $>2.78\times$ faster convergence in cases like ControlNet. For instance, in depth-map conditioning, LISA trained for only 4,000 iterations outperformed a standard ControlNet trained for 10,000 iterations. This was measured across FID (a metric for image quality), CLIP (a measure of text-image similarity), and RMSE (a measure of depth accuracy).

Beyond speed, the qualitative impact is visible in the model's ability to follow complex instructions. In, the authors show that LISA produces images with significantly higher condition fidelity.

For example, it produces more accurate poses compared to T2I-Adapter or vanilla ControlNet. Perhaps most interestingly, the authors find that this explicit alignment leads to "disentangled" features. Because the side network is forced to model only the likelihood score, its features become less entangled with the main network's unconditional priors. This manifests as a "bonus" in compositional control. As seen in, when you combine two independently trained side networks (such as one for pose and one for segmentation) by summing their features, LISA-trained networks compose much more cleanly than their counterparts.

Implications for controllable generation

If this approach of "role decomposition" scales, it suggests a fundamental shift in how we should design adapters for large-scale generative models. Rather than seeking better "teacher" encoders to align with, we should perhaps look for better ways to approximate the mathematical residues of the generative process itself.

There are two immediate implications I see: 1. Efficiency as a first-class citizen: For practitioners, LISA offers a way to reduce the massive GPU-hour costs associated with training new ControlNets or Adapters. Since the decoder is dropped during inference, there is zero extra cost at runtime. This makes it a "free" upgrade for deployment. 2. Architecture Agnosticism: The paper demonstrates that LISA works on both U-Net architectures (Stable Diffusion 2.1) and Diffusion Transformers (Stable Diffusion 3). This suggests the principle isn't tied to a specific way of injecting features. Instead, it is a property of the score-matching objective itself.

The paper does not, however, explore how this scaling behaves as the "main" model grows significantly larger than the side network. It also does not show if the "approximated" likelihood score remains a good proxy in extremely high-dimensional regimes. An obvious follow-up experiment would be to test if LISA can maintain its convergence advantages when applied to much larger, multi-modal foundation models. Such models might have a gap between the unconditional and conditional manifolds that is significantly more complex.

Figures from the paper