Representation Forcing for Bottleneck-Free Unified Multimodal Models

Most AI models that both understand and create images use a separate "translator"—a Variational Autoencoder (VAE). This layer turns abstract ideas into pixels. It compresses visual data into a manageable latent space. This helps manage computational load. However, it creates a structural bottleneck. The generative process is trapped in a latent space optimized for reconstruction rather than the model's actual goals.

Current Unified Multimodal Models (UMMs) bring language and image generation into a single transformer backbone. However, they almost always rely on a frozen, separately pretrained VAE. This dependency caps the model's ability to generate high-fidelity images. If you remove the VAE to generate directly in pixel space, the model often collapses. It struggles to learn both high-level semantic structure and fine-grained textures from raw pixels.

The Problem

The status quo relies on a decoupling that is fundamentally at odds with true unification. As shown in, prevailing UMMs use a frozen VAE encoder and decoder. This creates a "lossy" boundary. The generative model can only play within the sandbox defined by the VAE's latent space.

When engineers attempt to bypass this with a "naive" pixel-space approach, they hit a quality wall. Without the structural scaffolding of a pretrained latent space, the model lacks intermediate guidance. It tries to solve a massive multi-scale problem. It must decide where a mountain goes while simultaneously deciding the texture of a rock. The result is often incoherent compositions and distorted objects, a failure mode seen in .

How It Works

The authors propose Representation Forcing (RF) to bridge this gap. It makes representation prediction a native, internal capability of the model. Instead of using an external VAE, the model learns to generate its own "blueprints."

The mechanism works in three primary stages:

1. Internal Feature Extraction: The model uses its own understanding encoder (a DINOv3 ViT) to extract high-level visual features. These features capture object identity and spatial layout.
2. Online Vector Quantization: To make these features usable, they are discretized into "representation tokens." The authors use online vector quantization. They use an Exponential Moving Average (EMA) copy of the encoder to keep targets stable. This turns features into a discrete codebook of $K=16,384$ prototypes.
3. Autoregressive Forcing: During training, the decoder predicts these representation tokens autoregressively (predicting the next token in a sequence based on previous ones). It uses the same next-token prediction objective used for text. As seen in the training pipeline, these predicted tokens are placed back into the sequence.

They act as in-context conditioning for the subsequent pixel-space diffusion.

The pixel generation is handled via flow matching (a method for modeling the probability flow between noise and data). This occurs within the same shared transformer backbone. The representation tokens provide the "scaffold." This allows the diffusion process to focus purely on low-level rendering.

Results

These architectural mechanisms lead to significant performance gains. The core claim is that RF allows pixel-space models to match complex VAE-based systems. The authors report that their RF-Pixel model achieves a GenEval overall score of 0.84 (without an LLM rewriter). This outperforms the BAGEL baseline (0.82) and matches established unified models like BLIP3-o (0.84). Using an LLM rewriter brings the score to 0.88, matching state-of-the-art unified models.

The benefits are dual-purpose. RF improves performance across both VAE-based and pixel-space generation pathways. For pixel-space models, RF improves 6 out of 8 benchmarks. Notable gains include MMMU (+4.3) and MME (+3.6). Interestingly, pixel-space RF outperforms VAE-based variants on 6 of 8 benchmarks. This suggests that removing the VAE bottleneck actually tightens the coupling between perception and generation.

What's Missing

There are gaps that a production engineer should note. First, the model was initialized from a pretrained large language model (Qwen3-A3B). It was not trained from scratch. We do not know if these benefits hold without existing linguistic priors.

Second, the scope is limited to still-image generation. The authors have not extended RF to video or temporal modalities. Moving from static pixel-space diffusion to temporal consistency is a major challenge.

Finally, the paper focuses on high-level semantic understanding. RF saw slight reductions in performance on document-oriented tasks like DocVQA (-2.0) and ChartQA (-0.4). This suggests representation tokens might discard the granular information required for precise text and layout parsing.

Should You Prototype This

Yes, if you want to move away from the complexity of separate, frozen VAE pipelines. The ability to match VAE-based quality in pure pixel space is a significant win. The transition from "auxiliary alignment" (like REPA) to "direct prediction" (RF) is a clear winner in the ablations. Forcing the model to actually output representation tokens is far more effective than mere feature nudging.

Code is reportedly available; see the project page at https://yuqingwang1029.github.io/RepresentationForcing for the canonical link. If you have the compute for a Mixture-of-Transformers (MoT) setup, this is a viable path toward an end-to-end system.

Figures from the paper