Scaling Native Multimodal Pre-Training From Scratch

Scaling Native Multimodal Pre-Training: New Compute-Optimal Scaling Laws Discovered

Researchers have found that training AI models on both text and images from scratch follows specific mathematical rules. They discovered that while text learning is stable, learning from images requires shifting more resources toward data rather than just making the model bigger.

The field of multimodal machine learning aims to move beyond text-only large language models (LLMs) to create systems that perceive the physical world. Currently, the industry standard is "late fusion." This is a method where a pre-trained language model is coupled with a separate, pre-trained vision encoder via a projection layer. This is akin to hiring a translator to sit between a person who speaks only English and a person who speaks only French. While efficient, the two parties never truly share a common conceptual framework.

While native multimodal pre-training promises deeper integration, engineers have lacked a roadmap for how to scale it. Without knowing how to balance model size against the amount of data, training these massive models becomes a game of expensive guesswork. This study from researchers at The Chinese University of Hong Kong and Tencent fills that gap by characterizing the scaling laws of this paradigm.

The asymmetry of late fusion

The prevailing paradigm in multimodal AI relies on late fusion to save on computational costs. Developers reuse existing, high-performing weights from unimodal models. This avoids the massive expense of training from zero. However, the authors argue this introduces a fundamental optimization asymmetry. Because vision and language representations are learned independently, the model lacks a unified representational space.

This separation prevents the kind of deep cross-modal integration required for complex reasoning. Traditional scaling laws provide a guide for unimodal language models. These laws tell engineers how to distribute a fixed compute budget between model size ($N$) and token count ($D$). But these laws do not account for the tension created when two different types of data compete for the same shared parameters.

Decoupling the scaling objectives

To solve this, the authors investigate a "native" architecture. They use a decoder-only Transformer trained from scratch on a mixture of 250B text tokens and 75B multimodal tokens. Instead of a separate vision encoder, they use a single patch embedding layer. This projects images directly into continuous embeddings (mathematical representations of image segments). They also employ a Mixture of Experts (MoE) architecture. This uses specialized sub-networks to handle different types of information to improve efficiency.

The researchers' core methodology involves decoupling the optimization problem. They treat the language objective ($L_{text}$) and the multimodal objective ($L_{mm}$) as two separate but simultaneous goals. To find the "compute-optimal" point, they use two distinct estimators: IsoFLOP profiles and training-curve envelopes.

As shown in, the IsoFLOP profile method identifies the optimal model size. It does this by finding the "valley" in a loss curve where a specific compute budget is most efficient. They then use the training-curve envelope to verify these findings .

This envelope extracts the lower boundary of minimal loss across many training runs. This dual approach ensures that their discovered laws are inherent properties of the data.

Divergent paths for text and vision

The study's most significant finding is that language and multimodal objectives follow different scaling behaviors. The authors report that the language allocation law is "composition-invariant." This means the amount of parameter capacity needed to learn text remains stable. It stays consistent regardless of how much image data is added .

In stark contrast, the multimodal allocation law is "composition-variant." The authors find that as the ratio of multimodal data increases, the optimal allocation shifts. It moves heavily toward data scaling rather than parameter expansion. Essentially, as you add more visual complexity, the most efficient way to improve the model is to feed it more tokens. This is more effective than simply building a larger model .

This discovery leads to a "joint Pareto frontier." This is a mathematical boundary representing the optimal trade-off between language and multimodal performance . For instance, the paper notes that in a text-heavy mixture, models only become compute-efficient at much larger scales. This forces the optimal resource allocation toward greater model capacity.

Unexpected benefits for text reasoning

One might worry that forcing a model to learn visual patterns would hurt its ability to reason with pure text. However, the authors report that native multimodal pre-training preserves core language capabilities. In their evaluations across 16 text benchmarks, average accuracy remained consistent regardless of the multimodal data ratio .

Even more surprising is the evidence of "positive cross-modal transfer." The authors measured the model's performance on abstract spatial reasoning tasks that use only text . They found that models trained with multimodal data significantly outperformed text-only baselines. This advantage grew as the model scale increased. This suggests that spatial structures learned from images are encoded into the shared parameter space. The model can "understand" spatial concepts even when reading only text.

Finally, the researchers found that multimodal in-context learning emerges as a function of scale . In-context learning is the ability to learn a new task from a few examples provided in the prompt. While tiny models struggle to learn from examples, larger models show significant gains. Specifically, the A3B scale model showed a $+2.43$ point gain in 3-shot settings compared to 0-shot settings.

Limitations and the road ahead

While these findings provide a rigorous framework, the scope of the study is bounded. The authors note that their analysis is limited to models with up to 3B active parameters. In the world of foundation models, it remains to be seen if these power laws hold at much larger scales.

Additionally, the evaluation relied on a single family of image-text data. Different types of visual data might introduce different scaling pressures. Finally, the researchers used loss-based scaling proxies. While loss is a reliable indicator of learning, it does not always perfectly correlate with every downstream task.

The verdict

If you are designing a multimodal foundation model, the verdict is clear: depend on the data mixture. The era of "just make the model bigger" is ending for multimodal systems. The authors' work proves that as you increase the importance of visual data, your priority must shift. You must expand your token budget rather than just adding parameters.

This research provides the essential infrastructure for moving away from "late fusion" approaches. By following the derived Pareto frontier, engineers can now predictably allocate compute. This helps build models that are both linguistically competent and visually grounded.

Figures from the paper