EVA01: Unified Native 3D Understanding and Generation via Mixture-of-Transformers

Most current AI models treat 3D objects like separate files they simply "reconstruct" from 2D images. They act as stateless workers. You give them a picture, they spit out a mesh (a collection of vertices and faces representing a 3D shape), and the connection to the original intent is lost. This makes iterative design impossible. Every tiny tweak requires a full, expensive regeneration from scratch.

Until now, the field has relied on decoupling semantic understanding from geometric reasoning. We use frozen image encoders to interpret what an object is. Then, we use separate diffusion models to lift those pixels into 3D space. This leaves a gap between "knowing" what a character looks like and "reasoning" about its physical structure.

The SeeleAI team attempts to close this gap with EVA01. Instead of treating 3D as an external output, they treat it as a native modality. It is a first-class citizen in the model's sequence stream. This allows the model to generate, understand, and edit 3D meshes through a continuous, context-aware conversation.

The Problem

The status quo in 3D generation is dominated by diffusion-based reconstruction models. These models are essentially high-fidelity "lifters." They take dense 2D pixel priors and map them to 3D geometry. While they produce visually impressive results, they are fundamentally stateless. They lack a way to maintain geometric identity across sequential modifications. Therefore, they cannot perform true multi-turn editing. If you want to change a character's hat and then change their shoes, a stateless pipeline often forgets the original look after the first edit.

Current Multimodal Large Language Models (MLLMs) also treat 3D as an afterthought. They might output 3D tokens, but they do not systematically align the semantic latent space (the mathematical representation of meaning) with the geometric manifold (the underlying structure of the 3D shape). This decoupling means the model's "brain" does not actually understand the topology it is creating. This leads to fragmented surfaces and distorted geometries.

How It Works

EVA01 solves this via a Mixture-of-Transformers (MoT) architecture. This design separates "thinking" from "doing." As shown in, the model is split into two specialized experts: an Understanding Expert ($E_{und}$) and a Generation Expert ($E_{gen}$).

1. Expert Decoupling: $E_{und}$ acts as a semantic anchor. It is a pre-trained MLLM based on Qwen3-VL. It preserves the multimodal priors of the original model. $E_{gen}$ is a structurally mirrored expert dedicated to geometry synthesis. They are connected through shared global self-attention. This allows the generation expert to "query" the understanding expert for semantic cues without overwriting the stable reasoning of the backbone.
2. Structured Sparse Tokenization: EVA01 uses an "O-Voxel" representation instead of unordered point clouds. This anchors every token to a specific coordinate in a regular 3D grid. This spatial grounding is critical. It ensures the model knows exactly where a part exists in 3D space. This is a prerequisite for stable editing.
3. Stateful Sequence Modeling: To enable editing, the authors formulate 3D modification as a conditional sequence modeling task. Instead of regenerating the whole object, the model predicts the next geometric state. It does this by conditioning on the current instruction and the full historical context.
4. 3D Interleaved MRoPE: To prevent the model from losing track of 3D topology in a flattened sequence, they use a modified Rotary Positional Embedding (MRoPE). This repurposes standard $(T, W, H)$ embeddings for $(x, y, z)$ coordinates. This injects Euclidean inductive biases (mathematical assumptions about physical space) directly into the transformer layers.

Numbers

The authors report significant gains in generation fidelity and conversational intelligence. In single-turn text-to-3D generation, EVA01 achieves a CLIP score of 35.72 on the Toys4K benchmark. This outperforms the previous state-of-the-art, TRELLIS, which scored 30.80 .

This higher score indicates better semantic alignment between the text prompt and the resulting shape.

The most striking delta is in multi-turn editing. EVA01 achieves a 93.75% user preference score for context-aware editing. This dwarfs the 3.75% preference recorded for VoxHammer [Table 5]. This is notable because VoxHammer uses explicit 3D edit masks to protect unedited regions. EVA01 achieves superior identity preservation without any such masks. It relies entirely on its internal representation of the interaction history.

For mesh understanding, the authors measure the model's ability to caption 3D objects. The "final" version of the model achieves the highest scores under GPT-based judges. Specifically, it reaches a GPT-img score of 65.91 [Table 4]. This score evaluates how well the caption describes the actual rendered 3D object.

What's Missing

There are several technical gaps that a practitioner should note.

First, the model suffers from a representational asymmetry. The understanding branch relies on point-cloud features. Conversely, the generation branch uses sparse voxels. The authors admit this is a pragmatic choice. However, it means the model is not truly "native" in a single, unified 3D latent substrate. This split may limit the seamlessness of transferring understanding to generation.

Second, the scale is currently modest. The researchers trained the experts at a 2B parameter scale with a $512^3$ resolution. While this proves the MoT architecture works, the scaling behavior remains unproven. We do not know if the mixture-of-experts approach scales linearly toward 7B or 14B parameter regimes.

Finally, the model struggles with complex spatial reasoning. As shown in, EVA01 still fails at tasks involving exact counting or legible text on surfaces. It also struggles with highly unusual, out-of-distribution spatial compositions. It understands "a car," but it has not yet mastered "five cars stacked in a perfect pyramid."

Should You Prototype This

Yes, but only if your roadmap involves interactive 3D content creation. If you want a pure, high-speed reconstruction engine, stick with TRELLIS.2. It remains the specialist leader for single-image-to-3D.

However, if you are building a tool for iterative design, EVA01 provides a viable path. Users can say, "Make the dragon's wings larger, then change its scales to gold." The MoT decoupling is a practical pattern. It allows you to add new modalities to an existing LLM without breaking its core reasoning. Code is reportedly available; see the paper for the canonical link.

Figures from the paper