PAIWorld: A 3D-Consistent World Foundation Model for Robotic Manipulation

Solving the Ghost in the Machine: Why Robotic Simulators Struggle with 3D Consistency

Robots need to see the world from multiple angles to work effectively. However, current AI simulators struggle to keep these views consistent. If a robot's "imagined" future shows an apple in one spot from the top view but shifts slightly to the left from the side view, the robot's planned movements will fail in the real world. PAIWorld aims to solve this. It ensures that objects look the same and stay in the same place across all camera views. This creates a more reliable simulator for robot training.

The failure of flat concatenation

Current world foundation models (WFMs)—large-scale AI systems that simulate physical environments—predominantly operate in a single-view setting. While they can generate impressive videos, they often lack the multi-view 3D consistency that robotic manipulation demands. Most robots rely on several cameras simultaneously. These include wrist-mounted or eye-to-hand views to understand depth and reach.

To handle multiple views, existing models typically use a "flat concatenation" strategy. This involves treating tokens (the basic units of data processed by a transformer) from different viewpoints as if they were just more frames in a single timeline. The authors of the PAIWorld paper argue that this approach fails. It lacks an explicit mechanism for geometric reasoning. Without a dedicated way for views to communicate, each viewpoint essentially generates in isolation. This leads to "cross-view object drift," where an object appears to move between views. It also causes "texture misalignment," where surfaces do not match up. Such errors propagate directly into downstream planning and control. This makes the simulated environment unreliable for teaching a robot precise tasks.

Two pillars for a consistent world

The authors propose that solving multi-view inconsistency requires two simultaneous remedies. They need an architectural pathway for communication and a training objective that enforces 3D structure. They implement these through three modular components built on a Diffusion Transformer (DiT) backbone, as illustrated in .

First, they establish an inter-view communication pathway using two components: 1. Geometry-Aware Cross-View Attention: Instead of treating all tokens the same, these specialized blocks allow different viewpoints to exchange features explicitly. 2. Geometric Rotary Position Embedding (Geo-RoPE): This component encodes camera ray directions and extrinsic poses (the position and orientation of the camera) into the attention mechanism. Much like a GPS tells a driver where they are in relation to a city, Geo-RoPE tells the model where a specific pixel is located in 3D space. This biases the attention mechanism to route information between tokens that actually observe the same 3D point.

Second, they provide a geometric learning signal via Latent 3D-REPA. This component acts as a supervisor. It distills 3D-aware features from a frozen 3D foundation model called Depth Anything 3. Rather than forcing the model to predict exact pixel values, it teaches the model to mirror the relational structure. This refers to how different parts of a scene relate to one another in 3D. The authors argue that the pathway allows information to flow. Meanwhile, the objective ensures that the information being exchanged is geometrically meaningful.

Superiority in motion and geometry

The researchers evaluated PAIWorld on several benchmarks. They focused on its ability to generate realistic, action-conditioned videos. On the WorldArena benchmark, the authors report that PAIWorld ranks 1st overall with an EWMScore of 70.67%. This score represents the overall quality of the world model. A standout result is its Motion Quality score of 79.66. The paper identifies this as the best among all tested entries. High motion quality means the simulated movements are smoother and more realistic.

In the AgiBot-Challenge2026 benchmark, the paper finds that PAIWorld achieves a Scene Consistency score of 90.41%. This is the highest recorded among competitors. This metric is crucial because it measures how well the semantic and visual appearance remains coherent across different views. Qualitative results in and demonstrate that the model produces physically plausible dynamics.

Object interactions and motions closely track the commanded robot actions. Furthermore, in text-conditioned tests, the authors report that PAIWorld achieves the best MEt3R score (14.20). This metric quantifies 3D consistency via point cloud cross-projection. As shown in, this translates to significantly better object placement and depth structure compared to previous models like Genie-Envisioner.

Limits of the current framework

While the results are strong, the paper does not explore several critical frontiers. First, the model focuses heavily on geometric consistency. However, it does not yet incorporate complex physical interaction modeling. Examples include contact dynamics, deformable objects (like cloth), or fluid simulation. For a robot tasked with handling soft materials, geometric alignment alone might not suffice.

Second, the study does not address long-horizon stability. Maintaining 3D consistency is difficult for short clips. As the prediction horizon increases, errors in the "imagined" world may accumulate. Finally, the computational cost of training such a model is significant. The authors report conducting training on 200 NVIDIA H200 GPUs for approximately seven days. For practitioners looking to deploy these models locally, the hardware requirements for training these 14B parameter models remain a hurdle.

The verdict: A necessary step for embodied AI

Is PAIWorld ready for the lab? For researchers building high-fidelity simulators for robotic training, the answer is a qualified yes. The authors have successfully demonstrated a "super-additive" effect. This occurs when architectural changes and training objectives reinforce each other. By moving away from simple token concatenation toward explicit geometric reasoning, they have bridged a major gap in world modeling.

If you are developing a policy for a robot that relies on multi-camera setups, PAIWorld provides a more reliable "dream space" for training. It outperforms previous single-view or flat-concatenation models in geometric accuracy. However, until the framework incorporates more nuanced physics like friction and deformation, it should be viewed as a breakthrough in visual and geometric consistency. It is not yet a complete replacement for traditional physics engines.

Figures from the paper