MBench: A Comprehensive Benchmark on Memory Capability for Video World Models

Can AI Truly Remember the World?

Researchers have created a new way to test if AI video generators can actually "remember" things. Instead of just looking at pretty pictures, this test checks if objects, backgrounds, and physical rules stay consistent. This happens even when the camera moves away and comes back.

The field of video-based world models aims to create digital environments. These should not just look real. They must behave according to the laws of physics and logic. These models are essential for high-stakes applications like training autonomous vehicles or programming robots. In these fields, a machine must understand that an object still exists even when it is temporarily hidden behind a wall. Until now, however, the industry has focused almost entirely on "perceptual quality" (how high-fidelity and visually pleasing a single clip looks).

This focus has created a massive blind spot. A model might generate a stunningly realistic video of a car driving down a street. But if that car turns a corner and suddenly transforms into a truck, the model has failed. It has produced a beautiful hallucination rather than a stable simulation. The authors of the MBench study argue that we cannot claim to have functional world models until we can rigorously measure their long-term memory.

The illusion of visual consistency

Current evaluation protocols for video generation prioritize motion coherence (smoothness of movement) and text-video alignment (matching the prompt). These metrics essentially ask: "Does the video move smoothly?" and "Does the video match the prompt?" While useful for creative tools, these questions are insufficient for simulation. A functional world model must maintain a stable internal state over extended periods.

As the authors demonstrate, existing models often suffer from "forgetfulness" during long-horizon generation. When a camera moves through a scene or an object undergoes occlusion (the state of being hidden from view by another object), the model frequently loses track of the world's state. This manifests as "identity drift," where a person's face or an object's features subtly change over time. It also causes "geometric collapse," where the 3D structure of a room or scene drifts and loses its shape as the perspective shifts. Because current benchmarks do not systematically decompose these failures, developers lack a standardized way to distinguish between models that simply generate nice frames and those that actually simulate a predictable reality.

A three-tiered hierarchy of memory

To solve this, the authors propose MBench. This benchmark is built on a hierarchical taxonomy. It breaks memory down into three distinct, measurable dimensions. Rather than treating "consistency" as a single vague concept, they categorize it into:

1. Entity Consistency: Does the model remember the specific identity and attributes of individuals? This includes "Object Consistency" (keeping the texture of a bowl the same) and "Human Consistency" (ensuring a person's face and clothing do not morph).
2. Environment Consistency: Is the "stage" stable? This involves "Spatial Consistency" (maintaining the 3D layout of a room) and "Rendering Consistency" (keeping lighting and artistic style uniform).
3. Causal Consistency: Does the world follow logic? This tests "Self-evolution" (whether physical processes like an explosion progress logically) and "Interaction" (how the model responds to external commands).

The benchmark employs a unique "Exit-and-Reenter" paradigm for action-conditioned models. In this setup, the camera is forced to move away from a target object until it disappears. The camera then waits and returns to the exact starting position. This creates a "memory trigger" that forces the model to prove it hasn't forgotten what it saw earlier. To prevent models from "cheating"—such as by generating static, unchanging scenes to avoid making mistakes—the authors introduce a "Trigger-Conditioned Scoring" mechanism. This calculates an M-Score. This score uses a harmonic mean (a mathematical method used to balance two different metrics) of how often a model successfully attempts a challenge and how reliable it is when it does.

Measuring the drift

The results of the MBench audit reveal significant systemic vulnerabilities across state-of-the-art models. The authors report that even advanced models exhibit substantial temporal drift as the simulation horizon extends.

In the realm of text-conditioned models, the paper finds that while some models like Helios perform well on object geometry, they struggle with other forms of stability. Interestingly, the authors observe that even the strongest models show significantly lower scores in style consistency compared to lighting or epipolar geometry (mathematical constraints related to camera viewpoints). This suggests that current generators can maintain a stable visual tone while still losing the underlying 3D layout of the scene.

For action-conditioned models, the findings highlight a critical trade-off. The authors report that models like HY-WorldPlay achieve high scores in environment rendering and spatial consistency. However, their performance on "self-evolution" drops sharply. This identifies a common failure mode: models often "freeze" the scene to maintain high visual scores. They essentially sacrifice meaningful physical dynamics to avoid the risk of generating inconsistent movement. This discrepancy is visualized in, where high-scoring models correctly simulate physical destruction, while low-scoring models simply morph objects into something else entirely.

Limitations of the lens

While MBench provides a much-needed diagnostic tool, the authors acknowledge several technical hurdles. A primary limitation lies in the evaluators themselves. The benchmark relies on Vision Language Models (VLMs) to act as judges. However, the authors note that current VLMs have a limited ability to infer complex camera motions from sparse sets of frames. This can affect the accuracy of spatial evaluations.

Furthermore, there is a tension in the "action-conditioned" category. Because these models are designed to respond to specific movement commands, they are easier to trigger automatically. But they are also prone to "command collapse," where the model ignores parts of the control signal. Finally, the benchmark is highly dependent on the quality of the "trigger" events. If a model fails to execute the requested camera motion, it isn't necessarily a failure of memory. It might be a failure of instruction following. This is why the authors developed the dual-metric M-Score to decouple these two issues.

The verdict: Not ready for the real world

If you are looking for a model to power a high-fidelity robotics simulator or a truly interactive game engine, the answer is: not yet. The MBench study demonstrates that we are still in the era of "visually plausible hallucinations" rather than "reliable world simulations."

The gap between a video that looks real and a video that behaves predictably is the primary bottleneck facing the field. For practitioners, the takeaway is clear. Optimizing for frame-rate or pixel-perfect textures is not enough. True progress requires building better mechanisms for state tracking, action grounding, and latent causal simulation.

Code and project resources are available at the following links: * Project Page: https://peanutup.github.io/MBench-project/ * GitHub Repository: https://github.com/study-overflow/MBench * Leaderboard: https://huggingface.co/spaces/study-overflow/MBench_Leaderboard

Figures from the paper