Physics-IQ Verified

Physics-IQ Verified: Refining the Yardstick for Artificial World Models

Current video generative models (VGMs) are evolving from simple tools into sophisticated "world models." These systems attempt to simulate the causal and physical logic of our reality. To advance these models, researchers need reliable ways to test physical understanding. They must determine if an AI truly understands gravity or fluid dynamics. They need to know if the model understands physics or just produces motion that looks "plausible" to a human eye.

Measuring the Laws of Motion

The core challenge is distinguishing between visual realism and physical accuracy. A model might generate a beautiful video of a glass shattering. However, if the shards fly upward instead of downward, the model has failed to grasp gravity.

The Physics-IQ benchmark was developed to quantify this understanding. It compares AI-generated videos to real-world recordings of controlled physical experiments. But as the authors of "Physics-IQ Verified" argue, a benchmark is only as good as its precision. If the test questions are vague, the scores are unreliable. If the "answer key" (the ground-truth videos) contains distracting noise, the test fails.

The Background You Need

The benchmark uses a "prediction-from-context" paradigm. A model sees a starting image or clip. It must then generate the logical continuation of that scene.

Success is measured using four primary metrics. These act as a digital ruler for physical phenomena: 1. Spatial IoU (Intersection over Union): This measures where the action occurs by calculating the overlap between predicted and actual areas. 2. Spatiotemporal IoU: This measures where and when the action occurs. 3. Weighted Spatial IoU: This measures where and how intensely the action occurs. 4. Mean Squared Error (MSE): This is a pixel-level metric. It measures how the action occurs by comparing generated pixels to the ground truth.

Unlike older benchmarks that rely on human judgment, Physics-IQ uses real-world footage. This makes it sensitive to the "sim-to-real gap" (the difficulty of moving from idealized math to messy real-world visuals).

The Anatomy of a Systematic Audit

The researchers conducted a systematic audit of the original Physics-IQ benchmark. They identified three distinct "confounding factors" (variables that distort measurements). As shown in, they propose three refinements to sharpen the evaluation.

First, they addressed prompt quality. A prompt functions like an exam question. If it is ambiguous, even a brilliant model will fail. The authors found many original prompts were factually incorrect or lacked key information .

They implemented a six-field templater (using SETUP, SCENE, ACTION, CAM, STYLE, and SCOPE). This ensures models receive structured instructions.

Second, they tackled spurious metric activations, or "artifacts." In many ground-truth videos, lab equipment might move in the background. Because IoU metrics track any motion, these irrelevant movements create "noise" in the score. The authors categorized these into deterministic artifacts (predictable movements like a rotating base) and non-deterministic artifacts (random recording errors) . They used "frame freezing" (holding pixel values constant) to erase these artifacts . This ensures metrics focus only on the intended physical effect.

Third, they overhauled metric aggregation. The original Physics-IQ calculated one score for the entire dataset. This meant experiments with high natural variability could skew the results. The authors introduced a sample-level scoring system. This allows researchers to trace failures to specific individual samples.

What This Lets Us See

The impact of these refinements is significant. When the researchers applied "Physics-IQ Verified" to six image-to-video models, the rankings shifted. They report a Kendall’s $\tau$ (a metric measuring how well two rankings agree) of 0.46 .

This indicates moderate but meaningful changes in which models appear best.

The audit revealed that some models benefited from the changes while others did not. For example, the model Wan 2.2 saw a decrease in performance under the verified protocol .

This suggests its previous high scores were partly due to the artifacts the researchers removed. Conversely, models like Sora 2 showed large increases in scores . This happened because the improved prompts helped the models follow instructions more accurately.

By refining 57.6% of all samples and 34.8% of prompts, the researchers have created a clearer signal. This helps developers build models that prioritize actual physical reasoning over mere visual mimicry.

Where The Edges Are

The "Physics-IQ Verified" framework has limitations. The process of cleaning artifacts relies on manual annotations. This is labor-intensive and harder to scale. Furthermore, the benchmark is "reference-based." It compares AI output to one specific ground-truth video. A model might generate a version of an event that is physically legal but differs from the specific recording. Such a model would still be penalized. Finally, the study only evaluated six image-to-video models. The broader usefulness of these prompts for all future architectures remains unknown.

Figures from the paper