Benchmarking Visual State Tracking in Multimodal Video Understanding

Most Multimodal Large Language Models (MLLMs) are surprisingly good at telling you what is happening in a video. However, they are terrible at telling you what has changed. While current state-of-the-art models excel at semantic understanding—identifying that a person is playing basketball—they struggle with continuous state tracking. This means they cannot monitor how a latent state (an underlying variable like a score or position) evolves over time. Researchers have developed a new benchmark called VSTAT to expose this gap. They found that while these models can reason perfectly if given a text description, they fail to "see" the actual transitions in the video stream.

The Problem

The status quo in video understanding focuses on recognition. Most benchmarks ask, "What is in this frame?" or "What action is occurring?" These often allow models to cheat using visual shortcuts. A model might identify a final state or pick out a single salient moment to infer an answer. This ignores the necessity of visual state tracking. This is the ability to monitor how a state evolves throughout a procedure.

As the authors demonstrate, existing models lack the temporal integration required for real-world applications like robotics. If a model cannot track an object through an occlusion (when an object is temporarily hidden), it cannot be trusted. High performance on general video benchmarks does not guarantee the ability to maintain a persistent mental model of a changing scene.

How It Works

The researchers introduced VSTAT. It is a benchmark comprising 834 clips and 1,500 questions. It is designed specifically to prohibit visual shortcuts. The methodology uses a taxonomy that classifies tasks across two axes: state complexity and perceptual complexity.

1. State Complexity Mapping: Every task is categorized by the type of information being tracked. This includes count, location, or attribute. It also considers the mathematical structure, ranging from atomic values to complex dictionaries (mappings of entities to values).
2. Perceptual Challenge Injection: The authors curated videos with intentional difficulties. These include camera motion, homogeneity (where multiple objects look identical), and symbolic decoding (mapping visual patterns to characters).
3. Diagnostic Controlled Experiments: The authors performed a "perception vs. reasoning" split. They compared model performance on raw video against performance using a manual text transcription of the same events.

The benchmark uses a mix of data. This includes synthetic videos rendered in Blender and real-world footage from YouTube. This variety ensures models cannot simply memorize patterns from a single simulator.

Numbers

The results reveal a massive delta between artificial and biological intelligence. Humans achieve an average accuracy of 90.5% on VSTAT. In contrast, state-of-the-art MLLMs perform far below this. They only modestly exceed answer-prior baselines (the accuracy achieved by simply guessing the most frequent answer).

The most striking finding comes from the diagnostic experiments. MLLMs solve tasks almost perfectly when given a text transcription. However, when faced with actual video, performance drops to near-chance levels as video duration increases . This proves the bottleneck is perception, not reasoning.

Quantitatively, the authors find that more than 50% of all failures stem from event recognition errors .

Furthermore, increasing the "thinking budget" does not help. In some cases, it actually hurts performance. For example, in Table 3, some models showed a decline in accuracy when enabled with higher thinking levels. This happens because a larger thinking budget can increase the likelihood of hallucinations (generating false information).

What's Missing

The paper is a diagnostic tool, but it leaves a few gaps. First, the analysis of failures relies on "thinking traces." These are the textual logs of a model's internal monologue. These are only a proxy for actual visual processing. We do not see how intermediate visual embeddings (mathematical representations of images) are failing.

Second, the benchmark currently lacks coverage for extremely long-form video. The authors acknowledge this is due to current model context limits. A robust state tracker should eventually handle hour-long streams, such as a full soccer match.

Finally, the paper does not explore how these perceptual failures interact with quantization (reducing model precision to save memory). If you deploy a quantized model on the edge, you might encounter a steeper performance cliff.

Should You Prototype This

Not yet.

The VSTAT benchmark is an excellent diagnostic tool. The authors have made the code and data available; see the paper for the canonical links. However, the findings suggest that more compute or better prompting will not fix the underlying issue. Adding an agentic layer (using models as autonomous agents) or a "thinking" loop does not readily resolve these failures. Until we see a shift in how MLLMs integrate temporal visual signals, these tools are likely to yield diminishing returns. If you are building a system that requires high-fidelity state monitoring, treat these results as a warning. Current models are better at describing the world than they are at tracking it.

Figures from the paper