PaSBench-Video: A Streaming Video Benchmark for Proactive Safety Warning

PaSBench-Video: Exposing the Failure of MLLMs in Proactive Safety Warning

No tested MLLM exceeds 20.0% on the strictest safety metric. This finding disrupts the assumption that multimodal large language models (MLLMs)—models capable of processing both visual streams and text—are ready for safety-critical monitoring. Current models either miss the danger entirely or "cry wolf" by issuing constant false alarms. This creates a massive reliability gap for anyone hoping to use these models for real-world monitoring.

The Question

Can MLLMs actually function as proactive safety monitors? Specifically, the researchers wanted to know if these models can observe a continuous video stream. Can they identify the precise moment a risk becomes actionable? Can they issue a warning that is both timely and content-correct? A valid warning must land in the window between the first visible sign of danger and the moment an accident occurs. The goal was to see if models can reason about emerging harm rather than just identifying a "bad" scene.

Why The Old Answer Was Incomplete

Until now, industry benchmarks have treated video safety as a static classification problem. Many previous works used image sequences or offline QA (Question Answering). In those setups, the model sees the entire video at once. This ignores the fundamental reality of safety systems. They must operate in a streaming, causal manner. In production, a monitor cannot see the accident happen before it decides to warn. It must decide based only on the past and the present.

Furthermore, existing benchmarks frequently omitted the "no-risk" component. In a real-world deployment, a system that catches every accident but triggers a false alarm every ten minutes is useless. Frequent false alarms cause alarm fatigue (the tendency of users to ignore alerts due to high frequency). The authors argue that without measuring false positives on benign scenes, we aren't actually testing safety. We are just testing pattern matching.

What They Did

The authors developed PaSBench-Video, a benchmark comprising 740 videos. They curated 481 risk videos and 259 no-risk videos across four domains: driving, healthcare, daily life, and industrial production. They used a hybrid construction pipeline .

They pulled real-world footage from datasets like DADA and Nexar. They also synthesized complex scenarios using the Kling v3 Omni video generator.

The evaluation protocol is strictly streaming. As shown in, the video is partitioned into 3-second windows with a 0.3-second stride (the amount of time the window moves forward).

For every window, the model receives only the past and current frames. It also receives a compact state summary (a distilled history of the scene).

The researchers implemented a hierarchy of increasingly difficult requirements [Table 5]: 1. Any-Det: Did the model detect a risk at any point before the accident? 2. Any-Hit: Was the warning issued within the valid window $[r, a-\tau]$? 3. First-Hit: Was the very first warning issued within that window? 4. First-Act: Was the first warning both correctly timed AND content-correct? This requires identifying the right risk source and an avoidance action.

What They Found

The results were a sobering reality check. Testing 13 different models, the authors found that no model exceeded 20.0% on the strictest metric, First-Act@1.

Crucially, model performance is governed by a "sensitivity knob" rather than genuine reasoning. As shown in, there is a strong Pearson correlation ($\rho=0.64$) between recall and the false-positive rate.

Recall is the ability to find all risks. When you try to make a model more sensitive to catch more risks, it doesn't get smarter. It simply starts firing on safe clips. This is most evident in the driving domain. There, models struggle to distinguish routine traffic from imminent collisions.

The failure modes are highly specific. When models do manage to time a warning correctly, they often fail the "content" part of the test. Looking at, high-sensitivity models shift their mistakes from "missed" to "too-early." They scream about a danger before the visual evidence actually supports it.

Furthermore, the authors categorized failures where timing was correct but the reason was wrong [Table 6]. Over 70% of these errors stem from picking the wrong specific object or performing an "entity-type swap" (e.g., calling a vehicle a pedestrian).

What This Changes

The implications for engineers are clear. Current MLLMs are not ready for deployment in safety-critical tasks.

First, there is a fundamental capability gap. The data suggests that models are currently acting as "activity detectors" rather than "risk reasoners." They recognize that something is happening. However, they cannot reason about whether that activity will escalate into harm.

Second, the engineering constraints are massive. As demonstrated in, even the fastest proprietary models (like GPT-5.4 mini) have a median latency of 3.5s. This is an order of magnitude slower than the 0.3s window stride required for real-time streaming. When you combine this latency with the astronomical API costs, the math fails. Costs can reach up to \$12,000 per day for a single continuous camera feed.

If you are building a safety system today, do not look at general-purpose MLLMs as a silver bullet. They are currently too slow, too expensive, and too prone to crying wolf. Research should focus on models that can move beyond simple activity detection toward true temporal risk reasoning.

Figures from the paper