M^3Eval: Multi-Modal Memory Evaluation through Cognitively-Grounded Video Tasks

Most multi-modal models are marketed as having massive context windows. This implies they can "watch" and remember long-form video. But increasing context alone does not guarantee effective memory. Researchers have created a new way to test if AI models actually 'remember' what they see in videos. They use psychological triggers—like showing two videos at once or mixing them up—to see if the AI gets confused or forgets details.

Common wisdom suggests that if a model can ingest a 1-hour video, it can reason about it. This paper argues that we have been conflating perception and reasoning with actual memory. Current benchmarks mostly test if a model can find a "needle in a haystack" or describe a scene. They fail to isolate the mechanics of memory. These include what the model retains, how faithfully it preserves information, and how robust that information is when faced with interference (competing or distracting inputs).

Beyond the needle-in-a-haystack trap

Current video understanding benchmarks are heavily biased toward visual perception and reasoning. Whether it is long-form video QA or streaming evaluation, the goal is usually to identify an object or summarize a plot. As the authors note, these tasks treat memory as an implicit component rather than a measurable variable. This creates a blind spot. A model might successfully retrieve a specific frame because of a strong visual signal. However, it hasn't necessarily demonstrated the ability to maintain a stable, disentangled representation (keeping separate pieces of information distinct) of multiple concurrent events.

This gap is dangerous for real-world deployments. If you are building a system for autonomous driving or household robotics, the model won't just face one clean video stream. It will face parallel streams, overlapping objects, and interleaved sensory inputs. Standard benchmarks don't tell you if your model will suffer from "attention confusion." This happens when the presence of a second video stream causes the model to lose track of the identities or locations in the first.

The M3Eval cognitive framework

To solve this, the authors introduce M3Eval. This is a benchmark that translates established paradigms from cognitive psychology into the video domain .

Rather than generic QA, they use four specific "stress tests" to isolate individual memory dimensions:

1. Divided Attention: Using split-screen videos to test encoding under concurrent visual load. This checks if the model can maintain independent representations for two streams or if it suffers from source confusion .

2. Memory Interference: Concatenating semantically similar videos to measure proactive interference (old info disrupting new recall) and retroactive interference (new info disrupting old recall) .

3. Interleaved Events: Alternating segments from two different storylines into a single stream. This tests if the model can reconstruct the original temporal order .

4. N-Back: Adapting the classic N-back task to video clips. This probes symbolic memory (the ability to abstract a scene or action into a category). It compares a current state to a previous state across a temporal gap .

The framework is built on a dataset of 2,403 questions spanning approximately 403 hours of video. It uses materials from established datasets like HourVideo and Video-MME.

Discrepancies in model and human memory

The results are telling. The authors report that almost all tested models show a massive performance gap compared to humans. This is especially true in divided attention tasks. Most models hover near chance levels when forced to process parallel streams. The authors hypothesize this is due to attention maps becoming diffused and disorganized in split-screen settings .

One of the most striking findings involves how models handle interference. In humans, there is a clear asymmetry. Retroactive interference is significantly stronger than proactive interference. Multi-modal models, however, show nearly equal levels of interference for both [Table 2]. Furthermore, the authors found a surprising effect. Repeating either the target or the interfering video can actually improve model accuracy. This likely happens because the causal attention mechanism allows later frames to attend more effectively to earlier occurrences of the same content .

In symbolic N-back tasks, the divergence is even sharper. While human accuracy decays predictably as the temporal gap ($N$) increases, model accuracy remains flat or even improves . However, model performance crashes as the total sequence length ($K$) increases. This suggests that while Transformers are excellent at retrieving temporally distant tokens via global attention, they struggle to filter out irrelevant information from their "working memory."

Limitations and engineering blind spots

While M3Eval is a significant step forward, there are caveats to consider. First, the benchmark relies on an automated QA generation pipeline using Qwen3.5-27B. Even with manual verification, this introduces a layer of model-generated bias. If the generator favors certain linguistic structures, the benchmark might inadvertently measure parsing ability rather than pure memory.

Second, the scope is intentionally narrow. The paper focuses on specific cognitive dimensions like spatial, temporal, and symbolic memory. It does not explicitly address how memory scales with extremely high-resolution inputs. It also does not examine how memory behaves under varying compression artifacts. These are practical realities in video streaming pipelines. Finally, the "repetition" finding is a heuristic. Relying on repeating video segments to boost accuracy is a temporary fix. It does not solve the underlying structural attention problem.

The verdict

Is M3Eval worth your time? Yes. If you are working on anything involving multi-stream video—such as robotics, surveillance, or complex scene analysis—you should test against these specific failure modes. The findings suggest that simply increasing parameters or the KV cache (a memory storage for Transformer models) won't fix the underlying issue. Attention confusion and poor information filtering remain major hurdles.

Code and datasets are available at the canonical link. If you want to improve your model's robustness, do not just aim for higher accuracy on standard benchmarks. Look at your model's performance on interleaved and parallel streams. That is where the real production failures will occur.

Figures from the paper