SVI-Bench: A Dynamic Microworld for Strategic Video Intelligence

SVI-Bench: Bridging the Gap from Video Perception to Strategic Agency

Current multimodal models are excellent at describing what they see. They are remarkably poor at understanding why it happens or what might happen next. In video-language understanding, we have moved from simple action recognition to long-form video QA (question answering). Yet a fundamental gap remains. We lack a transition from mere perception to true strategic reasoning. Most existing benchmarks focus on short clips and superficial descriptions. They leave the question of whether AI can navigate complex, multi-agent environments with causal logic unanswered.

This paper introduces SVI-Bench. It is a massive new testing framework using professional team sports as a "dynamic microworld." By using the rigid rules and verifiable outcomes of basketball, soccer, and hockey, researchers created a way to test strategic intelligence. The goal is to see if an AI can move beyond saying "a player shoots a ball" to explaining "the defense collapsed because of a well-timed screen." The results are sobering. There is a massive "capability cliff" where model performance doesn't just dip—it collapses.

The Problem

The status quo in video intelligence is heavily skewed toward the perceptual layer. Current state-of-the-art models excel at parsing spatiotemporal primitives (basic units of space and time). They can identify who is where and what they are doing. However, they hit a wall when tasks require higher-order cognition. As shown in, there is a steep, consistent decline in performance. This occurs as you move from perception to reasoning, simulation, and finally agency.

Existing benchmarks fall into two inadequate camps. In-the-wild video datasets offer visual richness. However, they lack the verifiable ground truth needed to grade a model on why an event occurred. Conversely, synthetic environments (simulated worlds) provide perfect causal ground truth. But they operate in simplified, single-agent worlds. These lack the chaotic, adversarial complexity of real-world multi-agent systems. Neither allows us to measure the "Strategic Video Intelligence" (SVI) stack. This stack includes the ability to perceive, reason causally, simulate counterfactuals (alternative "what-if" scenarios), and act as an autonomous analyst.

How It Works

The core of the research is a sophisticated data engine. It turns raw, unstructured sports media into a dense, cross-referenced corpus. Rather than relying solely on manual labeling, the authors use a multi-stage pipeline. This pipeline aligns five distinct modalities: broadcast video, play-by-play logs, expert commentary, game reports, and box-score statistics.

The SVI-Bench evaluation follows a four-pillar hierarchy:

1. Dynamic Scene Understanding: This evaluates the perceptual floor. Tasks like T1 (Structured Play Description) require the model to generate dense captions for 10-second clips involving 10+ coordinated agents.
2. Causal Reasoning: This moves from "what" to "why." It requires analyzing much longer video segments (55–150 minutes). The model must explain strategic errors or forecast outcomes based on latent dynamics (hidden underlying factors).
3. Strategic Simulation: This tests the model's internal world model. In tasks like T7 (Motion-Conditioned Generation), the model receives a "player-removed" background. This is the court or pitch with players digitally erased via inpainting (filling in holes in images). The model must then generate a video where players follow specific, prescribed trajectories .

4. Agentic Synthesis: This is the highest tier. It tests if a model can act as an autonomous analyst over a massive corpus. In T9, the agent must plan a multi-step retrieval strategy. It must gather evidence from ~1.8M clips and ~33K documents to answer complex, multi-hop queries (questions requiring several steps of logic).

To isolate whether failures are due to poor "eyes" (perception) or poor "brains" (reasoning), the authors employ an "Oracle Mode." In this setup, they replace the raw video input with ground-truth textual descriptions of the game events. This shows if a model is bottlenecked by its inability to see or its inability to think.

Numbers

The most striking result is the performance delta between the easiest and hardest tasks. A fine-tuned LLaVA-Video-7B can achieve ~73.91% accuracy on fine-grained action QA. This represents a competent level of perception. However, the strongest model (GPT-5.2) manages only ~5% accuracy on the agentic reasoning task (T9). This drop signifies a total failure in autonomous reasoning.

The Oracle experiments provide crucial context. On the T9 agentic task, moving from raw video to oracle text boosts GPT-5.2's accuracy from 4.6% to 54.0%. This confirms that visual perception is a massive bottleneck. Even with "perfect" vision, the 54% ceiling proves that multi-step planning and evidence integration are unsolved problems. Similarly, for long-form narrative synthesis (T6), oracle access helps factual accuracy (rising to 87.19%). However, it barely touches the "saliency" problem. Saliency is the ability to judge which events actually matter. This remains at a dismal 20.60%.

What's Missing

While SVI-Bench is a massive leap forward, it has notable gaps:

• Domain Generalization: The authors frame this as a "microworld" of team sports. Camera conventions and agent behaviors in sports may not translate to "messier" domains. Examples include surgery or autonomous driving.
• Judge Reliability: Higher-level reasoning (T1, T4, T6) is often evaluated using an "LLM-as-a-judge" protocol. While the authors performed robustness checks, judge bias remains a potential confound (a variable that obscures results).
• Implementation Details: The paper focuses on capability rather than efficiency. The computational cost of querying 1.8 million clips is not reported. This makes it difficult to estimate the actual production costs.

Should You Prototype This

Not yet.

If you are building a system to describe video streams, current state-of-the-art is approaching human parity. However, do not attempt to build an autonomous agent for high-level strategic analysis yet. The benchmark shows that even frontier models like GPT-5.2 are essentially failing at agency.

The research is a signal. "More parameters" or "better vision encoders" will not solve the agency problem. The bottleneck is structural. We currently lack architectures capable of multi-step multimodal planning and long-horizon causal reasoning. Watch developments in "World Models" and "Agentic Tool-Use." Do not commit to a roadmap for autonomous strategic video analysis until the gap between perception and agency closes.

Code and data are available at github.com/texaser/svi-bench and huggingface.co/mvp-group/svi-bench.

Figures from the paper