Flat-Pack Bench: Evaluating Spatio-Temporal Understanding in Large Vision-Language Models through Furniture Assembly

Why Your Video-Language Models Can't Build IKEA Furniture

Large Vision-Language Models (LVLMs) excel at understanding static images. They also handle short video clips well. However, they struggle when the clock starts ticking on complex, multi-step tasks. Researchers find that while these models can caption a scene, they fall apart during physical interactions.

Current AI development focuses on expanding context windows (the amount of data a model can process at once). We want intelligent assistants to help us cook recipes or repair sinks. A critical question remains. Can these models track how objects move and connect over time? Or are they just making guesses based on static visual cues?

This paper introduces FLAT-PACK BENCH to answer that question. It uses the structured domain of furniture assembly. The researchers exposed a massive gap in fine-grained spatio-temporal reasoning (understanding how things move and interact in space and time).

The Problem

Most existing video understanding benchmarks are too coarse-grained. They focus on high-level semantics like action segmentation (categorizing actions like "running"). These tasks often involve easy entities like "a dog." Scenes are usually uncluttered.

True physical intelligence requires much more nuance. As shown in, an assistant must do more than identify a "leg." It must understand the temporal order of connections.

It must localize exactly when a state changes. It must track specific parts through occlusions (when an object is hidden from view). It must also recognize "mating"—the moment two parts physically meet. Current benchmarks fail to stress-test these dimensions. This leaves us with models that fail in cluttered, dynamic environments.

How It Works

The authors built FLAT-PACK BENCH using the IKEA-Manuals-at-Work (IMaW) dataset. They added dense, fine-grained annotations. To avoid linguistic shortcuts, they implemented a visual prompting strategy.

Instead of using only text, the benchmark provides an assembly video. It pairs this with one or two visual prompts. These prompts are segmented (outlined) and labeled images from the video .

This forces the model to ground its reasoning in specific spatial regions.

The benchmark evaluates four axes of understanding: 1. Temporal Ordering (TORD): Finding the correct sequence of connection events. 2. Temporal Localization (TLOC): Identifying the timing of events relative to a state. 3. Tracking (TRACK): Recovering part identities across different frames. 4. Mating (MATE): Deciding if two parts are actually connected.

The authors also tested a "Temporal Video Agent" (TVA). This agentic approach uses a Code LLM to orchestrate specialized tools. It uses a video object segmenter (built on SAM2) for tracking. It also uses a VLM-query function for contact reasoning .

Numbers

The results are a reality check. Humans achieve a micro-average accuracy of 94.18% [Table 2]. This proves the tasks are solvable. In contrast, GPT-5 achieved only 37.71% [Table 2]. This is barely better than the chance baseline.

The gap is wide in specific failure modes. The best open model, InternVL3-78B, reached 41.03% [Table 2]. However, models consistently struggled with TRACK and MATE tasks. The agentic TVA baseline performed even worse. It had an overall accuracy of only 11.79% [Table 5]. This happened because the underlying tools, like SAM2, struggle to maintain accurate tracks in these videos.

Ablation studies reveal that models "cheat" using static cues. When the authors removed the video, performance on TRACK dropped severely [Table 4]. Yet, models relied on image-based common sense for other categories. This indicates they do not use the video context effectively.

What's Missing

The paper is a rigorous diagnostic. However, it leaves some questions for practitioners.

First, the benchmark is limited to furniture assembly. The authors call this a "microcosm." However, furniture parts are rigid. They do not deform like food or fabric. I wonder how these failure modes translate to deformable object manipulation.

Second, the computational cost of the "ideal" solution is not fully explored. The benchmark uses long videos. The authors mention subsampling (reducing the number of frames) for different models. They do not provide a clear cost-benefit analysis. We need to know how much context is actually required.

Finally, the paper does not address real-time inference. Most evaluations are zero-shot (testing without prior training on the task) on pre-recorded clips. Latency and drift issues in a live assistant are not covered.

Should You Prototype This

Do not prototype using current off-the-shelf LVLMs for high-precision temporal tasks. The data shows they are unreliable at tracking identities. They also fail to verify physical contacts.

However, the methodology is worth adopting. Visual prompting is a proven way to reduce ambiguity. If you develop custom models for industrial inspection, use these techniques. Use the visual prompting and manual curation described here to build your own stress tests. You can find the benchmark resources at flat-pack-bench.github.io/viewer/. Code is reportedly available; see the paper for the canonical link.

Figures from the paper