SpatialAct: Probing Spatial Reasoning-to-Action Capabilities of VLM Agents in 3D Scenes

Can VLMs Actually Act on What They See?

Humans can effortlessly perceive spatial layouts. We form cognitive representations and reason about spatial relations. We then translate this reasoning into actions in everyday 3D environments. While recent vision-language models (VLMs) show promise in describing these spaces, it remains unclear if they can truly build a coherent understanding and act upon it. Specifically, can a model take an action, observe how that action changes the world, and then refine its next move based on that feedback?

The missing link in spatial intelligence

The researchers behind SpatialAct investigated whether current VLM agents possess "action-conditioned spatial reasoning." This goes beyond observation-conditioned reasoning (simply looking at a picture and describing it). Action-conditioned reasoning requires a model to predict how its own interventions—like moving, rotating, or scaling an object—will transform the environment. The model must then maintain a consistent "spatial belief" (a mental map of the environment) as that environment evolves. The central question is whether the leap from perception to execution is a smooth transition or a massive chasm.

The observer bias in current benchmarks

Until now, the field has largely treated spatial intelligence as a passive task. Most existing benchmarks evaluate models as observers. The model is given a static image or video. It is then asked to answer questions about what is already there. Even in more advanced "embodied" benchmarks, the evaluation often entangles high-level spatial reasoning with low-level motor control (the physical mechanics of movement).

As shown in the comparative analysis in [Table 1], most prior work focuses on "Understanding." SpatialAct aims for "Action." The authors argue that a "missing middle ground" exists. This space lies between passive question-answering and full-scale robotic embodiment. In this middle ground, we can test high-level semantic actions (e.g., "rotate the building 90 degrees") without simulating the granular physics of a robot's joints.

Diagnosing the reasoning-to-action gap

To investigate this, the authors developed SpatialAct. This is a simulator-grounded benchmark that uses a hierarchical approach. It isolates exactly where models fail. As illustrated in, the testing moves through three tiers.

It begins with "Basic Spatial Abilities" (like mental rotation). It moves to "Single-step Error Detection and Fix." Finally, it reaches the most difficult tier: "Multi-turn Interactive Refinement."

In the multi-turn setting, the model faces a 3D scene with intentional errors. These might include colliding objects or improper placements. The model must iteratively fix them. The model issues commands like Move(North, 0.7). The simulator executes them. Then, updated multi-view renderings are fed back to the model. This creates a closed-loop system, as seen in the workflow in .

The model must track the state of the world across multiple turns. To prevent "memorization" of static scenes, the authors used procedural generation (creating scenes via algorithms) and dynamically injected errors.

High scores on basics, collapse on interaction

The results reveal a stark "reasoning-to-action gap." Strong VLMs can achieve roughly 80% accuracy on isolated, basic spatial tasks. However, they struggle significantly during interaction. For example, the strongest model tested, Gemini-3.1 Pro, achieved a Repair Rate of 0.411. This means it only reduced the number of errors by about 41%. Its Scene Success Rate was only 0.206. This means it successfully cleared all errors in only about 20.6% of scenes [Table 2].

In contrast, human participants achieved a Repair Rate of 0.911 and a Scene Success Rate of 0.763 [Figure 5a]. Humans were much more effective at both fixing individual errors and completing entire scenes.

The failure modes are twofold. As demonstrated in, models suffer from "diagnosis errors." This occurs when they fail to perceive that an error exists.

They also suffer from "reasoning-to-action errors." This happens when they identify a problem but issue a command that fails to fix it. Sometimes, the command even introduces a new error. Furthermore, performance drops as initial scene complexity increases [Figure 6a]. Increasing the number of simultaneous errors makes it harder for models to manage the task. There is also a "diminishing return" regarding context. Larger context windows allow for more verbose reasoning. However, they do not improve the actual success rate of the repairs [Figure 6b]. This suggests the problem is not a lack of "thinking space." Instead, it is a lack of a reliable policy for translating thought into spatial change.

Implications for the path to embodiment

The findings suggest that current VLM architectures face a ceiling. Simply scaling up parameters or context windows may not suffice. If these results generalize, they imply that "spatial intelligence" in AI is currently divided. Models have learned the vocabulary of space (identifying objects and locations). However, they have not mastered the grammar of space (how movement affects object relations and boundaries).

There are two immediate implications. First, for researchers in embodied AI, improving visual perception alone is insufficient. The bottleneck is likely in "constraint-aware planning." This is the ability to model topological and boundary constraints during the planning phase. Second, for developers of autonomous agents, this highlights a reliability gap. A model might pass a visual exam but fail a multi-step task. Such a model lacks the ability to maintain a coherent state during interaction.

The paper does not explore specific methods to fix these issues. However, it provides a roadmap. A logical next step involves investigating how models can better track states across feedback loops. Solving this will be essential for creating dependable, spatially grounded agents.

Figures from the paper